# Robots.txt

User-agent: MegaIndex.ru/2.0
Disallow: /

User-agent: *
Disallow: /userfiles/newsletter/
Disallow: /userfiles/files/
Disallow: /userfiles/2015/
Disallow: /media/2015/
Disallow: /backend/

###### CRAWL BLOCKED for these user agents
User-Agent: Baiduspider
User-Agent: CazoodleBot
User-Agent: Fasterfox
User-Agent: Jyxobot
User-Agent: MJ12bot
User-Agent: ShopWiki
User-agent: AhrefsBot
User-agent: Sosospider
User-agent: Bingbot
Disallow: /
 
###### CRAWL DELAYS for these user agents
User-Agent: msnbot
Crawl-Delay: 20
 
User-Agent: Ingrid
Crawl-delay: 20
 
User-Agent: Slurp
Crawl-delay: 20
 
User-Agent: Yahoo! Slurp
Crawl-delay: 20
 
User-agent: rogerbot
Crawl-delay: 2