User-agent: * Disallow: /system Disallow: /admin Disallow: /feed User-agent: Yandex Crawl-delay: 60 User-agent: Slurp Crawl-delay: 150