User-agent: * Crawl-delay: 60 Allow: /index.cfm User-agent: Slurp Crawl-delay: 300 User-agent: Yahoo!-AdCrawler Disallow: / User-agent: twiceler Disallow: /