User-agent: * Crawl-Delay: 3 Disallow: /iac/archive/file/ Disallow: /iac/archive/tree/ Disallow: /iac/proceedings/ Disallow: /iac/author/ Disallow: /download/ # Some bots doesn't respect slash at the end, let's block such a buggy bots. # User-agent: dotbot User-agent: PetalBot Disallow: / # SoGou is very buggy # User-agent: sogou spider # Disallow: /