# Disallow "ExtateCrawler v2.0.1" in the site User-agent: ExtateCrawler v2.0.1 Disallow: / # Disallow "Wget/1.10.2 " in the site User-agent: Wget/1.10.2 Disallow: / # Disallow "Wget" in the site User-agent: Wget Disallow: / # Allow all other spiders User-Agent: * Disallow: /private Allow: /