# https://en.wikipedia.org/wiki/Robots_exclusion_standard User-Agent: * Disallow: /downloadables/ Crawl-delay: 3 # Block common crawler: