# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file User-agent: * Disallow: /signup/ Disallow: /activity/ Disallow: /main/ Disallow: /admin/ Disallow: /payment Disallow: /Dashboard # this is for these wonderful people opensiteexplorer.org/dotbot; help@moz.com Crawl-delay: 5