robots.txt Generator
Free robots.txt generator. Set your user-agent, allow and disallow paths, crawl-delay and sitemap URL to instantly build a valid robots.txt you can copy straight to your site root.
A robots.txt file lives at the root of your domain and tells search-engine
crawlers which parts of your site they may or may not request. Getting it right
matters: a stray Disallow: / can hide your whole site from Google,
while a missing rule can let bots crawl pages you would rather keep out of the
index. This generator builds the file for you from simple inputs so you avoid
syntax mistakes.
Add the user-agent you want to target, list the paths to allow and disallow
one per line, and optionally set a crawl-delay and your sitemap URL. The output
updates live and follows the standard robots exclusion format, so you can copy it
straight into a robots.txt file at the top level of your web server.
Frequently asked questions
Where do I put the robots.txt file?
It must sit at the root of your domain, reachable at https://yourdomain.com/robots.txt. Crawlers only look in that exact location.
What does User-agent: * mean?
The asterisk is a wildcard that applies your rules to every crawler. You can target a specific bot instead, such as Googlebot, by entering its name.
Is crawl-delay supported by Google?
Google ignores the crawl-delay directive and uses its own crawl-rate settings, but Bing and several other crawlers do respect it.
Does disallowing a page remove it from Google?
Not necessarily. Disallow stops crawling, but a blocked URL can still be indexed if linked elsewhere. Use a noindex meta tag to keep a page out of search results.
Add the robots.txt Generator to your site.
Ship it as a ranking page on your own domain in minutes.
Start free →