Robots.txt Generator
AI and answer-engine crawlers
Ticked means allowed. Unticked writes a Disallow rule asking that crawler to stay out.
Your robots.txt
Test a URL against these rules
What robots.txt actually does
It asks well-behaved crawlers not to fetch certain paths. That is all. It is not access control, it is not a way to hide anything, and — the part that catches people out — it does not remove a page from search results. A blocked URL can still be listed, just without a description.
Blocking versus hiding
- To stop a page appearing in search — let it be crawled and add
<meta name="robots" content="noindex">. A blocked page cannot be crawled, so the noindex is never seen. - To keep something private — put it behind a login. Anyone can read robots.txt, so listing a secret folder there advertises it.
- To save crawl budget — this is what robots.txt is genuinely good at: keeping crawlers out of faceted search, filters and endless parameter URLs.
The AI crawlers
These are separate from search crawlers and mostly honour robots.txt by policy rather than by force. Blocking them removes your content from AI training and, in some cases, from AI answers — which is increasingly where people start looking. Worth a deliberate decision rather than a default.
Share this tool with friends
Free to use, no sign-up, works on any phone.
Frequently Asked Questions
No, and this is the most expensive misunderstanding in SEO. It stops crawling, not indexing — a blocked page can still appear in results, listed without a description, if other sites link to it. To keep a page out, allow the crawl and use a noindex meta tag.
You can ask. The major crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended — publish user-agent names and say they honour robots.txt. It is a request, not a wall: nothing technically prevents a crawler that ignores the file.
Only if you understand the trade. It controls whether your content can be used to ground AI answers — including Google's own AI Overviews — so blocking it can remove you from a surface where people increasingly look for answers. It does not affect ordinary search ranking.
The root of the domain, at /robots.txt, and nowhere else. A file at /blog/robots.txt is ignored completely. Each subdomain needs its own.
Google ignores it outright. Bing and Yandex honour it. If Googlebot is genuinely overloading your server, set the crawl rate in Search Console instead.
It helps, and it costs nothing. Search engines read the Sitemap line even when they arrived without being told, and it applies to every crawler rather than one you submitted it to.