AI bots robots.txt generator

This page handles robots.txt from the angle of AI agents: who may train, who may cite and who may fetch on a person's request. It writes the entire file, general rules included, because robots.txt is a single file and shipping half of one would ship something broken. If your question is broader, whether the site as a whole is ready to be cited by AI, the right place is the GEO readiness checker, where crawler access is 1 item out of 16.

How to use it

1. Click a stance to fill the table in one move. The recommended one for publishers is block training and keep citation, which is already selected.
2. Adjust agent by agent across the three blocks. Each row carries the exact user-agent token, the vendor and the labels that change the consequence: classic search, training and answers, may ignore the file.
3. Write your private paths, one per line. They apply to everyone, including the agents you allowed, and you can paste a full URL: the tool extracts the path.
4. Fill in the sitemap. It is the cheapest line in the file and the only one that actively helps whoever crawls properly.
5. Read the alerts before you copy. They do not judge your choice, they show its consequence, such as blocking a token that is also the traditional search crawler.
6. Download and publish at yoursite.com/robots.txt, at the root and as plain text. Check it in a private window.

How it works: three purposes, not one block

The snippet going around the internet treats every AI bot as the same thing and sends Disallow to fifteen user-agents. The result is a site that vanishes from AI answers and gains nothing, because among those fifteen were the agents that cite with a link. Vendors split their agents by purpose, and an honest policy follows that split.

The same brand shows up in all three blocks under different tokens. OpenAI splits GPTBot, OAI-SearchBot and ChatGPT-User. Anthropic splits ClaudeBot, Claude-SearchBot and Claude-User. Perplexity splits PerplexityBot and Perplexity-User. Google and Apple do it differently: the training refusal token stands alone (Google-Extended, Applebot-Extended) and indexing stays with the usual crawler. That is why the table shows the vendor next to the token: the decision is by purpose, not by brand.

The grouping rule almost every generator gets wrong

A crawler reads robots.txt and obeys one group only, the most specific one matching its name. Every other group stops existing for it. That creates a silent trap: if you open a group for OAI-SearchBot with just Allow slash, the Disallow lines in your general group stop applying to that agent, and your checkout folder is open precisely to the crawler you allowed.

This tool repeats the private paths inside every allowed group for that reason. The Allow slash comes after and cancels nothing: under the longest match rule, /admin/ is 7 characters and beats a lone slash, which is 1. It is the same logic that makes a specific exception outweigh a generic ban.

Worked example (reproduces the default output)

The values that come prefilled apply the recommended stance to a site with three private paths and one sitemap. The generated file has 51 lines and 4 groups, and the summary shows:

• Training: all 12 of 12 training crawlers land in a single group with Disallow slash. One group may list several user-agents before the rules, which shortens the file without changing its meaning.
• Citation: 13 of 13 agents able to cite you stay allowed, 8 from search and 5 user requested. The citation surface stays whole.
• Each of those two allowed groups repeats the three private Disallow lines before the Allow slash, which is why the file has 4 groups instead of 2.
• Two alerts show up, and neither is an error: blocked Google-Extended warns that you also leave Gemini grounded answers, and Bytespider warns that the vendor makes no commitment to the file.

Click allow everything and the summary turns into 0 of 12 blocked with a note that the file changes nothing. Click close the door to AI and the citation surface drops to 3 of 13, which are exactly the classic search crawlers the tool leaves checked on purpose, because unchecking Googlebot does not close the door to AI, it closes the door to search itself.

How to read it and where the file misleads

The first trap is believing robots.txt protects content. It does not. It is a request published in a text file with no authentication at all, which reduces collection by whoever chooses to comply and does nothing else. It does not stop manual copying, it does not remove your text from datasets already distributed and it creates no right you did not have. If a block has to hold, it lives at the server, the CDN or the WAF, by user-agent and by the IP ranges the vendor publishes.

The second is blocking out of panic. Blocking training costs close to nothing, and blocking search costs traffic you will never watch disappear in a report, because there is no line called the visit that never happened. Before you close the citation door, look at how much of your topic is already answered on screen without a click: the method is in zero-click search optimization.

The third is confusing layers. Allowing the crawler does not produce citations: it still has to find a liftable passage when it arrives. Once access is settled, run the text through the AI citability checker and point at your canonical content with the llms.txt generator. Access, snippet and trust signal are three different problems, and solving the first alone moves nothing.

The fourth is forgetting that the list ages. A new agent shows up every quarter, and a file written in 2024 already misses half of what exists today. Put the robots.txt review on the same cadence as your site menu review.

From policy to evidence

Changing your bot policy is a bet on how third parties treat your content, and a bet without measurement becomes team folklore. The short, honest path:

• Keep the previous file and the date of the switch. Without a baseline, any later swing becomes free interpretation.
• Watch the server log by user-agent, not only the search console. That is where you can see who stopped knocking.
• Track mentions and citations in AI answers with the method in the GEO guide, and the FAQ structure that earns the most citations in FAQ content for AI citation.
• When the change also touches the page, not just the file, measure it with an A/B test before rolling it out. The link between optimizing for AI and converting humans is in does A/B testing affect AI citations.

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT?

Not the way most people assume. OpenAI uses different tokens for different jobs: GPTBot crawls for model training, OAI-SearchBot indexes for ChatGPT search and ChatGPT-User fetches the page when a user asks right then. Blocking GPTBot alone refuses training and keeps the two doors that bring cited links back to you. Whoever wants to actually disappear from the answers has to block all three, and that costs the traffic too. This is exactly the distinction the tool forces you to make before it writes the file.

What is the difference between Google-Extended and Googlebot?

Googlebot is the search crawler: it feeds the Google index, and the index feeds both the blue links and the AI generated overviews inside search. Google-Extended is not a crawler, it is a control token that says whether your content may be used for Gemini training and its grounded answers. The practical consequence is hard to swallow: blocking Google-Extended does not pull you out of search AI overviews, and blocking Googlebot to achieve that would pull you out of Google entirely. There is no separate control for overviews today.

Does it make sense to block training and allow citation?

It does, and it is the default stance for anyone who lives by publishing. The two have opposite economics: in training your text becomes model weights, with no visit and no credit; in citation it shows up as a source with a link, which is traffic and authority. Because most vendors split their agents, you can refuse one and accept the other in the same file. The honest exception is agents that do not split, such as Common Crawl CCBot: there it is all or nothing, and it is worth remembering that its corpus already circulates in datasets published before your block.

Do these bots really honor robots.txt?

The large vendors document that they do, and there is independent evidence that agents with a public token comply in practice. But the file is a request, not a lock: there is no authentication, and any client can ignore the rule or change its user-agent. Some user triggered agents, such as Perplexity-User, are documented as outside the scope of robots.txt precisely because a person fired the request. If a block has to hold, it has to exist at the server, the CDN or the WAF, by user-agent and by the IP ranges the vendor publishes.

Why does the tool repeat private paths inside every allowed group?

Because a crawler obeys one group only, the most specific one matching its name, and ignores every other group. That creates a silent trap: if you open a group for OAI-SearchBot with just Allow slash, the Disallow lines you wrote in the User-agent asterisk group stop applying to it, and your checkout folder is wide open precisely to the agent you allowed. Repeating the private paths in each group is the correct fix and the most common mistake in generic generators. The Allow slash comes after and cancels nothing, because the longest match rule makes /admin/ beat /.

Where do I publish the file and how do I check it works?

At the root of the domain, at yoursite.com/robots.txt, served as plain text. It applies per host and per protocol: a subdomain has its own file, and the www one does not cover the apex when both answer separately. After publishing, open the address in a private window and confirm the raw text shows up. To test rule by rule, the Search Console robots.txt report shows which line matches which URL for Googlebot. For AI agents there is no official tester: what is left is reading your server log and seeing who stopped knocking.

Does blocking AI protect my content from being copied?

No. robots.txt reduces automated collection by whoever chooses to comply, and nothing else. It does not stop manual copying, it does not stop an anonymous scraper, it does not remove your content from datasets already published and it creates no right you did not already have. Treating the file as legal protection or as a technical barrier is the most expensive mistake in this debate, because it trades real visibility for security that does not exist. Decide by the economics of your operation, not by the feeling of locking a door.

Keep going

With the file published, the next step is the layer it does not solve: liftable text, verifiable data and authorship signals. The full method is in the GEO guide, and what changes in conversion work when the answer arrives before the click is in CRO in the age of AI.