AI crawlers and robots.txt

The three jobs a crawler can be doing

Nearly all the confusion in this area comes from treating “AI crawler” as one thing. There are three distinct jobs, and most vendors use a different user agent for each:

• Training. Collecting text to train a future model. Blocking this affects models that do not exist yet, and nothing you see today.
• Search indexing. Building the index the assistant searches when somebody asks a question. Blocking this is how you actually disappear from AI answers.
• User-triggered fetch. A person pasted your URL, or the assistant decided to open it mid-conversation. Several vendors treat this as a user action rather than crawling, and say it does not follow the usual robots.txt rules.

The crawlers, and which job each does

The two traps

Trap one: blocking the training bot and expecting to disappear from answers. A site that disallows GPTBot and ClaudeBot and leaves the search bots alone has changed nothing about what an assistant says today. A site that does the reverse — blocks the search bots and allows training — has made itself uncitable while still contributing its text to the next model. Both happen constantly, and both are usually accidental.

Trap two: a blanket User-agent: * rule. Plenty of sites carry a broad disallow written for a crawler nobody remembers, or inherited from a plugin, a staging config or a template. It now applies to every AI crawler by default. If you have never read your own robots.txt line by line, read it before you conclude that you are invisible for any other reason.

Deciding what to allow

There is a real trade-off here and we are not going to pretend otherwise. Allowing training crawlers contributes your work to a model you get nothing from. Allowing search crawlers is the price of being in the answers. They are separable, so decide separately:

What ours says, and why

Bungad’s robots.txt allows everything and points at the sitemap. We are a business that wants to be found by assistants, we have nothing paywalled, and we would rather our pages be readable by every crawler that asks. That is the simplest honest position for most companies in our position, and it is the one we would suggest first.

The failures robots.txt will not explain

If your file is clean and you are still never retrieved, the block is somewhere else. In rough order of how often we see it:

1. Bot protection at the CDN. A managed challenge or a “block AI bots” toggle at the edge will stop crawlers that robots.txt happily allows, and it leaves no trace in your site config. Several CDNs turned such a toggle on by default. Check it.
2. JavaScript-only content. If the text only appears after a client render, a crawler that does not execute scripts sees an empty shell. Fetch your own page with scripting off and read what is actually there.
3. Login, email or consent walls. Anything a crawler cannot click through is a page it cannot read.
4. Facts locked in PDFs, images and video. A price in a picture cannot be quoted by anything, ever.
5. Slow or unstable responses. Crawlers give up. A page that takes ten seconds under load is a page that sometimes does not exist.

You can confirm most of this from your own server logs, for free, without any tool — the method is on checking whether AI mentions your brand.

Common questions