How to
AI crawlers and robots.txt
Whether to block GPTBot is a decision about model training, and it is yours to make. OpenAI says GPTBot gathers content that may be used to train its models, and that a separate crawler, OAI-SearchBot, is the one behind ChatGPT’s search features. Training crawlers and search crawlers have different names, and this page takes each one separately.
Updated 8 October 2026 · 6 minute read
Dated, and not independently verified. The table below was
compiled on 19 September 2026 from what each vendor publishes about its own
crawlers. On 8 October 2026 we re-read the OpenAI and Anthropic rows against
those companies’ own pages; the Perplexity, Google, Microsoft, Apple and Common
Crawl rows were not re-read that day. We have not audited any crawler from the
outside, and these names and behaviours change without much notice. Check the
vendor’s own current documentation before you edit a production
robots.txt on the strength of this page.
The three jobs a crawler can be doing
Nearly all the confusion in this area comes from treating “AI crawler” as one thing. There are three distinct jobs, and OpenAI and Anthropic each use a different user agent for each:
- Training. Collecting text that may be used to train a model. A block here is a request about training. OpenAI and Anthropic each name a crawler for this job that is separate from the one they use for search.
- Search indexing. Reading pages so they can be shown when the assistant searches. OpenAI and Anthropic both tie this job to visibility. OpenAI says a site opted out of its search crawler will not be shown in ChatGPT search answers, though it can still appear as a navigational link. Anthropic says blocking its search crawler may reduce your site’s visibility.
- User-triggered fetch. The assistant opens a page because a person asked it something. The operators differ here. OpenAI says that because a user starts these visits, robots.txt rules may not apply to ChatGPT-User. Anthropic says blocking Claude-User stops Claude retrieving your content for a user’s query.
The crawlers, and which job each does
| Crawler | Job | What a block means |
|---|---|---|
GPTBot, OpenAI | Training | OpenAI says this indicates your content should not be used in training its generative AI models. It is not the crawler OpenAI uses for ChatGPT search. |
OAI-SearchBot, OpenAI | Search indexing | OpenAI says a site opted out of it will not be shown in ChatGPT search answers, though it can still appear as a navigational link. |
ChatGPT-User, OpenAI | User-triggered fetch | The block may not hold. OpenAI says ChatGPT may visit a page when a user asks a question, and that because a user starts these visits, robots.txt rules may not apply. |
PerplexityBot, Perplexity | Search indexing | You tell the crawler Perplexity reads pages with not to read yours. A crawler that follows the rule does not read them, so it brings Perplexity nothing from them to quote. Perplexity can still name you from what other sites say. Not re-read on 8 October 2026. |
Perplexity-User, Perplexity | User-triggered fetch | When we compiled this table, Perplexity documented this as acting for a user, so robots.txt may not stop it. |
ClaudeBot, Anthropic | Training | Anthropic says your site’s future materials should then be excluded from its AI model training datasets. |
Claude-SearchBot, Anthropic | Search indexing | Anthropic says this stops your content being indexed for search, which may reduce your site’s visibility. |
Claude-User, Anthropic | User-triggered fetch | Anthropic says this stops Claude retrieving your content in response to a user’s query, which may reduce your site’s visibility. |
Googlebot, | Search indexing, and AI Overviews | You tell Google not to crawl your pages for Search. AI Overviews are part of Search, so Googlebot governs them too. Not re-read on 8 October 2026. |
Google-Extended, | A permission token, not a crawler | Controls whether Google may use your content for Gemini apps and Vertex AI. It does not affect Google Search, and it does not affect AI Overviews. Not re-read on 8 October 2026. |
Bingbot, Microsoft | Search indexing, and Copilot | You tell Microsoft not to crawl your pages for Bing. When we compiled this table, Copilot drew on the same crawl. |
Applebot-Extended, Apple | A permission token | When we compiled this table, Apple documented this as opting your content out of its generative training without affecting Applebot’s search crawl. |
CCBot, Common Crawl | Open corpus | You ask to be left out of Common Crawl, a public dataset many models are built from. Whether to block it is your decision. Not re-read on 8 October 2026. |
The two traps
Trap one: blocking one kind of crawler and expecting the effect of the
other. A site that disallows GPTBot and ClaudeBot
and leaves the search bots alone has made a request about training only; OpenAI and
Anthropic each use a different crawler for search.
More detail
A site that does the reverse, blocking the search bots and allowing training, has opted out of the crawlers those two companies tie to visibility while still offering its text for training.
Either can be a deliberate choice. Either can also be an accident, so check which one your file actually says.
Trap two: a blanket User-agent: * rule. Some sites
carry a broad disallow written for a crawler nobody remembers, or inherited from a
plugin, a staging config or a template.
It applies to every crawler the file does
not name, and that includes the AI crawlers. If you have never read your own
robots.txt line by line, read it before you look for any other reason
you are not being cited.
Deciding what to allow
There is a real trade-off here and we are not going to pretend otherwise. Letting a training crawler in offers your work for use in training a model.
More detail
Letting a search crawler in is what OpenAI and Anthropic tie to visibility in their search. The two are separable, so decide them separately. The table sets out what each kind of site is weighing. The choice is yours.
| If you are… | Training crawlers | Search and answer crawlers |
|---|---|---|
| A business that wants recommending | Your decision. OpenAI and Anthropic each use a separate crawler for search, so this one is about training. | A block works against the goal. OpenAI says an opted-out site will not be shown in ChatGPT search answers, and Anthropic says a block may reduce your site’s visibility. |
| A publisher selling content | A block asks that your work is not used for training. Licensing it is the other route. | You are weighing referral traffic against answers that replace the visit. |
| A docs or support site | Your decision. The case for allowing is that a model may then know your product; nothing guarantees it. | The case for allowing is that an assistant that may read your docs can answer from them. |
| Anything private or paywalled | A block is only a request. Do not rely on this file for it. | The same. Enforce it with a real login. |
robots.txt is a request, not a control. It is honoured voluntarily. Anything that must not be read by a machine needs authentication, not a directive in a text file. Treat robots.txt as a preference expressed to well-behaved crawlers, and nothing more.
What ours says, and why
Bungad’s robots.txt allows everything and points at the sitemap. We are a business that wants to be found by assistants, we have nothing paywalled, and we would rather our pages be readable by every crawler that asks, training crawlers included. That is our choice for our own site. It is not advice: your trade-off may be different.
The failures robots.txt will not explain
If your file is clean and you are still never retrieved, the block is somewhere else. The places to look:
- Bot protection at the CDN. A managed challenge or a “block AI bots” toggle at the edge can stop crawlers that robots.txt allows, and it leaves no trace in your site config. It may be on without anyone on your team having chosen it. Check it.
- JavaScript-only content. If the text only appears after a client render, a crawler that does not execute scripts sees an empty shell. Fetch your own page with scripting off and read what is actually there.
- Login, email or consent walls. Anything a crawler cannot click through is a page it cannot read.
- Facts locked in PDFs, images and video. A price in a picture cannot be quoted by anything, ever.
- Slow or unstable responses. Crawlers give up. A page that takes ten seconds under load is a page that sometimes does not exist.
You can confirm most of this from your own server logs, for free, without any tool — the method is on checking whether AI mentions your brand.
Common questions
Should I block GPTBot in robots.txt?
That is a decision about model training, and it is yours to make. OpenAI says GPTBot crawls content that may be used in training its generative AI foundation models, and that disallowing it indicates your content should not be used for that. GPTBot is a different crawler from OAI-SearchBot, which OpenAI says is used to surface websites in ChatGPT's search features, and from ChatGPT-User, which may visit a page when a user asks ChatGPT a question. If your goal is to be recommended by assistants, the search crawlers are the ones OpenAI and Anthropic tie to visibility: OpenAI says a site opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though it can still appear as a navigational link.
Does Google-Extended stop my site appearing in AI Overviews?
No. Google-Extended controls whether Google may use your content for Gemini apps and Vertex AI. It does not affect Google Search, and it does not affect AI Overviews, which are part of Search and governed by Googlebot. A rule for Google-Extended therefore does not take you out of AI Overviews.
Will blocking AI crawlers protect my content?
Not reliably. robots.txt is a request that a crawler chooses to honour, not a lock. OpenAI says robots.txt rules may not apply to ChatGPT-User, because those visits are started by a user. Anything that genuinely must not be read by a machine needs authentication. Treat robots.txt as a preference, not a control.
My robots.txt is fine and I am still never cited. What now?
Check the edge before the origin: a bot-protection rule at your CDN can block crawlers that robots.txt allows. After that, check whether your content exists without JavaScript, whether anything sits behind a login or consent wall, and whether your key facts are trapped in PDFs or images. If all of that is clean, the likelier cause is that no page of yours answers the exact question being asked.
More guides
Read next
Check that you are actually reachable
We ask Perplexity three of your buyers’ questions and show you the sources it cites. We also read your robots.txt and tell you if it tells PerplexityBot not to read your home page. We check that rule for the home page only, and Perplexity is the only engine we measure today.
3 questions on Perplexity, free. A domain checked in the last week is served from that stored run rather than asked again, and the daily free allowance resets at midnight UTC.