Start free

How to

AI crawlers and robots.txt

Whether to block GPTBot is a decision about model training, and it is yours to make. OpenAI says GPTBot gathers content that may be used to train its models, and that a separate crawler, OAI-SearchBot, is the one behind ChatGPT’s search features. Training crawlers and search crawlers have different names, and this page takes each one separately.

Updated 8 October 2026 · 6 minute read

Dodong, the Bungad mascot, on a meeting-room desk in a high-rise office while colleagues work on laptops behind him
Dodong, the Bungad mascot, studying a stack of blank cards through a magnifying glass on a gold background

Dated, and not independently verified. The table below was compiled on 19 September 2026 from what each vendor publishes about its own crawlers. On 8 October 2026 we re-read the OpenAI and Anthropic rows against those companies’ own pages; the Perplexity, Google, Microsoft, Apple and Common Crawl rows were not re-read that day. We have not audited any crawler from the outside, and these names and behaviours change without much notice. Check the vendor’s own current documentation before you edit a production robots.txt on the strength of this page.

robots.txt / Three jobsDecide each
TrainingA block here is a request about trainingSeparate
Search indexingThe job tied to being shown in assistant searchVisibility
User-triggered fetchA page fetched when someone asksSeparate
Three jobs, three decisions. Training and search are separable.
Dodong, the Bungad mascot, on a navy studio table among sketchbooks and fabric swatches while designers talk at desks behind him

The three jobs a crawler can be doing

Nearly all the confusion in this area comes from treating “AI crawler” as one thing. There are three distinct jobs, and OpenAI and Anthropic each use a different user agent for each:

  • Training. Collecting text that may be used to train a model. A block here is a request about training. OpenAI and Anthropic each name a crawler for this job that is separate from the one they use for search.
  • Search indexing. Reading pages so they can be shown when the assistant searches. OpenAI and Anthropic both tie this job to visibility. OpenAI says a site opted out of its search crawler will not be shown in ChatGPT search answers, though it can still appear as a navigational link. Anthropic says blocking its search crawler may reduce your site’s visibility.
  • User-triggered fetch. The assistant opens a page because a person asked it something. The operators differ here. OpenAI says that because a user starts these visits, robots.txt rules may not apply to ChatGPT-User. Anthropic says blocking Claude-User stops Claude retrieving your content for a user’s query.
Dodong, the Bungad mascot, on a walnut desk beside an open card-catalogue drawer of blank index cards, a card lifted by its gold tab

The crawlers, and which job each does

Crawlers / robots.txtJob
Training crawlerA block is a request about trainingTraining
Search crawlerA block can keep you out of that company’s search answersSearch
User-triggered fetcherFetches a page when a user asksUser fetch
A company can run separate crawlers for separate jobs. The table names them.
CrawlerJobWhat a block means
GPTBot,
OpenAI
TrainingOpenAI says this indicates your content should not be used in training its generative AI models. It is not the crawler OpenAI uses for ChatGPT search.
OAI-SearchBot,
OpenAI
Search indexingOpenAI says a site opted out of it will not be shown in ChatGPT search answers, though it can still appear as a navigational link.
ChatGPT-User,
OpenAI
User-triggered fetchThe block may not hold. OpenAI says ChatGPT may visit a page when a user asks a question, and that because a user starts these visits, robots.txt rules may not apply.
PerplexityBot,
Perplexity
Search indexingYou tell the crawler Perplexity reads pages with not to read yours. A crawler that follows the rule does not read them, so it brings Perplexity nothing from them to quote. Perplexity can still name you from what other sites say. Not re-read on 8 October 2026.
Perplexity-User,
Perplexity
User-triggered fetchWhen we compiled this table, Perplexity documented this as acting for a user, so robots.txt may not stop it.
ClaudeBot,
Anthropic
TrainingAnthropic says your site’s future materials should then be excluded from its AI model training datasets.
Claude-SearchBot,
Anthropic
Search indexingAnthropic says this stops your content being indexed for search, which may reduce your site’s visibility.
Claude-User,
Anthropic
User-triggered fetchAnthropic says this stops Claude retrieving your content in response to a user’s query, which may reduce your site’s visibility.
Googlebot,
Google
Search indexing, and AI OverviewsYou tell Google not to crawl your pages for Search. AI Overviews are part of Search, so Googlebot governs them too. Not re-read on 8 October 2026.
Google-Extended,
Google
A permission token, not a crawlerControls whether Google may use your content for Gemini apps and Vertex AI. It does not affect Google Search, and it does not affect AI Overviews. Not re-read on 8 October 2026.
Bingbot,
Microsoft
Search indexing, and CopilotYou tell Microsoft not to crawl your pages for Bing. When we compiled this table, Copilot drew on the same crawl.
Applebot-Extended,
Apple
A permission tokenWhen we compiled this table, Apple documented this as opting your content out of its generative training without affecting Applebot’s search crawl.
CCBot,
Common Crawl
Open corpusYou ask to be left out of Common Crawl, a public dataset many models are built from. Whether to block it is your decision. Not re-read on 8 October 2026.
Dodong, the Bungad mascot, beside miniature navy garden gates, a path of blank cards running through the open gate and stopping short of a closed gate

The two traps

Trap one: blocking one kind of crawler and expecting the effect of the other. A site that disallows GPTBot and ClaudeBot and leaves the search bots alone has made a request about training only; OpenAI and Anthropic each use a different crawler for search.

More detail

A site that does the reverse, blocking the search bots and allowing training, has opted out of the crawlers those two companies tie to visibility while still offering its text for training.

Either can be a deliberate choice. Either can also be an accident, so check which one your file actually says.

Trap two: a blanket User-agent: * rule. Some sites carry a broad disallow written for a crawler nobody remembers, or inherited from a plugin, a staging config or a template.

It applies to every crawler the file does not name, and that includes the AI crawlers. If you have never read your own robots.txt line by line, read it before you look for any other reason you are not being cited.

Example / robots.txt, line by lineTwo traps
Search bots blocked, training allowedOut of the crawlers tied to visibility, still offered for trainingCheck
User-agent: * with a disallowApplies to every crawler the file does not name, AI crawlers includedTrap
Example: both can be deliberate, and both can be an accident. Read your own file.
Dodong, the Bungad mascot, beside a brass balance scale weighing a small wooden toolbox against a stack of blank paper

Deciding what to allow

There is a real trade-off here and we are not going to pretend otherwise. Letting a training crawler in offers your work for use in training a model.

More detail

Letting a search crawler in is what OpenAI and Anthropic tie to visibility in their search. The two are separable, so decide them separately. The table sets out what each kind of site is weighing. The choice is yours.

If you are…Training crawlersSearch and answer crawlers
A business that wants recommending Your decision. OpenAI and Anthropic each use a separate crawler for search, so this one is about training. A block works against the goal. OpenAI says an opted-out site will not be shown in ChatGPT search answers, and Anthropic says a block may reduce your site’s visibility.
A publisher selling content A block asks that your work is not used for training. Licensing it is the other route. You are weighing referral traffic against answers that replace the visit.
A docs or support site Your decision. The case for allowing is that a model may then know your product; nothing guarantees it. The case for allowing is that an assistant that may read your docs can answer from them.
Anything private or paywalled A block is only a request. Do not rely on this file for it. The same. Enforce it with a real login.

robots.txt is a request, not a control. It is honoured voluntarily. Anything that must not be read by a machine needs authentication, not a directive in a text file. Treat robots.txt as a preference expressed to well-behaved crawlers, and nothing more.

Dodong, the Bungad mascot, on a desk beside a printed page of grey lines, a line near the top marked with a gold highlighter

What ours says, and why

Bungad’s robots.txt allows everything and points at the sitemap. We are a business that wants to be found by assistants, we have nothing paywalled, and we would rather our pages be readable by every crawler that asks, training crawlers included. That is our choice for our own site. It is not advice: your trade-off may be different.

Bungad / robots.txtbungad.com
Every crawlerTraining crawlers includedAllowed
SitemapListed in the fileLinked
Our choice for our own site, not advice for yours.
Dodong, the Bungad mascot, on a workbench beside a navy tool caddy holding a hammer, a pencil, a putty knife and gold tape

The failures robots.txt will not explain

If your file is clean and you are still never retrieved, the block is somewhere else. The places to look:

Diagnose / Other blocksCheck
Bot protection at the CDNCan stop crawlers the file allowsCheck
JavaScript-only contentA plain fetch sees an empty pageCheck
The section lists the other places to look.
  1. Bot protection at the CDN. A managed challenge or a “block AI bots” toggle at the edge can stop crawlers that robots.txt allows, and it leaves no trace in your site config. It may be on without anyone on your team having chosen it. Check it.
  2. JavaScript-only content. If the text only appears after a client render, a crawler that does not execute scripts sees an empty shell. Fetch your own page with scripting off and read what is actually there.
  3. Login, email or consent walls. Anything a crawler cannot click through is a page it cannot read.
  4. Facts locked in PDFs, images and video. A price in a picture cannot be quoted by anything, ever.
  5. Slow or unstable responses. Crawlers give up. A page that takes ten seconds under load is a page that sometimes does not exist.

You can confirm most of this from your own server logs, for free, without any tool — the method is on checking whether AI mentions your brand.

Dodong, the Bungad mascot, standing beside a glowing gold speech bubble on a navy background

Common questions

Should I block GPTBot in robots.txt?

That is a decision about model training, and it is yours to make. OpenAI says GPTBot crawls content that may be used in training its generative AI foundation models, and that disallowing it indicates your content should not be used for that. GPTBot is a different crawler from OAI-SearchBot, which OpenAI says is used to surface websites in ChatGPT's search features, and from ChatGPT-User, which may visit a page when a user asks ChatGPT a question. If your goal is to be recommended by assistants, the search crawlers are the ones OpenAI and Anthropic tie to visibility: OpenAI says a site opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though it can still appear as a navigational link.

Does Google-Extended stop my site appearing in AI Overviews?

No. Google-Extended controls whether Google may use your content for Gemini apps and Vertex AI. It does not affect Google Search, and it does not affect AI Overviews, which are part of Search and governed by Googlebot. A rule for Google-Extended therefore does not take you out of AI Overviews.

Will blocking AI crawlers protect my content?

Not reliably. robots.txt is a request that a crawler chooses to honour, not a lock. OpenAI says robots.txt rules may not apply to ChatGPT-User, because those visits are started by a user. Anything that genuinely must not be read by a machine needs authentication. Treat robots.txt as a preference, not a control.

My robots.txt is fine and I am still never cited. What now?

Check the edge before the origin: a bot-protection rule at your CDN can block crawlers that robots.txt allows. After that, check whether your content exists without JavaScript, whether anything sits behind a login or consent wall, and whether your key facts are trapped in PDFs or images. If all of that is clean, the likelier cause is that no page of yours answers the exact question being asked.

More guides

Read next

Check that you are actually reachable

We ask Perplexity three of your buyers’ questions and show you the sources it cites. We also read your robots.txt and tell you if it tells PerplexityBot not to read your home page. We check that rule for the home page only, and Perplexity is the only engine we measure today.

3 questions on Perplexity, free. A domain checked in the last week is served from that stored run rather than asked again, and the daily free allowance resets at midnight UTC.

Dodong, the Bungad mascot, waving on a gold background