A founder, wary of AI companies scraping their content to train models for free, adds a blanket robots.txt rule blocking every bot with 'GPT' or 'Claude' in its name. Reasonable instinct, wrong outcome: months later, competitors show up in ChatGPT search results for the founder's own product category and the founder's own site never does, even though the site ranks fine on Google. The rule that felt privacy-protective also silently unplugged the site from AI search entirely, because the block was aimed at the wrong bot.
Why do OpenAI and Anthropic each run more than one crawler?
Because training a model and answering a live search query are different jobs with different consequences if you block them. OpenAI's own crawler documentation separates GPTBot, which collects content as potential training material for future models, from OAI-SearchBot, which is used specifically to surface and link to sites in ChatGPT search results, and from ChatGPT-User, the live fetch triggered when a person clicks a citation inside a ChatGPT conversation.
Anthropic runs the same three-way split: ClaudeBot for training data collection, Claude-SearchBot for indexing content for search-style answers, and Claude-User for real-time fetches a live user's question triggers. Each one reads a separate line in robots.txt, and blocking one has zero effect on the others.
What actually happens if you block the wrong one?
| User-agent | Job | If you block it |
|---|---|---|
| GPTBot | Trains future OpenAI models on your content | Content excluded from training data. No measurable effect on ChatGPT search visibility. |
| OAI-SearchBot | Crawls and indexes your site for ChatGPT search results | You disappear from ChatGPT's search-style answers and citations entirely. |
| ChatGPT-User | Live fetch when a ChatGPT user clicks through to your page | That specific click-through fails for the user, breaking the citation experience. |
| ClaudeBot | Trains future Anthropic models on your content | Content excluded from training data. No measurable effect on Claude's answer citations. |
| Claude-SearchBot | Indexes your content for Claude's search-style answers | You disappear from Claude's cited search results. |
| Claude-User | Live fetch when a Claude user's query needs your page | That specific real-time fetch fails. |
Each user-agent is controlled independently in robots.txt. A wildcard rule or a single named block does not extend to the others.
So what is the actual right robots.txt strategy for 2026?
- Decide separately whether you want your content used for model training versus whether you want to appear in AI-generated search answers, since these are genuinely different tradeoffs with different upside, not one setting.
- If you want AI search visibility, explicitly allow OAI-SearchBot, Claude-SearchBot and PerplexityBot by their exact user-agent strings, don't rely on a generic 'allow all' rule, since some CDN and bot-management layers apply their own blanket AI-bot blocks underneath your robots.txt that you have to separately check.
- If you specifically want to opt out of model training while keeping search visibility, disallow GPTBot and ClaudeBot by name while explicitly allowing OAI-SearchBot, Claude-SearchBot and ChatGPT-User, exactly the granular split OpenAI and Anthropic both document as independently controllable.
- Verify with a real fetch, not an assumption: request a key page with the exact crawler user-agent string set and confirm you get your real content back with a 200 status, not a CAPTCHA wall, a redirect loop, or a blank shell from unrendered JavaScript.
Why is this mistake so easy to make without noticing?
Because the failure is invisible in the tools most site owners already check. Google Search Console reports nothing about GPTBot, OAI-SearchBot or ClaudeBot access, so a site can look completely healthy in the dashboard a founder actually looks at every week, while quietly returning nothing to every AI crawler that matters for the newer half of search traffic.
It compounds with a second common mistake covered in setting up llms.txt and structured data: a site can have a perfect llms.txt file and flawless JSON-LD schema and still be invisible to AI search if the underlying robots.txt is quietly blocking the exact crawler that would have read them.
How AIBOOTSTRAPPER solved this for a client
PropLock, the UK real estate platform AIBOOTSTRAPPER built, reached 12,000 organic visitors a month within 90 days of launch, and part of that number depends on a detail as unglamorous as robots.txt being configured correctly for every AI crawler role from day one, not just Googlebot. A GEO-optimized site with excellent structured listing data delivers zero AI-search value if the crawler responsible for indexing it into AI search results is blocked by an overly broad rule nobody audited.
That is the same discipline behind tracking which pages are actually earning AI citations: visibility has to be verified at the crawler level, not assumed from a clean Google Search Console report.
How AIBOOTSTRAPPER helps
Every site AIBOOTSTRAPPER builds ships with an audited robots.txt that explicitly allows the search-facing AI crawlers by name, as part of the same GEO-optimized build process that covers structured data and llms.txt.
If you are not sure whether your current robots.txt is quietly blocking ChatGPT or Claude search visibility, book a call and we will run the crawler-level check before you spend another month wondering why competitors get cited and you don't.
Want this done for you?
Book a free strategy call and we'll show you how to build and market your business with AI.