AI crawlers and robots.txt: which bots to allow for AI search
Short answer
To appear in AI search answers, allow the search crawlers of each AI assistant in robots.txt and in your firewall: OAI-SearchBot (ChatGPT), Claude-SearchBot (Claude), PerplexityBot (Perplexity), Googlebot (Google AI Overviews, AI Mode and Gemini) and Bingbot (Copilot). You can still opt out of AI model training by blocking the training crawlers, such as GPTBot, ClaudeBot and CCBot, and the Google-Extended and Applebot-Extended controls, without affecting AI search.
Three kinds of AI crawler
AI companies now run separate crawlers for separate jobs. Knowing which is which lets you stay visible in AI answers while making your own choice about training.
- Search crawlers build the index an assistant searches when it answers a question. Block these and you are left out of AI search answers.
- User fetchers open a specific page because a person asked the assistant to, for example by pasting your link.
- Training crawlers collect content to train future AI models. Blocking them does not remove you from AI search.
The crawlers that matter
| User-agent | Company | Used for | Allow to appear in AI answers? |
|---|---|---|---|
OAI-SearchBot | OpenAI | ChatGPT search results | Yes |
ChatGPT-User | OpenAI | Pages a ChatGPT user asks it to open | Recommended |
GPTBot | OpenAI | Training OpenAI models | Your choice |
Claude-SearchBot | Anthropic | Claude search results | Yes |
Claude-User | Anthropic | Pages a Claude user asks it to open | Recommended |
ClaudeBot | Anthropic | Training Anthropic models | Your choice |
PerplexityBot | Perplexity | Perplexity search index | Yes |
Perplexity-User | Perplexity | Pages fetched for a user’s question | Recommended |
Googlebot | Google Search, AI Overviews, AI Mode | Yes | |
Google-Extended | A robots.txt switch (not a separate crawler) for Gemini training and grounding | Your choice | |
Bingbot | Microsoft | Bing and Microsoft Copilot | Yes |
Applebot | Apple | Siri and Spotlight suggestions | Yes |
Applebot-Extended | Apple | A robots.txt switch for Apple AI training | Your choice |
CCBot | Common Crawl | Open web archive used to train many models | Your choice |
Names come from each company’s own documentation. AI companies add and rename crawlers from time to time, so check their pages when you review your rules.
A robots.txt that keeps you visible
For most businesses the simplest rule is the best: allow everyone, and list your sitemap.
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml
If you want to stay in AI search but opt out of model training, add groups for the training crawlers only:
User-agent: *
Allow: /
# Opt out of AI model training (AI search stays allowed)
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
Sitemap: https://www.example.com/sitemap.xml
Keep private areas such as admin, cart, checkout and account pages disallowed for everyone as you normally would.
Common mistakes
- A leftover
Disallow: /underUser-agent: *from when the site was being built. It hides the whole site from every search engine and assistant. - Blocking every bot with “AI” in its name. Copy-pasted block lists often include the search crawlers, not only the training ones.
- A crawler-specific group that forgets the basics. A crawler follows only the most specific group that names it. If you add a
User-agent: OAI-SearchBotgroup, the rules under*no longer apply to it. - Long Crawl-delay values. Google ignores Crawl-delay, but Bing and some AI crawlers respect it, and a large value slows how quickly they see changes.
- robots.txt that returns an error. If the file returns a server error (5xx), Google and most crawlers stop crawling the site until it loads again.
Check the firewall too
robots.txt is a request, not a gate. Many sites allow AI crawlers in robots.txt but block them in practice:
- Cloudflare and other CDNs offer settings to block AI crawlers. Since July 2025 Cloudflare asks new customers whether to allow AI crawlers, with blocking as the default. Check the bot and AI crawler settings, and make sure the search crawlers you want are allowed.
- Security plugins (on WordPress, for example) may rate-limit or block unfamiliar user agents.
- Hosting firewalls sometimes block whole data-centre IP ranges, which is where crawlers run from.
The test is simple: request a page with each crawler’s user agent and see whether you get your page or an error or challenge page. A site audit can do this for you across all crawlers at once.
Also check these page-level settings
noindexkeeps a page out of Google, and therefore out of AI Overviews and AI Mode.nosnippetandmax-snippet:0stop Google from showing text from the page, including in its AI features.- If the main text only appears after JavaScript runs, many AI crawlers see an empty page, because they do not run scripts.
Questions
If I block GPTBot, will I disappear from ChatGPT?
No. GPTBot collects content for training OpenAI’s models. ChatGPT search uses a different crawler, OAI-SearchBot. OpenAI says sites that block OAI-SearchBot will not be shown in ChatGPT search answers, so allow that one if you want to be found.
Does blocking Google-Extended remove me from AI Overviews?
No. Google says Google-Extended controls use of your content for training Gemini models and for grounding in the Gemini apps and Vertex AI, and does not affect inclusion in Google Search. AI Overviews and AI Mode use Googlebot. Google offers separate controls, such as nosnippet and a Search Console setting, for its AI search features.
How long until a robots.txt change takes effect?
Crawlers re-read robots.txt regularly, typically within a day. OpenAI says changes take about 24 hours to apply to its search systems.
Do “user” agents like ChatGPT-User follow robots.txt?
Not necessarily. They fetch a page because a person asked the assistant to open it. OpenAI notes robots.txt rules may not apply to ChatGPT-User, and blocking them can stop assistants from reading your page when a customer shares the link.
My robots.txt allows everything. Why is ChatGPT still not reading my site?
Check the firewall or CDN. Bot protection, “block AI bots” options and security plugins can reject AI crawlers with an error or challenge page even when robots.txt allows them.