?AEO Tool

AI crawlers and robots.txt: which bots to allow for AI search

Short answer

To appear in AI search answers, allow the search crawlers of each AI assistant in robots.txt and in your firewall: OAI-SearchBot (ChatGPT), Claude-SearchBot (Claude), PerplexityBot (Perplexity), Googlebot (Google AI Overviews, AI Mode and Gemini) and Bingbot (Copilot). You can still opt out of AI model training by blocking the training crawlers, such as GPTBot, ClaudeBot and CCBot, and the Google-Extended and Applebot-Extended controls, without affecting AI search.

Three kinds of AI crawler

AI companies now run separate crawlers for separate jobs. Knowing which is which lets you stay visible in AI answers while making your own choice about training.

The crawlers that matter

User-agentCompanyUsed forAllow to appear in AI answers?
OAI-SearchBotOpenAIChatGPT search resultsYes
ChatGPT-UserOpenAIPages a ChatGPT user asks it to openRecommended
GPTBotOpenAITraining OpenAI modelsYour choice
Claude-SearchBotAnthropicClaude search resultsYes
Claude-UserAnthropicPages a Claude user asks it to openRecommended
ClaudeBotAnthropicTraining Anthropic modelsYour choice
PerplexityBotPerplexityPerplexity search indexYes
Perplexity-UserPerplexityPages fetched for a user’s questionRecommended
GooglebotGoogleGoogle Search, AI Overviews, AI ModeYes
Google-ExtendedGoogleA robots.txt switch (not a separate crawler) for Gemini training and groundingYour choice
BingbotMicrosoftBing and Microsoft CopilotYes
ApplebotAppleSiri and Spotlight suggestionsYes
Applebot-ExtendedAppleA robots.txt switch for Apple AI trainingYour choice
CCBotCommon CrawlOpen web archive used to train many modelsYour choice

Names come from each company’s own documentation. AI companies add and rename crawlers from time to time, so check their pages when you review your rules.

A robots.txt that keeps you visible

For most businesses the simplest rule is the best: allow everyone, and list your sitemap.

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

If you want to stay in AI search but opt out of model training, add groups for the training crawlers only:

User-agent: *
Allow: /

# Opt out of AI model training (AI search stays allowed)
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

Keep private areas such as admin, cart, checkout and account pages disallowed for everyone as you normally would.

Common mistakes

Check the firewall too

robots.txt is a request, not a gate. Many sites allow AI crawlers in robots.txt but block them in practice:

The test is simple: request a page with each crawler’s user agent and see whether you get your page or an error or challenge page. A site audit can do this for you across all crawlers at once.

Also check these page-level settings

Questions

If I block GPTBot, will I disappear from ChatGPT?

No. GPTBot collects content for training OpenAI’s models. ChatGPT search uses a different crawler, OAI-SearchBot. OpenAI says sites that block OAI-SearchBot will not be shown in ChatGPT search answers, so allow that one if you want to be found.

Does blocking Google-Extended remove me from AI Overviews?

No. Google says Google-Extended controls use of your content for training Gemini models and for grounding in the Gemini apps and Vertex AI, and does not affect inclusion in Google Search. AI Overviews and AI Mode use Googlebot. Google offers separate controls, such as nosnippet and a Search Console setting, for its AI search features.

How long until a robots.txt change takes effect?

Crawlers re-read robots.txt regularly, typically within a day. OpenAI says changes take about 24 hours to apply to its search systems.

Do “user” agents like ChatGPT-User follow robots.txt?

Not necessarily. They fetch a page because a person asked the assistant to open it. OpenAI notes robots.txt rules may not apply to ChatGPT-User, and blocking them can stop assistants from reading your page when a customer shares the link.

My robots.txt allows everything. Why is ChatGPT still not reading my site?

Check the firewall or CDN. Bot protection, “block AI bots” options and security plugins can reject AI crawlers with an error or challenge page even when robots.txt allows them.