AI crawlers

AI companies use different bots for different things. Some fetch a page when someone asks ChatGPT or Claude a question, some crawl for search results, and some collect pages to train new models. Here's what each one does and what happens if you block it.

Updated

The list

BotCompanyWhat it's forPublished IP list
ChatGPT-UserOpenAIAnswering user questionsYes
OAI-SearchBotOpenAISearch resultsYes
GPTBotOpenAIModel trainingYes
Claude-UserAnthropicAnswering user questionsNo
Claude-SearchBotAnthropicSearch resultsNo
ClaudeBotAnthropicModel trainingNo
Perplexity-UserPerplexityAnswering user questionsYes
PerplexityBotPerplexitySearch resultsYes
Google-AgentGoogleAnswering user questionsYes
Google-ExtendedGooglerobots.txt settingNot needed
Applebot-ExtendedApplerobots.txt settingNot needed
meta-externalagentMetaModel trainingNo
CCBotCommon CrawlModel trainingNo
BytespiderByteDanceModel trainingNo
AmazonbotAmazonModel trainingNo

When a company publishes its bot's IP addresses, you can check whether a request using that name is real.

Which ones to allow

If you want your site to show up in ChatGPT, Claude or Perplexity, keep their search crawlers (OAI-SearchBot, Claude-SearchBot and PerplexityBot) allowed, along with the bots they use to fetch a page when someone asks a question (ChatGPT-User, Claude-User and Perplexity-User).

Training crawlers like GPTBot, ClaudeBot and CCBot are a separate decision. Blocking them keeps your pages out of future models, and it doesn't remove you from the answers people see now.

Frequently asked questions

Does blocking GPTBot remove me from ChatGPT?

No. GPTBot only collects training data. ChatGPT search uses OAI-SearchBot, and ChatGPT uses ChatGPT-User to open a page when someone asks a question. Each has its own line in robots.txt.

Is robots.txt enough to block an AI crawler?

For the big companies, usually yes. Their crawlers follow it. If you need to be certain, add a firewall rule, such as Cloudflare's AI crawler settings.

How can I see which AI crawlers visit my site?

Connect the Citerank WordPress plugin or Cloudflare Worker and open AI Traffic. It shows every visit from AI bots, checks them against each company's IP list where there is one, and counts the people who click through from AI answers.

See which AI bots visit your site

AI Traffic shows every visit from ChatGPT, Claude, Perplexity and the rest, and the people who click through from their answers.

Open AI Traffic