AI crawlers
AI companies use different bots for different things. Some fetch a page when someone asks ChatGPT or Claude a question, some crawl for search results, and some collect pages to train new models. Here's what each one does and what happens if you block it.
Updated
The list
| Bot | Company | What it's for | Published IP list |
|---|---|---|---|
| ChatGPT-User | OpenAI | Answering user questions | Yes |
| OAI-SearchBot | OpenAI | Search results | Yes |
| GPTBot | OpenAI | Model training | Yes |
| Claude-User | Anthropic | Answering user questions | No |
| Claude-SearchBot | Anthropic | Search results | No |
| ClaudeBot | Anthropic | Model training | No |
| Perplexity-User | Perplexity | Answering user questions | Yes |
| PerplexityBot | Perplexity | Search results | Yes |
| Google-Agent | Answering user questions | Yes | |
| Google-Extended | robots.txt setting | Not needed | |
| Applebot-Extended | Apple | robots.txt setting | Not needed |
| meta-externalagent | Meta | Model training | No |
| CCBot | Common Crawl | Model training | No |
| Bytespider | ByteDance | Model training | No |
| Amazonbot | Amazon | Model training | No |
When a company publishes its bot's IP addresses, you can check whether a request using that name is real.
Which ones to allow
If you want your site to show up in ChatGPT, Claude or Perplexity, keep their search crawlers (OAI-SearchBot, Claude-SearchBot and PerplexityBot) allowed, along with the bots they use to fetch a page when someone asks a question (ChatGPT-User, Claude-User and Perplexity-User).
Training crawlers like GPTBot, ClaudeBot and CCBot are a separate decision. Blocking them keeps your pages out of future models, and it doesn't remove you from the answers people see now.
Frequently asked questions
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot only collects training data. ChatGPT search uses OAI-SearchBot, and ChatGPT uses ChatGPT-User to open a page when someone asks a question. Each has its own line in robots.txt.
Is robots.txt enough to block an AI crawler?
For the big companies, usually yes. Their crawlers follow it. If you need to be certain, add a firewall rule, such as Cloudflare's AI crawler settings.
How can I see which AI crawlers visit my site?
Connect the Citerank WordPress plugin or Cloudflare Worker and open AI Traffic. It shows every visit from AI bots, checks them against each company's IP list where there is one, and counts the people who click through from AI answers.