Claude-SearchBot
Claude-SearchBot crawls pages to improve the search results Claude uses.
Updated
What happens if you block Claude-SearchBot
Your pages are less likely to come up when Claude searches the web.
# Allow Claude-SearchBot
User-agent: Claude-SearchBot
Allow: /
# Block Claude-SearchBot
User-agent: Claude-SearchBot
Disallow: /
Add one of these to your robots.txt. If your site is connected to Citerank Deploy, it can add the lines for you and take them out again later.
Other Anthropic bots
Anthropic also uses Claude-User (answering user questions) and ClaudeBot (model training). Each one has its own line in robots.txt, so you can allow some and block others.
Checking that it's really Claude-SearchBot
Anthropic doesn't publish a list of IP addresses for Claude-SearchBot, so there's no reliable way to confirm a request really came from them.
Frequently asked questions
What is Claude-SearchBot?
Claude-SearchBot crawls pages to improve the search results Claude uses.
Should I block Claude-SearchBot?
Your pages are less likely to come up when Claude searches the web. If you want to show up in ChatGPT, Claude or Perplexity answers, keep their search and user-question bots allowed. Training bots are a separate decision.
How do I block Claude-SearchBot?
Add
User-agent: Claude-SearchBotandDisallow: /to your robots.txt. If you need to be certain, a firewall rule will also stop anything that ignores robots.txt.How can I see when Claude-SearchBot visits my site?
Connect the Citerank WordPress plugin or Cloudflare Worker and open AI Traffic. It shows each visit from Claude-SearchBot, which pages it looked at, and when it last came by.