CCBot

CCBot builds Common Crawl, a free public copy of the web. A lot of AI models are trained on it.

Updated

What happens if you block CCBot

Your pages are left out of future Common Crawl snapshots, and so out of models trained on them.

# Allow CCBot
User-agent: CCBot
Allow: /

# Block CCBot
User-agent: CCBot
Disallow: /

Add one of these to your robots.txt. If your site is connected to Citerank Deploy, it can add the lines for you and take them out again later.

Checking that it's really CCBot

Common Crawl doesn't publish a list of IP addresses for CCBot, so there's no reliable way to confirm a request really came from them.

Frequently asked questions

What is CCBot?

CCBot builds Common Crawl, a free public copy of the web. A lot of AI models are trained on it.

Should I block CCBot?

Your pages are left out of future Common Crawl snapshots, and so out of models trained on them. If you want to show up in ChatGPT, Claude or Perplexity answers, keep their search and user-question bots allowed. Training bots are a separate decision.

How do I block CCBot?

Add User-agent: CCBot and Disallow: / to your robots.txt. If you need to be certain, a firewall rule will also stop anything that ignores robots.txt.

How can I see when CCBot visits my site?

Connect the Citerank WordPress plugin or Cloudflare Worker and open AI Traffic. It shows each visit from CCBot, which pages it looked at, and when it last came by.

See which AI bots visit your site

AI Traffic shows every visit from ChatGPT, Claude, Perplexity and the rest, and the people who click through from their answers.

Open AI Traffic