Bytespider

Bytespider is ByteDance's web crawler, the company behind TikTok.

Updated

What happens if you block Bytespider

You're asking ByteDance not to crawl your site.

# Allow Bytespider
User-agent: Bytespider
Allow: /

# Block Bytespider
User-agent: Bytespider
Disallow: /

Add one of these to your robots.txt. If your site is connected to Citerank Deploy, it can add the lines for you and take them out again later.

Checking that it's really Bytespider

ByteDance doesn't publish a list of IP addresses for Bytespider, so there's no reliable way to confirm a request really came from them.

Frequently asked questions

What is Bytespider?

Bytespider is ByteDance's web crawler, the company behind TikTok.

Should I block Bytespider?

You're asking ByteDance not to crawl your site. If you want to show up in ChatGPT, Claude or Perplexity answers, keep their search and user-question bots allowed. Training bots are a separate decision.

How do I block Bytespider?

Add User-agent: Bytespider and Disallow: / to your robots.txt. If you need to be certain, a firewall rule will also stop anything that ignores robots.txt.

How can I see when Bytespider visits my site?

Connect the Citerank WordPress plugin or Cloudflare Worker and open AI Traffic. It shows each visit from Bytespider, which pages it looked at, and when it last came by.

See which AI bots visit your site

AI Traffic shows every visit from ChatGPT, Claude, Perplexity and the rest, and the people who click through from their answers.

Open AI Traffic