Google-Agent

Google-Agent is used by Google's AI agents when they do something on a person's behalf, like browsing a site for them.

Updated

What happens if you block Google-Agent

Google's agents can't use your site when they're acting for someone.

One thing to know: Google-Agent fetches a page because a person asked for it, and some vendors do not apply robots.txt to that. Check Google's documentation, and use a firewall rule if you need to be sure.

# Allow Google-Agent
User-agent: Google-Agent
Allow: /

# Block Google-Agent
User-agent: Google-Agent
Disallow: /

Add one of these to your robots.txt. If your site is connected to Citerank Deploy, it can add the lines for you and take them out again later.

Other Google bots

Google also uses Google-Extended (robots.txt setting). Each one has its own line in robots.txt, so you can allow some and block others.

Checking that it's really Google-Agent

Google publishes the IP addresses Google-Agent uses (list). If a request says it's Google-Agent but comes from another address, it isn't Google. Citerank's AI Traffic report checks every visit against this list.

Frequently asked questions

What is Google-Agent?

Google-Agent is used by Google's AI agents when they do something on a person's behalf, like browsing a site for them.

Should I block Google-Agent?

Google's agents can't use your site when they're acting for someone. If you want to show up in ChatGPT, Claude or Perplexity answers, keep their search and user-question bots allowed. Training bots are a separate decision.

How do I block Google-Agent?

Add User-agent: Google-Agent and Disallow: / to your robots.txt. Some vendors do not apply robots.txt to bots that fetch pages for a person, so check Google's documentation, and use a firewall rule to be sure.

How can I see when Google-Agent visits my site?

Connect the Citerank WordPress plugin or Cloudflare Worker and open AI Traffic. It shows each visit from Google-Agent, which pages it looked at, and when it last came by.

See which AI bots visit your site

AI Traffic shows every visit from ChatGPT, Claude, Perplexity and the rest, and the people who click through from their answers.

Open AI Traffic