Skip to content

See where you stand

Change it

Works withWordPressShopifyWebflowGitHubGoogleSlackAll integrations
FixesFixes now ship to WordPress, Shopify, Webflow and GitHub

By team

By goal

Done for you

Not sure where to start? Take a guided tour.Book a demo
Customer storyHow the voice AI company Futuro grew its traffic3×AI traffic in 60 days

Learn

Data and research

FreeAI Site GraderAI Query Scannerllms.txt GeneratorAll free tools
Open dataWhere AI sends people, across more than 6,000 sites

GoogleAI search crawler

Googlebot

Googlebot is a Google Search crawler used to discover, crawl, render, and index pages for Google Search.

Needed to be found and cited in Gemini.

Get the robots.txt ruleGoogle documentation
Is Googlebot allowed on your site?4 Google bots

Reads your live robots.txt and applies the same rules these crawlers use. Nothing is stored.

robots.txt token
Googlebot
Follows robots.txt
Yes, documented
Seen on
Not measured
Last checked
11 June 2026

Should you block Googlebot?

Not if you want to be found in Gemini. Blocking Googlebot can remove pages from Google Search and eligibility for Google AI features that depend on Search indexing.

What blocking each Google bot does
Google botJobIf you block it
Google-ExtendedControl tokenDisallowing it limits future Gemini training and grounding uses covered by the token. Google says it does not affect Search inclusion or ranking.
GooglebotThis pageSearchBlocking Googlebot can remove pages from Google Search and eligibility for Google AI features that depend on Search indexing.
Google-AgentUser fetchBlocking it can prevent a Google agent from retrieving a page while completing a user-directed task. It is separate from Google Search crawling.
Google-GeminiNotebookUser fetchA hit usually means a person added your page as a source, not that Google is crawling your site broadly.
Google-CloudVertexBotOtherA hit usually relates to a configured Vertex AI workflow rather than broad public search crawling.
GoogleOtherOtherBlocking it may affect Google product-specific crawling but is separate from primary Search indexing policy.
GoogleOther-ImageOtherBlocking it can limit image-oriented Google product crawls without targeting regular Googlebot directly.
GoogleOther-VideoOtherBlocking it can limit video-oriented Google product crawls without targeting regular Googlebot directly.

From Google’s documentationGooglebot feeds Google Search, including AI features that draw on the Search index.

robots.txt
User-agent: Googlebot
Disallow: /

Asks Googlebot not to crawl any page. Other Google bots are unaffected. Add it to the robots.txt file at the root of your site, above any wider rules.

Ignored robots.txt?

Googlebot access decides whether Google can fetch your pages. Whether those pages get used in generated answers is a separate question, so track your presence in Google's AI results rather than treating crawl volume as visibility.

Also seen under Google’s name, but not in its current documentation: GoogleAgent-Mariner, Gemini-Deep-Research, Google-NotebookLM, Google-Gemini-CLI, Google-Firebase. Rules naming them control no documented Google crawler.

Spot it in your logs.

Match the name, then check where the request came from. Anyone can copy a user-agent.

Smartphone crawlersource
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Version numbers change. Match the highlighted name, not the whole string.

Desktop crawlersource
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36

Version numbers change. Match the highlighted name, not the whole string.

  1. 1

    Match the name

    Search your server or CDN log for Googlebot, case-insensitive. A match is a clue, not proof.

  2. 2

    Check the published IP ranges and reverse DNS

    Google also documents reverse DNS: a genuine common crawler’s hostname ends in googlebot.com or google.com and resolves back to the same address. developers.google.com/static/search/apis/ipranges/googlebot.json

Is this really Googlebot?

Addresses and reverse DNS

No log handy?

Paste an address that claims to be Googlebot. It is checked against Google’s published list and its reverse DNS. Nothing is stored.

Check any crawler’s address

It runs JavaScript

Google documents a rendering stage that runs JavaScript after crawling when resources permit.

Rendering source

Questions, answered.

Googlebot is a Google Search crawler used to discover, crawl, render, and index pages for Google Search.

Sources.

Officially documented. Operator facts and Trakkr’s measurements keep separate dates.

  1. [1]
    Google crawler documentation

    Officially documented · checked 11 June 2026

    developers.google.com/search/docs/crawling-indexing/googlebot

First verified 11 June 2026. Last source check 11 June 2026.

Changes

Google refreshed its crawler referenceThe current reference separates common crawlers, special-case crawlers, user-triggered fetchers, and product control tokens.
All AI crawlers

See which AI crawlers read your site.

Trakkr shows which bots, Googlebot included, visit which pages each day, and whether the answers they feed mention you.

14-day free trial. Cancel anytime.