What is Googlebot? AI crawler guide
Googlebot is a crawler or fetcher associated with Google. See source-graded user-agent, robots.txt, verification, and observation guidance.
What is Googlebot?
Googlebot is a documented Google crawler or fetcher. Google Search crawler used to discover, crawl, render, and index pages for Google Search.
Evidence status
| Field | Value |
|---|---|
| Evidence | Officially documented |
| Lifecycle | Active |
| Purpose | search |
| robots.txt posture | honors |
| Source checked | 2026-06-11 |
Documented user-agent
Googlebot
Googlebot access decides whether Google can fetch your pages. Whether those pages get used in generated answers is a separate question, so track your presence in Google's AI results rather than treating crawl volume as visibility.
robots.txt allow example
User-agent: Googlebot Allow: /
robots.txt block example
User-agent: Googlebot Disallow: /
What the rule can and cannot do
Robots.txt expresses an access policy to compliant automated crawlers. It does not authenticate the sender, remove content already collected, or guarantee that a model will use or not use content.
How to verify a request
- Match the complete documented user-agent where one exists. A match is only a clue.
- Look for operator-published IP ranges, reverse DNS, or signed-request guidance. None is attached to this record.
- Keep the observed request, operator documentation, and any inference as separate fields.
JavaScript behavior
Google documents a rendering stage that runs JavaScript after crawling when resources permit.
Related bots
- Bravebot: Also tracked as a search crawler.
- OAI-SearchBot: Also tracked as a search crawler.
- Bingbot: Also tracked as a search crawler.
- PetalBot: Also tracked as a search crawler.
- Amzn-SearchBot: Also tracked as a search crawler.
- atlassian-bot: Also tracked as a search crawler.
- Kagibot: Also tracked as a search crawler.
- KlaviyoAIBot: Also tracked as a search crawler.
- MistralAI-Index: Also tracked as a search crawler.
- Google DeepMind: Googlebot connects this operator term to its crawler behavior.
- Gemini 2.0: Googlebot gives crawler context for Gemini 2.0.
- ShapBot: Also tracked as a search crawler.
Frequently Asked Questions
What is Googlebot?
Googlebot is a officially documented crawler or fetcher record associated with Google.
What user-agent does Googlebot use?
Googlebot. Check the evidence status before attributing a matching request.
Can Googlebot be blocked in robots.txt?
The record identifies Googlebot as the token to review. Robots.txt is a request policy, not proof of model use or non-use.
How can I verify a Googlebot request?
Treat the user-agent as a clue and look for operator-published IP, reverse-DNS, or signature guidance.
Data & Sources
- Google documentation - Primary source for Googlebot crawler details.