What is Claude-Web? AI crawler guide
Claude-Web is a crawler or fetcher associated with Anthropic. See source-graded user-agent, robots.txt, verification, and observation guidance.
What is Claude-Web?
Claude-Web is a reported crawler or fetcher string associated with Anthropic, but current operator evidence is incomplete. Reported Anthropic-related token seen in public crawler registries and Trakkr detection, but absent from Anthropic's current bot documentation.
Evidence status
| Field | Value |
|---|---|
| Evidence | Reported, not operator verified |
| Lifecycle | Status uncertain |
| Purpose | interaction |
| robots.txt posture | unverified |
| Source checked | 2026-07-21 |
Reported user-agent
Claude-Web
Claude-Web fires when a real user's question pulls in your page. To know how often Claude reaches for your brand at all, track your brand mentions in Claude.
robots.txt allow example
User-agent: Claude-Web Allow: /
robots.txt block example
User-agent: Claude-Web Disallow: /
What the rule can and cannot do
Robots.txt expresses an access policy to compliant automated crawlers. It does not authenticate the sender, remove content already collected, or guarantee that a model will use or not use content.
How to verify a request
- Match the complete documented user-agent where one exists. A match is only a clue.
- Look for operator-published IP ranges, reverse DNS, or signed-request guidance. None is attached to this record.
- Keep the observed request, operator documentation, and any inference as separate fields.
JavaScript behavior
The cited operator material does not verify JavaScript rendering. Serve useful HTML before client-side JavaScript where possible.
Related bots
- Google-Gemini-CLI: Also tracked as a live fetcher.
- Claude-Code: Another Anthropic live fetcher to compare.
- AmazonBuyForMe: Also tracked as a live fetcher.
- GoogleAgent-Mariner: Also tracked as a live fetcher.
- ChatGPT Agent: Also tracked as a live fetcher.
- Google-Agent: Also tracked as a live fetcher.
- cohere-ai: Also tracked as a live fetcher.
- Claude-User: Another Anthropic live fetcher to compare.
- Gemini-Deep-Research: Also tracked as a live fetcher.
- Anthropic-AI: Claude-Web connects this operator term to its crawler behavior.
- AI Crawlers: Claude-Web is a concrete crawler example for this concept.
Frequently Asked Questions
What is Claude-Web?
Claude-Web is a reported, not operator verified crawler or fetcher record associated with Anthropic.
What user-agent does Claude-Web use?
Claude-Web. Check the evidence status before attributing a matching request.
Can Claude-Web be blocked in robots.txt?
The record identifies Claude-Web as the token to review. Robots.txt is a request policy, not proof of model use or non-use.
How can I verify a Claude-Web request?
Treat the user-agent as a clue and look for operator-published IP, reverse-DNS, or signature guidance.
Data & Sources
- Claude-Web source reference - Source used to verify Claude-Web.