What is anthropic-ai? AI crawler guide
anthropic-ai is a crawler or fetcher associated with Anthropic. See source-graded user-agent, robots.txt, verification, and observation guidance.
What is anthropic-ai?
anthropic-ai is a reported crawler or fetcher string associated with Anthropic, but current operator evidence is incomplete. Legacy Anthropic robots.txt token that predates the current ClaudeBot, Claude-User and Claude-SearchBot names.
Evidence status
| Field | Value |
|---|---|
| Evidence | Reported, not operator verified |
| Lifecycle | Status uncertain |
| Purpose | training |
| robots.txt posture | unverified |
| Source checked | 2026-08-11 |
Reported user-agent
anthropic-ai
robots.txt allow example
User-agent: anthropic-ai Allow: /
robots.txt block example
User-agent: anthropic-ai Disallow: /
What the rule can and cannot do
Robots.txt expresses an access policy to compliant automated crawlers. It does not authenticate the sender, remove content already collected, or guarantee that a model will use or not use content.
How to verify a request
- Match the complete documented user-agent where one exists. A match is only a clue.
- Use the operator-published check: Anthropic published IP ranges.
- Keep the observed request, operator documentation, and any inference as separate fields.
JavaScript behavior
The cited operator material does not verify JavaScript rendering. Serve useful HTML before client-side JavaScript where possible.
Related bots
- Applebot-Extended: Also tracked as a training crawler.
- Google-Extended: Also tracked as a training crawler.
- GPTBot: Also tracked as a training crawler.
- Meta-ExternalAgent: Also tracked as a training crawler.
- ClaudeBot: Another Anthropic training crawler to compare.
- DeepSeekBot: Also tracked as a training crawler.
- cohere-training-data-crawler: Also tracked as a training crawler.
- img2dataset: Also tracked as a training crawler.
- PanguBot: Also tracked as a training crawler.
- Anthropic-AI: Anthropic-AI is the glossary definition behind this crawler guide.
- AI Training Opt-Out: anthropic-ai is a training crawler tied to this policy decision.
Frequently Asked Questions
What is anthropic-ai?
anthropic-ai is a reported, not operator verified crawler or fetcher record associated with Anthropic.
What user-agent does anthropic-ai use?
anthropic-ai. Check the evidence status before attributing a matching request.
Can anthropic-ai be blocked in robots.txt?
The record identifies anthropic-ai as the token to review. Robots.txt is a request policy, not proof of model use or non-use.
How can I verify a anthropic-ai request?
Start with the full user-agent, then use anthropic published ip ranges from the operator. A user-agent match alone is not proof.
Data & Sources
- Anthropic documentation - Primary source for anthropic-ai crawler details.
- anthropic-ai source reference - Source used to verify anthropic-ai.
- anthropic-ai verification: Anthropic published IP ranges - Trakkr measured 20,197 anthropic-ai requests carrying a source IP across its monitored sites in the 30 days to 11 August 2026. None fell inside Anthropic's published ranges, while the same test matched 48% of ClaudeBot and 24% of Claude-User requests.