Skip to content

See where you stand

Change it

Works withWordPressShopifyWebflowGitHubGoogleSlackAll integrations
FixesFixes now ship to WordPress, Shopify, Webflow and GitHub

By team

By goal

Done for you

Not sure where to start? Take a guided tour.Book a demo
Customer storyHow the voice AI company Futuro grew its traffic3×AI traffic in 60 days

Learn

Data and research

FreeAI Site GraderAI Query Scannerllms.txt GeneratorAll free tools
Open dataWhere AI sends people, across more than 6,000 sites

AnthropicTraining crawler

anthropic-ai

anthropic-ai is a legacy Anthropic robots.txt token that predates the current ClaudeBot, Claude-User and Claude-SearchBot names.

Anthropic's current documentation does not list anthropic-ai. Use the current ClaudeBot, Claude-SearchBot, and Claude-User names for policy.

Get the robots.txt ruleRecord source
Is anthropic-ai allowed on your site?4 Anthropic bots

Reads your live robots.txt and applies the same rules these crawlers use. Nothing is stored.

robots.txt token
anthropic-ai
Follows robots.txt
Not documented
Last checked
11 August 2026

Should you block anthropic-ai?

There is little to gain. Anthropic does not document anthropic-ai, so a rule naming it controls none of its documented crawlers. Write rules for ClaudeBot, Claude-SearchBot and Claude-User instead.

What blocking each Anthropic bot does
Anthropic botJobIf you block it
anthropic-aiThis pageTrainingAnthropic’s current guidance does not list this token, so a rule naming it is unlikely to control any documented Anthropic crawler.
ClaudeBotTrainingDisallowing ClaudeBot expresses a training-data opt-out for future compliant requests. A crawler log cannot confirm whether content entered a training dataset.
Claude-SearchBotSearchDisallowing it can reduce Claude search discovery. Allowing it does not guarantee indexing, ranking, citation, or referral traffic.
Claude-UserUser fetchDisallowing it asks the documented fetcher not to retrieve covered paths. A matching request does not prove a page was cited in a Claude answer.

About robots.txtAnthropic has not documented whether anthropic-ai follows robots.txt, so a rule is a request, not a guarantee.

robots.txt
User-agent: anthropic-ai
Disallow: /

Asks anthropic-ai not to crawl any page. Other Anthropic bots are unaffected. Add it to the robots.txt file at the root of your site, above any wider rules.

Ignored robots.txt?

Spot it in your logs.

Match the name, then check where the request came from. Anyone can copy a user-agent.

Reported user-agentsource
anthropic-ai

This string comes from a public crawler registry or observed naming, not current operator documentation.

  1. 1

    Match the name

    Search your server or CDN log for anthropic-ai, case-insensitive. A match is a clue, not proof.

  2. 2

    Check the anthropic published IP ranges

    Trakkr measured 20,197 anthropic-ai requests carrying a source IP across its monitored sites in the 30 days to 11 August 2026. None fell inside Anthropic’s published ranges, while the same test matched 48% of ClaudeBot and 24% of Claude-User requests. claude.com/crawling/bots.json

Is this really anthropic-ai?

Published addresses

Paste an address that claims to be anthropic-ai. It is checked against Anthropic’s published list. Nothing is stored.

Check any crawler’s address

Serve the content in HTML

The cited operator material does not verify JavaScript rendering. Serve useful HTML before client-side JavaScript where possible.

Widely still present in robots.txt files. Traffic using this token should be verified before it is treated as Anthropic.

How active is anthropic-ai?

Measured across the sites connected to Trakkr, 27 July 2026 to 25 October 2026.

7%

of connected sites saw anthropic-ai in the four weeks to 25 October 2026. That is 7 of 104.

100%75%50%25%3%
anthropic-ai visited between 0% and 16% of connected sites each week from 27 July 2026 to 25 October 2026.

Share of connected sites it visited each week

Rank by reach
25 of 26
Typical site, per week
1 requests
Sites reached, 13 weeks
21 of 135

Most widespread AI crawlers

Share of connected sites each one visited, last four weeks.

  1. 1OAI-SearchBot84%
  2. 2ClaudeBot84%
  3. 3GPTBot67%
  4. 4Amazonbot61%
  5. 5ChatGPT-User56%
  6. 6PerplexityBot56%
  7. 7Applebot51%
  8. 25anthropic-ai7%
All 26 measured crawlers
How this is measured

Finalized daily crawler summaries from sites connected to Trakkr, grouped by the crawler named in each request. Trakkr’s own site is excluded. A site counts as reached in a week when it logged at least one request from anthropic-ai. The typical figure is the median weekly requests among the sites it reached.

  • This is a sample of connected sites, not the whole web.
  • A matching user-agent does not prove the named operator sent the request.
  • Requests are not unique pages, training use or citations.
  • Crawlers seen on fewer than 10 sites are not shown. Classic search crawlers such as Googlebot are not collected on every site, so they are left out.

Questions, answered.

anthropic-ai is a legacy Anthropic robots.txt token that predates the current ClaudeBot, Claude-User and Claude-SearchBot names. This record has no current Anthropic source, so treat a matching request as unverified.

Sources.

Reported, not operator verified. Operator facts and Trakkr’s measurements keep separate dates.

  1. [1]
    Reported crawler reference

    Reported, not operator verified · checked 11 August 2026

    support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
  2. [2]
    Anthropic current crawler documentation

    Officially documented · checked 18 August 2026

    support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler

Last source check 11 August 2026. Activity measured to 25 October 2026.

Changes

Anthropic confirmed its current three-bot splitAnthropic documents ClaudeBot for training, Claude-SearchBot for search, and Claude-User for user-directed retrieval.
All AI crawlers

See which AI crawlers read your site.

Trakkr shows which bots, anthropic-ai included, visit which pages each day, and whether the answers they feed mention you.

14-day free trial. Cancel anytime.