ClaudeBot

Training crawler

Anthropic's crawler for pages that may be used to train its models. Seen on 107 of 135 sites in 13 weeks.

ClaudeBot reached 75 of 92 sites last week

Sites that got at least one ClaudeBot request each week, against every site reporting that week.

75%84%78%85%87%77%81%82%80%82%80%81%82%

Reach84%2nd of 2687 of 104 sitesLast 4 weeks

Requests2.9M7% of all crawler requests13 weeks, 107 sites

Per site169a weekMedian where it visitsLast 4 weeks

Pages a visit5.1GPTBot reads 60.5Snapshot, 1 Feb 2026

How ClaudeBot reads a site

22,074 ClaudeBot visits to 47 brand sites. Snapshot, 1 Feb 2026.

Depth

41%

of its visits land three folders deep, like /blog/2024/post. 16% touch the homepage.

When

−8%

fewer visits on a weekend day than a weekday. Busiest hour 23:00 UTC, but no hour is quiet.

Blocking ClaudeBot is a training choice. Claude search uses Claude-SearchBot.

Anthropic says ClaudeBot follows robots.txt. A rule changes future crawls, not pages it already collected.

Block ClaudeBot

User-agent: ClaudeBot
Disallow: /

Allow ClaudeBot

User-agent: ClaudeBot
Allow: /

User agentMozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)

Is it really ClaudeBot?

Anyone can send the ClaudeBot name. Requests we could check against Anthropic's addresses, 1 Aug to 14 Oct 2026.

Fakes

7.4%

of requests signed ClaudeBot did not come from Anthropic's published addresses.

Verified, 92.6%Fake, 7.4%

How to check

1Match the IP to Anthropic's listAnthropic publishes ClaudeBot's addresses at claude.com/crawling/bots.json.

2Read the user agent closelyOne common fake opens a bracket after Mozilla/5.0 and never closes it.

See every fake crawler

Common questions

What is ClaudeBot?

ClaudeBot is Anthropic's training crawler. Anthropic crawler for public web content that could contribute to Claude model training.

How do I block ClaudeBot?

Add a directive to your robots.txt: "User-agent: ClaudeBot" followed by "Disallow: /". Anthropic says ClaudeBot follows robots.txt. A rule changes future crawls, not pages it already collected.

Does blocking ClaudeBot hide me from AI?

No single bot rule hides a site from every AI product. Disallowing ClaudeBot expresses a training-data opt-out for future compliant requests. A crawler log cannot confirm whether content entered a training dataset.