Skip to content

Claude-SearchBot

Search crawler

Anthropic's crawler for its search index. Seen on 25 of 135 sites in 13 weeks.

Claude-SearchBot reached 8 of 92 sites last week

Sites that got at least one Claude-SearchBot request each week, against every site reporting that week.

Reach8%24th of 268 of 104 sitesLast 4 weeks
Requests376K1% of all crawler requests13 weeks, 25 sites
Per site7a weekMedian where it visitsLast 4 weeks

Blocking Claude-SearchBot can keep your pages out of Claude answers.

Anthropic says Claude-SearchBot follows robots.txt. A rule changes future crawls, not pages it already collected.

Block Claude-SearchBot
User-agent: Claude-SearchBot
Disallow: /
Allow Claude-SearchBot
User-agent: Claude-SearchBot
Allow: /
User agentMozilla/5.0 (compatible; Claude-SearchBot/1.0; +claudebot@anthropic.com)

Is it really Claude-SearchBot?

Anyone can send the Claude-SearchBot name. Here is how to check a request that carries it.

How to check
1Match the IP to Anthropic's listAnthropic publishes Claude-SearchBot's addresses at claude.com/crawling/bots.json.
2Never count a secret-file requestA request for /.env or /.git is not a crawl, whatever name it carries.
See every fake crawler

Common questions

What is Claude-SearchBot?

Claude-SearchBot is Anthropic's search crawler. Anthropic search crawler that indexes content to improve Claude search result relevance and accuracy.

How do I block Claude-SearchBot?

Add a directive to your robots.txt: "User-agent: Claude-SearchBot" followed by "Disallow: /". Anthropic says Claude-SearchBot follows robots.txt. A rule changes future crawls, not pages it already collected.

Does blocking Claude-SearchBot hide me from AI?

No single bot rule hides a site from every AI product. Disallowing it can reduce Claude search discovery. Allowing it does not guarantee indexing, ranking, citation, or referral traffic.