Not sure where to start? Take a guided tour.
AI Site Grader
AI Query Scanner
llms.txt GeneratorAI crawlers,
with the evidence.
Know which AI crawlers are real, what they do, and how to handle them. A source-checked reference to user-agents, controls and observed requests.
98 of 98 crawlers
A
13 recordsAI2Bot
Allen Institute for AI crawler used to find web content for open language model datasets.
AI2BotTraining crawler · Honors robots.txtAi2Bot-Dolma
AI2 crawler token associated with Dolma/open language model dataset collection.
Ai2Bot-DolmaTraining crawler · Honors robots.txtAIWebIndex
Crawler that builds Lyrenth, an index of public web pages served to AI systems and developers as extracted, attributed documents.
AIWebIndex/2.0 (+https://lyrenth.com/bot; AI-readable web index)AI search crawler · Honors robots.txtamazon-kendra
Amazon Kendra crawler token for intelligent enterprise search over configured content sources.
amazon-kendraAI search crawler · Honors robots.txtAmazonbot
Amazon crawler used to improve its products and services and potentially train Amazon AI models.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36Other crawler · Honors robots.txtAmazonBuyForMe
Amazon agent token reported for Buy for Me shopping actions directed by customers.
AmazonBuyForMeLive fetcher · Compliance unverifiedAmzn-SearchBot
Amazon crawler used to improve search experiences in Amazon products and services, such as Alexa.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/W.X.Y.Z Safari/537.36AI search crawler · Honors robots.txtAmzn-User
Amazon fetcher that retrieves a specific page because a person asked an Amazon assistant about it.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-User/0.1) Chrome/W.X.Y.Z Safari/537.36Live fetcher · Partial or user-triggeredanthropic-ai
Legacy Anthropic robots.txt token that predates the current ClaudeBot, Claude-User and Claude-SearchBot names.
anthropic-aiTraining crawler · Compliance unverifiedApifyBot
Token associated with crawlers run on the Apify scraping platform by its customers.
ApifyBotOther crawler · Compliance unverifiedApplebot
Apple crawler for search experiences across Spotlight, Siri, Safari, and related Apple surfaces.
Mozilla/5.0 (Device; OS_version) AppleWebKit/WebKit_version (KHTML, like Gecko) Version/Safari_version Safari/WebKit_version (Applebot/Applebot_version; +http://www.apple.com/go/applebot)AI search crawler · Honors robots.txtApplebot-Extended
Robots.txt control token for whether Applebot-crawled content may be used to train Apple foundation models.
Applebot-Extended: control token, no HTTP crawlerTraining crawler · Honors robots.txtatlassian-bot
Atlassian Rovo crawler used to index connected website content for AI search, assistants, and agents.
atlassian-botAI search crawler · Honors robots.txtB
5 recordsbedrockbot
Amazon Bedrock web crawler connector token for customer-configured AI applications.
bedrockbotOther crawler · Honors robots.txtBingbot
Microsoft Bing crawler used to crawl and index pages for Bing and Microsoft search-powered experiences.
bingbotAI search crawler · Honors robots.txtBravebot
Reported name for Brave Search crawling. Brave says its crawler does not advertise a differentiated user-agent.
No official user-agent publishedAI search crawler · Partial or user-triggeredBrightbot
Bright Data’s declared crawler for collecting public web data for its own datasets.
Brightbot 1.0Other crawler · Compliance unverifiedBytespider
ByteDance crawler associated with training and powering AI products.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Bytespider; spider-feedback@bytedance.comTraining crawler · Partial or user-triggeredC
12 recordsD
3 recordsE
1 recordF
3 recordsG
15 recordsI
5 recordsK
5 recordsL
2 recordsM
8 recordsN
1 recordO
4 recordsP
6 recordsS
6 recordsT
3 recordsV
1 recordW
1 recordX
1 recordY
3 recordsWho is crawling right now.
Share of the 104 sites connected to Trakkr that each crawler visited in the four weeks to 25 October 2026. Check your own logs.
| # | Crawler | Job | Sites visited | Typical site, per week | 13 weeks |
|---|---|---|---|---|---|
| 1 | AI search crawler | 84% | 246 | ||
| 2 | Training crawler | 84% | 169 | ||
| 3 | Training crawler | 67% | 352 | ||
| 4 |  AmazonbotAmazon | Other crawler | 61% | 188 | |
| 5 | Live fetcher | 56% | 156 | ||
| 6 | AI search crawler | 56% | 400 | ||
| 7 | AI search crawler | 51% | 127 | ||
| 8 | Training crawler | 48% | 394 | ||
| 9 | Live fetcher | 46% | 47 | ||
| 10 | AI search crawler | 39% | 63 | ||
| 11 | Live fetcher | 39% | 24 | ||
| 12 | Live fetcher | 38% | 20 | ||
| 13 | Training crawler | 37% | 17 | ||
| 14 | Training crawler | 37% | 18 | ||
| 15 | Training crawler | 36% | 40 | ||
| 16 | Other crawler | 33% | 12 | ||
| 17 | Live fetcher | 32% | 16 | ||
| 18 | Live fetcher | 27% | 33 | ||
| 19 | Training crawler | 18% | 235 | ||
| 20 | Live fetcher | 18% | 10 | ||
| 21 | Other crawler | 13% | 7 | ||
| 22 | Other crawler | 12% | 42 | ||
| 23 | AI search crawler | 12% | 17 | ||
| 24 | AI search crawler | 8% | 7 | ||
| 25 | Training crawler | 7% | 1 | ||
| 26 | Live fetcher | 3% | – |
Classic search crawlers such as Googlebot are not collected on every site, so they are left out. Crawlers seen on fewer than 10 sites are not shown. A matching user-agent is not proof of the sender.
One robots.txt for every AI company.
Each company runs separate bots for training, search and user fetches. Choose what you want, then copy one file.
# AI crawlers: allow search and user fetches, opt out of training
# From trakkr.ai/bots. Review existing rules before you publish.
# OpenAI
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
Allow: /
# Anthropic
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /
# Google
User-agent: Google-Extended
Disallow: /
User-agent: Google-Agent
User-agent: Google-GeminiNotebook
Allow: /
# Perplexity
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
# Apple
User-agent: Applebot-Extended
Disallow: /
# Meta
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: Meta-ExternalFetcher
Allow: /
# Amazon
User-agent: Amazonbot
Disallow: /
User-agent: Amzn-SearchBot
User-agent: Amzn-User
Allow: /
# Mistral AI
User-agent: MistralAI-Training
Disallow: /
User-agent: MistralAI-Index
User-agent: MistralAI-User
Allow: /
# Moonshot AI
User-agent: KimiBot
Disallow: /
User-agent: Kimi-SearchBot
User-agent: Kimi-User
Allow: /
# Common Crawl
User-agent: CCBot
Disallow: /Opts out of training at each company while letting its search and user fetches through, so your pages can still be found and cited. Googlebot, Bingbot and Applebot are left alone.
Check your site against every AI crawlerWho runs which bot.
Training, search and a visit a person asked for use different names. A rule for one does not cover the others.
| Company | Training | Search | User fetch |
|---|---|---|---|
| – | |||
| – | |||
| Amazon | |||
| – | – |
Purpose before policy.
Training, search and a visit requested by a user have different jobs.
Training crawler
Collects or controls access to content that can feed AI training data.
AI search crawler
Indexes pages so AI search products can retrieve, rank, cite, or summarize them.
Live fetcher
Fetches pages because a user or agent asked for a specific URL or task.
A safer allow or block decision
- 01Start with purpose. Search, training, and user-triggered retrieval have different costs when blocked.
- 02Use a documented token. Target the narrowest official robots.txt token. If none is published, do not invent one.
- 03Verify traffic separately. A robots rule sets policy; IP, reverse DNS, or signatures help identify the sender.
Recent verified changes.
Google documented Google-GeminiNotebook
Google lists Google-GeminiNotebook as the current fetcher and Google-NotebookLM as the former name supported until August 2026.
Google refreshed its crawler reference
The current reference separates common crawlers, special-case crawlers, user-triggered fetchers, and product control tokens.
Anthropic confirmed its current three-bot split
Anthropic documents ClaudeBot for training, Claude-SearchBot for search, and Claude-User for user-directed retrieval.
What the labels mean.
Evidence stays attached to each fact.
Official
The operator publishes the name, purpose, user-agent or control token. Every verified field links to that source and has a checked date.
Observed
Trakkr counted matching request signatures in a stated time window and site sample. User-agents can be spoofed, so observation is not operator verification.
Reported, not inferred
A public crawler list or historical record names the string, but current operator documentation is missing. We do not infer ownership from the name; treat it as a detection clue.
See which AI crawlers read your site.
Trakkr shows which bots visit which pages each day, and whether the answers they feed mention you.
14-day free trial. Cancel anytime.