AI crawler request share.
A dated panel view of which crawler labels appear most often in Trakkr's connected-site request data. It is not global market share or verified operator traffic.
Which crawler labels appear most in this sample?
39 rows · updated June 8, 2026
| Crawler label | Documented or cautious purpose | Matched requests | Observed request share | Sites seen |
|---|---|---|---|---|
| Bytespider | Discovery or unknown | 8,033,121 | 49.0% | 82 |
| ClaudeBot | Training crawler | 1,736,590 | 10.6% | 86 |
| ChatGPT-User | User-requested fetch | 1,540,508 | 9.4% | 40 |
| Applebot | AI search index | 1,504,844 | 9.2% | 28 |
| Amazonbot | Discovery or unknown | 1,015,896 | 6.2% | 38 |
| OAI-SearchBot | AI search index | 901,386 | 5.5% | 101 |
| GPTBot | Training crawler | 868,367 | 5.3% | 122 |
| Scrapy | Unknown | 339,806 | 2.1% | 12 |
| Meta-ExternalFetcher | User-requested fetch | 197,356 | 1.2% | 50 |
| PerplexityBot | AI search index | 140,159 | 0.9% | 36 |
| GoogleOther | Unknown | 16,606 | 0.1% | 1 |
| Claude-User | User-requested fetch | 11,841 | 0.1% | 20 |
| Meta-ExternalAgent | Training or product crawler | 11,760 | 0.1% | 9 |
| Bingbot | Unknown | 11,375 | 0.1% | 9 |
| Claude-SearchBot | AI search index | 11,204 | 0.1% | 12 |
| CCBot | Training crawler | 11,193 | 0.1% | 13 |
| ChatGPT | Unknown | 9,388 | 0.1% | 8 |
| GrokBot | Unknown | 8,300 | 0.1% | 12 |
| YouBot | AI search index | 7,834 | 0.0% | 12 |
| Perplexity-User | User-requested fetch | 3,492 | 0.0% | 13 |
| DeepSeekBot | Unknown | 2,051 | 0.0% | 19 |
| Claude | Unknown | 1,661 | 0.0% | 8 |
| MistralAI-User | User-requested fetch | 1,431 | 0.0% | 43 |
| Googlebot | Unknown | 911 | 0.0% | 1 |
| Perplexity | Unknown | 752 | 0.0% | 6 |
| Bing | Unknown | 682 | 0.0% | 3 |
| Amazon | Unknown | 645 | 0.0% | 3 |
| Meta AI | Unknown | 503 | 0.0% | 3 |
| cohere-ai | Unknown | 225 | 0.0% | 13 |
| Claude-Web | Unknown | 203 | 0.0% | 7 |
| Diffbot | Unknown | 177 | 0.0% | 10 |
| facebookexternalhit | Unknown | 171 | 0.0% | 1 |
| Google-Agent | Agent | 88 | 0.0% | 5 |
| AI2Bot | Unknown | 82 | 0.0% | 4 |
| anthropic-ai | Unknown | 82 | 0.0% | 6 |
| Timpibot | Unknown | 37 | 0.0% | 2 |
| MistralBot | Unknown | 21 | 0.0% | 7 |
| Mistral | Unknown | 4 | 0.0% | 1 |
| Omgili | Unknown | 3 | 0.0% | 2 |
Observed crawler-label request share
Share of matched requests inside Trakkr's connected-site panel
How this benchmark is built
The denominator is all included requests that matched a published crawler label in Trakkr's connected-site panel. It is not global internet traffic.
Case variants of the same known label are grouped. Site reach uses the largest reported value instead of summing potentially overlapping sites.
Purpose labels follow official operator documentation where available. Unknown and multi-purpose labels stay cautious rather than being inferred from the name.
Google-Extended is excluded because Google documents it as a robots.txt product token, not a separate HTTP user-agent.
Snapshot updated 2026-06-08T13:00:11.373572+00:00. Open the source study for its collection window and exclusions.
User-agent labels can be spoofed. These rows are request-signature observations, not verified operator identity.
Customer mix, install coverage, cache behavior, WAF order, sampling, retries and classification changes can shape request share.
A request does not prove indexing, model use, ranking, citation or a later human visit.
Related benchmarks
Published as an open dataset under CC BY 4.0. Fetch the live numbers as JSON above.
What counts as an AI crawler visit?
The benchmark counts requests that match published crawler labels in Trakkr connected-site data. User-agent strings can be spoofed, so a match is not proof of operator identity.
What is the difference between crawler types?
Training crawlers may collect data for model improvement. Search crawlers build retrieval indexes. User-requested fetchers retrieve a page after an action. Some labels are multi-purpose or undocumented, so they remain cautious or unknown.
Does a higher request share mean more citations?
No. Request share describes this panel and time window. It does not prove indexing, ranking, citation, referral traffic or model use. Those outcomes must be measured separately.
Monitor requests, then measure outcomes separately
Connect page-level crawler activity to separately observed citations, AI referrals, access checks, and next actions.