Not sure where to start? Take a guided tour.
AI Site Grader
AI Query Scanner
llms.txt Generator
In short
Four findings, strongest first- 72%
of matched requests carried OpenAI labels (GPTBot and OAI-SearchBot). The denominator is this connected-site panel, not global crawler traffic.
Chapter 02 - 19% vs 3%
of ClaudeBot-labelled and GPTBot-labelled requests were for homepages. That alone doesn’t explain discovery method or intent.
Chapter 03 - 88.5%
of pages had one recorded request inside the study window. It says nothing about frequency outside the window.
Chapter 06 - 21%
of ChatGPT Search-labelled sessions in the linked sample began on blog pages: an observed landing mix, not a citation preference.
Chapter 05
AI crawler monitoring starts with a request, not an assumption.
Some documented bots may collect data for model improvement. Others build search indexes or fetch a page after a user action. Those purposes use different tokens and should not be combined into one policy or one performance claim.
This study analysed 575,788 crawler visits across 7 crawler labels. Recorded requests also skewed towards shallower URLs: more than half landed on pages within three site-depth steps, though URL structure, templates, popularity and customer mix may all contribute.
Crawler access is evidence of retrieval, not answer inclusion. Next, track AI Overview selection across a controlled query set.
The study measures this across many sites. For one worked instance with dates, a crawler-only 404 found, fixed at the second attempt, and cited twelve days later follows a single page from fault to citation.
Training bots and search bots do different jobs.
Operators document these crawlers as collecting data that may be used to improve models. A request does not prove that a page entered a dataset or changed a later answer.
ChatGPT Training
Claude TrainingOperators document these crawlers as supporting AI search discovery. A successful fetch does not guarantee indexing, ranking, retrieval or citation.
ChatGPT SearchThe distinction shapes access policy and measurement. GPTBot and OAI-SearchBot use separate controls. Keep request evidence separate from later citation and referral observations.
OpenAI labels led this sample.
OpenAI labels led this sample
GPTBot and OAI-SearchBot labels represented about 72% of matched requests in this panel and window. ClaudeBot represented 3.8%.
Bytespider represented 9.2% of matched requests. Provider purpose and identity cannot be inferred from a user-agent label alone.
ChatGPT TrainingChatGPT Search
Meta
AmazonClaude TrainingHomepage rates differ by label.
Different training philosophies
Claude visits homepages 7x more often than ChatGPT Training. It wants to understand who you are. ChatGPT's training crawler skips straight to your content.
Claude TrainingChatGPT TrainingThe homepage rates differ in this sample. Treat that as a hypothesis about request paths, not proof of crawler strategy or a content recommendation.
Crawlers keep their own hours.
Weekend request ratios differed by label
GPTBot and OAI-SearchBot labels had higher weekend request averages in this sample. ClaudeBot had an 8% lower weekend average. The logs do not reveal why the schedules differed.
ChatGPT TrainingChatGPT SearchClaude TrainingWeekday publishes may get crawled faster by Anthropic. Weekend publishes may be picked up faster by OpenAI.
Blog pages show up as entry points.
Blog pages appeared as entry points
In the linked session sample, 21% of OAI-SearchBot-labeled entry pages were blogs. The request data does not reveal a user's question or prove that the page appeared in an answer.
Page type, internal links, URL inventory, demand and customer mix may all contribute. Compare the same page types with separate citation and referral evidence before drawing a product conclusion.
ChatGPT SearchEntry Point DistributionMost pages were requested once.
In this bounded window, 2.4% of URLs had at least three recorded requests. Activity outside the window or at an unobserved layer is not measured.
Requests skew towards shallow URLs.
/2.7%/about10.3%/blog/post19.6%/blog/2024/post51.7%/docs/api/auth12.0%/docs/api/v1/...3.7%Requests skewed toward mid-depth URLs
In this sample, GPTBot-labeled requests appeared most often on mid-depth pages and less than 3% were for homepages. The data does not prove how the URLs were discovered or why deeper URLs had fewer requests.
Compare important deep pages with access, internal-link, citation and referral evidence before changing site structure.
Reach and depth vary by crawler.
Reach vs depth
ChatGPT Search prioritizes breadth - visiting 76% of sites in our dataset. ChatGPT Training prioritizes depth - fewer sites but 5,586 visits per site on average. Claude is the most selective at just 470 visits each.
OAI-SearchBot labels appeared on 76% of sites in this panel and GPTBot labels appeared on 70%. Reach describes matching requests, not search eligibility, training depth or citation likelihood.
Questions to investigate.
Turn these observations into checks on your own request data before you change site structure.
Compare entry-page mix
21% of sessions in the linked ChatGPT Search sample began on blog pages. Check whether your own referrals show the same pattern.
Use a complete time window
88.5% of pages had one request inside this study window. Avoid turning a bounded frequency into a permanent crawler rule.
Audit important deep pages
52% of matched requests were at measured depth three or less. Compare access, internal links, content type and demand.
Separate purpose classes
The 72% of OpenAI-labelled requests combined a training crawler and a search crawler. Keep their policies and outcomes separate.
Treat label differences as hypotheses
A 7× homepage-rate ratio between two labels in this sample is a question to test on your data, not a rule.
