Not sure where to start? Take a guided tour.
AI Site Grader
AI Query Scanner
llms.txt GeneratorWhen AI Comes to Your Website
Crawler share, entry patterns, revisit behavior, and bot-specific session styles. Answer pages, reference facts, and live trackers drawn from this study.
Published 26 February 2026 · Updated 1 February 2026
22 results
Which AI crawler shows up most often?
GPTBot shows up most often by a wide margin. It accounts for 57.2% of observed crawler visits in the study, far ahead of OAI-SearchBot at 15.1%.
Do AI crawlers start on the homepage?
Usually no, especially for GPTBot. GPTBot hits the homepage only 2.75% of the time, while ClaudeBot is much more homepage-friendly at 19.18%.
Are most AI crawler visits one and done?
Yes. 88.5% of observed crawler sessions were single-visit sessions, while only 0.3% reached 10 or more visits.
Does GPTBot behave differently from ClaudeBot?
Yes, very differently. GPTBot dominates raw volume and rarely touches the homepage, while ClaudeBot is smaller but more homepage-oriented and behaviorally diverse.
Do AI crawlers read sites deeply?
Usually not. The study is dominated by shallow sessions, with 88.5% ending after one visit and only 0.3% reaching 10 or more requests.
Are weekend patterns real for AI crawlers?
Yes. GPTBot runs 1.29x hotter on weekends than weekdays, and OAI-SearchBot shows an even stronger 1.41x weekend lift.
Why do entry pages matter so much for AI bots?
Because most crawler sessions are shallow and many bots never see your homepage. The entry page often decides whether the bot finds a citeable explanation or leaves with nothing useful.
Should you optimize for GPTBot and OAI-SearchBot separately?
Yes. GPTBot and OAI-SearchBot differ in scale and reading pattern, so a page architecture that works for one does not automatically capture the other at the same efficiency.
What does the crawler data say about site architecture?
It says AI crawlers are biased toward easy-to-hit pages and often do not traverse deeply enough to rescue weak structure. The architecture has to expose useful answers early.
Can you treat AI crawler traffic like normal SEO bot traffic?
Not safely. AI crawlers show different homepage preferences, session depths, and weekend patterns than classic search-engine assumptions would suggest.
What is the main operational takeaway from the crawler study?
Operational takeaways from the When AI Comes to Your Website crawler study.
Why is ClaudeBot worth watching even though its share is smaller?
Because its behavior is meaningfully different from the larger OpenAI bots. ClaudeBot hits the homepage far more often and explores a more diverse mix of page types despite only 3.8% share.
GPTBot controls more than half of observed AI crawler traffic
GPTBot is the largest crawler family in the observed dataset.
OAI-SearchBot is the second-largest crawler in the study
Search-oriented OpenAI traffic is substantial in its own right.
GPTBot rarely uses the homepage as an entry point
GPTBot overwhelmingly lands deeper in the site.
ClaudeBot is far more homepage-friendly than GPTBot
A comparative analysis of AI crawler entry points based on the study When AI Comes to Your Website.
Single-visit sessions dominate AI crawler behavior
AI crawlers are usually shallow readers.
Deep repeat sessions are extremely rare
Deep session depth is the exception, not the rule.
GPTBot and SearchBot both show weekend lift
Observed crawler timing is not evenly distributed through the week.
The study tracks more than 300,000 unique URLs
The dataset is wide enough to capture real site-architecture differences.
AI crawler traffic share by bot
Observed crawler traffic share across major AI bot families.
Crawler depth and homepage entry benchmark
Research tracker page for the study When AI Comes to Your Website, detailing the crawler depth and homepage entry benchmark.