Not sure where to start? Take a guided tour.
AI Site Grader
AI Query Scanner
llms.txt GeneratorWhat is the main operational takeaway from the crawler study?
Mostly, discovery is shallow and crawler-specific. Teams should prioritize answer-dense entry pages, strong links, and crawler-aware refresh timing over generic assumptions about deep crawling.
The key numbers.
- One-visit sessions
- Most AI crawler visits are one-and-done sessions.
- 88.5%
- GPTBot share
- GPTBot is the dominant crawler by observed visits.
- 57.2%
- OAI-SearchBot share
- SearchBot is the second-largest crawler in the study.
- 15.1%
If a bot only takes one or two meaningful steps, the page it lands on has to do much more work to ensure content is indexed.
The evidence.
| Measure | Result |
|---|---|
| One-visit sessions Most AI crawler visits are one-and-done sessions. | 88.5% |
| GPTBot share GPTBot is the dominant crawler by observed visits. | 57.2% |
| OAI-SearchBot share SearchBot is the second-largest crawler in the study. | 15.1% |
Population and method
Built from 575,788 AI crawler visits across 84 brands and 314,501 unique URLs observed between June 11, 2025 and February 1, 2026.
Where the evidence stops
This study measures observable crawler traffic rather than hidden fetches or internal cache reads.
One dominant e-commerce brand was excluded to keep the page-type patterns more generalizable.
Visit share and homepage preference vary sharply by bot family, so averages can hide crawler-specific behavior.
What to do with this
- Optimize entry pages and internal links for the crawlers that actually show up in your logs.
- Treat crawler families differently because they do not read the same site architecture the same way.
- Use low-depth answer pages to improve the odds that a one-visit crawler still finds something citeable.
This turns study findings into an operating rule teams can use when deciding what to publish, refresh, or measure next, especially given that one-visit sessions account for 88.5% of crawler activity.
Questions about this answer.
Cite this answer.
Source study: Crawler Behavior“Mostly, discovery is shallow and crawler-specific. Teams should prioritize answer-dense entry pages, strong links, and crawler-aware refresh timing over generic assumptions about deep crawling.”
Mack Grenfell · What is the main operational takeaway from the crawler study?
When AI Comes to Your Website · Trakkr Research · Updated 1 February 2026