Crawlers
What AI crawlers actually do when they reach a website - who shows up, when, and how deep they read - built from real server logs across the brands Trakkr tracks. Then two live experiments on whether Markdown or llms.txt change how AI reads you.
The daily crawl rhythm
When GPTBot fetches over a day, in UTC. It peaks at 14:00 and goes quietest around 02:00.
Who's crawling
Every AI bot we see in the logs, ranked by share of all crawler visits. Filter by what each one is actually for.
| # | Crawler | Share of visits |
|---|---|---|
| 1 | 57.2% | |
| 2 | 15.1% | |
| 3 | 9.2% | |
| 4 | 7.9% | |
| 5 |  Amazonbot | 6.7% |
| 6 | 3.8% | |
| 7 | 0.1% | |
| 8 | <0.1% | |
| 9 | <0.1% | |
| 10 | <0.1% | |
| 11 | <0.1% | |
| 12 | <0.1% |
How each bot reads you
The big three behave nothing alike. Pick a crawler to see how deep it goes, how much it grabs per visit, and how widely it reaches.
GPTBot crawls hard and deep - long sessions, many pages, mostly 2-4 clicks in. The training workhorse.
One and done
How many times a page gets re-crawled after the first visit. Most pages are read exactly once.
The page has to be ready before the crawler arrives - because it usually won't come back. First-crawl quality beats ongoing tweaks.
How many bots reach you
Of the three biggest crawlers, how many a typical site actually sees.
Being visible to AI isn't one bot's job. Most sites that get found are crawled by OpenAI, Anthropic and the search bot alike - the “triple crown”.
Does serving Markdown change how AI reads you?
We serve half of our eligible pages as clean Markdown and half as HTML, randomly, then watch which version each bot fetches.
The median Markdown page is 79% lighter than its HTML twin - about 33% fewer bytes shipped to bots overall.
Markdown vs HTML, by bot
How much more of each version a bot fetched. Left = it preferred HTML, right = Markdown.
Does llms.txt earn you more citations?
llms.txt hands AI a clean map of your site. We matched adoption against citations across tens of thousands of domains - and the short answer is the honest one.
13.3% of scanned domains publish one, but pages with llms.txt and pages without earn the same median citations - no measurable lift (p=0.85). Today it's good hygiene, not a ranking lever.
Full llms.txt adoption - by tier, category and who's shipping itBehavior comes from identified AI-crawler requests in the server logs of the brands Trakkr tracks - 576K visits across 84 sites, Jun 11, 2025 - Feb 1, 2026. Dominant e-commerce brand excluded to ensure generalizable patterns. The Markdown and llms.txt findings are live, randomized experiments - they measure crawl and retrieval behavior, not whether an answer ultimately cited the page. For what all this crawling actually converts into, see Content.
Common questions
Which AI crawler is most active?
OpenAI’s crawlers (GPTBot and OAI-SearchBot) and Anthropic’s ClaudeBot are consistently among the most active across the 700K+ visits Trakkr observes, though shares shift over time. The dataset ranks every named AI crawler by visit volume.
Does llms.txt change how AI crawls your site?
Trakkr runs a live experiment on exactly this. So far, publishing an llms.txt file shows no statistically significant lift in AI citations - the dataset tracks the result as more data accumulates.
Do AI crawlers read Markdown differently from HTML?
That is the second live experiment on this page: serving bots Markdown versus HTML and measuring crawl coverage. Results are reported as they come in.