Crawlers
share of sites each AI crawler reached, last 4 weeks
Who crawls
26 crawlers seen on 10 or more sites in 13 weeks. Requests are all 13 weeks; reach is the last 4.
OAI-SearchBot works the night shift. GPTBot never stops.
Share of each crawler's visits by hour of day, UTC. From 575,788 visits to 84 brand sites. Snapshot, 1 Feb 2026.
GPTBot reads 60 pages a visit. ClaudeBot reads 5.
Where each crawler's visits land, by folders below the homepage. Snapshot, 1 Feb 2026.
Revisits
Snapshot, 1 Feb 2026of pages a crawler fetched got one visit and no more. Fewer than 1 in 100 got six or more.
Overlap
75 brand sitesof brand sites saw all three of GPTBot, OAI-SearchBot and ClaudeBot. 20% saw just one.
Serving Markdown changed nothing, except for GPTBot.
Share of 9,033 pages each crawler reached, with each page randomly served as HTML or Markdown. Snapshot, 5 Aug 2026.
On best-for pages the gap was wider: GPTBot reached 67% of the HTML pages and 5% of the Markdown ones.
Sites with llms.txt were cited no more often.
AI citations per site across 37,894 cited sites, split by whether they publish llms.txt. Scan, 14 Mar 2026.
Dots mark the average, the black line the middle site. Among the 5,000 most-cited sites the difference is too small to tell from chance (p = 0.85).
How we measure
Reach and requests come from daily crawler summaries on the 135 sites connected to Trakkr, grouped by the user agent each request carries. Trakkr's own site is left out. The rhythm, depth and revisit bands come from Study 003; the Markdown test from Study 010; llms.txt from Study 005.
A sample of connected sites, not the whole web. A matching user agent does not prove who sent the request. Counts are requests, not pages, training use or citations. Crawlers seen on fewer than 10 sites are left out.
Which AI crawler reaches the most sites?
In the four weeks to 25 Oct 2026, OAI-SearchBot (84%), ClaudeBot (84%) and GPTBot (67%) reached the most of the 104 sites connected to Trakkr. That is reach within Trakkr's sample, not global market share, and user-agent labels do not prove operator identity.
Does llms.txt change how AI crawls your site?
We have not tested crawling against llms.txt. We compared citations: in a scan of 37,894 AI-cited sites on 14 Mar 2026, sites with llms.txt averaged 6.8 AI citations and sites without averaged 6.7, with 3 for the middle site either way. Among the 5,000 most-cited sites the difference is too small to tell from chance (p = 0.85). The scan compares sites; it does not test cause.
Do AI crawlers read Markdown differently from HTML?
Mostly not. In Study 010 (snapshot 5 Aug 2026), 9,033 pages were randomly served as HTML or Markdown. ClaudeBot, OAI-SearchBot, ChatGPT-User and PerplexityBot reached about the same share of each. GPTBot was the exception: it reached 61% of the HTML pages and 45% of the Markdown ones.