Skip to content

See where you stand

Change it

Works withWordPressShopifyWebflowGitHubGoogleSlackAll integrations
FixesFixes now ship to WordPress, Shopify, Webflow and GitHub

By team

By goal

Done for you

Not sure where to start? Take a guided tour.Book a demo
Customer storyHow the voice AI company Futuro grew its traffic3×AI traffic in 60 days

Learn

Data and research

FreeAI Site GraderAI Query Scannerllms.txt GeneratorAll free tools
Open dataWhere AI sends people, across more than 6,000 sites

Common CrawlTraining crawler

CCBot

CCBot is Common Crawl’s crawler for building public web crawl datasets used by researchers and AI builders.

Training only. It does not decide whether you appear in answers.

Get the robots.txt ruleCommon Crawl documentation
Is CCBot allowed on your site?
  • CCBotTraining
    Not checked

Reads your live robots.txt and applies the same rules these crawlers use. Nothing is stored.

robots.txt token
CCBot
Follows robots.txt
Yes, documented
Last checked
11 June 2026

Should you block CCBot?

Only if you want your pages kept out of training data. Blocking CCBot keeps your pages out of future Common Crawl datasets, which can affect downstream training and discovery uses.

robots.txt
User-agent: CCBot
Disallow: /

Asks CCBot not to crawl any page. Add it to the robots.txt file at the root of your site, above any wider rules.

Ignored robots.txt?

CCBot collects pages for model training. Training inclusion is not the same as being cited, so measure where AI answers actually cite your site before drawing conclusions from crawl logs.

To see why CCBot access is worth checking rather than assuming, read a real crawl-to-citation chain with dates on every step.

Spot it in your logs.

Match the name, then check where the request came from. Anyone can copy a user-agent. Search your own logs for CCBot.

User-agentsource
CCBot/2.0 (https://commoncrawl.org/faq/)

Version numbers change. Match the highlighted name, not the whole string.

  1. 1

    Match the name

    Search your server or CDN log for CCBot, case-insensitive. A match is a clue, not proof.

  2. 2

    Check the published IP ranges and reverse DNS

    Common Crawl also documents reverse DNS for IPv4: a genuine CCBot hostname ends in crawl.commoncrawl.org and resolves back to the same address. index.commoncrawl.org/ccbot.json

Is this really CCBot?

Addresses and reverse DNS

No log handy?

Paste an address that claims to be CCBot. It is checked against Common Crawl’s published list and its reverse DNS. Nothing is stored.

Check any crawler’s address

Serve the content in HTML

The cited operator material does not verify JavaScript rendering. Serve useful HTML before client-side JavaScript where possible.

How active is CCBot?

Measured across the sites connected to Trakkr, 27 July 2026 to 25 October 2026.

36%

of connected sites saw CCBot in the four weeks to 25 October 2026. That is 37 of 104.

100%75%50%25%32%
CCBot visited between 17% and 37% of connected sites each week from 27 July 2026 to 25 October 2026.

Share of connected sites it visited each week

Rank by reach
15 of 26
Typical site, per week
40 requests
Sites reached, 13 weeks
47 of 135

Most widespread AI crawlers

Share of connected sites each one visited, last four weeks.

  1. 1OAI-SearchBot84%
  2. 2ClaudeBot84%
  3. 3GPTBot67%
  4. 4Amazonbot61%
  5. 5ChatGPT-User56%
  6. 6PerplexityBot56%
  7. 7Applebot51%
  8. 15CCBot36%
All 26 measured crawlers
How this is measured

Finalized daily crawler summaries from sites connected to Trakkr, grouped by the crawler named in each request. Trakkr’s own site is excluded. A site counts as reached in a week when it logged at least one request from CCBot. The typical figure is the median weekly requests among the sites it reached.

  • This is a sample of connected sites, not the whole web.
  • A matching user-agent does not prove the named operator sent the request.
  • Requests are not unique pages, training use or citations.
  • Crawlers seen on fewer than 10 sites are not shown. Classic search crawlers such as Googlebot are not collected on every site, so they are left out.

Questions, answered.

CCBot is Common Crawl's crawler for building public web crawl datasets used by researchers and AI builders.

Sources.

Officially documented. Operator facts and Trakkr’s measurements keep separate dates.

  1. [1]
    Common Crawl crawler documentation

    Officially documented · checked 11 June 2026

    commoncrawl.org/ccbot

First verified 11 June 2026. Last source check 11 June 2026. Activity measured to 25 October 2026.

All AI crawlers

See which AI crawlers read your site.

Trakkr shows which bots, CCBot included, visit which pages each day, and whether the answers they feed mention you.

14-day free trial. Cancel anytime.