Not sure where to start? Take a guided tour.
AI Site Grader
AI Query Scanner
llms.txt GeneratorDiffbotOther crawler
Diffbot
Diffbot is a Diffbot crawler for extracting structured web data and maintaining its knowledge graph. This record has no current Diffbot source, so treat a matching request as unverified.
Not tied to answers in an AI assistant.
- DiffbotOtherNot checked
Reads your live robots.txt and applies the same rules these crawlers use. Nothing is stored.
- robots.txt token
Diffbot- Follows robots.txt
- Yes, documented
- How to verify it
- Not published
- Seen on
- 13% of sites · #21
- Last checked
- 11 June 2026
Should you block Diffbot?
You can, but it may not listen. Diffbot has not documented Diffbot, so treat any rule as a request. Your content can flow indirectly into data products and AI systems that license Diffbot data.
User-agent: Diffbot
Disallow: /Asks Diffbot not to crawl any page. Add it to the robots.txt file at the root of your site, above any wider rules.
Allowing Diffbot only creates the possibility of retrieval. To find out whether your pages are selected, measure which answer engines actually cite your pages using a stable query set.
Spot it in your logs.
Match the name, then check where the request came from. Anyone can copy a user-agent.
Mozilla/5.0 (compatible; Diffbot/0.1; +http://www.diffbot.com)This string comes from a public crawler registry or observed naming, not current operator documentation.
- 1
Match the name
Search your server or CDN log for
Diffbot, case-insensitive. A match is a clue, not proof. - 2
Check the source
Diffbot publishes no IP ranges, reverse DNS or signatures for Diffbot in the sources here, so it cannot be verified. Treat matches as unconfirmed.
Serve the content in HTML
The cited operator material does not verify JavaScript rendering. Serve useful HTML before client-side JavaScript where possible.
How active is Diffbot?
Measured across the sites connected to Trakkr, 27 July 2026 to 25 October 2026.
of connected sites saw Diffbot in the four weeks to 25 October 2026. That is 13 of 104.
Share of connected sites it visited each week
- Rank by reach
- 21 of 26
- Typical site, per week
- 7 requests
- Sites reached, 13 weeks
- 18 of 135
Most widespread AI crawlers
Share of connected sites each one visited, last four weeks.
- 1
OAI-SearchBot84%
- 2
ClaudeBot84%
- 3
- 4
Amazonbot61%
- 5
- 6
PerplexityBot56%
- 7
Applebot51%
- 21
Diffbot13%
How this is measured
Finalized daily crawler summaries from sites connected to Trakkr, grouped by the crawler named in each request. Trakkr’s own site is excluded. A site counts as reached in a week when it logged at least one request from Diffbot. The typical figure is the median weekly requests among the sites it reached.
- This is a sample of connected sites, not the whole web.
- A matching user-agent does not prove the named operator sent the request.
- Requests are not unique pages, training use or citations.
- Crawlers seen on fewer than 10 sites are not shown. Classic search crawlers such as Googlebot are not collected on every site, so they are left out.
Questions, answered.
Diffbot is a Diffbot crawler for extracting structured web data and maintaining its knowledge graph. This record has no current Diffbot source, so treat a matching request as unverified.
Sources.
Reported, not operator verified. Operator facts and Trakkr’s measurements keep separate dates.
- [1]Reported crawler reference
Reported, not operator verified · checked 11 June 2026
github.com/ai-robots-txt/ai.robots.txt/blob/main/table-of-bot-metrics.md
Last source check 11 June 2026. Activity measured to 25 October 2026.
Crawlers like Diffbot.
All AI crawlersSee which AI crawlers read your site.
Trakkr shows which bots, Diffbot included, visit which pages each day, and whether the answers they feed mention you.
14-day free trial. Cancel anytime.