Diffbot
Diffbot’s crawler. Builds a structured knowledge graph from web pages that many AI and data products license.
What Diffbot is for
General-purpose or knowledge-graph crawling with a broader purpose.
Your content can flow into downstream AI products indirectly through Diffbot’s knowledge graph rather than a single assistant.
Crawler guide
Evergreen identity, source, and robots.txt handling guidance.
Allow or block Diffbot
Honours robots.txt. Add one of these to your robots.txt.
User-agent: Diffbot Disallow: /
User-agent: Diffbot Allow: /
Other AI crawlers
Every bot Trakkr tracks is a doorway.
Common questions
What is Diffbot?
Diffbot is an AI web crawler operated by Diffbot. Diffbot’s crawler. Builds a structured knowledge graph from web pages that many AI and data products license.
How do I block Diffbot?
Add a directive to your robots.txt: "User-agent: Diffbot" followed by "Disallow: /". Honours robots.txt.
Does blocking Diffbot hide me from AI?
Your content can flow into downstream AI products indirectly through Diffbot’s knowledge graph rather than a single assistant.
Identity and robots.txt facts come from Diffbot's published bot documentation. Behaviour is measured from identified Diffbot requests in the server logs of the brands Trakkr tracks - 576K crawler visits across 84 sites, recounted as new data arrives.
Telemetry updated Feb 1, 2026.