Skip to content

Diffbot

Other crawler

Diffbot crawler for extracting structured web data and maintaining its knowledge graph. Seen on 18 of 135 sites in 13 weeks.

Diffbot reached 8 of 92 sites last week

Sites that got at least one Diffbot request each week, against every site reporting that week.

Reach13%21st of 2613 of 104 sitesLast 4 weeks
Requests3.5K1% of all crawler requests13 weeks, 18 sites
Per site7a weekMedian where it visitsLast 4 weeks

Diffbot is not tied to answers in an AI assistant.

Diffbot says Diffbot follows robots.txt. A rule changes future crawls, not pages it already collected.

Block Diffbot
User-agent: Diffbot
Disallow: /
Allow Diffbot
User-agent: Diffbot
Allow: /
User agentMozilla/5.0 (compatible; Diffbot/0.1; +http://www.diffbot.com)

Is it really Diffbot?

Anyone can send the Diffbot name. Here is how to check a request that carries it.

How to check
1There is no address list to matchDiffbot does not publish addresses for Diffbot, so a matching name is a clue, not proof.
2Never count a secret-file requestA request for /.env or /.git is not a crawl, whatever name it carries.
See every fake crawler

Common questions

What is Diffbot?

Diffbot is Diffbot's other crawler. Diffbot crawler for extracting structured web data and maintaining its knowledge graph.

How do I block Diffbot?

Add a directive to your robots.txt: "User-agent: Diffbot" followed by "Disallow: /". Diffbot says Diffbot follows robots.txt. A rule changes future crawls, not pages it already collected.

Does blocking Diffbot hide me from AI?

No single bot rule hides a site from every AI product. Your content can flow indirectly into data products and AI systems that license Diffbot data.