Skip to content

Bytespider

Training crawler

ByteDance's crawler for pages that may be used to train its models. Seen on 71 of 135 sites in 13 weeks.

Bytespider reached 42 of 92 sites last week

Sites that got at least one Bytespider request each week, against every site reporting that week.

Reach48%8th of 2650 of 104 sitesLast 4 weeks
Requests3.2M8% of all crawler requests13 weeks, 71 sites
Per site394a weekMedian where it visitsLast 4 weeks

Blocking Bytespider is a training choice.

Crawler registries and Trakkr's seed registry flag inconsistent robots.txt compliance.

Block Bytespider
User-agent: Bytespider
Disallow: /
Allow Bytespider
User-agent: Bytespider
Allow: /
User agentMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Bytespider; spider-feedback@bytedance.com

Is it really Bytespider?

Anyone can send the Bytespider name. Here is how to check a request that carries it.

How to check
1There is no address list to matchByteDance does not publish addresses for Bytespider, so a matching name is a clue, not proof.
2Never count a secret-file requestA request for /.env or /.git is not a crawl, whatever name it carries.
See every fake crawler

Common questions

What is Bytespider?

Bytespider is ByteDance's training crawler. ByteDance crawler associated with training and powering AI products.

How do I block Bytespider?

Add a directive to your robots.txt: "User-agent: Bytespider" followed by "Disallow: /". Crawler registries and Trakkr's seed registry flag inconsistent robots.txt compliance.

Does blocking Bytespider hide me from AI?

No single bot rule hides a site from every AI product. Blocking it is a training-data policy choice, but ByteDance compliance has been reported as inconsistent by crawler trackers.