What is Brightbot? AI crawler guide
Brightbot is a crawler or fetcher associated with Bright Data. See source-graded user-agent, robots.txt, verification, and observation guidance.
What is Brightbot?
Brightbot is a documented Bright Data crawler or fetcher. Bright Data's declared crawler for collecting public web data for its own datasets.
Evidence status
| Field | Value |
|---|---|
| Evidence | Officially documented |
| Lifecycle | Active |
| Purpose | other |
| robots.txt posture | unverified |
| Source checked | 2026-08-11 |
Documented user-agent
Brightbot 1.0
robots.txt allow example
User-agent: Brightbot Allow: /
robots.txt block example
User-agent: Brightbot Disallow: /
What the rule can and cannot do
Robots.txt expresses an access policy to compliant automated crawlers. It does not authenticate the sender, remove content already collected, or guarantee that a model will use or not use content.
How to verify a request
- Match the complete documented user-agent where one exists. A match is only a clue.
- Use the operator-published check: Declared source subnet.
- Keep the observed request, operator documentation, and any inference as separate fields.
JavaScript behavior
The cited operator material does not verify JavaScript rendering. Serve useful HTML before client-side JavaScript where possible.
Related bots
- ImagesiftBot: Also tracked as a general crawler.
- omgili: Also tracked as a general crawler.
- ApifyBot: Also tracked as a general crawler.
- Google-Firebase: Also tracked as a general crawler.
- Amazonbot: Also tracked as a general crawler.
- GoogleOther-Video: Also tracked as a general crawler.
- Panscient: Also tracked as a general crawler.
- OAI-AdsBot: Also tracked as a general crawler.
- Scrapy: Also tracked as a general crawler.
- AI Crawlers: Brightbot is a concrete crawler example for this concept.
- Robots.txt: Robots.txt is the control file used to allow or block Brightbot.
Frequently Asked Questions
What is Brightbot?
Brightbot is a officially documented crawler or fetcher record associated with Bright Data.
What user-agent does Brightbot use?
Brightbot 1.0. Check the evidence status before attributing a matching request.
Can Brightbot be blocked in robots.txt?
The record identifies Brightbot as the token to review. Robots.txt is a request policy, not proof of model use or non-use.
How can I verify a Brightbot request?
Start with the full user-agent, then use declared source subnet from the operator. A user-agent match alone is not proof.
Data & Sources
- Bright Data documentation - Primary source for Brightbot crawler details.
- Brightbot verification: Declared source subnet - Bright Data declares the single subnet 82.97.199.0/24 for Brightbot.