What is xAI-SearchBot? AI crawler guide
xAI-SearchBot is a crawler or fetcher associated with xAI. See source-graded user-agent, robots.txt, verification, and observation guidance.
What is xAI-SearchBot?
xAI-SearchBot is a reported crawler or fetcher string associated with xAI, but current operator evidence is incomplete. Crawler token observed retrieving pages for xAI's Grok search and answer features.
Evidence status
| Field | Value |
|---|---|
| Evidence | Reported, not operator verified |
| Lifecycle | Status uncertain |
| Purpose | search |
| robots.txt posture | unverified |
| Source checked | 2026-08-11 |
Reported user-agent
Mozilla/5.0 (compatible; xAI-SearchBot/1.0; +https://x.ai)
robots.txt allow example
User-agent: xAI-SearchBot Allow: /
robots.txt block example
User-agent: xAI-SearchBot Disallow: /
What the rule can and cannot do
Robots.txt expresses an access policy to compliant automated crawlers. It does not authenticate the sender, remove content already collected, or guarantee that a model will use or not use content.
How to verify a request
- Match the complete documented user-agent where one exists. A match is only a clue.
- Look for operator-published IP ranges, reverse DNS, or signed-request guidance. None is attached to this record.
- Keep the observed request, operator documentation, and any inference as separate fields.
JavaScript behavior
The cited operator material does not verify JavaScript rendering. Serve useful HTML before client-side JavaScript where possible.
Related bots
- Kimi-SearchBot: Also tracked as a search crawler.
- Amzn-SearchBot: Also tracked as a search crawler.
- Kagibot: Also tracked as a search crawler.
- YandexAdditional: Also tracked as a search crawler.
- MistralAI-Index: Also tracked as a search crawler.
- OAI-SearchBot: Also tracked as a search crawler.
- Bravebot: Also tracked as a search crawler.
- Claude-SearchBot: Also tracked as a search crawler.
- ExaSearchBot: Also tracked as a search crawler.
- Robots.txt: Robots.txt is the control file used to allow or block xAI-SearchBot.
- AI Crawlers: xAI-SearchBot is a concrete crawler example for this concept.
- YandexAdditionalBot: Also tracked as a search crawler.
Frequently Asked Questions
What is xAI-SearchBot?
xAI-SearchBot is a reported, not operator verified crawler or fetcher record associated with xAI.
What user-agent does xAI-SearchBot use?
Mozilla/5.0 (compatible; xAI-SearchBot/1.0; +https://x.ai). Check the evidence status before attributing a matching request.
Can xAI-SearchBot be blocked in robots.txt?
The record identifies xAI-SearchBot as the token to review. Robots.txt is a request policy, not proof of model use or non-use.
How can I verify a xAI-SearchBot request?
Treat the user-agent as a clue and look for operator-published IP, reverse-DNS, or signature guidance.
Data & Sources
- xAI-SearchBot source reference - Source used to verify xAI-SearchBot.