Not sure where to start? Take a guided tour.
AI Site Grader
AI Query Scanner
llms.txt GeneratorFree tool94 AI crawlersNo signup
AI robots.txt checker
See which AI crawlers your robots.txt lets in, from 49 companies at once. Training, search and user fetches are judged separately, with the exact lines to change.
Block training and stay in AI answers, or block it all. Your choice, checked.
- Training23 crawlers · Collects pages that may train AI modelsNot checked
- Search28 crawlers · Indexes pages so AI answers can cite themNot checked
- User fetch19 crawlers · Reads a page when a person asks about itNot checked
- Other24 crawlers · Ads, data products and AI toolsNot checked
Reads your live robots.txt once and applies the same rules each crawler uses. Nothing is stored.
Every AI crawler it checks.
Each company runs separate crawlers for training, search and user fetches. A rule for one does not cover the others.
| Company | Training | Search | User fetch | Other |
|---|---|---|---|---|
|
| – | ||
|
| |||
| – | – | |||
| – | – | ||
 Amazon |
|
| ||
| – |
| – | – | |
| – | ||||
| – | ||||
| – | – | – | ||
| – | – |
| |
| – | – | – | ||
| – | – | – |
| |
| – | – | – | ||
| – | – | – | ||
| – | – | – | ||
| – | – | – | ||
| – |
| – | |
| – | – | – | |
| – | – | – |
| |
| – | – |
| – | |
| – | – | – | ||
| – | – | – |
| |
|
| – | – | |
| – |
| – | – | |
| – |
| – | – | |
| – | – | – | ||
| – | – | – | ||
| – | – | – | ||
| – | – | – | ||
| – | – | – | ||
| – | – | – | ||
| – | – | – | ||
| – | – | – | ||
| – | – | – | |
| – | – | – | ||
| – | – | – | ||
| – |
| – | – | |
| – | – | – | ||
| – | – | – | ||
| – | – | – | ||
| – |
| – | – | |
| – | – | – | ||
| – | – | |||
|
| – | – | |
| – | – | – | ||
| – |
| – | – | |
| – | – | – |
|
“Reported” crawlers appear in public crawler lists but not in their operator’s current documentation. Web search crawlers are shown because blocking them removes a site from search.
How the check works.
The same rules the crawlers use, applied to your live file.
Read robots.txt once
It fetches /robots.txt over HTTPS and follows redirects. No file means every crawler may read every page. A file it can’t read is reported as unknown, not guessed.
Match each crawler
Each crawler obeys the groups that name it, or the * group when none do. The longest matching path wins, and Allow wins a tie (RFC 9309).
Compare with your goal
Pick “stay in AI answers, skip training” or “block AI crawlers”. It lists the crawlers that differ and writes only the lines you need.
Robots.txt asks; it doesn’t prove who visited. To check whether a request in your logs really came from GPTBot, Googlebot or another crawler, use the AI crawler verifier. To see which crawlers visit and how often, try the AI crawler log analyser.
Questions, answered.
94 crawlers from 49 companies: every crawler with a robots.txt name in Trakkr’s AI crawler directory. That covers training crawlers such as GPTBot and ClaudeBot, search crawlers such as OAI-SearchBot and PerplexityBot, and user fetchers such as ChatGPT-User. Googlebot, Bingbot and Applebot are included because blocking them removes a site from search.
See which AI crawlers actually visit.
Trakkr shows which crawlers read which pages each day, and whether the answers they feed mention you.
14-day free trial. Cancel anytime.