Know which AI crawlers are real, what they do, and how to handle them

Allen Institute for AI crawler used to find web content for open language model datasets.

AI2 crawler token associated with Dolma/open language model dataset collection.

Amazon Kendra crawler token for intelligent enterprise search over configured content sources.

Amazon crawler used to improve its products and services and potentially train Amazon AI models.

Amazon agent token reported for Buy for Me shopping actions directed by customers.

Amazon search crawler that indexes pages so they can be retrieved and cited in Amazon search and assistant answers.

Amazon fetcher that retrieves a specific page because a person asked an Amazon assistant about it.

Legacy Anthropic robots.txt token that predates the current ClaudeBot, Claude-User and Claude-SearchBot names.

Token associated with crawlers run on the Apify scraping platform by its customers.

Apple crawler for search experiences across Spotlight, Siri, Safari, and related Apple surfaces.

Robots.txt control token for whether Applebot-crawled content may be used to train Apple foundation models.

Atlassian Rovo crawler used to index connected website content for AI search, assistants, and agents.

Amazon Bedrock web crawler connector token for customer-configured AI applications.

Microsoft Bing crawler used to crawl and index pages for Bing and Microsoft search-powered experiences.

Reported name for Brave Search crawling. Brave says its crawler does not advertise a differentiated user-agent.

Bright Data's declared crawler for collecting public web data for its own datasets.

ByteDance crawler associated with training and powering AI products.

Common Crawl's crawler for building public web crawl datasets used by researchers and AI builders.

OpenAI agent used when ChatGPT navigates websites for user-directed tasks.

User-triggered OpenAI fetcher for ChatGPT and Custom GPT actions.

Claude Code related agent token seen in crawler/user-agent lists.

Anthropic search crawler that indexes content to improve Claude search result relevance and accuracy.

Anthropic user-triggered fetcher for Claude answers that need a specific web page.

Reported Anthropic-related token seen in public crawler registries and Trakkr detection, but absent from Anthropic's current bot documentation.

Anthropic crawler for public web content that could contribute to Claude model training.

Cloudflare AutoRAG crawler used to index configured content for AI search applications.

Cohere token reported for retrieving data in response to user-initiated prompts.

Cohere training-data crawler token reported for downloading web data for enterprise language models.

DeepSeek crawler token reported for training language models and improving AI products.

Diffbot crawler for extracting structured web data and maintaining its knowledge graph.

DuckDuckGo AI assistant fetcher used by DuckAssist to retrieve content for real-time answers.

Crawler for Exa's search index, which AI products and agents query to retrieve source pages.

Meta crawler historically documented for Facebook crawling and AI-related training uses.

Meta link preview crawler used when content is shared on Meta family apps.

Firecrawl agent token for AI scraping and web-to-LLM data extraction workflows.

Gemini Deep Research agent token reported for collecting and scanning resources used in research answers.

Google user-triggered fetcher used by agents that act on a person's request, including Project Mariner.

Google crawler used for site-owner-requested crawls related to Vertex AI Agents.

Robots.txt product token that controls eligible use of Google-crawled content for Gemini training and grounding.

Google Firebase AI product token reported for app-related fetches.

Gemini CLI related token listed in AI crawler registries for coding-agent activity.

Google fetcher that reads a URL because someone added it as a source inside Gemini or NotebookLM.

Superseded name for the Google fetcher that reads a URL when someone adds it as a source in Gemini or NotebookLM.

Google AI agent token associated with browser-style task execution.

Google Search crawler used to discover, crawl, render, and index pages for Google Search.

Google generic crawler used by product teams for publicly accessible content fetches outside core Googlebot.

Google product-specific image crawler token for publicly accessible content fetches.

Google product-specific video crawler token for public content fetches.

OpenAI crawler for content that may be used to improve generative AI foundation models.

xAI crawler token listed by public crawler directories for Grok-related crawling.

iAsk crawler used to provide answers to user queries.

Ibou crawler for building a graph representation of the web used in search.

NICT crawler for data used in artificial intelligence technologies and third-party research/commercial uses.

ImageSift crawler for public image and page data used in web intelligence products.

Open-source image dataset downloader token used to collect images for machine learning datasets.

Kagi search crawler that indexes pages for its search index and the answers built on it.

Moonshot AI search crawler that indexes pages so they can be retrieved and cited in Kimi answers.

Moonshot AI fetcher that retrieves a specific page because a person asked Kimi about it.

Moonshot AI crawler for content that may be used to improve its Kimi models.

Klaviyo AI crawler for indexing configured content to tailor AI experiences and recommendations.

LAION downloader token used in machine learning research dataset collection.

Meta crawler documented alongside its other external agents, used for advertising related page reads.

Meta crawler for indexing content directly for AI model training and product improvement use cases.

Meta user-requested fetcher for AI and link features across Meta products.

Meta crawler for improving Meta AI search result quality and source linking.

Mistral automated crawler for indexing content used by Mistral AI search in Vibe.

Mistral crawler for content that may be used to train its models, kept separate from its search index crawler.

Mistral user-action fetcher for Vibe responses that need a source page.

Amazon Nova Act agent token reported for browser-style task execution.

OpenAI crawler that reviews the safety and relevance of pages submitted as ChatGPT ad landing pages.

OpenAI search crawler for indexing pages that can appear in ChatGPT search results.

Webz.io crawler for collecting web data sold through APIs and datasets.

Legacy Omgili/Webz.io crawler token for web data collection.

Huawei crawler token reported for training data collection for the PanGu multimodal LLM.

Panscient crawler for collecting and structuring business data with AI and machine learning.

User-triggered Perplexity fetcher for pages needed to answer a specific question.

Perplexity crawler for surfacing and linking websites in Perplexity search results.

Huawei crawler used for recommendations, assistant features, and AI search services.

Phind crawler token associated with AI-enhanced developer search.

SB Intuitions crawler for data used in AI development and information analysis.

Scrapy framework user-agent commonly used for web scraping, including AI and machine learning data extraction.

Semrush crawler for the ContentShake AI content tool.

Semrush crawler for SEO Writing Assistant URL checks.

Parallel user-triggered fetcher that reads content at a user's direction rather than crawling the web automatically.

Parallel crawler for discovering and indexing websites for Parallel web APIs.

Ceramic AI crawler token for downloading data used to train LLMs.

ByteDance/TikTok crawler token reported alongside AI crawling lists.

Timpi crawler reported for scraping data used in search and AI model training contexts.

Velen crawler for business datasets and machine learning models.

Webz.io token covering whether crawled content may be included in the datasets it resells for AI and machine learning use.

Crawler token observed retrieving pages for xAI's Grok search and answer features.

Yandex crawler token for data used in YandexGPT quick answers and additional analysis.

Yandex additional crawler token for YandexGPT-related answer and analysis features.

Reported You.com crawler string associated with web search and AI answer retrieval.