AI crawlers, search bots, and training agents
A compact reference of the crawler identities OpenRevenue checks: what they are for, how they identify themselves, and where their verification sources live.
62 crawlers found
AI answers
User-triggered fetches when an assistant opens a page to answer someone.
ChatGPT-User
IP verifiedUser-triggered fetches when ChatGPT opens a page to answer a person.
Claude-User
IP verifiedUser-triggered fetches when Claude opens a page to answer a person.
Perplexity-User
IP verifiedUser-triggered fetches for Perplexity answers and source citations.
Google-Agent
IP verifiedGoogle user-triggered fetcher used by AI or product experiences.
Google-NotebookLM
IP verifiedGoogle fetcher associated with NotebookLM-style answer workflows.
Google-Read-Aloud
IP verifiedGoogle fetcher associated with read-aloud and assistant experiences.
GoogleAgent
IP verifiedGoogle user-triggered fetcher used by AI or product experiences.
MistralAI-User
IP verifiedUser-triggered fetcher from Mistral.
Copilot
UA matchUser-triggered Microsoft/Copilot fetches for AI answers.
Amzn-User
UA matchUser-triggered Amazon fetches for fresh answers in products such as Alexa.
DuckAssistBot
IP verifiedDuckDuckGo real-time crawler for AI-assisted answers with citations.
xAI-SearchBot
UA matchUser-triggered fetcher from xAI.
Grok-DeepSearch
UA matchUser-triggered deep-search fetcher from xAI.
meta-externalfetcher
UA matchMeta fetcher used when a person requests or shares a specific URL.
Kimi-User
IP verifiedUser-triggered fetcher from Moonshot AI.
Qwen-User
UA matchUser-triggered fetcher from Alibaba.
Search indexes
Discovery crawlers that refresh content for search and answer indexes.
OAI-SearchBot
IP verifiedOpenAI crawler used to surface sites in ChatGPT search results.
Claude-SearchBot
IP verifiedAnthropic crawler used to discover and refresh content for Claude search.
PerplexityBot
IP verifiedPerplexity crawler used to keep its answer and search index fresh.
Google-InspectionTool
IP verifiedSearch or answer-index crawler from Google.
Googlebot
IP verifiedGoogle crawler used for Search indexing and discovery. Cloudflare notes that Googlebot traffic can also support AI-related search features such as AI Overviews.
MistralAI-Index
IP verifiedSearch or answer-index crawler from Mistral.
Bingbot
IP verifiedMicrosoft crawler used for Bing Search indexing.
msnbot
IP verifiedLegacy Microsoft crawler used for search indexing.
Amzn-SearchBot
UA matchAmazon crawler used to make content eligible for Amazon search experiences.
meta-webindexer
UA matchMeta crawler used to improve Meta AI search result quality.
Kimi-SearchBot
IP verifiedSearch or answer-index crawler from Moonshot AI.
TikTokSpider
UA matchSearch or answer-index crawler from ByteDance.
Baiduspider
UA matchSearch or answer-index crawler from Baidu.
YouBot
UA matchSearch or answer-index crawler from You.com.
Training crawlers
Public-content crawlers that may improve models, datasets, or products.
GPTBot
IP verifiedOpenAI crawler for collecting public content that may improve future models.
ClaudeBot
IP verifiedAnthropic crawler for public content that may be used to improve Claude.
GoogleOther
IP verifiedGeneric Google crawler used by product teams for public content fetches, including internal research and development.
Google-Extended
UA matchRobots token controlling whether Google may use crawled content for Gemini model training and grounding; it is not a separate HTTP crawler user agent.
Google-CloudVertexBot
IP verifiedGoogle crawler used for targeted crawls requested by site owners building Vertex AI agents.
Applebot-Extended
IP verifiedRobots token controlling whether Apple may use crawled content for AI training.
Applebot
IP verifiedApple crawler Cloudflare classifies across search and training use cases.
Amazonbot
UA matchAmazon crawler used to improve Amazon products and services.
meta-externalagent
UA matchMeta crawler for indexing or improving Meta products and AI systems.
KimiBot
IP verifiedPublic-content crawler from Moonshot AI.
Bytespider
UA matchPublic-content crawler from ByteDance.
ERNIEBot
UA matchPublic-content crawler from Baidu.
QwenBot
UA matchPublic-content crawler from Alibaba.
ChatGLM-Spider
UA matchPublic-content crawler from Zhipu AI.
DeepSeekBot
UA matchPublic-content crawler from DeepSeek.
cohere-ai
UA matchPublic-content crawler from Cohere.
cohere-training-data-crawler
UA matchPublic-content crawler from Cohere.
AI2Bot
UA matchAllen Institute crawler used to find documents for AI research systems.
CCBot
IP verifiedCommon Crawl crawler that builds open web crawl datasets.
Other AI bots
Recognised AI bots that don't fit the categories above.
OAI-AdsBot
UA matchOpenAI crawler associated with ad or landing-page fetch workflows.
GrokBot
UA matchOther AI bot from xAI.
xAI-Bot
UA matchOther AI bot from xAI.
xAI-Grok
UA matchOther AI bot from xAI.
xAI-Web-Crawler
UA matchOther AI bot from xAI.
Grok
UA matchOther AI bot from xAI.
meta-externalads
UA matchMeta crawler used for advertising and business product use cases.
facebookexternalhit
UA matchMeta crawler used for shared link previews on Facebook, Instagram, and Messenger.
FacebookBot
UA matchMeta crawler traffic associated with Facebook bot fetches.
Doubaobot
UA matchOther AI bot from ByteDance.
YiyanBot
UA matchOther AI bot from Baidu.
TongyiBot
UA matchOther AI bot from Alibaba.
AliyunBot
UA matchOther AI bot from Alibaba.
See which AI bots crawl your site
OpenRevenue tracks every crawl server-side, verifies it against the provider's official IP ranges, and groups it by AI answers, indexing, and training.