AI Crawler & Agent User-Agents — live list
The user-agent signatures of AI crawlers, assistants, and autonomous-agent frameworks, with their operators. This is the live matching table used by the Agent Observatory classifier — machine-readable at GET https://agent-observatory-flame.vercel.app/ai-crawlers.json (CORS-open, free).
Want to know how your client is classified? Test it →
AI crawlers, assistants & agent frameworks · 30 signatures
| UA substring | Label | Operator |
|---|---|---|
| GPTBot | GPTBot | OpenAI |
| ChatGPT-User | ChatGPT-User | OpenAI |
| OAI-SearchBot | OAI-SearchBot | OpenAI |
| ClaudeBot | ClaudeBot | Anthropic |
| Claude-User | Claude-User | Anthropic |
| Claude-Web | Claude-Web | Anthropic |
| anthropic-ai | anthropic-ai | Anthropic |
| PerplexityBot | PerplexityBot | Perplexity |
| Perplexity-User | Perplexity-User | Perplexity |
| Google-Extended | Google-Extended | |
| Google-CloudVertexBot | Vertex | |
| GoogleAgent-Mariner | Project Mariner | |
| Gemini | Gemini | |
| meta-externalagent | Meta External Agent | Meta |
| FacebookBot | FacebookBot | Meta |
| Bytespider | Bytespider | ByteDance |
| Amazonbot | Amazonbot | Amazon |
| cohere-ai | cohere-ai | Cohere |
| CCBot | CCBot | Common Crawl |
| Diffbot | Diffbot | Diffbot |
| YouBot | YouBot | You.com |
| DuckAssistBot | DuckAssistBot | DuckDuckGo |
| Applebot-Extended | Applebot-Extended | Apple |
| MistralAI | MistralAI-User | Mistral |
| Operator | Operator | OpenAI |
| Devin | Devin | Cognition |
| browser-use | browser-use | framework |
| LangChain | LangChain | framework |
| LlamaIndex | LlamaIndex | framework |
| AutoGPT | AutoGPT | framework |
Ordinary crawlers & HTTP clients (non-AI) · 41 signatures
Search engines, link previewers, monitors, and raw HTTP libraries — non-human, but not AI agents. Kept separate so AI-traffic stats aren’t polluted.
GooglebotbingbotSlurpDuckDuckBotBaiduspiderYandexBotSogouExabotfacebookexternalhitLinkedInBotTwitterbotDiscordbotTelegramBotWhatsAppSlackbotcurl/Wgetpython-requestspython-httpxaiohttpGo-http-clientnode-fetchaxiosJava/okhttplibwww-perlPostmanRuntimeinsomniaHeadlessChromePhantomJSPlaywrightPuppeteerUptimeRobotPingdomStatusCakeSemrushBotAhrefsBotDotBotMJ12botPetalBotDataForSeoBot
Usage notes
- Matching is case-insensitive substring against the
User-Agentheader. - To welcome AI crawlers, serve an llms.txt; to refuse them, add
User-agent:blocks per label to robots.txt. - UA strings are self-reported and spoofable — for stronger classification see the detection signals this site scores.
- List evolves during the Observatory census; fetch the JSON for the current version.