AI bot directory

27 AI crawlers and fetchers, with copy-ready rules for each.

Model training and datasets

GPTBotOpenAICollects public pages to train OpenAI models.
ClaudeBotAnthropicCollects public pages to train Claude.
anthropic-aiAnthropicOlder Anthropic name, still listed by many sites. Harmless to keep.
Google-ExtendedGoogleOpt-out token for Gemini training. Does not change Google Search.
Applebot-ExtendedAppleOpt-out token for Apple's AI training. Siri and Spotlight indexing continue.
CCBotCommon CrawlBuilds the open Common Crawl archive that many models train on.
BytespiderByteDanceByteDance's web crawler. Third parties report it often ignores robots.txt.
Meta-ExternalAgentMetaCollects web content for Meta's AI models and products.
FacebookBotMetaListed by many sites as a Meta crawler. Not confirmed in Meta's own documentation here.
cohere-aiCohereListed by many sites as a Cohere crawler. No official documentation of this token was found.
DiffbotDiffbotExtracts web data for Diffbot's knowledge graph and data products.
OmgilibotWebz.ioWebz.io crawler that collects forum and news data it sells. Webz.io may have renamed it.
AI2BotAllen Institute for AICollects data for open research models such as OLMo.
ImagesiftBotHiveCrawls images for the Imagesift image search index.
PanguBotHuaweiListed by many sites as a Huawei crawler. No official documentation was found.

OAI-SearchBotOpenAIIndexes pages for ChatGPT search results. Not used for training.
Claude-SearchBotAnthropicIndexes pages to improve the quality of Claude's search answers.
PerplexityBotPerplexityIndexes pages so Perplexity can cite them in answers.
DuckAssistBotDuckDuckGoFetches pages to write DuckAssist answers.
AmazonbotAmazonAmazon's web crawler. Its purposes are listed on Amazon's crawler page.
YouBotYou.comIndexes pages for You.com's AI search.
PetalBotHuaweiHuawei's Petal Search crawler.

ChatGPT-UserOpenAIOpens a page when a ChatGPT user asks about it.
Claude-UserAnthropicOpens a page when a Claude user asks about it.
Perplexity-UserPerplexityOpens a page when a Perplexity user clicks or asks.
MistralAI-UserMistralOpens a page when a Le Chat user asks about it.
Meta-ExternalFetcherMetaFetches links on a user's request. Meta says it may bypass robots.txt.