Agents
AI Search Crawlers
Every known artificial agent (bot) on the internet. You can track their activity on your website with agent analytics, or control their behavior with automatic robots.txt.
AI Search Crawlers
AddSearchBot
AddSearchBot is a web crawler that indexes website content for AddSearch's AI-powered site search solution, collecting data to provide fast and accurate search results.
AI Search Crawler
See More →
Amazonbot
Amazonbot is Amazon's web crawler that collects web content to improve services like Alexa and other AI-powered features, while respecting robots.txt and standard crawling protocols.
AI Search Crawler
See More →
Applebot
Applebot is Apple's web crawler that gathers data to power search features across Apple's ecosystem including Spotlight, Siri, and Safari search functionality.
AI Search Crawler
See More →
Claude-SearchBot
Claude-SearchBot is used to create an index of websites that can be surfaced as results in Anthropic's Claude AI assistant search feature.
AI Search Crawler
See More →
meta-webindexer
meta-webindexer is Meta's web crawler that browses the internet to improve search results for Meta AI users. It analyzes online content to make Meta AI's responses more relevant and accurate. When the crawler accesses your website, it allows Meta AI to properly cite and link back to your content when answering user questions, ensuring you get credit while helping the AI provide better information.
AI Search Crawler
See More →
OAI-SearchBot
OAI-SearchBot is OpenAI's web crawler that indexes websites for SearchGPT, collecting and analyzing web content to power AI-driven search results and real-time information retrieval in OpenAI's search applications.
AI Search Crawler
See More →
PerplexityBot
PerplexityBot is the web crawler for Perplexity AI that indexes web content to power the AI search engine's real-time information retrieval and answer generation capabilities, enabling conversational search experiences.
AI Search Crawler
See More →
PetalBot
PetalBot is Huawei's web crawler that indexes PC and mobile websites to build search databases for Petal Search engine and provide AI-powered content recommendations for Huawei Assistant and other Huawei AI services.
AI Search Crawler
See More →
YouBot
YouBot is a web crawler used by You.com to index search results that allow their AI Assistant to answer user questions. The assistant's answers normally contain references to the website as inline sources.
AI Search Crawler
See More →
ZanistaBot
ZanistaBot is an AI search crawler operated by Zanista. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
AI Search Crawler
See More →