Agents
AI Search Crawlers
Every known artificial agent (bot) on the internet. You can track their activity on your website with Agent Analytics, or control their behavior with Robots.txt Categories.
AI Search Crawlers
AddSearchBot
AddSearchBot is a web crawler that indexes website content for AddSearch's AI-powered site search solution, collecting data to provide fast and accurate search results.
AI Search Crawler
See More →
Amazonbot
Amazonbot is Amazon's web crawler that collects web content to improve services like Alexa and other AI-powered features, while respecting robots.txt and standard crawling protocols.
AI Search Crawler
See More →
Anomura
Anomura is Direqt's web crawler that discovers and indexes links and metadata from websites for inclusion in Direqt's AI search results.
AI Search Crawler
See More →
Applebot
Applebot is Apple's web crawler that gathers data to power search features across Apple's ecosystem including Spotlight, Siri, and Safari search functionality.
AI Search Crawler
See More →
atlassian-bot
atlassian-bot is Atlassian's AI web crawler that indexes custom websites for Rovo, allowing the indexed content to appear in Rovo Search results and be used by Rovo Chat and Agents.
AI Search Crawler
See More →
Bravebot
Bravebot is Brave Search's web crawler that discovers and indexes new pages for Brave's AI search engine.
AI Search Crawler
See More →
Channel3Bot
Channel3Bot is Channel3's web crawler that visits publicly accessible product detail pages to index product information into a universal product catalog. The bot collects product details such as images, titles, descriptions, prices, and availability to enable AI apps and agents to discover products and direct shoppers to the original websites.
AI Search Crawler
See More →
Claude-SearchBot
Claude-SearchBot is used to create an index of websites that can be surfaced as results in Anthropic's Claude AI assistant search feature.
AI Search Crawler
See More →
Cloudflare-AutoRAG
Cloudflare-AutoRAG is Cloudflare's web crawler that indexes websites for AI Search (formerly AutoRAG), a managed retrieval-augmented generation service. The crawler automatically ingests and continuously indexes website content to enable natural language search and AI-powered applications without requiring manual reprocessing.
AI Search Crawler
See More →
Google-CloudVertexBot
Google-CloudVertexBot is a web crawler operated by Google that collects content for Google Cloud's Vertex AI Search service. This crawler indexes web pages to power enterprise search solutions that organizations use for custom search experiences in their websites and applications.
AI Search Crawler
See More →
LinkupBot
LinkupBot is a web crawler operated by Linkup that indexes enterprise web content for its AI search platform. The bot visits websites to collect relevant business information that's used to power Linkup's services.
AI Search Crawler
See More →
meta-webindexer
meta-webindexer is Meta's web crawler that browses the internet to improve search results for Meta AI users. It analyzes online content to make Meta AI's responses more relevant and accurate, allowing proper citation and linking back to source content.
AI Search Crawler
See More →
OAI-SearchBot
OAI-SearchBot is OpenAI's web crawler that indexes websites for SearchGPT, collecting and analyzing web content to power AI-driven search results and real-time information retrieval in OpenAI's search applications.
AI Search Crawler
See More →
PerplexityBot
PerplexityBot is the web crawler for Perplexity AI that indexes web content to power the AI search engine's real-time information retrieval and answer generation capabilities, enabling conversational search experiences.
AI Search Crawler
See More →
PetalBot
PetalBot is Huawei's web crawler that indexes PC and mobile websites to build search databases for Petal Search engine and provide AI-powered content recommendations for Huawei Assistant and other Huawei AI services.
AI Search Crawler
See More →
YouBot
YouBot is a web crawler used by You.com to index search results that allow their AI Assistant to answer user questions. The assistant's answers normally contain references to the website as inline sources.
AI Search Crawler
See More →
ZanistaBot
ZanistaBot is an AI search crawler operated by Zanista. If you think this is incorrect or can provide additional detail about its purpose, please let us know.
AI Search Crawler
See More →