Agents
AI Assistants
Every known artificial agent (bot) on the internet. You can track their activity on your website with agent analytics, or control their behavior with automatic robots.txt.
AI Assistants
Ai2Bot-DeepResearchEval
Ai2Bot-DeepResearchEval is operated by Ai2, a non-profit AI research institute. It's used to collect and scan resources used in deep research queries performed by Ai2's open source AI models.
AI Assistant
See More →
bigsur.ai
bigsur.ai is a web crawler operated by Big Sur AI that fetches website content to enable AI-powered web agents, sales assistants, and content marketing solutions for businesses.
AI Assistant
See More →
ChatGPT-User
ChatGPT-User is OpenAI's web crawler that visits websites when ChatGPT users request information. This enables ChatGPT to include links in its responses.
AI Assistant
See More →
Claude-User
Claude-User is dispatched by Anthropic's Claude AI assistant in response to user prompts, when it needs to fetch content to include in its answers.
AI Assistant
See More →
Devin
Devin is a software engineering AI assistant that can browse websites and perform web-based tasks, functioning as a collaborative AI teammate for engineering teams.
AI Assistant
See More →
DuckAssistBot
DuckAssistBot is a web crawler that scans websites to collect content for DuckDuckGo's AI-assisted answers feature, which generates brief responses to search queries using natural language technology and Wikipedia sources.
AI Assistant
See More →
Gemini-Deep-Research
Gemini-Deep-Research is the agent responsible for collecting and scanning resources used in Google Gemini's Deep Research feature, which acts as a personal research assistant.
AI Assistant
See More →
Google-NotebookLM
Google-NotebookLM is an AI-powered research and note-taking assistant that helps users synthesize information from their own uploaded sources, such as documents, transcripts, or web content. It can generate summaries, answer questions, and highlight key themes from the materials you provide, acting like a personalized research companion built on Google's Gemini model. Google-NotebookLM fetches source URLs when users add them to their notebooks, enabling the AI to access and analyze those pages for context and insights.
AI Assistant
See More →
LinerBot
LinerBot is the web crawler used by Liner AI assistant to gather information from academic sources and websites to provide accurate answers with line-by-line source citations for research and scholarly work.
AI Assistant
See More →
meta-externalfetcher
meta-externalfetcher is used by Meta to perform user-initiated fetches of individual links from AI assistant product functions.
AI Assistant
See More →
MistralAI-User
MistralAI-User is Mistral's AI assistant bot that performs web browsing and data gathering tasks for users in Le Chat, including opening web pages and retrieving information to answer user queries.
AI Assistant
See More →
Perplexity-User
Perplexity-User supports user actions within Perplexity. When users ask Perplexity a question, it might visit a web page to help provide an accurate answer and include a link to the page in its response.
AI Assistant
See More →
QualifiedBot
QualifiedBot is Qualified's web crawler that analyzes customer websites to provide contextual information for their AI-powered chatbots and conversational marketing platform.
AI Assistant
See More →