Global AI & Bot Traffic
Insights into the hidden ecosystem of AI agents, crawlers, scrapers, and other bots swarming across the web, as well as traffic driven from AI chat and search LLMs. This anonymized data comes from thousands of websites leveraging Dark Visitors.
Overview
Bot vs. Human Traffic
37%
Artificial agents (bots) make up 37% of all website visits
LLM Referral vs. Other Traffic
0.10%
0.10% of all human website visits are AI chat & search referrals
LLM Referral Click Rate (CTR)
2.59%
2.59% of all AI chat & search mention links were actually clicked
How Do I Get These Insights for My Website?
Use the WordPress plugin, Node.js package, or API to get started in seconds.
Crawlers & Scrapers
Traffic by Agent Type
AI Agent
AI Assistant
AI Data Scraper
AI Search Crawler
Archiver
Developer Helper
Fetcher
Headless Agent
Intelligence Gatherer
Scraper
Search Engine Crawler
Security Scanner
SEO Crawler
Uncategorized
Undocumented AI Agent
The overall volume of internet traffic coming from each agent type.
Most Active Agents
The most active agents on the internet.
Top Originating Countries
The countries that operate the most agents.
AI Agents
Time Spent Browsing
39 minutes
Average session duration
Pages Browsed
5.7
Average visits per session
Sessions from autonomous AI agents who intelligently navigate and interact with websites.
AI Chat & Search LLMs
Traffic by LLM Referrer
ChatGPT
Claude
Copilot
DeepSeek
Gemini
Meta AI
Mistral
Perplexity
The overall volume of internet traffic coming from each LLM referrer.
Top Mentioned Website Categories
The types of websites most frequently mentioned by LLMs in their responses.
Top Clicked (Referred) Website Categories
The types of websites in LLM responses most frequently clicked on by users.
Top Referred Countries
The countries where humans are most frequently referred to websites by LLMs.
Top Website Robots.txts
Blocked AI Data Scrapers Over Time
How many of the world's top 1000 websites are blocking various AI data scrapers in their robots.txt.
Breakdown by Agent
All AI-Related Agents
|
Agent
The name of the agent
|
Type
The category of the agent
|
Company
The company that operates the agent
|
Country
The country where visits normally originate
|
% Blocked
The percentage of top websites that block the agent
|
M/M Change
The monthly change in blocked percentage
|
% Traffic
The percentage of all traffic coming from the agent
|
M/M Change
The monthly change in traffic percentage
|
|---|---|---|---|---|---|---|---|
| ChatGPT Agent |
AI Agent
|
OpenAI | 🇺🇸 US |
0%
|
↓ 6%
|
0.0110%
|
↓ 20%
|
| GoogleAgent-Mariner |
AI Agent
|
N/A |
1%
|
↓ 6%
|
0.0000%
|
↑ 0%
|
|
| NovaAct |
AI Agent
|
Amazon | 🇺🇸 US |
2%
|
↓ 6%
|
0.0000%
|
↑ 94%
|
| AI2Bot-DeepResearchEval |
AI Assistant
|
Ai2 | 🇺🇸 US |
0%
|
↑ 0%
|
0.0000%
|
↑ 0%
|
| bigsur.ai |
AI Assistant
|
Big Sur AI | N/A |
1%
|
↑ 6%
|
0.0000%
|
↑ 0%
|
| ChatGPT-User |
AI Assistant
|
OpenAI | 🇺🇸 US |
15%
|
↓ 1%
|
2.5343%
|
↓ 5%
|
| Claude-User |
AI Assistant
|
Anthropic | 🇺🇸 US |
6%
|
↑ 11%
|
0.0012%
|
↑ 19%
|
| Devin |
AI Assistant
|
Devin AI | 🇺🇸 US |
2%
|
↓ 6%
|
0.0000%
|
↑ 337%
|
| DuckAssistBot |
AI Assistant
|
DuckDuckGo | 🇺🇸 US |
6%
|
↓ 6%
|
0.0218%
|
↑ 34%
|
| Gemini-Deep-Research |
AI Assistant
|
🇺🇸 US |
2%
|
↑ 16%
|
0.0027%
|
↓ 32%
|
|
| Google-NotebookLM |
AI Assistant
|
🇺🇸 US |
1%
|
↓ 6%
|
0.0029%
|
↑ 303%
|
|
| LinerBot |
AI Assistant
|
Liner | 🇺🇸 US |
2%
|
↑ 20%
|
0.0001%
|
↑ 36%
|
| meta-externalfetcher |
AI Assistant
|
Meta | 🇺🇸 US |
7%
|
↑ 6%
|
0.0000%
|
↑ 177%
|
| MistralAI-User |
AI Assistant
|
Mistral | 🇸🇪 SE |
5%
|
↓ 6%
|
0.0003%
|
↓ 22%
|
| Perplexity-User |
AI Assistant
|
Perplexity | 🇺🇸 US |
6%
|
↑ 10%
|
0.0102%
|
↑ 9%
|
| Poggio-Citations |
AI Assistant
|
Poggio | 🇺🇸 US |
0%
|
↑ 0%
|
0.0000%
|
↑ 0%
|
| QualifiedBot |
AI Assistant
|
Qualified | 🇺🇸 US |
2%
|
↓ 6%
|
0.0043%
|
↑ 16,088%
|
| Ai2Bot-Dolma |
AI Data Scraper
|
Ai2 | 🇺🇸 US |
4%
|
↓ 33%
|
0.0038%
|
↑ 137,531%
|
| Applebot-Extended |
AI Data Scraper
|
Apple | 🇺🇸 US |
14%
|
↓ 4%
|
0.0000%
|
↑ 0%
|
| Bytespider |
AI Data Scraper
|
ByteDance | 🇸🇬 SG |
16%
|
↓ 2%
|
0.2342%
|
↓ 9%
|
| CCBot |
AI Data Scraper
|
Common Crawl | 🇺🇸 US |
22%
|
↓ 5%
|
0.0551%
|
↑ 12%
|
| ChatGLM-Spider |
AI Data Scraper
|
Zhipu AI | N/A |
2%
|
↑ 0%
|
0.0000%
|
↑ 0%
|
| ClaudeBot |
AI Data Scraper
|
Anthropic | 🇺🇸 US |
18%
|
↓ 3%
|
0.3263%
|
↓ 17%
|
| CloudVertexBot |
AI Data Scraper
|
N/A |
0%
|
↓ 6%
|
0.0000%
|
↑ 0%
|
|
| cohere-training-data-crawler |
AI Data Scraper
|
Cohere | N/A |
4%
|
↓ 9%
|
0.0000%
|
↑ 0%
|
| Cotoyogi |
AI Data Scraper
|
Research Organization of Information and Systems | 🇯🇵 JP |
2%
|
↓ 6%
|
0.0011%
|
↑ 0%
|
| Datenbank Crawler |
AI Data Scraper
|
netEstate | N/A |
1%
|
↓ 6%
|
0.0000%
|
↑ 0%
|
| Diffbot |
AI Data Scraper
|
Diffbot | 🇺🇸 US |
12%
|
↓ 6%
|
0.0003%
|
↑ 81%
|
| FacebookBot |
AI Data Scraper
|
Meta | 🇩🇪 DE |
12%
|
↓ 5%
|
0.0000%
|
↑ 1,646%
|
| Google-Extended |
AI Data Scraper
|
🇩🇪 DE |
18%
|
↓ 6%
|
0.0000%
|
↑ 0%
|
|
| GoogleOther |
AI Data Scraper
|
🇺🇸 US |
3%
|
↓ 10%
|
0.1843%
|
↓ 38%
|
|
| GPTBot |
AI Data Scraper
|
OpenAI | 🇺🇸 US |
21%
|
↓ 7%
|
0.7711%
|
↓ 35%
|
| ICC-Crawler |
AI Data Scraper
|
NICT | 🇯🇵 JP |
3%
|
↓ 9%
|
0.0000%
|
↑ 20%
|
| imageSpider |
AI Data Scraper
|
ByteDance | N/A |
0%
|
↑ 0%
|
0.0000%
|
↑ 0%
|
| Kangaroo Bot |
AI Data Scraper
|
Kangaroo LLM | N/A |
3%
|
↓ 13%
|
0.0000%
|
↑ 0%
|
| laion-huggingface-processor |
AI Data Scraper
|
LAION | N/A |
0%
|
↑ 0%
|
0.0000%
|
↑ 0%
|
| LCC |
AI Data Scraper
|
University of Leipzig | 🇩🇪 DE |
0%
|
↑ 0%
|
0.0001%
|
↑ 0%
|
| meta-externalagent |
AI Data Scraper
|
Meta | 🇺🇸 US |
12%
|
↓ 4%
|
2.0372%
|
↓ 13%
|
| netEstate Imprint Crawler |
AI Data Scraper
|
netEstate | N/A |
1%
|
↓ 6%
|
0.0000%
|
↑ 0%
|
| omgili |
AI Data Scraper
|
Webz.io | 🇮🇱 IL |
13%
|
↓ 4%
|
0.0000%
|
↑ 0%
|
| PanguBot |
AI Data Scraper
|
Huawei | 🇨🇳 CN |
5%
|
↓ 11%
|
0.0000%
|
↓ 3%
|
| SBIntuitionsBot |
AI Data Scraper
|
SB Intuitions | 🇯🇵 JP |
1%
|
↑ 0%
|
0.0000%
|
↑ 0%
|
| Spider |
AI Data Scraper
|
Spider | N/A |
0%
|
↑ 0%
|
0.0000%
|
↑ 0%
|
| Timpibot |
AI Data Scraper
|
Timpi | 🇫🇷 FR |
7%
|
↓ 6%
|
0.0078%
|
↓ 86%
|
| VelenPublicWebCrawler |
AI Data Scraper
|
Velen | 🇧🇪 BE |
3%
|
↑ 41%
|
0.0205%
|
↑ 6%
|
| webzio-extended |
AI Data Scraper
|
Webz.io | 🇺🇸 US |
6%
|
↓ 10%
|
0.0000%
|
↑ 0%
|
| AddSearchBot |
AI Search Crawler
|
AddSearch | 🇺🇸 US |
1%
|
↓ 6%
|
0.0000%
|
↑ 0%
|
| Amazonbot |
AI Search Crawler
|
Amazon | 🇺🇸 US |
11%
|
↓ 1%
|
1.1164%
|
↓ 10%
|
| Applebot |
AI Search Crawler
|
Apple | 🇺🇸 US |
5%
|
↓ 1%
|
0.6386%
|
↑ 37%
|
| Claude-SearchBot |
AI Search Crawler
|
Anthropic | 🇺🇸 US |
6%
|
↑ 10%
|
1.3286%
|
↑ 1,864%
|
| Google-CloudVertexBot |
AI Search Crawler
|
🇺🇸 US |
4%
|
↑ 6%
|
0.0005%
|
↓ 57%
|
|
| LinkupBot |
AI Search Crawler
|
Linkup | 🇫🇷 FR |
0%
|
↑ 0%
|
0.0006%
|
↑ 0%
|
| meta-webindexer |
AI Search Crawler
|
Meta | 🇺🇸 US |
1%
|
↑ 48%
|
0.5802%
|
↑ 1,120%
|
| OAI-SearchBot |
AI Search Crawler
|
OpenAI | 🇺🇸 US |
10%
|
↑ 2%
|
0.9460%
|
↓ 7%
|
| PerplexityBot |
AI Search Crawler
|
Perplexity | 🇺🇸 US |
15%
|
↓ 4%
|
0.1269%
|
↑ 1%
|
| PetalBot |
AI Search Crawler
|
Huawei | 🇸🇬 SG |
8%
|
↓ 5%
|
0.7168%
|
↑ 9%
|
| YouBot |
AI Search Crawler
|
You.com | 🇺🇸 US |
8%
|
↑ 7%
|
0.0000%
|
↑ 0%
|
| ZanistaBot |
AI Search Crawler
|
Zanista | N/A |
1%
|
↑ 0%
|
0.0000%
|
↑ 0%
|
Are you looking for a particular insight? Contact us.