Claude-User

What is Claude-User?

About

Claude-User is dispatched by Anthropic's Claude AI assistant in response to user prompts, when it needs to fetch content to include in its answers. You can see how often Claude-User visits your website by setting up Dark Visitors agent analytics.

Expected Behavior

AI assistants make one-off visits to websites based on user requests, rather than crawling the web in any automatic fashion. This allows the underlying LLM to incorporate up-to-date information outside of its training data in its answers, using a technique called retrieval-augmented generation (RAG).

Type

Detail

Insights

The percentage of all internet traffic coming from AI Assistants

Top Visited Website Categories

Robots.txt

Should I Block Claude-User?

Probably not. AI assistants visit websites directly on behalf of human users, so blocking them will effectively block those users. This could lead to a poor user experience and possible negative sentiment about your website. Not blocking AI assistants will allow more human users to use your website as they choose.

How Do I Block Claude-User?

You can block Claude-User or limit its access by setting user agent token rules in your website's robots.txt. Set up Dark Visitors agent analytics to check whether it's actually following them.

# robots.txt
# This should block Claude-User

User-agent: Claude-User
Disallow: /