Set Up a Robots.txt

Hosting a robots.txt with the agent list will help protect your website from unwanted access by artificial agents. There are 3 ways to do it.

Option 1: Use the API

You can generate your robots.txt using the free API. The agent list is updated continuously, so calling the endpoint periodically will automatically keep your robots.txt up to date. Sign up to get started.

Option 2: Use the Wordpress Plugin

The Wordpress plugin will also keep your robots.txt up to date automatically. Setting it up only takes a few seconds.

Option 3: Use the Example

In this example, all currently known AI data scrapers and undocumented AI agents are blocked. You can use it as a starting point and manually customize it using the agent list. If you sign up, you’ll get notified when new agents are added. However, using the API or Wordpress plugin is a more effective strategy since the agent list is updated continuously.

# Dark Visitors Robots.txt

# AI Data Scraper
# https://darkvisitors.com/agents/bytespider

User-agent: Bytespider
Disallow: /

# AI Data Scraper
# https://darkvisitors.com/agents/ccbot

User-agent: CCBot
Disallow: /

# AI Data Scraper
# https://darkvisitors.com/agents/diffbot

User-agent: Diffbot
Disallow: /

# AI Data Scraper
# https://darkvisitors.com/agents/facebookbot

User-agent: FacebookBot
Disallow: /

# AI Data Scraper
# https://darkvisitors.com/agents/google-extended

User-agent: Google-Extended
Disallow: /

# AI Data Scraper
# https://darkvisitors.com/agents/gptbot

User-agent: GPTBot
Disallow: /

# AI Data Scraper
# https://darkvisitors.com/agents/omgili

User-agent: omgili
Disallow: /

# Undocumented AI Agent
# https://darkvisitors.com/agents/anthropic-ai

User-agent: anthropic-ai
Disallow: /

# Undocumented AI Agent
# https://darkvisitors.com/agents/claude-web

User-agent: Claude-Web
Disallow: /

# Undocumented AI Agent
# https://darkvisitors.com/agents/claudebot

User-agent: ClaudeBot
Disallow: /

# Undocumented AI Agent
# https://darkvisitors.com/agents/cohere-ai

User-agent: cohere-ai
Disallow: /