Serve an Automatic Robots.txt From Your PHP Backend

Overview

Use the REST API to generate and serve an automatically updating robots.txt from your website's PHP backend. Please contact us if you need help.

Step 1: Generate Your Robots.txt

Define a function that makes an HTTP request to the REST API with your project's access token. Select which AgentTypes you want to block, and a string specifying which URLs are disallowed (e.g. "/" to disallow all paths). Allowed agent types include:

• AI Agent
• AI Assistant
• AI Data Scraper
• AI Search Crawler
• Archiver
• Developer Helper
• Fetcher
• Headless Agent
• Intelligence Gatherer
• Scraper
• SEO Crawler
• Search Engine Crawler
• Security Scanner
• Undocumented AI Agent
• Uncategorized

Paste in this code:

function generate_dark_visitors_robots_txt() {
    $curl = curl_init('https://api.darkvisitors.com/robots-txts');

    curl_setopt_array($curl, [
        CURLOPT_POST => true,
        CURLOPT_HTTPHEADER => [
            'Authorization: Bearer YOUR_ACCESS_TOKEN',
            'Content-Type: application/json',
        ],
        CURLOPT_POSTFIELDS => json_encode([
            'agent_types' => [
                'AI Data Scraper',
                'Scraper',
                'Intelligence Gatherer',
                'SEO Crawler',
            ],
            'disallow' => '/'
        ], JSON_UNESCAPED_SLASHES),
        CURLOPT_RETURNTRANSFER => true,
    ]);

    $response = curl_exec($curl);

    if ($response === false) {
        $error = curl_error($curl);
        curl_close($curl);
        throw new RuntimeException('Error fetching robots.txt: ' . $error);
    }

    $status = curl_getinfo($curl, CURLINFO_RESPONSE_CODE);
    curl_close($curl);

    if ($status < 200 || $status >= 300) {
        throw new RuntimeException('Invalid response code fetching robots.txt: ' . $status);
    }

    return $response;
}

• Then, navigate to the Projects page and open your project
• Copy your access token from the Settings page
• Back in your code, swap in in your access token where it says YOUR_ACCESS_TOKEN

Here's how to use it:

$robots_txt = generate_dark_visitors_robots_txt();

The return value is a plain text robots.txt string.

Step 2: Serve Your Robots.txt

Generate a $robots_txt periodically (e.g. once per day), then cache and serve it from your website's /robots.txt endpoint.