Set Up Robots.txt Categories for Your PHP Backend

Overview

Use the REST API to connect your PHP website to Robots.txt Categories in just a few seconds. Please contact us if you need help getting set up.

Step 1: Generate Your Robots.txt

Define a function that makes an HTTP request to the REST API with your project's access token. Select which AgentTypes you want to block, and a string specifying which URLs are disallowed (e.g. "/" to disallow all paths). Allowed agent types include:

• AI Agent
• AI Assistant
• AI Data Scraper
• AI Search Crawler
• Archiver
• Developer Helper
• Fetcher
• Automated Agent
• Intelligence Gatherer
• Scraper
• SEO Crawler
• Search Engine Crawler
• Security Scanner
• Undocumented AI Agent
• Uncategorized

Paste in this code:

function generate_dark_visitors_robots_txt() {
    $curl = curl_init('https://api.darkvisitors.com/robots-txts');

    curl_setopt_array($curl, [
        CURLOPT_POST => true,
        CURLOPT_HTTPHEADER => [
            'Authorization: Bearer YOUR_ACCESS_TOKEN',
            'Content-Type: application/json',
        ],
        CURLOPT_POSTFIELDS => json_encode([
            'agent_types' => [
                'AI Data Scraper',
                'Scraper',
                'Intelligence Gatherer',
                'SEO Crawler',
            ],
            'disallow' => '/'
        ], JSON_UNESCAPED_SLASHES),
        CURLOPT_RETURNTRANSFER => true,
    ]);

    $response = curl_exec($curl);

    if ($response === false) {
        $error = curl_error($curl);
        curl_close($curl);
        throw new RuntimeException('Error fetching robots.txt: ' . $error);
    }

    $status = curl_getinfo($curl, CURLINFO_RESPONSE_CODE);
    curl_close($curl);

    if ($status < 200 || $status >= 300) {
        throw new RuntimeException('Invalid response code fetching robots.txt: ' . $status);
    }

    return $response;
}

• Navigate to the Dark Visitors Projects page and open your project
• Copy your access token from the Settings page
• Back in your code, swap in your access token where it says YOUR_ACCESS_TOKEN

Here's how to use it:

$robots_txt = generate_dark_visitors_robots_txt();

The return value is a plain text robots.txt string.

Step 2: Serve Your Robots.txt

Generate a $robots_txt periodically (e.g. once per day), then cache and serve it from your website's /robots.txt endpoint.