Set Up Robots.txt Categories for Your Fastly Website

Overview

Use a Fastly Compute service to append Robots.txt Categories to your existing robots.txt file. Please contact us if you need help getting set up.

Step 1: Create a Compute Service

• Install and configure the Fastly command line tool on your computer
• Run this command to create and open a new folder for your project:

mkdir dark-visitors-robots-txt && cd dark-visitors-robots-txt

• Run this command to create a new project

fastly compute init

• Choose JavaScript as the language, and Empty starter for JavaScript as the template
• Open the dark-visitors-robots-txt folder in a code editor
• In the src/index.js file, paste the following code:

/// <reference types="@fastly/js-compute" />

const FASTLY_CDN_SERVICE_BACKEND_NAME = "cdn_service"
const FASTLY_DARK_VISITORS_API_BACKEND_NAME = "dark_visitors_api"

const DARK_VISITORS_ACCESS_TOKEN = "YOUR_ACCESS_TOKEN" // TODO: Swap in your access token
const ROBOTS_TXT_DISALLOW_PATH = "/"
const ROBOTS_TXT_AGENT_TYPES = [
    // TODO: Add blocked agent types
]

async function handleRequest(request) {
    const url = new URL(request.url)

    if (url.pathname === "/robots.txt") {
        const thisHeaders = new Headers(request.headers)
        thisHeaders.delete("accept-encoding")
        const thisRequest = new Request(request, {
            headers: thisHeaders
        })

        const [thisResponse, thatResponse] = await Promise.all([
            fetch(thisRequest, {
                backend: FASTLY_CDN_SERVICE_BACKEND_NAME
            }),
            fetchRobotsTXT()
        ])

        const [thisRobotsTXT, thatRobotsTXT] = await Promise.all([
            thisResponse.ok ? thisResponse.text() : "",
            thatResponse.ok ? thatResponse.text() : ""
        ])

        const robotsTXT = [
            thisRobotsTXT.trim(),
            "# BEGIN Dark Visitors Managed Content",
            thatRobotsTXT.trim(),
            "# END Dark Visitors Managed Content",
        ].join("\n\n")
    
        return new Response(robotsTXT, {
            headers: {
                "Content-Type": "text/plain"
            },
        })
    }

    return fetch(request, {
        backend: FASTLY_CDN_SERVICE_BACKEND_NAME
    })
}

async function fetchRobotsTXT() {
    return fetch("/robots-txts", {
        backend: FASTLY_DARK_VISITORS_API_BACKEND_NAME,
        method: "POST",
        headers: {
            "Authorization": `Bearer ${DARK_VISITORS_ACCESS_TOKEN}`,
            "Content-Type": "application/json",
        },
        body: JSON.stringify({
            agent_types: ROBOTS_TXT_AGENT_TYPES,
            disallow: ROBOTS_TXT_DISALLOW_PATH,
        }),
    })
}

addEventListener("fetch", (event) =>
    event.respondWith(handleRequest(event.request))
)

• Navigate back to the Dark Visitors Projects page and open your project
• Copy your access token from the Settings page
• Back in your code, swap in your access token where it says YOUR_ACCESS_TOKEN
• Where it says // TODO: Add blocked agent types, add the agent types you want to block, and a string specifying which URLs are disallowed (e.g. "/" to disallow all paths). Allowed agent types include:
  • "AI Agent"
  • "AI Assistant"
  • "AI Data Scraper"
  • "AI Search Crawler"
  • "Archiver"
  • "Developer Helper"
  • "Fetcher"
  • "Headless Agent"
  • "Intelligence Gatherer"
  • "Scraper"
  • "SEO Crawler"
  • "Search Engine Crawler"
  • "Security Scanner"
  • "Undocumented AI Agent"
  • "Uncategorized"
• Run this command to create a new Compute service for this project

fastly compute publish

Step 2: Set Up Service Chaining

Service chaining is how Fastly services connect to each other. This section assumes you don't already have any service chaining set up. If you do, you'll need to make adjustments to account for your existing services. You can see another example in the Fastly tutorial.

First, we need to add the Fastly backends referenced in the code.

• Run this command to add the Dark Visitors API backend:

fastly backend create --name=dark_visitors_api --address=api.darkvisitors.com --version=latest --autoclone --use-ssl

• Open the Fastly dashboard and select your website's CDN service
• Click Service configuration, then Domains in the sidebar
• If you don't already see one, create a new internal domain (e.g. YOUR_WEBSITE_NAME.global.ssl.fastly.net)
• Swap this domain in for both places where it says YOUR_CDN_SERVICE_DOMAIN, then run this command:

fastly backend create --name=cdn_service --address=YOUR_CDN_SERVICE_DOMAIN --override-host=YOUR_CDN_SERVICE_DOMAIN --version=latest --autoclone --use-ssl

Next, we need to update the external domain to point to the Compute service, rather than your CDN service.

• Open the Fastly dashboard and select the Compute service you just created (dark-visitors-robots-txt)
• Click Service configuration, then Domains in the sidebar
• Click Add domain
• Select Link an existing domain, and choose the external domain currently linked to your CDN service
• Click Save and activate the latest version

Step 3: Test Your Integration

If your website is correctly connected, you should see the new rules in your website's robots.txt.