Set Up Agent Analytics

Overview

Track the activity of all known artificial agents scraping, crawling, archiving, and gathering intelligence on your website. These visits likely account for 50% of your traffic.

In addition to realtime insight, agent analytics enables:

You can connect your website using the API or the WordPress plugin.

1. Create a New Project

Sign up and create a new project for your website if you haven't already.

2. Copy Your Access Token

3. Start Sending Visit Events

There are 2 ways to send visit events from your website to your project.

Option 1: Using the WordPress Plugin

Use this method for WordPress websites. Adding the plugin is quick and easy.

Option 2: Using the API

Use this method for server-side rendered websites. Simply make a request to the Visits endpoint every time a page view request is made to your website.

The Request

Endpoint
URL https://api.darkvisitors.com/visits
HTTP Method POST
Headers
Authorization A bearer token with your project's access token (e.g. Bearer 48d7dcbd-fc44-4b30-916b-2a5955c8ee42).
Content-Type This needs to be set to application/json
Body
request_path The URL path of your visitor's request
request_method The HTTP method of your visitor's request (e.g. GET, POST, etc.)
request_headers The HTTP headers of your visitor's request, as a key-value object

Tips

Example

This cURL example sends a visit. If the visit is from an artificial agent, it will show up in agent analytics.

curl -X POST https://api.darkvisitors.com/visits \
-H "Authorization: Bearer ${ACCESS_TOKEN}" \
-H "Content-Type: application/json" \
-d '{
    "request_path": "'"${request.path}"'",
    "request_method": "'"${request.method}"'",
    "request_headers": "'"${request.headers}"'"
}'

Here's an example of how to use this in practice for a Node.js backend:

// For each page view, send a visit without awaiting

fetch("https://api.darkvisitors.com/visits", {
    method: "POST",
    headers: {
        "Authorization": "Bearer " + ACCESS_TOKEN,
        "Content-Type": "application/json"
    },
    body: JSON.stringify({
        request_path: request.path,
        request_method: request.method,
        request_headers: request.headers
    })
}).catch((error) => {
    // Silently log the error
})

// Then render and return the page's HTML response

You can follow these examples to call the API in any language.

4. Test Your Integration

If your website is correctly sending visit events to the project, you should see a new visit from the Dark Visitor agent in the project's realtime timeline.