Set Up Agent Analytics
Overview
Track the activity of all known artificial agents scraping, crawling, archiving, and gathering intelligence on your website. These visits likely account for 50% of your traffic.
In addition to realtime insight, agent analytics enables:
- Spike alerts: Get notified when an agent's traffic starts surging.
- Agent verification: See whether agent visits are legitimate or spoofed.
- Report exporting: Download CSV reports of all agent visits to your website.
You can connect your website using the API or the WordPress plugin.
1. Create a New Project
Sign up and create a new project for your website if you haven't already.
- Navigate to the Projects page
- Click the New Project button
- Enter your website details
- Click Create
2. Copy Your Access Token
- Click on your project
- Click Settings
- Copy your access token
3. Start Sending Visit Events
There are 2 ways to send visit events from your website to your project.
Option 1: Using the WordPress Plugin
Use this method for WordPress websites. Adding the plugin is quick and easy.
- Log in to your website's WordPress dashboard
- Click Plugins in the sidebar
- Search for "Dark Visitors" or download the plugin directly
- Click Install Now
- Click Activate
- Click Dark Visitors in the sidebar
- Paste your access token
- Click Save Changes
Option 2: Using the API
Use this method for server-side rendered websites. Simply make a request to the Visits endpoint every time a page view request is made to your website.
The Request
Endpoint | |
---|---|
URL | https://api.darkvisitors.com/visits |
HTTP Method | POST |
Headers | |
Authorization |
A bearer token with your project's access token (e.g. Bearer 48d7dcbd-fc44-4b30-916b-2a5955c8ee42 ). |
Content-Type |
This needs to be set to application/json |
Body | |
request_path |
The URL path of your visitor's request |
request_method |
The HTTP method of your visitor's request (e.g. GET , POST , etc.) |
request_headers |
The HTTP headers of your visitor's request, as a key-value object |
Tips
- Add this call in middleware to cover all of your pages from one place.
- Don't wait for the API response (i.e. make it non-blocking) to avoid adding latency to your page response.
- Ensure that errors are handled in a way that does not make your page response fail too.
Example
This cURL example sends a visit. If the visit is from an artificial agent, it will show up in agent analytics.
curl -X POST https://api.darkvisitors.com/visits \
-H "Authorization: Bearer ${ACCESS_TOKEN}" \
-H "Content-Type: application/json" \
-d '{
"request_path": "'"${request.path}"'",
"request_method": "'"${request.method}"'",
"request_headers": "'"${request.headers}"'"
}'
Here's an example of how to use this in practice for a Node.js backend:
// For each page view, send a visit without awaiting
fetch("https://api.darkvisitors.com/visits", {
method: "POST",
headers: {
"Authorization": "Bearer " + ACCESS_TOKEN,
"Content-Type": "application/json"
},
body: JSON.stringify({
request_path: request.path,
request_method: request.method,
request_headers: request.headers
})
}).catch((error) => {
// Silently log the error
})
// Then render and return the page's HTML response
You can follow these examples to call the API in any language.
4. Test Your Integration
- Navigate to the Projects page
- Click on your project
- Click Settings
- Click Visit With Dark Visitor
- Click realtime
If your website is correctly sending visit events to the project, you should see a new visit from the Dark Visitor agent in the project's realtime timeline.