Set Up Automatic Robots.txt

Overview

Protect sensitive content from unwanted access and scraping. Generate a continuously updating robots.txt that stays up to date with all current and future bots in the specified categories automatically.

To get started, choose from the integration options below based on how your website is built. Please reach out to us if you're not sure which one to pick. We're happy to answer any questions you have.

Option 1: Connect Using Your Platform

Looking for Vercel, Fastly, Akamai, AWS, or Netlify? Let us know.

Option 2: Connect From Your Backend