Set Up Automatic Robots.txt

Overview

Protect sensitive content from unwanted access and scraping. Generate a continuously updating robots.txt that stays up to date with all current and future bots in the specified categories automatically.

Choose from the integration options below based on how your website is built. If you don't see an option that works for you, please ask us for help.

Option 1: Connect Using Your Platform

Option 2: Connect From Your Backend