Set Up Automatic Robots.txt
Overview
Protect sensitive content from unwanted access and scraping. Generate a continuously updating robots.txt that stays up to date with all current and future bots in the specified categories automatically.
Choose from the integration options below based on how your website is built. If you don't see an option that works for you, please ask us for help.