The robots.txt file is a plain text file that provides instructions to search engine crawlers (such as Googlebot and Bingbot) about which parts of your website they are allowed or restricted from crawling. Managing the file through BOLD CMS allows publishers to update crawler directives without requiring any code changes or deployments.

Steps to Update the robots.txt File

1. Log in to BOLD CMS.
2. Navigate to Content → Static Pages.
3. Create a new static page or open the existing robots.txt page for editing.
4. Configure the following fields:
   • Source Path – Set the path to /robots.txt.
   • MIME Type – Set the MIME type to text/plain.
   • Content – Enter the required robots.txt directives.
5. Click Save.
6. Publish the page.

Once published, the updated robots.txt file will be available at:

https://yourdomain.com/robots.txt

Replace yourdomain.com with your website's domain.

Benefits

• Manage the robots.txt file directly from BOLD CMS.
• No code changes or deployments are required.
• Control how search engines crawl and index your website.
• Publish updates quickly through the CMS.
• Maintain crawler directives from a single, centralized location.