Robots.txt Generation

The robots.txt file tells search engine crawlers which pages they can access. When enabled, Reverb 2.0 copies a template robots.txt to your output directory and completes its Sitemap: line with the Sitemap Base URL, because crawlers require a full URL there.

Default robots.txt Content

The default template allows all crawlers to access all content and points them to the sitemap. With the Sitemap Base URL set to https://docs.example.com/help/, the generated file reads:

User-agent: *
Allow: /

Sitemap: https://docs.example.com/help/sitemap.xml

When Sitemap Base URL is empty, or Generate Sitemap is disabled, the Sitemap: line is left out.

Customizing robots.txt

To customize the robots.txt file, create a Format Override:

  1. In ePublisher Designer, expand your Reverb 2.0 target

  2. Navigate to Format > Pages > robots.txt

  3. Right-click and select Create Override

  4. Edit the file with your custom rules

ePublisher copies your rules as written. A relative Sitemap: line, such as Sitemap: sitemap.xml, is completed with the Sitemap Base URL the same way as in the default template. An absolute or protocol-relative Sitemap: line is kept as is.

Last modified date: 11/05/2026