Navigate to Train → Generative AI → Documents → Upload Document.

Select Sitemap as the content source.

Provide the Sitemap XML URL of the website.

Example:

The system will crawl all URLs listed inside the sitemap.

Select how the URLs should be processed from the sitemap.

Available option: Include all, Exclude some

This option crawls all URLs by default and allows you to exclude specific pages using regex rules.

Other options such as Exclude all, Include some and Custom will be available in future releases.

Regex rules allow you to exclude specific URLs from being indexed.

Example sitemap entry:

Example regex pattern:

You can create multiple rules using Add Regex Rule to Exclude.

Category: Select the appropriate category.

Language: Choose the language used on the website.

You can enable Auto Refresh to automatically re-index the sitemap content at regular intervals.

When Auto Refresh is enabled, the system will crawl the sitemap again and update the indexed content based on the configured refresh interval.

Specify how frequently the sitemap should be refreshed.

Example: 7 days – The system will re-crawl the sitemap and update the indexed pages every 7 days.

This ensures that any new or updated webpages in the sitemap are automatically included in the AI knowledge base.

Verify the sitemap URL, regex rules, and document details.

Click Save & Start Indexing to begin crawling the website

Once the sitemap indexing status shows Ready to Search, configure the agent workflow so the AI can retrieve knowledge from the indexed website content.

➡️ Continue with 3. Path SetupPath setup for Agents