Sitemap Setup Guide

step 1 — select sitemap source navigate to train → generative ai → documents → upload document select sitemap as the content source step 2 — enter sitemap url provide the sitemap xml url of the website example 1\ \<https //help engati ai/platform tutorials/sitemap xml> the system will crawl all urls listed inside the sitemap step 3 — configure processing options select how the urls should be processed from the sitemap available option include all, exclude some this option crawls all urls by default and allows you to exclude specific pages using regex rules other options such as exclude all, include some and custom will be available in future releases step 4 — configure regex rules (optional) regex rules allow you to exclude specific urls from being indexed example sitemap entry \<loc>\<https //help engati ai/google sheets>\</loc> example regex pattern ^https \\\\/\\\\/help\\\\ engati\\\\ ai\\\\/google sheets\\\\/?$ you can create multiple rules using add regex rule to exclude category select the appropriate category language choose the language used on the website step 5 set up indexing & sync you can enable auto refresh to automatically re index the sitemap content at regular intervals when auto refresh is enabled, the system will crawl the sitemap again and update the indexed content based on the configured refresh interval refresh interval specify how frequently the sitemap should be refreshed example 7 days – the system will re crawl the sitemap and update the indexed pages every 7 days this ensures that any new or updated webpages in the sitemap are automatically included in the ai knowledge base step 5 — review setup verify the sitemap url, regex rules, and document details click save & start indexing to begin crawling the website 📚 next steps once the sitemap indexing status shows ready to search , configure the agent workflow so the ai can retrieve knowledge from the indexed website content ➡️ continue with docid\ g jippcsr0vjwz4richmf for agents