# See https://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file # HINT: See https://developers.cloudflare.com/fundamentals/get-started/reference/cdn-cgi-endpoint/#disallow-using-robotstxt # 'web crawlers may also mistakenly crawl these endpoints' user-agent: * Disallow: /cdn-cgi/ Disallow: /chocoBRAIN/