AI Crawler Policy

AI Crawler Policy covers governance and technical controls over whether AI agents may fetch, index, or train on a site's content: robots.txt directives, rate limits, licensing considerations, and distinguishing training crawlers from search crawlers, without presenting control as perfectly enforceable. When access rules give way to improving the clarity and retrievability of content that permitted systems can access, see the GEO/AEO Fundamentals hub.