robots.txt for AI Crawlers: Build an Allow/Block Policy Bot by Bot

A robots.txt file is a UTF-8 plain-text policy at an origin’s top-level /robots.txt URL. It tells compliant crawlers which paths they may fetch. For AI access, the useful unit of policy is not “AI bots” as one class: it is each documented agent, its purpose, and the outcome you want. That lets a site permit search discovery while declining model-training crawls, without pretending that robots.txt secures private content or binds undeclared scrapers.

The Robots Exclusion Protocol applies to one scheme, host, and port. Rules at https://example.com/robots.txt do not automatically govern https://docs.example.com, http://example.com, or another port. The file is public, and a conforming crawler reads its applicable User-agent, Allow, and Disallow rules before fetching site paths.

There is no robots.txt formula and no meaningful benchmark for the “right” percentage of AI crawlers to allow. The practical artifact is a policy ledger: one row per documented token, with a purpose, decision, scope, consequence, enforcement layer, owner, source, and review trigger.

Separate crawling, indexing, training, and retrieval first#

An allow/block list becomes unsafe when it collapses several activities into “AI access.” These are different controls:

OpenAI makes the search-versus-training distinction unusually clear in its crawler documentation: a publisher can allow OAI-SearchBot for ChatGPT search while disallowing GPTBot for potential foundation-model training. Anthropic similarly documents three separate agents: Claude-SearchBot, ClaudeBot, and Claude-User.

Google requires a different mental model. Google-Extended is a robots.txt product token, not a separate HTTP crawler. Google’s crawler reference says it controls specified Gemini training and grounding uses and does not affect Google Search or Search ranking. For Google AI Overviews and AI Mode, Google says normal Googlebot crawl access, indexing, and snippet eligibility apply.

Apple has a parallel distinction. Applebot supports search across Spotlight, Siri, and Safari, while Applebot-Extended controls specified foundation-model training use of data Applebot collected. Apple states that Applebot-Extended does not itself crawl, and disallowing it does not remove pages from Apple search results.

robots.txt also differs from indexing and access control. Google warns that a disallowed URL can still appear in Search without a snippet if other pages reveal it. A crawler must be allowed to fetch a page before it can read a noindex meta tag or header. Authentication, authorization, and server or edge rules are what keep private material private; robots.txt merely publishes a preference to clients that choose to comply.

Build the bot ledger before editing the file#

Start with the outcome the site owner wants, not a copied blocklist. Give every row these fields:

The “unknown” purpose is important. A user-agent string is easy to invent, and community blocklists often outlive the crawler names and descriptions they copied. If an operator does not document a token well enough to connect it to an outcome, label it unknown and handle observed traffic at the server or edge according to security and capacity policy. Do not invent a product consequence from the bot’s name.

A verified bot-by-bot starting matrix#

The matrix below is a policy input, not a universal verdict. It includes agents whose roles were documented by their operators at this article’s evidence cutoff.

This table deliberately omits undocumented aliases and guesses. It is also a snapshot, not a permanent registry. Bot names, purposes, IP ranges, and product consequences can change; the source URL and review trigger are part of the policy, not editorial decoration.

Decide by content zone, not only by domain#

A public B2B SaaS site rarely has one sensible answer for every URL. Split the inventory into operational zones:

• Canonical public pages such as product explanations, documentation, and evidence-backed articles can be available to search agents when discovery and citations are desired.
• Licensed, paywalled, or partner-supplied material needs an explicit rights and product-use decision. Public reach does not automatically settle training permission.
• Accounts, exports, admin tools, previews, and customer-specific URLs require authentication or authorization. Add crawl exclusions for hygiene, but never rely on them for secrecy.
• Internal search results, faceted combinations, duplicate parameters, and crawl traps can be disallowed to reduce waste even when their canonical parent content remains available.

For each zone, record the smallest policy that achieves the outcome. Blocking an entire search agent because one parameter space is wasteful is usually too broad. Allowing a training agent across the whole origin because the marketing section is public may be broader than the rights decision. Path-level scope is where the ledger becomes an operating policy rather than a list of opinions.

Compile the ledger into valid robots.txt groups#

The safest edit starts from the site’s existing rules. Preserve its search, rendering, sitemap, and crawl-trap decisions; then add only the groups needed to express a real difference.

One protocol detail causes many accidental leaks: a crawler with a specifically matching group does not also inherit the User-agent: * group. The wildcard is the fallback when no specific group matches. If you create a specific group for an allowed AI search bot, repeat the baseline restrictions that must still apply to it.

Within one applicable group, the longest matching path rule wins. A longer Allow can therefore open a deliberate subtree beneath a broader Disallow, but path values are case-sensitive. Test exact URLs instead of reading the file by eye.

Here is an illustrative policy for a public marketing and documentation site, not a recommendation for every publisher. It keeps public content available to three documented AI search crawlers, repeats the site’s private and low-value path exclusions for those crawlers, and declines selected model-development, extended-use, and shared-corpus access.

# Baseline for crawlers without a more specific matching group.
User-agent: *
Disallow: /account/
Disallow: /admin/
Disallow: /internal-search/
Disallow: /preview/

# AI search discovery: public paths remain allowed by default.
# Repeat baseline exclusions because this specific group does not inherit *.
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /account/
Disallow: /admin/
Disallow: /internal-search/
Disallow: /preview/

# Model-development crawlers.
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

# Data-use control tokens; these are not standalone HTTP crawlers.
User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

# Shared open-web corpus crawler.
User-agent: CCBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

There is no need to add Allow: / merely to look explicit: when no applicable path rule matches, the protocol allows the URI. Explicit groups earn their maintenance cost only when they express a distinct policy or make an approved decision auditable.

Do not assume Crawl-delay is portable. Anthropic says its bots support the non-standard record, while Google does not support it for Google crawlers. If request volume must be enforced, use capacity-aware controls at the server or edge and verify genuine operator traffic before allowlisting or rate limiting it.

Test the policy as a routing table#

Treat the file like production configuration. The acceptance test is not “the syntax looks right.” It is a set of representative agent-and-URL pairs with expected outcomes.

For the illustrative policy above, the minimum test table is:

Run the verification in four layers.

1. Verify every origin serves the intended file#

Fetch /robots.txt from the apex, www, documentation host, and any other public origin in scope. Confirm a stable 200 response, plain-text body, expected encoding, and the exact file produced after CMS, CDN, and hosting transformations.

curl -i https://www.example.com/robots.txt
curl -i https://docs.example.com/robots.txt

Do not use an error response as policy. Under RFC 9309, a 4xx response can be treated as an unavailable file that permits crawling, while a network error or 5xx requires a conforming crawler to assume a complete disallow initially. Cached rules and operator-specific behavior make either failure mode a poor control.

2. Parse representative URLs for every specific group#

Use a parser that implements RFC 9309 plus any operator-specific behavior you deliberately rely on. Test an allowed public URL, every disallowed prefix, one case-variant path, one query-pattern path if present, and both sides of any Allow exception. Store the expected matrix beside the source file so a later edit cannot silently widen access.

Google Search Console can help test Google-owned crawling behavior, but it does not prove how another operator parses its groups. The owner documentation and your protocol-level tests serve different purposes; keep both.

3. Test the actual HTTP response path#

A robots parser cannot see a WAF challenge, redirect loop, origin authorization rule, or CDN feature that serves a different response by user agent. Send harmless requests through the production edge for each desired response class:

curl -I -A 'OAI-SearchBot' https://www.example.com/docs/getting-started/
curl -I -A 'GPTBot' https://www.example.com/blog/research-method/

These commands test your own user-agent handling; they do not authenticate the caller. A scraper can spoof GPTBot, CCBot, or any browser string. Where an operator publishes IP ranges or reverse-DNS verification, combine that evidence with the user agent before trusting, allowlisting, or attributing a request.

4. Observe genuine traffic and product consequences#

After release, check server logs for the documented agents, paths, response codes, request rates, and source verification. Compare observed behavior with the ledger rather than with analytics alone: crawlers may not run JavaScript, so client-side analytics can miss them.

Allow for caching and recrawl delay. RFC 9309 permits caching, and OpenAI says its search systems may take about 24 hours to adjust after a robots.txt change. Search discovery, indexing, and answer citations can take longer and are never guaranteed. Define a rollback condition before release—for example, wanted search agents receiving denials, server load exceeding the approved capacity boundary, or a bot purpose changing in operator documentation.

Review the policy when the meaning changes#

A bot policy decays even when the file does not. Review it when an operator adds or retires a token, combines product uses, changes its robots.txt commitment, revises IP ranges, or introduces a new search or user-retrieval path. Reopen the decision when content rights, paywall behavior, infrastructure cost, or the site’s discovery strategy changes.

Avoid a calendar-only ritual that re-approves stale assumptions. The most useful review receipt is a diff with four attachments: the updated ledger, links to current operator documentation, the agent-and-URL test matrix, and a small log sample showing what the production edge actually returned. That is enough for content, SEO, platform, security, and legal owners to see the same decision.

The durable rule is simple: allow or block an outcome, not a logo. robots.txt is useful when it is the compiled output of that decision. As a copied list of bot names, it is only a stale policy waiting to create the wrong consequence.