How to Create a Robots.txt File Without Blocking the Wrong URLs

A robots.txt file tells cooperative crawlers which URLs they may fetch. It does not secure a URL, and it does not guarantee that a URL will stay out of search results. For most B2B sites, the safe approach is to allow crawling by default, disallow only verified low-value paths, publish the file at the exact site’s /robots.txt location, and test the response that crawlers actually receive.

The underlying standard is the Robots Exclusion Protocol in RFC 9309. Its rules look simple, but a mistake in the host, crawler group, or path can change the result across thousands of URLs.

Confirm that robots.txt is the right control#

Choose the control according to the outcome you need:

This distinction prevents a common failure: blocking a page in robots.txt and assuming it has disappeared from search. Google says it may still index a blocked URL when it discovers the URL elsewhere. It also cannot read a page’s noindex directive if robots.txt prevents the page from being fetched. Its official guidance explains both the limits of robots.txt and how indexing directives depend on crawl access.

Never put a confidential path in robots.txt as a security measure. The file is public, so the entry can advertise the path to people and bots that ignore the policy. RFC 9309 explicitly states that the protocol is not access authorization.

Build the file from real URL patterns#

Before writing rules, list the site’s actual origins and representative URLs. A policy served from https://www.example.com/robots.txt applies to that protocol and host; it does not automatically cover http://www.example.com, https://example.com, or https://app.example.com. Each origin that needs a policy must serve its own file.

Then classify paths by what crawlers should do. On a typical B2B site, public product, solution, pricing, documentation, and editorial pages normally need to remain crawlable. Internal search results, faceted duplicates, cart flows, or other low-value URL spaces may be candidates for exclusion—but only after checking the site’s real routing and SEO requirements.

This conservative starter file allows the public site and blocks three example directories:

User-agent: *
Allow: /
Disallow: /internal-search/
Disallow: /cart/
Disallow: /checkout/

Sitemap: https://www.example.com/sitemap.xml

Replace or remove every example path before deployment. User-agent: * targets crawlers without a more specific matching group. Allow and Disallow contain paths relative to the origin. The optional Sitemap line uses an absolute URL and helps supporting crawlers discover the sitemap; Google documents this format in its robots.txt creation guide.

If the site has nothing to exclude, an allow-all rule is valid but not necessary for permission: URLs are crawlable when no applicable rule blocks them. A minimal file may therefore contain only a sitemap declaration, or the site may have no robots.txt file at all. Keep an explicit allow-all group if it makes ownership and operational intent clearer.

Check how the rules will match#

Crawlers first select the group matching their product token. If no specific group matches, they use the * group when one exists. A specific group does not inherit rules from the wildcard group. For example:

User-agent: *
Disallow: /reports/

User-agent: ExampleBot
Allow: /

Under the standard, ExampleBot follows its specific group and may crawl /reports/; the wildcard restriction is not added to that group. If that is not the intended result, repeat the restriction inside the specific group.

Path matching begins at the start of the URL path, is normally case-sensitive, and uses the most specific matching rule. That makes slashes and spelling consequential:

User-agent: *
Disallow: /resources/
Allow: /resources/public/

Here, /resources/private/report.pdf is disallowed, while /resources/public/guide.pdf is allowed because the longer rule is more specific. By contrast, Disallow: /resources would also match paths such as /resources-old; use the trailing slash when the intent is a directory boundary.

Avoid clever patterns when explicit prefixes will do. RFC 9309 defines * as a wildcard and $ as an end-of-match marker, but complex patterns are harder to review and easier to misapply. Build test cases for every wildcard rather than relying on visual inspection.

Test the delivered file, not just the repository copy#

The file must be named robots.txt, encoded as UTF-8, served as plain text, and available at the lowercase top-level path. After deployment, inspect the live response:

curl -i https://www.example.com/robots.txt

Confirm that the response is successful, the body contains the intended rules rather than an application error page, and the content type is text/plain. Check every relevant host separately.

Response failures can change crawler behavior even when the source file is correct. RFC 9309 allows crawlers to treat an unavailable file, such as many 4xx responses, as having no restrictions; it requires a crawler to assume complete disallow when the file is unreachable because of server or network errors. Individual crawlers document additional handling. For example, Google treats most 4xx responses as no restrictions, reacts differently to 429, and may use a previously cached file during server failures. Google also says it generally caches robots.txt for up to 24 hours. The current details are in Google’s robots.txt parser and status-code documentation.

Before release, test at least one URL expected to be allowed and one expected to be disallowed for each affected crawler group. For Google, the robots.txt report in Search Console can test a file that is already accessible. Also verify that the policy does not block CSS, JavaScript, images, or other resources needed to render important public pages.

Record the previous file, the tested URLs, the policy owner, and the rollback condition. After release, watch crawler logs and indexing diagnostics. A correct launch is not merely a syntactically valid text file; it is a live response whose behavior matches the intended URLs on every relevant origin.