Robots.txt File:
Set Rules by Host and Path

A robots.txt file tells cooperating crawlers which URL paths they may crawl, as Google’s introduction explains. This plain text file belongs at the site’s root, such as https://example.com/robots.txt. Blocking a web page’s crawl does not guarantee that its URL disappears from search results.

Where to find a robots.txt file#

Add /robots.txt after the site’s protocol and host. The file placement rules limit its scope to that protocol, host, and port.

A file on the bare domain does not govern its subdomains or HTTP version, and /blog/robots.txt cannot control /blog/, under Google’s location requirements.

The filename must be lowercase robots.txt; RFC 9309 requires UTF-8 encoding and the text/plain media type.

How to Create a Robots.txt File Without Blocking the Wrong URLs#

Use a plain text editor and save the file as UTF-8, following Google’s creation instructions. A basic syntax example is:

User-agent: *
Disallow: /search/
Disallow: /downloads/
Allow: /downloads/public/

Sitemap: https://example.com/sitemap.xml

Replace the paths and sitemap address with the URLs the file needs to describe. The supported fields have these jobs:

Comments begin with #, which the protocol treats as a comment marker.

An empty Disallow sets no restriction; Disallow: / blocks every path for the applicable group under Google’s rule interpretation. These two files therefore request opposite behavior:

# No crawl restrictions for the fallback group
User-agent: *
Disallow:

# Block every path for the fallback group
User-agent: *
Disallow: /

How crawler groups and path matching work#

Google uses the most specific matching user-agent group, combines matching specific groups, and does not merge them with the wildcard group.

User-agent: *
Disallow: /search/

User-agent: Googlebot
Disallow: /preview/

Under Google’s group-selection rules, Googlebot is blocked from /preview/, while /search/ remains allowed. Repeat /search/ in the Googlebot group to block both.

Within the applicable group, RFC 9309’s matching rules compare paths from the beginning, use the longest match, favor an equivalent Allow over Disallow, and permit unmatched URLs. The earlier /downloads/public/ exception therefore permits that branch inside the broader /downloads/ block.

The protocol’s wildcard syntax gives * the meaning of zero or more characters and $ the end of the match. Path spelling and boundaries matter:

Google treats path values as case-sensitive, so /Docs/ differs from /docs/. Test both blocked URLs and nearby allowed URLs before publishing a rule.

What the Sitemap line does#

The Sitemaps protocol allows full sitemap URLs in robots.txt, independently of user-agent groups, and permits multiple Sitemap lines.

Sitemap: https://example.com/sitemap.xml

A sitemap pointer helps crawlers discover URLs. It does not override a Disallow: RFC 9309 separates other records from crawl rules. Check crawl permission independently for URLs listed in the sitemap.

Robots.txt versus noindex and password protection#

For web pages, Google distinguishes blocking a crawl from excluding a URL from search. A blocked URL can still appear if Google discovers it through links.

To exclude a publicly accessible HTML page, put a supported noindex meta tag in its HTML head:

<meta name="robots" content="noindex" />

For non-HTML resources, such as PDFs, the response can carry an HTTP indexing directive:

X-Robots-Tag: noindex

Google must crawl the resource to read either instruction, so blocking that crawl can prevent Google from seeing noindex; Google does not support a noindex directive inside robots.txt itself.

Protect confidential content on the server: Google recommends password protection for private files, since a robots.txt request cannot force every crawler to comply.

How to edit, publish, and test the file#

1. Keep the existing rules. Download the current file before editing. Google’s update guide describes downloading, changing, and uploading the file at the root. For hosted platforms, use the provider’s supported settings.
2. Check the public response. Open the exact /robots.txt URL without signing in and read the returned text. Google recommends a private browsing window for this check. Verify that the response contains the intended rules.
3. Test crawler and URL combinations. Google’s open-source parser accepts a file, crawler token, and full URL, returning whether crawling is allowed. Include directory boundaries, allowed exceptions, capitalization, and query strings in the checks. Treat a local result as a check of the supplied file; compare it separately with the remote fetched version. The library excludes some crawler behavior outside parsing and matching.
4. Inspect Google’s fetched copy. For Domain properties and URL-prefix properties without a path, the Search Console robots.txt report shows fetch status, the last check, parsing issues, and the last fetched file. Compare that copy with the public response before diagnosing an ineffective rule.

Why a published rule may not take effect immediately#

Retrieval errors and caching affect Google’s use of the file. Under Google’s robots.txt fetch handling:

• A 2xx response supplies rules to process.
• A 4xx response other than 429, including 404 or 403, means no crawl restrictions.
• A 5xx response initially stops crawling for 12 hours; if fetching still fails, Google uses the last good copy for the next 30 days, or assumes no restrictions if none exists.

Google normally caches the file for up to 24 hours, but fetch failures can extend caching. For a critical fix, Search Console offers Request a recrawl; this does not guarantee an immediate crawl of newly unblocked pages.

Frequently asked questions#