Google Web Crawler: Access Is Not Indexing
A server log shows a visitor calling itself Googlebot. At the same time, a page you meant to hide still appears in Google Search, and someone suggests adding Disallow: / to robots.txt. These can look like one problem, but they call for different decisions. The name in a request does not establish who sent it; a crawl rule does not necessarily remove a URL from search results; and Google’s AI-related setting is separate from its main Search crawler. Google describes Googlebot as a crawler used by Search and warns that the HTTP user-agent string can be spoofed. Google’s crawler list and Google’s robots.txt guide set out those distinctions.
The useful starting question is therefore what you want to change. Do you want Google to stop fetching a set of URLs, prevent a public page from appearing in Search, check whether traffic really came from Google, or control a named use of content in Google’s AI products? Each goal has a different control. Choosing the wrong one can leave a URL visible, keep Google from seeing an instruction you need it to see, or block search access you intended to retain.
Identify the Google request before changing access
For ordinary web search, Googlebot is the name to recognize. Google’s crawler documentation lists both smartphone and desktop HTTP user-agent strings containing Googlebot and identifies Googlebot as the corresponding token for robots.txt. Google says preferences addressed to that token affect Google Search, including Search features and Discover, along with other named Google products. It also lists more specific tokens, including Googlebot-Image and Googlebot-Video, for image and video uses. The choice of token matters when the desired change is narrower than all Googlebot access. Google’s common-crawlers documentation describes the tokens and affected products.
A user-agent token in robots.txt is a rule selector. An HTTP User-Agent string is a request header that a server can record. They are related, but they are not interchangeable proof of identity. Anyone can send a header that says Googlebot. If a sudden run of requests is straining your site, changing crawl rules based only on that label may leave the actual source of the traffic untouched. Google explicitly cautions that the header can be spoofed and provides a method for checking the request’s origin. Google’s crawler list and verification guide explain the distinction.
Nor does every request from Google have the same relationship to robots.txt. Google says its common crawlers respect the file for automatic crawls. Its special-case crawlers may or may not respect it, while Google-controlled fetchers triggered by a user ignore it. That makes “block all Google traffic with one robots rule” an unreliable operational goal. First identify the request category and the product you mean to affect; then choose the appropriate response. Google’s verification guide separates those categories.
Use robots.txt to choose which URLs a crawler may fetch
The Robots Exclusion Protocol is a public set of crawl preferences. A site places robots.txt at the top level of an origin, such as https://www.example.com/robots.txt, and groups rules under crawler product tokens. A User-agent line selects the crawler; Allow and Disallow lines name URL paths. Under the protocol, a crawler uses the most specific matching path rule, and an equal Allow and Disallow match favors Allow. The file tells a conforming crawler which paths it may request; it does not authorize access or protect private content. RFC 9309 defines the mechanism and its limits.
Suppose an illustrative site publishes articles under /guides/ and creates many low-value internal URLs under /filters/. If the aim is to keep Googlebot from repeatedly fetching those filter URLs while leaving articles available to Search, the relevant group would contain User-agent: Googlebot followed by Disallow: /filters/. That is a crawl decision about a path prefix. It is not a statement that every /filters/ URL will disappear from search results, and it does nothing to make those URLs private. Google’s guide describes robots.txt as a way to manage crawler traffic or avoid crawling unimportant and similar pages, while warning against using it to hide web pages from Search. Google’s robots.txt guide supports that narrower use.
The scope of the rule deserves attention before publishing it. A broad Disallow: / under User-agent: Googlebot asks Googlebot to avoid the whole site, including the pages you may want in Search. A path rule also applies by its matching path, not by the editor’s intention: a hypothetical Disallow: /draft can catch URLs beginning with that sequence, while Disallow: /draft/ expresses a directory prefix more clearly. If some path inside a blocked area must remain crawlable, a more specific Allow can express that exception under the protocol’s longest-match rule. Check representative URLs against the actual rule text, including the pages that should remain open. RFC 9309 describes path matching and precedence.
There is also a practical cost to broad crawl blocks. If Googlebot cannot fetch a page, it cannot read the page’s content or its page-level indexing instruction. Blocking a whole section may reduce requests to that section, but it also removes your ability to tell Google about individual pages there through a visible noindex directive. Use a broad block when avoiding those fetches is the real priority and the loss of page-level control is acceptable. For public pages that must leave Search, use the indexing route instead. Google’s noindex documentation states that a blocked crawler cannot see the directive.
Keep crawling and search indexing separate
A disallowed URL can still appear in Google Search if Google learns about it through links elsewhere. Google says it may show the URL and other publicly available information, such as link text, even without fetching the blocked page. That is why a robots.txt entry can seem to “fail” when the actual goal was removal from results. The file can be working as a crawl preference while the URL remains known. Google’s robots.txt guide makes this distinction explicitly.
For an accessible public page that should not appear in Google Search, Google’s documented mechanism is noindex, delivered as an HTML meta tag or an HTTP response header. Googlebot must be allowed to fetch the resource to read that instruction. If you place noindex on a page and simultaneously disallow that page in robots.txt, the crawl block may prevent Google from seeing the very instruction intended to remove it. Google says it does not support putting a noindex rule in robots.txt. Google’s noindex guide describes the tag, header, and access requirement.
If the material is actually private, neither a crawl preference nor a search-indexing directive is the access control to rely on. robots.txt is publicly readable and can expose the paths you list there; RFC 9309 recommends a real security measure, such as HTTP authentication, for restricted resources. Google also names password protection as an option for keeping a page out of Search. The direct choice for a private document is to restrict access at the server or application. A crawler instruction can then serve a separate publishing purpose, but it must not carry the confidentiality requirement. RFC 9309 and Google’s robots.txt guide draw that boundary.
Consider a second illustrative case: a public event page has expired, yet its URL still appears in Search. If visitors may still see the page but you want it excluded from Google results, make the page crawlable and deliver noindex on it. If the event page contains information that visitors must no longer be able to obtain, remove or protect that content at its source. Merely disallowing Googlebot would leave open the possibility of a URL-only result and would not stop other visitors from requesting the page. These decisions differ because crawl access, search appearance, and human access are separate properties. Google’s noindex guide and robots.txt guide support the distinction.
Verify a claimed Googlebot before responding to traffic
If the problem begins in server logs, first preserve the requesting IP address, timestamp, requested URL, and stated user agent for the requests in question. The user-agent text tells you what the client claimed. To check whether the request actually came from Google, Google describes two routes: perform reverse DNS on the IP address and then forward DNS on the returned hostname, or compare the IP with Google’s published ranges for the relevant crawler or fetcher category. For a manual check, the forward lookup must resolve back to the original IP. Google’s verification instructions give the steps and category-specific ranges.
The hostname category matters as well. Google’s documentation lists googlebot.com patterns for common crawlers, google.com patterns for some special-case and user-triggered requests, and googleusercontent.com patterns for other user-triggered fetching. Its published IP lists are offered by category, so a copied address list can grow stale. Verify against the current published information when making an access decision. An origin check answers whether a request belongs to Google; it does not tell you which content uses you have agreed to, whether a URL should be indexed, or which rule will apply to another platform’s bot. Google’s verification guide provides the patterns and live lists.
Once the source is known, the logs can answer the narrower operational question: which paths are being requested? If verified Googlebot traffic concentrates on an unimportant generated path, a path-specific crawl rule may be appropriate. If the requests are for the pages you want searchable, a blanket block would trade away useful access. If the traffic only claims to be Googlebot, changing Googlebot’s robots.txt group is unlikely to govern it, because a nonconforming client can ignore the file. The protocol is a request to conforming crawlers, not a server-side access barrier. RFC 9309 establishes that limit.
Treat Google-Extended as a separate product choice
The phrase “Google crawler” can also hide a different decision: whether content fetched from a site may be used for the Google AI uses covered by Google-Extended. Google identifies Google-Extended as a robots.txt product token with no separate HTTP user-agent string; crawling uses existing Google user-agent strings. Google says the token lets publishers manage specified uses involving future Gemini model training and grounding in named Gemini and Vertex AI products. Google also states that the Google-Extended setting does not affect inclusion or ranking in Google Search. Google’s common-crawlers documentation names the token, uses, and Search boundary.
For a publisher that wants ordinary Google Search visibility but wants to decline the uses Google assigns to Google-Extended, a Google-Extended rule addresses that stated product choice without disallowing Googlebot. The cost is narrower than cutting off Search crawling, but the promise is narrower too: Google’s documentation describes the affected Google products, not every Google service or every way content might be encountered. Do not look for a separate Google-Extended header in the logs as proof that this setting is being applied; Google says there is no such separate HTTP user-agent string. Google’s crawler list is the source for that distinction.
This is why a single rule copied from a discussion of “AI crawlers” is a poor substitute for a publishing decision. A Googlebot block changes access relevant to Search; a Google-Extended setting addresses the Google product uses Google lists; a page-level noindex targets Search appearance when the crawler can fetch it. Decide which outcome matters for each class of URL before writing the corresponding rule. Google’s crawler list and noindex guide describe those separate controls.
Apply the same questions to other crawlers, not the same rule
The logic extends beyond Google, but the names and promises change by operator. OpenAI’s publisher guidance distinguishes OAI-SearchBot access for ChatGPT search summaries and snippets from a GPTBot setting for content a publisher wishes to exclude from potential training. That is a distinction in OpenAI’s documented controls, not a claim that Google’s Google-Extended rule manages ChatGPT. If being cited in ChatGPT search matters to you, the discovery setting needs its own review. OpenAI’s publisher FAQ describes the two controls.
Perplexity likewise documents PerplexityBot as a crawler intended to surface and link websites in its search results, while Perplexity-User supports a fetch triggered by a user’s question. Perplexity says that user-triggered fetcher generally ignores robots.txt. Google also distinguishes its own user-triggered fetchers from automatic crawlers. The lesson is operational: read the specific operator’s current description before assuming that a robots.txt rule covers every request bearing its name. Perplexity’s crawler documentation and Google’s verification guide document those differences.
For a site that depends on public search discovery, the sound default is to keep valuable public pages fetchable by Googlebot, use targeted robots.txt rules for paths whose crawling is genuinely unwanted, and use noindex for accessible pages that should leave Google Search. Protect private material with actual access controls. Verify suspicious “Googlebot” traffic before attributing it to Google, and make a separate decision about Google-Extended if Google’s named AI uses matter to your publishing terms. The central question is not whether to “allow the Google web crawler” in the abstract. It is which request, URL, and product outcome you mean to change—and which mechanism can actually change it.