moderate change What the guidance says has shifted
Google added a sentence to the 'URL matching based on path values' section clarifying that while Google parses raw UTF-8 characters correctly, using percent-encoding in robots.txt rule paths can make processing easier for third-party tools and services.
Lincoln Digital's AI Overview
Site owners may want to use percent-encoded UTF-8 strings in robots.txt allow and disallow paths, as it could improve interoperability with third-party robots.txt validators and tools. Google already treats raw and percent-encoded forms identically, so this does not change how Google interprets the rule.
Generated by AI from the change below: a guide to what it could mean, not a certainty.
Word for word
red was taken outgreen was put in
In “URL matching based on path values”
Google uses the path value in the allow and disallow rules as a basis to determine whether or not a rule applies to a specific URL on a site. This works by comparing the rule to the path component of the URL that the crawler is trying to fetch. The path in the rules can be either raw UTF-8 characters or percent-encoded UTF-8 strings (per RFC 3986). As defined in RFC 9309 Section 2.2.2, Google's crawlers compare rules against URLs using their percent-encoded forms, and automatically canonicalize raw UTF-8 rule paths into percent-encoded forms. This means that, for example, Disallow: /foo/bar/ツ and Disallow: /foo/bar/%E3%83%84 are treated identically by the parser.[added: While Google's systems parse raw UTF-8 characters properly, using percent-encoding for rule paths can make it easier to process these files with third-party tools and services.]
On Google's page
The "before" is the same page with the earlier wording put back.
Changed between our checks at and (UK time)Google's “last updated” date moved from 31 Aug 2026 to 23 Oct 2026Posted on X Link to this change