moderate change What the guidance says has shifted
Google has updated the 'Reduce the Google crawl rate' emergency instructions to add guidance on using the Retry-After HTTP header with 503 or 429 status codes, with examples, and restructured the existing advice into bullet points.
Lincoln Digital's AI Overview
This looks like it gives site owners more precise control when temporarily throttling Googlebot: if you return a 503 or 429, you can now tell Google's crawlers exactly when to retry, which could reduce guesswork and make short-term crawl rate reductions more predictable. It doesn't change the underlying mechanism or any ranking effect, but it's a useful operational addition.
Generated by AI from the change below: a guide to what it could mean, not a certainty.
Word for word
red was taken outgreen was put in
At the top of the page
Google's crawler infrastructure has sophisticated algorithms to determine the optimal crawl rate for a site. Our goal is to crawl as many pages from your site as we can on each visit without overwhelming your server. In some cases, Google's crawling of your site might [removed: be causing ][added: cause ]a critical load on your infrastructure, or[removed: cause] unwanted costs during an outage. To alleviate this, you may decide to reduce the number of requests made by Google's crawlers.
In “Understand the cause of the sharp increase in crawling”
[removed: Sharp ][added: A sharp ]increase in crawling may be caused by inefficiencies in your site's structure or [added: other ]issues with your site[removed: otherwise]. Based on the reports we've received in the past, the most common causes are:
Inefficient configuration of URLs on the site, which is typically [removed: casued][added: caused] by [removed: a ]specific [removed: functionality of the ]site[added: functionality]:
Faceted navigation or other sorting and filtering functionality[removed: of the site]
We strongly recommend that you check with your hosting company and look at [added: your server's ]recent access logs [removed: of your server ]to understand the source of the traffic, and see if it [removed: fits in the aformentioned][added: matches any of these] common causes of [removed: the][added: a] sharp increase in crawling. Then, check our guides about managing crawling of faceted navigation URLs and optimizing crawling efficiency.
In “Urgently reduce crawler traffic (for emergencies)”
Warning: When considering reducing [removed: the ]Google's crawl rate, keep in mind that this will have broad effects. For Search, Googlebot will discover fewer new pages, [removed: and ]existing pages will be refreshed less frequently (for example, prices and product availability may take longer to be reflected in Search), and removed pages may stay in the index longer. For Google Ads, your campaigns may be [removed: cancelled][added: canceled] or paused, and your ads may not serve.
[removed: If you need to urgently reduce the crawl rate for short period of time (for example, a couple of hours, or 1-2 days), then return 500, 503, or 429 HTTP response status code instead of 200 to the crawl requests. Google's crawling infrastructure reduces your site's crawling rate when it encounters a significant number of URLs with 500, 503, or 429 HTTP response status codes (for example, if you disabled your website). The reduced crawl rate affects the whole hostname of your site (for example, subdomain.example.com), both the crawling of the URLs that return errors, as well as the URLs that return content. Once the number of these errors is reduced, the crawl rate will automatically start increasing again.]
[removed: Warning: We don't recommend that you do this for a long period of time (meaning, longer than 1-2 days) as it may have a negative effect on how your site appears in Google products. For example, in case of Search, if Googlebot observes these status codes on the same URL for multiple days, the URL may be dropped from Google's index.]
[added: To urgently reduce the crawl rate for a short period of time (for example, a couple of hours, or 1-2 days), return a 500, 503, or 429 HTTP response status code instead of 200 to the crawl requests.]
[added: When Google's crawling infrastructure encounters a significant number of URLs with 500, 503, or 429 HTTP response status codes (for example, if you disabled your website):]
[added: It reduces your site's crawl rate across the whole hostname (for example, subdomain.example.com), including both the URLs that return errors and the URLs that return content.]
[added: It automatically increases the crawl rate again once the number of these errors is reduced.]
…and 7 more changed paragraphs. The rest is on the page itself.
On Google's page
The "before" is the same page with the earlier wording put back.
Changed between our checks at and (UK time)Google's “last updated” date moved from 18 Dec 2025 to 6 Oct 2026Posted on X Link to this change