AI Summary
Home Depot Canada applied a brute force block that returned Access Denied to anyone reaching homedepot.ca from outside Canada, and Sistrix recorded a cliff edge drop in the site's UK search visibility. The technical lesson is that Googlebot crawls from United States infrastructure, so an IP block on foreign traffic is functionally a sitewide noindex.
- Luke Carthy described the cause as a brute force block on visitors from outside Canada, returning Access Denied.
- Sistrix showed a cliff edge drop in UK visibility while Canadian visibility was unaffected.
- Rankings for the brand query survived, which is the classic signature of a crawl access failure rather than a penalty.
- The source post does not publish exact dates or percentage figures, and none are invented here.
- The fix is to allowlist reverse DNS verified crawlers before any geographic deny rule runs.
This case is short on published numbers and long on transferable principle, so it is worth reading for the mechanism rather than the metrics. Luke Carthy documented that homedepot.ca began serving an Access Denied response to visitors arriving from outside Canada, and that Sistrix data showed the site's United Kingdom visibility falling off a cliff while its Canadian visibility carried on untouched.
The intent behind a rule like that is almost always defensible. Non domestic traffic on a domestic only ecommerce site converts at close to zero, it inflates bot and scraper load, and it distorts analytics. Blocking it looks like housekeeping. The problem is that the single most valuable non domestic visitor any site has is Googlebot, and Googlebot does not arrive from the country you sell to.
Where the crawler actually comes from
Google publishes its crawler ranges as machine readable JSON, and the overwhelming majority of the addresses in it are United States datacentre space. Google does operate a locale aware crawler that issues requests from other countries, but it is applied selectively and is not a substitute for general crawl coverage. If you are building an edge rule, assume the crawl originates in the United States.
# The authoritative crawler ranges, published by Google
curl -s https://developers.google.com/static/search/apis/ipranges/googlebot.json \
| python3 -c 'import json,sys; d=json.load(sys.stdin); print(len(d["prefixes"]), "ranges")'
# Companion files for the other Google fetchers
# special-crawlers.json : AdsBot, Google Read Aloud, APIs Google
# user-triggered-fetchers.json : Google Site Verifier, feed fetchersThe corollary matters for anyone running Cloudflare, Akamai, Fastly or an AWS WAF: a country rule written in the dashboard as anything other than Canada gets denied will, unless you sequence it correctly, catch the crawler on its very first request.
Why brand rankings survived and everything else did not
The detail that the site held position one for its own brand name while general visibility collapsed is the diagnostic tell. It rules out a manual action or a quality demotion, both of which suppress brand terms too. What it points to instead is a pure access failure.
Google can answer a navigational brand query from the entity graph, from external references and from historical data, without needing a fresh fetch. It cannot answer a non brand commercial query that way, because matching a page to a query about a specific product requires having read that page recently. So the non brand tail evaporates while the brand head sits there looking healthy, which is exactly why this class of failure so often goes unnoticed on an executive dashboard that tracks brand terms.
Four ways to block, and what each one does to your index
| Blocking method | What the crawler receives | Effect on indexing | Detectability in Search Console |
|---|---|---|---|
| IP deny returning 403 | 403 Forbidden on every URL | Pages progressively dropped from the index | High: Page indexing report fills with Blocked due to access forbidden |
| IP deny with connection reset | No response at all, request times out | Crawl rate throttled back, then deindexing | Medium: shows as Server error or a crawl stats collapse |
| Geo redirect to a country splash page | 302 from every URL to a single destination | Entire URL set collapses to one page | Medium: mass Page with redirect classification |
| Blanket robots.txt disallow | Nothing is fetched | URLs may persist as indexed without content | High: Indexed though blocked by robots.txt |
| Verified crawler allowlist before the deny rule | 200 with the full rendered page | No impact, index preserved | Nothing to see, which is the point |
The last row is the only configuration that achieves the business goal without collateral indexing damage.
The correct implementation
Order of operations is everything. The crawler allowlist has to be evaluated before the geographic rule, not after it, and it has to verify rather than trust. Trusting the user agent string is not viable because anyone can send it, so the standard is a reverse DNS lookup that resolves to googlebot.com or google.com followed by a forward lookup that returns the original IP.
# Manual verification of a single requesting IP
host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1
# Both directions agree, so the request is genuine Googlebot.# Cloudflare WAF, expressed as a custom rule
# Rule 1, action SKIP (evaluate first)
(cf.client.bot) or (ip.src in $verified_search_crawlers)
# Rule 2, action BLOCK (evaluate second)
(ip.geoip.country ne "CA")On Cloudflare specifically, the cf.client.bot field is already populated from Cloudflare's own verified bot list, which removes the need to maintain the IP ranges yourself. On an origin level Apache or nginx configuration you are back to maintaining the JSON range list, which means a scheduled job that refetches it, because Google does change the ranges.
How to detect that this is happening to you
The failure is silent from inside the country you operate in, which is why it runs for weeks. Every check below is deliberately something you can run from your own machine or from Search Console without waiting on the infrastructure team.
| Signal | Where to look | What a problem looks like |
|---|---|---|
| Crawl request outcome | Search Console, Settings, Crawl stats, By response | A sudden share of 403, 5xx or timeouts appearing overnight |
| Live fetch parity | Search Console URL Inspection, Test Live URL | Live test fails or renders an error page while the browser is fine |
| Rendered HTML | URL Inspection, View crawled page, HTML tab | The word Denied, Forbidden or a challenge page instead of product markup |
| Index coverage trend | Search Console Page indexing report | Valid pages falling while Blocked due to access forbidden climbs |
| Brand vs non brand split | Search Console Performance, filtered by query | Brand clicks flat while non brand clicks fall off a cliff |
The brand versus non brand split in the last row is the fastest single confirmation that you are looking at an access problem rather than an algorithmic one.
What's changed since
Checking the site from a United States client on 20 July 2026, requests to www.homedepot.ca complete the TLS handshake and then have the HTTP/2 stream reset by the origin without any HTTP response being returned. That is a single observation from one non browser client and it is not evidence about how Googlebot is treated today, so no conclusion about current rankings should be drawn from it. What it does confirm is that some form of edge filtering on non browser or non Canadian requests is still in place.
The wider context has moved on too. In the years since this was written the same mechanism has become a live question for AI crawlers as well as search crawlers. A rule that denies non domestic datacentre IPs now also excludes the fetchers behind AI answer surfaces, so the visibility being traded away is broader than it was. That is a strategic choice a business is entitled to make, but it should be a choice, not an accident of a WAF rule that someone shipped to cut bot traffic.
Related reading on this site
For the directives that govern crawling deliberately rather than accidentally, see the complete robots.txt reference. For a migration where the redirects rather than the firewall were the fault, the Clarks international consolidation case is the closest parallel, and the full migrations case study index collects the rest.
FAQ
Yes, and far more than most teams expect. Googlebot crawls overwhelmingly from United States based infrastructure regardless of which country your site targets, so an IP block on non domestic traffic blocks the crawler itself. The crawler receives a 403 or a timeout instead of your content, and pages that consistently return an error on recrawl are eventually dropped from the index.
Google publishes its crawler IP ranges in a JSON file at googlebot.json on the Google developers domain, and the overwhelming majority of those ranges are United States based. Google also runs a smaller locale aware crawler that requests from other countries, but it is used selectively and cannot be relied on to cover a whole site. Plan for a US originating crawl as the default.
Brand queries are the last thing to go. Google can satisfy a navigational query like the Home Depot Canada brand name from the historical record, external references and the sheer strength of the entity, even when it can no longer fetch the pages. Non brand commercial queries, which depend on being able to read the actual page content, disappear first.
Allowlist verified search engine crawlers ahead of any deny rule. Verification means a reverse DNS lookup on the requesting IP that resolves to googlebot.com or google.com, followed by a forward lookup back to the same IP, rather than trusting the user agent string. Apply your geographic or rate based rules only to traffic that fails that check.
No, it is usually worse. A redirect that sends every foreign request to a single country splash page means Googlebot sees your entire URL set collapse into one destination. From the crawler's point of view the site has become a one page site, which removes far more from the index than a 403 on a handful of URLs would.
Recovery starts as soon as the crawler can fetch pages again, but it is not instant. Google has to re-crawl the URL set, and recrawl frequency for URLs that have been serving errors is reduced. Submitting an updated sitemap with fresh lastmod values and requesting indexing on the highest value templates accelerates it. Expect weeks rather than days for a large catalogue.
Source: https://lukecarthy.com/blog/dear-home-depot-youve-killed-your-ca-site-overnight/
Not sure what your edge rules are doing to Googlebot?
An advanced audit tests crawler access from the outside and reads the crawl stats from the inside.
Claude Vincent is a technical SEO consultant focused on crawlability, rendering, and AI-search visibility. He writes the field guides and case studies at SEO ProCheck, with a bias toward the durable, unglamorous work that decides whether search engines and AI answer engines can actually read and cite a site.
About SEO ProCheck
Technical SEO consulting and GEO strategy with 20 years of enterprise experience. Case studies, resources, and tools for search and AI visibility.
Work With Me
Technical SEO audits, GEO strategy, site migrations, and international SEO. Hourly consulting for teams who need hands-on support, not just reports.
Subscribe to our newsletter!
