What Goes Wrong When a Site Moves to a CDN?
A CDN is a network of edge servers that caches and serves your pages close to visitors, so your own server, the origin, handles fewer requests. The problem is timing. Google’s November 2026 post, by Gary Illyes and Cherry Prommawin, explains that Google’s crawlers speed up when responses get faster: “When response times increase, our crawlers slow down so we don’t overwhelm your server” (Google Search Central blog). A CDN makes responses faster at once, but its cache is empty, so the extra crawling lands on the origin.
The second failure is the opposite: CDN bot protection that challenges every automated client, including Googlebot. Google warns that leaving such protection on without exempting verified crawlers can hurt a site’s visibility in Search.
How to Move to a CDN at a Glance
| Step | Action | Expected result |
|---|---|---|
| 1 | List the URLs to pre-warm | Top traffic, upcoming promotional and most-crawled pages identified |
| 2 | Pre-warm the edge cache at a controlled pace | Edge serves those pages without hitting the origin |
| 3 | Turn on the CDN’s verified-bot exemption | Googlebot and other verified crawlers skip challenges |
| 4 | Block unwanted bots selectively | Scrapers are limited without catching search crawlers |
| 5 | If the origin struggles, return 429 or 503 briefly | Crawling slows for hours, not days |
| 6 | Watch crawl and server logs for two weeks | Crawl rate and errors settle |
Time: a day of preparation, then two weeks of monitoring. Tools: CDN dashboard, server logs, Google Search Console. Cost: none beyond the CDN plan. Skill level: intermediate.
Step 1: Decide Which URLs to Pre-Warm
Pick the pages that will draw the most requests in the first days. Google names three groups: your highest-traffic pages, pages for any upcoming peak or promotion, and your most frequently crawled URLs (Google Search Central blog). The last group is in Search Console’s Crawl Stats report and in your server logs. For a media or PR site, add your newsroom, latest press releases and any page you expect coverage to link to that week.
Step 2: Pre-Warm the Cache Before You Switch Traffic
Request those URLs through the CDN, slowly, before you point DNS at it. Google’s wording is “Pre-warm your caches before opening the floodgates.” A cache entry exists only after the first request for each URL, so a scripted fetch of your list at a steady rate fills the edge without spiking the origin. Our recommendation, not Google’s: throttle the script to a rate your origin already handles on a normal day.
Step 3: Exempt Verified Crawlers From Bot Challenges
Use the CDN’s managed “verified bots” rule rather than your own allowlist. Google says to “allow verified bots in your CDN and firewall settings” and notes most major CDNs and web application firewalls have a toggle or managed rule for it (Google Search Central blog). A user-agent string is not proof; anyone can send “Googlebot.” Google publishes two ways to verify: reverse DNS (the hostname must end in googlebot.com, google.com or googleusercontent.com, and a forward lookup must return the same IP) and its JSON IP range files (Google Crawling docs).
We counted those files on November 12, 2026. Together they list 2,201 IP prefixes, and they are regenerated regularly; all five carried a November 11 timestamp. That volume is why Google points you to managed rules. Our explainer How Do You Verify a Crawler Is What It Claims to Be? covers the same checks for Bing and AI crawlers.
Number of IPv4 and IPv6 prefixes per file, files dated November 11, 2026
Source: Google, crawler IP range JSON files linked from “Verifying Google’s crawlers and fetchers” (creationTime November 11, 2026). Counted November 12, 2026.
A prefix is one CIDR block. Counts change when Google updates the files; check the creationTime field before relying on a cached copy.
The chart shows why a hand-made allowlist fails: Googlebot alone uses 317 ranges, and the user-triggered fetchers that power tools such as site verification use more than three times as many.
Step 4: Block Unwanted Bots Selectively
Target named bad actors instead of all automation. Google suggests robots.txt or targeted rules for specific unwanted bots instead of blocking every automated client. If you also want to limit AI crawlers, do it by name; our guide to whether blocking a crawler removes you from search explains which tokens affect Google Search and which do not.
Step 5: Use 429 or 503 Only as a Short Brake
If the origin still strains, return 429 or 503 for a few hours to a day or two, then stop. The November post recommends “a 429 (Too Many Requests) or 503 (Service Unavailable)” response for a short window. Google’s crawl-rate guide adds 500 to that list, says the slowdown applies “across the whole hostname,” and warns that a URL returning these codes for multiple days “may be dropped from Google’s index” (Google Crawling docs). Do not use 403 or 404 as a brake; they are not on Google’s list.
| Response | Listed by Google to reduce crawl rate? | Safe duration per Google |
|---|---|---|
| 429 Too Many Requests | Yes | A couple of hours to 1-2 days |
| 503 Service Unavailable | Yes | A couple of hours to 1-2 days |
| 500 Internal Server Error | Yes (crawl-rate guide only) | A couple of hours to 1-2 days |
| 403 or 404 | Not listed | Do not use as a brake |
How to Verify It Worked
Check three things in the first two weeks. First, Search Console’s Crawl Stats report should show crawl requests rising without a jump in server errors or average response time. Second, your CDN’s firewall log should show no challenges or blocks for requests from Google’s published ranges. Third, the URL Inspection tool’s live test should fetch key pages with a 200 status. If the move overlaps Google’s November 2026 core update, separate the two using our guide to measuring the core update in Search Console.
Common Problems and Fixes
| Problem | Likely cause | Fix |
|---|---|---|
| Origin CPU spikes after launch | Cold cache plus faster crawling | Pre-warm; brief 429 or 503 if needed |
| Crawl requests drop to near zero | Bot challenge catching Googlebot | Enable verified-bot exemption |
| Pages fall out of the index | 429 or 503 left on for days | Remove the rule; return 200 |
| “Googlebot” traffic still heavy after fixes | Fake user agents | Verify by reverse DNS or IP range, block fakes |
What Not to Do
Do not switch all traffic to an empty cache. Consequence: the origin takes the full crawl at once. Correction: pre-warm first. Do not allowlist by user agent. Consequence: scrapers walk through. Correction: use verified-bot rules. Do not leave a 503 on over a weekend. Consequence: URLs may leave the index. Correction: lift it within a day or two, as Google advises. Google’s note on crawl budget explains why large sites feel this most.
Frequently Asked Questions
Does moving to a CDN hurt SEO?
Not by itself. Google says faster responses can increase crawling. The risks are an origin overloaded while the cache is cold and bot protection that blocks Googlebot, both of which you can prevent (Google Search Central, November 2026).
Why does Googlebot crawl more after a CDN switch?
Google’s crawlers adjust to how fast a site responds. Google says “Fast and stable responses speed up crawling,” so a faster CDN can trigger more requests, many of which reach the origin until the cache fills.
How long can I return 503 to slow Googlebot?
Google’s crawl-rate guide suggests a couple of hours or 1-2 days and warns against longer, because URLs that keep returning these codes for multiple days may be dropped from the index.
Should I use 403 to slow down Googlebot?
No. Google lists 500, 503 and 429 as the responses that reduce crawl rate. 403 and 404 are not on that list.
How do I know a request is really from Googlebot?
Run a reverse DNS lookup on the IP, check that the hostname ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup and confirm it returns the same IP. Or match the IP against Google’s published JSON range files.
How many IP ranges does Google publish for its crawlers?
By our count on November 12, 2026, Google’s five crawler and fetcher JSON files list 2,201 prefixes, including 317 for common crawlers such as Googlebot. The files change, so check them directly.
Sources and Methodology
This guide is built from Google’s November 2026 Search Central post on CDN moves and Google’s crawling documentation, read November 12, 2026, plus our own count of Google’s five IP range files and a reverse and forward DNS lookup we ran that day. The step order, timing suggestions marked as ours and the problem table are house recommendations, not Google rules. We did not test any CDN vendor’s settings.
- platform documentation Google Search Central, Moving to a CDN without overwhelming your origin (or blocking Googlebot) (November 2026). Crawl-speed behavior, cold cache, verified bots, pre-warming, 429/503. Read November 12, 2026.
- platform documentation Google Crawling docs, Reduce the Google crawl rate. 500/503/429, duration warning, index drop. Read November 12, 2026.
- platform documentation Google Crawling docs, Verifying Google’s crawlers and fetchers. DNS steps, JSON range files. Read November 12, 2026.
- platform documentation Google Crawling docs, Crawl budget management. Background on crawl capacity. Read November 12, 2026.
- live read Google, crawler IP range JSON files (five files, creationTime November 11, 2026). Our prefix count. Read November 12, 2026.
- trade press Search Engine Roundtable, Google: How Moving to a CDN Can Accidentally Overwhelm Your Website (November 11, 2026). News context. Read November 12, 2026.