Guide
Finding the pages you lost in a site migration
After a redesign or platform move, the traffic you lose is mostly on URLs nobody remembered. Here's how to rebuild the list and redirect what matters.
Most migrations lose traffic in the same way. The team redirects the pages it knows about: the main navigation, the top blog posts, the product categories. Then the new site launches and a long tail of URLs that nobody listed starts returning 404. Each one brought in a few visits a month. Together they were a fifth of the traffic.
We call these lost entry pages: URLs that used to be a way into the site and now lead nowhere. Finding them is mostly a matter of collecting old URLs from every source you can, then testing them against the new site.
Step 1: build the list of old URLs
No single source has every URL. Combine several.
Google Search Console
The Pages report lists URLs under "Not found (404)", with the date Google last crawled each one. After a migration this list grows for weeks as Google recrawls old URLs. The Performance report, with a date range before the migration, gives you the pages that actually had search clicks, which is the list that matters most. Export both.
Your analytics
A landing page report for the three to twelve months before the migration shows which URLs people arrived on, from any source. This catches pages that got traffic from social posts, newsletters or other sites rather than search.
Server logs
If you still have access logs from the old server, every URL that got a request is in there. Filter for status 200 responses to HTML pages and count requests per URL. Logs also show you which URLs crawlers were hitting, which can differ from what people visited.
Backlink data
Links from other sites point at specific URLs, often old ones. Search Console's Links report lists your top linked pages. Any link pointing at a 404 is wasted.
The old sitemap
If you saved a copy, it's the closest thing to an official list of what the old site had. If you didn't, the Internet Archive's Wayback Machine may have captured it, along with many old URLs.
The old CMS export
A database dump or export from the old platform gives you every post and page slug, including ones nobody linked to.
Merge everything into one column, strip query strings that don't change the content (tracking parameters, session IDs), and remove duplicates.
Step 2: test every old URL against the new site
Request each old URL on the live site and record the status code and, for redirects, where it ends up. A command-line loop works for a few thousand URLs:
while read -r url; do
curl -s -o /dev/null -w "%{http_code} %{redirect_url} $url\n" --max-time 10 "$url"
done < old-urls.txt > results.txt
Sort the results into four groups:
| Result | Meaning | Action |
|---|---|---|
| 200 | The URL still works | Nothing, unless it's now a different page |
| 301 to a relevant page | Redirected correctly | Nothing |
| 301 to the home page | Mass redirect | Usually treated as a soft 404. Map to a closer page if one exists |
| 404 or 410 | Lost | Map it, or decide it should stay gone |
Mass redirects to the home page deserve a mention. They look tidy, but Google generally treats a redirect to an unrelated page as a soft 404, so the old page's value doesn't carry over. Visitors who wanted a specific article and land on the home page leave.
Step 3: map the lost URLs
Start with the URLs that had traffic or links, because those are worth the effort. For each one, find the closest page on the new site: the same article under a new slug, the category it moved into, or a newer page on the same topic. Write the mapping as old path, new path.
Some patterns save time:
- Date-based blog URLs (
/2019/04/my-post/) moving to plain slugs (/blog/my-post/) can usually be handled with one pattern rule. - Category or tag renames can be redirected folder by folder.
- Uppercase, trailing slash and
.htmldifferences are often one rule each.
Pages that are truly gone, with no equivalent, can stay 404, or return 410 if you want to say clearly that they were removed. That's fine for pages with no traffic or links. A 404 isn't a penalty; it's an honest answer.
Step 4: implement and keep the redirects
Put the mapping in place as permanent 301 redirects, one hop each, straight to the final URL. Avoid chaining through an old redirect layer. If the old site already redirected /a to /b, and the new site moves /b to /c, update the first rule to go to /c directly.
Keep redirects for a long time. Google's guidance for site moves is to keep them for at least a year, and links from other sites keep pointing at old URLs for much longer than that. Removing redirects two years later to "clean up" is one of the quieter ways sites lose traffic.
Step 5: watch the 404 report
For the first two months after a migration, check Search Console's 404 list weekly. New old URLs will keep surfacing as Google recrawls them. Spot-check redirects with a tool that shows the full chain, such as the Robotto checker for the home page and main hosts, or curl -I -L for individual URLs.
Before the next migration
Most of this work disappears if you prepare beforehand. Before launch, export a full URL list with traffic and links from every source above, write the redirect map, and test it on the staging site. After launch, run the same test on production the same day.
Robotto monitoring will track known URLs and alert you when one that used to work starts returning 404. Until then, the steps above are the reliable way to find them.