Guide
Sitemap "couldn't fetch"? How to find and fix a broken sitemap
A sitemap that returns your home page, a 403 or a 60 MB file is worse than none. Here is how to test yours the way a crawler does.
Short answer
A sitemap fails when the URL doesn't return a real XML file with status 200, so request it yourself, check the status code and content type, make sure robots.txt and your firewall let crawlers reach it, and keep each file under 50,000 URLs and 50 MB.
Key takeaways
- The sitemap URL must return status 200 with real XML, not an HTML page, a redirect loop or a login screen.
- Each sitemap file can hold up to 50,000 URLs or 50 MB uncompressed; split larger sites with a sitemap index.
- Use absolute URLs on the same host and protocol as the site's canonical version.
- Google ignores priority and changefreq and uses lastmod only when it is consistently accurate.
- List the sitemap in robots.txt so every crawler can find it, not only the ones you submit to.
You submit a sitemap in Google Search Console and the status says "Couldn't fetch", or "Has errors", or the discovered page count stays at zero. The file looks fine when you open it in a browser. The problem is usually what a crawler gets, which can differ from what you see.
Why does Search Console say "Couldn't fetch"?
Google could not download a valid file from that URL, most often because of a non-200 status, a block or a response that isn't XML.
Freshly submitted sitemaps sometimes show "Couldn't fetch" before Google has tried at all. Give it a day. If the status stays, the causes are usually one of these:
| Cause | What the crawler gets |
|---|---|
| Wrong path or typo | 404 |
| Firewall, bot rules or a login wall | 403, a challenge page or a 401 |
| Single-page app routing | 200 with your home page HTML |
| Server error under load | 5xx or a timeout |
| Redirect to another host or protocol | A file whose URLs no longer match its own location |
| robots.txt disallows the path | No request at all |
How do you test the sitemap the way a crawler sees it?
Request the URL from outside your network and check the status code, the content type and the first lines of the body.
curl -sI https://example.com/sitemap.xml
curl -s https://example.com/sitemap.xml | head -5
You want 200, a content type such as application/xml or text/xml, and a body that starts with <?xml followed by <urlset or <sitemapindex. If you see <!DOCTYPE html>, the server is sending a web page. That is the most common failure on JavaScript frameworks, where every unknown path falls through to the app.
The Robotto checker does this for you: it reads the Sitemap: line in robots.txt, or tries /sitemap.xml and /sitemap_index.xml, and only reports a sitemap as found when the response really is one. A home page served at the sitemap address shows up as broken.
What if a firewall or bot protection blocks it?
Allow verified search engine crawlers through to the sitemap and robots.txt, and test from a clean connection.
Bot protection, country blocks and rate limits often treat crawlers as suspicious. You won't notice from your own browser, because you pass the challenge. Check your CDN or firewall logs for blocked requests to /sitemap.xml. Most providers have a setting for verified bots that lets Googlebot and Bingbot through without a challenge. If you block AI crawlers on purpose, make sure the rule doesn't catch search crawlers too; our guide on letting AI crawlers in covers how those rules overlap.
What limits does a sitemap have?
One sitemap file can list up to 50,000 URLs and be up to 50 MB uncompressed, and it must be UTF-8.
Bigger sites split their URLs into several files and list those in a sitemap index. Each file in the index has the same limits. Gzip compression is fine, but the 50 MB limit counts the uncompressed size.
Other format rules that trip people up:
- URLs must be absolute:
https://example.com/page/, not/page/. - Special characters must be escaped. A bare
&in a URL breaks the XML; write&. - All URLs should use the canonical host and protocol. A sitemap on
https://example.comlistinghttp://www.example.com/pages sends crawlers through redirects for every entry.
What should go in the sitemap?
Only canonical, indexable URLs that return 200: no redirects, no noindex pages, no blocked paths.
A sitemap is a list of pages you want in search. Each entry that redirects, returns 404 or carries noindex makes the list less trustworthy. CMS plugins sometimes include tag archives, attachment pages or test pages. Review what yours generates every few months.
Google says it ignores priority and changefreq and uses lastmod only when it is consistently and verifiably accurate. A lastmod that changes on every build, for every page, tells Google nothing. Set it from the real content date or leave it out.
How do crawlers find the sitemap?
Submit it in Search Console and Bing Webmaster Tools, and also list it in robots.txt so every other crawler finds it.
A single line at the end of robots.txt is enough:
Sitemap: https://example.com/sitemap.xml
You can list several. The Sitemap: line works for any crawler that reads robots.txt, which includes search engines you never registered with. Make sure robots.txt itself loads cleanly; our list of robots.txt mistakes covers the ways it can fail.
A five-minute sitemap check
curl -sIthe sitemap URL: 200 and an XML content type.- Look at the first lines:
<urlsetor<sitemapindex, not HTML. - Count the URLs; split at 50,000.
- Spot-check ten entries: each returns 200 with no redirect.
- Confirm the
Sitemap:line in robots.txt. - Resubmit in Search Console and watch the status for a few days.
Questions
What does 'Couldn't fetch' mean in Search Console?
Google could not download the file, or has not tried yet. A newly submitted sitemap can show this for a while. If it persists, request the URL yourself and check the status code, firewall rules and robots.txt.
Do I need a sitemap at all?
Small, well-linked sites can be crawled fine without one. A sitemap helps most on large sites, new sites with few links, and sites with pages that are hard to reach through navigation.
Can a sitemap be gzipped?
Yes. A .xml.gz file is fine. The 50 MB limit applies to the uncompressed size.
Should the sitemap include pages blocked by robots.txt or marked noindex?
No. Only list canonical URLs you want indexed. Listing blocked or noindex pages sends mixed signals and shows up as errors in Search Console.