Guide
Ten robots.txt mistakes that quietly kill traffic
A leftover staging rule, a blocked CSS folder, a file that returns your home page. Most robots.txt damage comes from a short list of mistakes.
Robots.txt is one of the few files where a single character can take a site out of search. It's also a file nobody reads after launch. That combination is why the same mistakes show up again and again when we run checks across portfolios. Here are the ones that cost the most traffic, roughly in order of how often we see them.
1. Disallow: / left over from staging
User-agent: *
Disallow: /
Staging servers block crawlers, which is sensible. Then the site launches, someone copies the whole directory to production, and the block comes along. Pages don't vanish overnight. Google stops recrawling them, and over a few weeks they fall out of the index or show up with no description.
Fix it by deleting the rule, then make the deploy process generate robots.txt per environment instead of copying it. The Robotto checker shows this case as a red "blocks all" in the robots.txt column.
2. Thinking robots.txt removes pages from search
Disallow stops crawling. It doesn't stop indexing. If other sites link to a blocked URL, Google can still list it, usually as a bare URL with "No information is available for this page." Worse, a blocked page can't be crawled, so Google never sees a noindex tag on it.
If you want a page out of search, let it be crawled and add <meta name="robots" content="noindex">, or send an X-Robots-Tag: noindex header. Google stopped honoring noindex lines inside robots.txt in 2019, so those do nothing.
3. Blocking CSS, JavaScript or images
Old advice said to block /wp-includes/, /assets/ or /static/ to save crawl budget. Google renders pages, and it needs the CSS and JavaScript to see what users see. Block them and Google may decide the page is broken or not mobile friendly. Leave asset folders crawlable unless they contain something private.
4. A robots.txt that isn't a text file
On many frameworks every unknown path is routed to the application. Ask for /robots.txt and you get the home page, an HTML error page, or a JSON response, all with status 200. Crawlers try to parse it as robots.txt, find no valid rules, and crawl everything. That's harmless until the day you add a real rule to a file nobody is serving.
The checker reports the content type and warns when the body looks like HTML or JSON. The fix is a real static file at the root with Content-Type: text/plain.
5. Server errors on robots.txt
A missing robots.txt (404) is fine: crawlers take it as permission to crawl everything. A robots.txt that returns 500 or 503, or times out, is different. Under RFC 9309, crawlers should assume complete disallow while the file is unreachable because of a server error. Google follows that and holds back crawling of the whole site until it can read the file again. If the errors last for weeks, Google eventually falls back to the last copy it had.
This shows up on sites behind aggressive bot protection, where the firewall challenges the request for robots.txt itself. Make sure /robots.txt is reachable by everything, including bots you otherwise block.
6. Wildcards that match more than you meant
Google and most major crawlers support * (any sequence) and $ (end of URL). They're easy to get wrong:
| Rule | Blocks | Also blocks |
|---|---|---|
Disallow: /*? |
Any URL with a query string | Paginated archives, filtered category pages you wanted indexed |
Disallow: /blog |
/blog and /blog/ |
/blog-tips/, /blogroll.html |
Disallow: /*.pdf |
PDFs | /guides.pdf-alternatives/ (no $, so it's a prefix match) |
Disallow: /*.pdf$ |
URLs ending in .pdf | Nothing else, which is usually what you meant |
Rules are prefix matches. If you mean a folder, end the rule with a slash.
7. Forgetting that groups don't combine
When a crawler finds a group with its own name, it uses that group and ignores User-agent: *. So this:
User-agent: *
Disallow: /admin/
Disallow: /cart/
User-agent: Googlebot
Disallow: /search/
lets Googlebot crawl /admin/ and /cart/. People add a specific group to block one extra path and accidentally unblock everything in the general group. Repeat the shared rules in every named group.
The same thing happens with AI crawlers. A User-agent: GPTBot group with one Allow line means GPTBot ignores your * rules entirely. Robotto's details panel shows which group each AI crawler matched.
8. Allow and Disallow fights
When an Allow and a Disallow both match, Google uses the most specific rule, meaning the longest path. If they're the same length, Allow wins. Order in the file doesn't matter. This trips people who write rules in order and expect the first match to win, as in some old parsers.
User-agent: *
Disallow: /shop/
Allow: /shop/sale/
Here /shop/sale/ is crawlable and the rest of /shop/ isn't. That's correct under the RFC, but some smaller crawlers still use first-match logic. Keep important pages out of these overlaps if you can.
9. A huge file
Google reads the first 500 KiB of robots.txt and ignores the rest. Generated files with thousands of lines, one per product or tag, can cross that line, and the rules after the cut-off quietly stop working. The checker reports the file size. If yours is anywhere near 100 KB, look for patterns you can replace with a wildcard.
10. Blocking the sitemap or forgetting to list it
A Sitemap: line in robots.txt is the simplest way to tell every crawler where your sitemap is. It must be an absolute URL, and it shouldn't point into a disallowed folder. We often see Sitemap: /sitemap.xml (relative, which the standard doesn't allow) or a sitemap URL that's on the old domain after a migration.
A quick audit routine
Once a quarter, or after every launch or migration:
- Run your domains through the checker. Look for red in the robots.txt column, a content type other than text/plain, and a changed hash you didn't expect.
- Open the details for a few domains and read which group each AI crawler matched. If the answer surprises you, the file is more complicated than it needs to be.
- Test your most important URLs against the live file in Google Search Console's robots.txt report, which also shows when Google last fetched it.
- Keep robots.txt in version control, next to the code that deploys it.
The file should be short enough that someone new on the team can read it in a minute and say what it does. If it isn't, that's the first thing to fix.