Every check, and why it's there

Redirects and status codes

Live in the free checker

We request the home page four ways: http://example.com, http://www.example.com, https://example.com and https://www.example.com. Redirects are followed by hand, one hop at a time, up to five, so you see every status code and every Location header instead of just the end result.

A site that changes hosting, CDN or CMS often picks up an extra hop without anyone noticing. A chain like http to https to www to a trailing-slash version costs a round trip each time, and every hop is one more place for a rule to break. Search engines follow several hops, but they don't promise to follow forever, and link signals are cleaner when the first redirect lands on the final address.

The checker flags temporary redirects (302 and 307) used for site-wide moves, chains of more than two redirects, loops, and variants that end in an error.

www vs non-www canonical setup

Live in the free checker

All four versions of your domain should end on one address. When www serves a page and the bare domain serves the same page without redirecting, you have two copies of the site. Links split between them, and search engines have to guess which one you meant.

The verdict picks the address most versions end on and calls it canonical. It goes red when the versions end on different hosts, and amber when the canonical version is plain http while https works. It also tells you when www has no DNS record at all, which is fine as long as nobody links to it. Our guide to www redirects covers which to choose and how to set it up.

robots.txt

Live in the free checker

We fetch /robots.txt from the canonical host and report the status code, size, content type and a short SHA-256 hash. The hash lets you spot an edit without reading the file: if it changes between runs, the file changed.

A robots.txt that returns 404 is fine. Crawlers read that as permission to crawl everything. A 5xx or a timeout is worse than no file, because Google treats an unreachable robots.txt as a reason to slow down or stop crawling until it can read it again. The checker flags both cases, plus a file that comes back as an HTML error page or as JSON, which happens more often than you'd think on frameworks that route every path to the app.

The loudest warning is for Disallow: / under User-agent: *. That one line, left over from a staging server, removes a site from search over a few weeks. Ten robots.txt mistakes that cost traffic.

AI crawler access

Live in the free checker

Your robots.txt is read for five AI crawler tokens:

For each one we report allowed or blocked, whether the decision came from its own group or the User-agent: * fallback, and the exact rule that matched. The parser follows RFC 9309: the most specific matching rule wins and Allow wins a tie.

Several of these companies run more than one crawler, for example separate agents for search features and for fetching a page when a user asks about it. We test the five tokens most robots.txt files mention. The AI crawlers guide lists the others.

AI bot blocking at the edge

Live in the free checker

Robots.txt is a request; a firewall is an order. Cloudflare and other CDNs can block AI crawlers before the request ever reaches your server, and since July 2025 Cloudflare has asked owners of newly added domains whether to allow AI crawlers, with blocking as the default. Site owners often don't remember choosing.

So after the normal request, we request your home page once more with GPTBot's user agent string. If the normal request gets a page and the GPTBot one gets 401, 403, 429 or 503, or a cf-mitigated challenge header, we report an edge block and note whether the response came from Cloudflare.

This is a basic test. A real GPTBot request comes from OpenAI's published IP ranges, and some firewalls treat verified bots differently from a user agent string sent from somewhere else. A block here means "requests that look like GPTBot are refused". A pass is a good sign but doesn't prove the real crawler gets through.

Sitemap

Live in the free checker

If robots.txt has a Sitemap: line, we fetch the first one. If it doesn't, we try /sitemap.xml and then /sitemap_index.xml. A sitemap only counts as found when the response really is one: an XML urlset or sitemapindex, a gzip file, or a plain-text list of URLs. A 200 that turns out to be your home page, a common result on single-page apps, is reported as broken.

HTTPS and TLS

Live in the free checker

The https versions of your domain must complete a TLS handshake that Cloudflare's network trusts. If the certificate is expired, self-signed or issued for another name, the request fails and we show the error. We also report whether the final page sends Strict-Transport-Security.

Cloudflare Workers can't read certificate details, so the free checker doesn't show the issuer or expiry date of the certificate itself. Certificate expiry warnings are planned for monitoring.

Domain expiry

Live in the free checker

We look up the registered domain through RDAP, using the rdap.org bootstrap service to find the right registry. RDAP replaced WHOIS as the official lookup for generic top-level domains in January 2025, and it returns structured dates instead of free text, so the expiry date doesn't depend on parsing a registrar's format.

You get the expiration date, days left, the registrar and the registry status codes. Under 30 days turns amber and under 8 turns red. Some country-code registries don't offer RDAP or don't publish expiry dates; those show as unknown.

Monitoring will send reminders at 30, 15, 7 and 2 days. How to track expiry across a portfolio.

Page load timing

Live in the free checker

For the canonical home page we record DNS lookup time, time to first byte, total download time and the extra time redirects add. These are server-side numbers taken from Cloudflare's edge, so they show how quickly your server answers, not how long a browser takes to render the page.

The full breakdown the old monitoring tools reported (connection, DNS, redirect, download and full load) needs repeated samples over time to mean anything. It's part of the monitoring plan.

Meta robots noindex

Live in the free checker

We read the robots and googlebot meta tags in the first 512 KB of the home page, and the X-Robots-Tag response header. Either one containing noindex or none turns the column red. A noindex on the home page is almost always a mistake carried over from a staging build or a "discourage search engines" checkbox.

Subdomain discovery

Coming with monitoring

Forgotten subdomains are where old staging sites, abandoned landing pages and dangling DNS records live. Monitoring will list the subdomains it can find for each domain, from certificate transparency logs and DNS, and tell you when a new one appears.

Lost entry pages

Coming with monitoring

A lost entry page is a URL that used to bring in visitors and now returns 404. They pile up after migrations, slug changes and deleted categories. Monitoring will keep a list of your known URLs and alert you when one that used to work starts failing. Until then, this guide shows how to find them by hand.

Traffic and engagement signals

Coming with monitoring

Organic versus non-organic visits, engagement on landing pages and crawl messages from search consoles all need access to your accounts. We plan to read them through official APIs, with your permission, and line them up against the technical checks so you can see whether a redirect change and a traffic drop happened in the same week.

Malware and blocklist status

Coming with monitoring

A domain on a malware or phishing blocklist shows a warning page in browsers and loses traffic overnight. Monitoring will check the major public lists and alert you when one of your domains appears.