RobottoCheck, the bot behind the checker

What a check requests

RobottoCheck is not a crawler. It only visits a site when a person enters that domain into the checker and presses the button. A single check sends at most these requests:

• The home page at http:// and https://, with and without www, following up to five redirects for each
• /robots.txt on the main host
• One sitemap URL (from robots.txt, or /sitemap.xml and /sitemap_index.xml)
• The home page once more with an AI crawler's user agent, to see whether the site treats AI bots differently
• A DNS lookup through Cloudflare's resolver, and an RDAP lookup at the domain's registry

That's usually 8 to 15 HTTP requests to your server, spread over a few seconds. It never follows links, submits forms, or requests anything beyond the list above.

User agents

Normal requests use:

Mozilla/5.0 (compatible; RobottoCheck/1.0; +https://robotto.org/bot/)

The AI crawler comparison sends a single home page request with a GPTBot-style user agent. It's labeled as GPTBot because the point is to see how your site answers that user agent. It comes from Cloudflare's network, not from OpenAI, and it is not OpenAI's crawler.

Where requests come from

Checks run on Cloudflare Workers, so requests come from Cloudflare's IP ranges and carry Cloudflare's usual CF-Worker header naming robotto.org.

Does it obey robots.txt?

It reads robots.txt to report on it, but it doesn't skip pages because of it. A person asked for a one-time check of a few specific URLs, much like opening them in a browser. RobottoCheck doesn't crawl and doesn't store your content.

Asking us to stop

Email contact@robotto.org from an address at your domain, or tell us how to verify you run it, and we'll add it to a list the checker refuses to check.