https://robotto.org/blog/ Practical guides on robots.txt mistakes, AI crawler access, www redirects, redirect chains, domain expiry tracking and finding pages lost in a migration. en https://robotto.org/blog/reading-a-redirect-chain/ https://robotto.org/blog/reading-a-redirect-chain/ Wed, 04 Nov 2026 09:00:00 +0000 What each redirect status code means, how to read a chain hop by hop, which patterns cause trouble, and how to shorten a chain to a single clean redirect. https://robotto.org/blog/finding-lost-pages-after-migration/ https://robotto.org/blog/finding-lost-pages-after-migration/ Sat, 31 Oct 2026 09:00:00 +0000 How to find URLs that used to bring traffic and now return 404 after a migration, using Search Console, logs, analytics and archives, then map them to new pages. https://robotto.org/blog/tracking-domain-expiry/ https://robotto.org/blog/tracking-domain-expiry/ Wed, 28 Oct 2026 09:00:00 +0000 How domain expiry works after the renewal date, why RDAP is the source to trust, and a simple routine for keeping dozens or hundreds of names from dropping. https://robotto.org/blog/www-vs-non-www-redirects/ https://robotto.org/blog/www-vs-non-www-redirects/ Sat, 24 Oct 2026 09:00:00 +0000 Whether to use www or the bare domain, why both must not serve pages, and how to set up one clean 301 from every variant on common servers and CDNs. https://robotto.org/blog/letting-ai-crawlers-in/ https://robotto.org/blog/letting-ai-crawlers-in/ Wed, 21 Oct 2026 09:00:00 +0000 Which AI crawlers exist, what each one does, how to allow them in robots.txt, and how to find out whether your CDN or firewall is blocking them anyway. https://robotto.org/blog/robots-txt-mistakes/ https://robotto.org/blog/robots-txt-mistakes/ Wed, 14 Oct 2026 09:00:00 +0000 The robots.txt errors we see most on real sites, from a leftover Disallow-all to blocked CSS and 5xx errors, with how to spot each one and what to change.