Letting AI crawlers in on purpose: a GEO checklist

Generative engine optimization (GEO) is the work of getting your pages read, understood and cited by AI assistants and AI search. Most of the advice is about content. Before any of that matters, the crawlers have to be able to fetch your pages, and on a surprising number of sites they can't. Sometimes that's a decision. Often it's an accident.

This guide covers who the crawlers are, how to write robots.txt rules for them, and how to check that nothing upstream overrides those rules.

Know which crawler does what

AI companies now run several bots each, with different jobs. Blocking one doesn't block the others, and the difference matters:

Two things in that table catch people out. First, Google-Extended doesn't crawl anything. Googlebot does the crawling, and Google-Extended only tells Google whether it may use that content for Gemini models. Blocking Google-Extended doesn't remove you from Google Search, and it doesn't affect AI Overviews, which are part of Search. Google lists all of this in its crawler overview.

Second, the user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) act on behalf of a person who asked for a page. Operators describe these differently from autonomous crawlers, and some say robots.txt rules may not apply to them in the same way. Check each operator's current documentation if this matters to you, because the details change.

Decide what you actually want

There's no single right answer. A useful way to frame it is to separate training from retrieval.

Training crawlers (GPTBot, ClaudeBot, CCBot, the Google-Extended and Applebot-Extended tokens) feed model training. Allowing them doesn't send visitors directly. Some publishers block them as a matter of policy; others allow them because they want their brand and facts to be part of what models know.

Search and retrieval crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) decide whether you show up, with a link, when someone asks an AI assistant a question. If you want AI referral traffic, these are the ones to let in.

A common middle position is to allow search crawlers and block training crawlers:

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: *
Disallow: /admin/

Several User-agent lines in a row form one group, so this file is shorter than listing each bot separately. Remember that a bot with its own group ignores the * group. If you want the search bots kept out of /admin/ too, add that line to their group.

Check that the firewall agrees

Robots.txt is advice to well-behaved bots. A CDN or firewall rule is enforcement, and it wins. The most common case is Cloudflare. Its "Block AI bots" setting refuses known AI crawlers at the edge. Since July 2025, Cloudflare asks owners of newly added domains whether to allow AI crawlers, and the default answer is to block them. Cloudflare can also manage robots.txt for you, adding its own AI rules to the file you serve.

Security plugins on WordPress and other platforms do similar things with their own bot lists. The result is a site whose robots.txt says "welcome" while every request from GPTBot gets a 403.

The Robotto checker tests both layers. It reads your robots.txt rules for GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot, then requests your home page twice, once normally and once with GPTBot's user agent. If the second request is refused and the first isn't, the AI crawlers column shows an edge block and says whether the response came from Cloudflare.

One caution about that test: real AI crawlers come from their operators' published IP ranges, and some firewalls let verified bots through while blocking anything that only claims to be one. A block in our test means requests that look like GPTBot are refused. A pass is a good sign, and your server logs are the final word.

Make pages easy to read without JavaScript

Most AI crawlers fetch HTML and don't run JavaScript the way Googlebot's renderer does. If your content is loaded by client-side scripts, a crawler may see an empty shell. Check a page with JavaScript disabled, or fetch it with curl, and make sure the main text is in the HTML.

Other things that help retrieval: a clear title and description, headings that say what each section covers, answers stated plainly near the top, and dates on anything time-sensitive. None of this is special to AI. It's what made pages easy for search engines to understand before AI assistants existed.

What about llms.txt?

llms.txt is a proposed convention: a Markdown file at your site root that lists your most useful pages for language models. It's cheap to add, and some tools read it. As of this writing none of the big AI operators has said its crawlers rely on it, so treat it as a nice extra rather than a replacement for robots.txt and crawlable HTML.

A short checklist

1. Write down which AI crawlers you want in, and which you don't, as a team decision.
2. Express that in robots.txt with grouped User-agent lines. Keep shared rules in every group.
3. Check your CDN and security plugin settings for AI bot blocking and make them match.
4. Run the checker and confirm robots.txt and the edge test agree.
5. Look at your server logs for the crawler user agents over the following weeks.
6. Revisit every few months. New crawlers appear and old ones change names.