Which AI crawlers does your robots.txt let in?

The tokens

Search crawlers build the index an answer engine queries. User crawlers fetch a page because someone asked about it right now. Training crawlers collect corpora. Blocking training is a policy choice. Blocking search or user crawlers means you can't be cited by that assistant. Vendor documentation for each token is linked from the methodology.

How a crawler reads your robots.txt

A crawler doesn't read your whole file. It picks one group and ignores the rest. That's the part most people get wrong, so it's worth slowing down on.

Each group starts with one or more User-agent lines, followed by Allow and Disallow rules. A crawler looks for a group that names its own token. If it finds one, it uses that group and nothing else. If it doesn't, it falls back to the * group. So a named group doesn't add to the * rules; it replaces them.

Inside the chosen group, the longest matching rule wins. Here's a short example in words. Say the * group disallows the whole site with a single slash, and also allows a folder called /guides/. A crawler with no group of its own asks about a page inside /guides/. Both rules match, but the Allow path is longer than the one-character Disallow, so the page is open. Ask about the homepage and only the Disallow matches, so it's closed. Now add a group that names GPTBot and says nothing except Disallow for a /private/ folder. GPTBot ignores the * group completely. The rest of the site is open to it, even though the * group looked locked down.

That's exactly how our AI-crawler robots.txt check works: it evaluates every AI crawler token the way that crawler reads the file, its own group if it has one, otherwise the * group, longest rule wins.

Search, user and training crawlers

Not all AI crawlers do the same job, so they shouldn't get the same treatment. Think of them as three families.

Search crawlers build the index an answer engine queries. OAI-SearchBot, Claude-SearchBot and PerplexityBot sit here. If they can't read a page, that page isn't in the index the assistant draws on.

User crawlers fetch a page because someone asked about it right now. ChatGPT-User, Claude-User and Perplexity-User are the examples. A person pastes a question, the assistant goes and looks, and your robots.txt decides whether it can.

Training crawlers collect corpora. GPTBot, ClaudeBot and Google-Extended belong to this family. Blocking them is a policy choice, and a legitimate one either way.

Blocking search or user crawlers is different. It means that assistant cannot cite you. Full stop.

Common mistakes

Most problems come down to three things.

The first is a blanket Disallow for all bots. Someone wanted to stop training, wrote a * group with Disallow: /, and shut out every search and user crawler along with it. It feels tidy. It's also the quickest way to vanish from answers.

The second is blocking user crawlers by accident. This usually happens through a named group that's broader than intended, or a copied template that lists every AI token in one block. Remember the group rule: once a crawler has its own group, the * rules stop applying to it, for better or worse. Check each token, not just the file as a whole.

The third is forgetting that a CDN or firewall can block independently. Your robots.txt might say yes while a bot-protection rule says no. The file is a request; the firewall is a wall. A clean robots.txt result tells you what you've asked for, not what actually gets through. Also worth checking: whether your security settings treat unfamiliar user agents as threats by default.

What to do with the result

Start with the search and user crawlers. If any of them are blocked and you didn't mean that, fix it first, because that's the one decision that directly stops an assistant from citing you.

Then decide on training, separately. Some businesses are happy to be in the corpora; others aren't, and that's fine. Write the choice down so the next person who edits the file knows it was deliberate.

Once the file says what you want, test the layer beneath it. Ask your hosting or security team whether the same bots reach the page. And remember what this tool can't do: it reads rules, it doesn't measure outcomes. Whether assistants then name and cite you is a separate question, and the answer can change from one day to the next.

The check takes seconds and needs no account. Run it, read the verdict for each token, and make the call on purpose.