AI crawlers and robots.txt: tokens and what they control

0:00 AI crawlers and robots.txt

Here's a mistake I see every month. A company reads a headline about AI companies training on websites, copies a list of bot names into robots.txt, and blocks all of them. Months later, they ask why ChatGPT and Claude never cite their guides. They blocked the search bots along with the training bots. In this lecture you'll learn the difference between training crawlers, search crawlers and user-initiated fetchers, what each robots.txt token actually controls, and how to test your file so it matches the policy you meant to set.

0:39 Three kinds of AI bot

Here's the mental model. AI companies run up to three kinds of bot. Training crawlers collect content that may be used to train future models. Search crawlers build the index an answer engine retrieves from when it answers questions. And user-initiated fetchers visit a page because a person in a chat just asked the assistant to look at it. Think of a library. One visitor photocopies books for an archive, one writes the catalogue, and one fetches a single book because a reader asked for it. Blocking one doesn't block the others. And blocking the cataloguer means you never appear in the catalogue.

1:24 The main tokens

Let's map the main tokens. OpenAI: GPTBot for training, OAI-SearchBot for ChatGPT search, and ChatGPT-User for user requests. Anthropic: ClaudeBot for training, Claude-SearchBot for search, and Claude-User for user requests. Perplexity: PerplexityBot for its index and Perplexity-User for user requests. Apple: Applebot-Extended, a control token for training Apple's models. Common Crawl's CCBot feeds many training datasets. And Google, which needs its own slide, because it's the one people get wrong most often.

1:55 Google: two switches, two jobs

Google-Extended is not a crawler. It's a control token that governs whether content Google crawls can be used for training future Gemini models and for grounding in the Gemini apps and Vertex AI. It does not remove you from Google Search, and it does not remove you from AI Overviews or AI Mode. Those run on Googlebot and Search's own controls. In twenty twenty-six, Google added a separate lever for them: the Search generative AI control in Search Console settings, which excludes you from AI Overviews, AI Mode and AI Overviews in Discover while keeping you in regular results. So remember: two switches, two jobs.

2:41 User-initiated fetchers

Now the tricky bit: user-initiated fetchers. Providers treat them differently, and their documentation has changed. At the time of writing, OpenAI says robots.txt rules may not apply to ChatGPT-User, because a person initiated the action. Perplexity describes its user fetcher similarly. Google's user-triggered fetchers, including a new one called Google-Agent added in March twenty twenty-six for AI agents, generally ignore robots.txt. Anthropic, by contrast, says all its bots, including Claude-User, honour robots.txt. So if you need hard enforcement for sensitive content, robots.txt isn't enough. Use authentication, firewall rules or rate limits.

3:21 Two rules to get right

Two robots.txt rules to get right. First, grouping. You can list several user-agent lines above one set of rules; that's valid under the standard, RFC ninety-three-oh-nine. Second, and this one catches experts too, a bot obeys only its most specific matching group. If you add a group for OAI-SearchBot that says allow everything, it no longer inherits the rules in the star group, like disallow slash account. So repeat any shared rules inside each specific group. Otherwise you may accidentally open your account pages to the very bots you just configured.

4:01 Content Signals

There's a newer layer too: Content Signals. robots.txt controls whether a bot may fetch a page, not what it may do with it afterwards. Cloudflare's Content Signals Policy adds a line to robots.txt stating your preferences for three uses: search, AI input, meaning using content in answers at query time, and AI training. For example, search yes, AI input yes, AI train no. Cloudflare applies search yes and AI train no by default on its managed robots.txt. But be clear with clients: these are signals of preference, not blocks. It's up to each crawler operator to honour them.

4:44 Example 1: Dubai design studio

Worked example one, simple. A Dubai interior design studio wants to appear in ChatGPT and Perplexity answers, but doesn't want its portfolio text used to train models. The rules are straightforward. Allow OAI-SearchBot, Claude-SearchBot and PerplexityBot. Block GPTBot, ClaudeBot, CCBot, Google-Extended and Applebot-Extended. Keep Googlebot and Bingbot untouched. Repeat the account and cart disallows in each specific group. Then test it, which brings us to the hands-on.

5:13 Hands-on: the robots grid

The lesson text includes a short Python script using only the standard library. It reads your live robots.txt and prints a grid: every AI token down the side, key paths across the top, and allow or block in each cell. One caveat: Python's parser applies rules in file order, not Google's longest-match precedence, so use it for per-bot checks and confirm tricky allow and disallow overlaps in Search Console's robots.txt report. Compare the grid with your written policy. Any mismatch is either a decision nobody recorded, or a bug.

5:52 Watch me do it: robots.txt vs policy

Watch me do it. I'll audit a live robots.txt against a written policy. The policy says: allow search and user-initiated AI bots on public pages, block training crawlers, keep account and cart private. Step one: I open the site's robots.txt and read it top to bottom. There's a star group with disallows for account and cart, and a new group for OAI-SearchBot with allow slash. Step two: I run the grid script from the lesson with thirteen tokens and four paths. The grid shows OAI-SearchBot allowed on slash account. That's the most-specific-group rule biting: the new group doesn't inherit the star group's disallows. Step three: GPTBot shows allowed everywhere, but the policy says block. Nobody added a GPTBot group. Step four: I draft the corrected file. I copy the account and cart disallows into the OAI-SearchBot group, and add a grouped block for GPTBot, ClaudeBot, CCBot, Google-Extended and Applebot-Extended with disallow slash. I add a Content-Signal line stating ai-train equals no. Step five: I run the grid again on the draft. Every cell now matches the policy. I attach the before and after grids to the ticket, and schedule a CDN check, because robots.txt isn't the only layer.

7:18 Example 2: the 'block all AI' mistake (illustrative)

Worked example two, with illustrative details. A London agency copied a block every AI bot list into a client's robots.txt after a news story. Three months later the client asked why ChatGPT and Claude never cited its guides. The grid showed the search bots blocked alongside the training crawlers. The corrected policy allowed search and user-initiated access, kept the training crawlers blocked because the client preferred that, and recorded the reasoning with a six-month review date. And the team also checked the CDN, because edge settings can override what robots.txt implies.

7:58 Recap and try this now

Recap. Separate training, search and user-initiated bots. Google-Extended is about Gemini training and grounding, not AI Overviews; the Search Console control handles AI surfaces. User fetchers may ignore robots.txt, so use authentication for anything truly private. Mind the most-specific-group rule, and use Content Signals to state preferences. Try this now. Run the robots grid script on your site, put it next to your written AI policy, and fix or document every mismatch.