AI search, answer engines and llms.txt: what actually matters

0:00 AI search: what actually matters

Every week there's a new acronym. GEO, AEO, LLMO. And every week someone sells a new trick: add an llms dot txt file, chunk your content into tiny pieces, rewrite everything in AI style. So what actually matters for AI search? In this lecture you'll learn how AI features and answer engines use web content, what Google's own guide says to do and to ignore, the truth about llms dot txt and AI crawler controls, and how to measure AI visibility.

0:35 Why it matters

Why does this matter? Because people now get answers from AI features inside search engines, like Google's AI Overviews and AI Mode and Microsoft Copilot in Bing, and from assistants with web search like ChatGPT, Perplexity, Gemini and Claude. Some simple informational queries are answered directly, which can reduce clicks. But being cited can drive highly qualified visits and brand awareness. Getting this right protects budgets from snake oil and puts effort where it pays.

1:08 The key idea

Here's the key idea. For Google, AI Overviews and AI Mode are built on the same crawling, indexing and ranking systems as Search. Google's guide, published in May twenty twenty-six, describes AEO and GEO as still SEO. Think of AI answers as a skilled editor writing a summary from the best sources in a library. The editor can only quote books that are on the shelves, clearly written and trustworthy. Your job is to be one of those books.

1:42 Google says do

What does Google's guide tell you to do? Create valuable, non commodity content with a unique point of view and first hand insight. Organise it so readers can use it, with good images and video. Keep pages crawlable and indexable. And use structured data as part of normal SEO. There's no special schema for AI features. It also explains query fan-out: for complex questions, the model runs several related searches and combines them, so a deep page can be cited for one sub question.

2:19 Google says ignore

And what does it say you can ignore for Google Search? llms dot txt and other special AI files or markup. Chunking content into tiny fragments, because Google's systems understand multi topic pages. Rewriting in an AI style or for every long tail variation, because AI features understand synonyms. And chasing inauthentic mentions across the web. In fact, Google's spam documentation, also updated in May twenty twenty-six, says attempts to manipulate generative AI responses in Search are covered by its spam policies.

2:55 llms.txt, honestly

So what is llms dot txt? It's a proposed convention from twenty twenty-four: a Markdown file at your site root that gives language models a concise overview and links to key content. It's not an official web standard, and Google says Google Search doesn't use it. Creating one neither helps nor harms visibility in Google, including AI Overviews. Some developer tools and AI products do read it, especially for documentation sites. So treat it as an optional, low cost extra for specific audiences, never as an SEO lever.

3:33 Crawler controls

Now crawler controls, where many teams make an expensive mistake. AI companies publish crawler names, like GPTBot and OAI SearchBot from OpenAI, ClaudeBot from Anthropic, and PerplexityBot. You can allow or block them in robots.txt. For Google, there's the Google Extended token, which controls use for Gemini training and grounding in some Gemini products. But blocking it does not remove you from AI Overviews or AI Mode. For those, Google offers snippet controls, and since twenty twenty-six, a Search generative AI control in Search Console.

4:10 Example 1 (simple)

Worked example one, simple. A small UK recipe site panics about AI and adds an llms dot txt file, then blocks Google Extended, expecting to protect its recipes from AI Overviews. Neither does what they expected. Google Search ignores the file, and Google Extended doesn't control AI Overviews. What actually helps them is making recipes genuinely non commodity: tested variations, step photos, and the reasons behind each step. If they truly want out of Google's AI features, the Search Console control is the tool, with the trade off of losing citations.

4:50 Example 2 (illustrative)

Worked example two, realistic and illustrative. A UK accountancy firm finds AI assistants answer how to register as self employed without citing it. It restructures its guide with a direct summary answer, a step by step list, and links to the official GOV dot UK pages. It adds an author bio for a named chartered accountant, updates dates and figures, and adds Organization and Person structured data. It then tracks AI assistant referrals in GA4, checks the Generative AI performance report in Search Console, and runs a monthly prompt panel.

5:29 Watch me do it: answer the AI question with evidence

Watch me do it. A UK accountancy firm asks me: should we add llms dot txt, block AI crawlers, and rewrite for AI? I'll answer with evidence in half an hour. First, I check how they appear today. I test five real client questions in AI Mode and two assistants, and record which sources are cited. Their guide on registering as self employed isn't cited; GOV dot UK and two competitors are. Next, the controls. I open their robots dot txt. Someone has blocked Google Extended, believing it removes them from AI Overviews. I explain that it controls Gemini training and grounding, not AI Overviews, and that the Search Console generative AI control is the tool if they ever want to opt out, with the trade off of losing citations. They decide to stay in. Then llms dot txt. I explain Google Search doesn't use it, but their developer tools partner might read one, so it's optional and low priority. Then the page itself. I restructure the guide with an answer first summary, a step list, links to the official pages, a named chartered accountant, and updated figures. Finally, I set up measurement: the GA4 AI assistants channel, the Generative AI performance report in Search Console, and a monthly prompt panel.

7:01 Measure AI visibility

How do you measure AI visibility? Search Console includes AI feature clicks and impressions within normal web search totals, and the Generative AI performance report, rolled out to all sites by the end of August twenty twenty-six, shows impressions for AI Overviews, AI Mode and AI features in Discover by page, country and device. In GA4, create a channel for AI assistants using a regex on session source. And keep a monthly prompt panel of real questions. Note that clicks from Google's own AI features still show as Google organic.

7:40 Mistakes + recap

Common mistakes. Treating llms dot txt as a ranking factor. Blocking Google Extended and thinking you've left AI Overviews. Blocking all crawlers without weighing visibility. Publishing generic content with nothing an AI summary would need to cite. And ignoring accuracy, which AI answers can amplify. Recap: AI features in Google run on Search foundations, so create non commodity, crawlable, trustworthy content. Ignore the gimmicks, understand the controls, and measure with first party data. Try this now. Test three real questions in two assistants and note which sources are cited. For the deep dive, take AI Search Optimization next.