---
title: "AI search, answer engines and llms.txt: what actually…"
description: "Search is changing, not disappearing People now get answers from AI-powered features inside search engines (such as Google's AI Overviews and AI Mode…"
url: https://optimizeall.com/learn/seo-and-content-strategy/ai-search-and-llms-txt
updated: 2026-10-05
---

SEO & Content Strategy · AI search and measuring SEO · lesson 15 of 17 · 12 min

# AI search, answer engines and llms.txt: what actually matters

## Search is changing, not disappearing

People now get answers from **AI-powered features** inside search engines (such as Google's AI Overviews and AI Mode, and Microsoft Copilot in Bing) and from **AI assistants and answer engines** such as ChatGPT search, Perplexity, Gemini and Claude. These systems summarise information from multiple sources and sometimes cite and link to them.

This affects SEO in several ways:

- Some informational queries are answered directly, which can reduce clicks for simple facts.
- Being **cited** in AI answers can drive highly qualified visits and brand awareness.
- Content that is clear, trustworthy, well-structured and genuinely original is more likely to be used as a source.

Terms such as **GEO (generative engine optimisation)** and **AEO (answer engine optimisation)** describe this work. In practice, most of it builds on SEO fundamentals — Google's own guide to optimising for its generative AI features (May 2026) says as much, describing AEO/GEO as still SEO.

## How to be a good source for AI answers

1. **Be crawlable and indexable.** AI features in search engines generally rely on the search index. If search engines cannot crawl and index a page, it is unlikely to be used.
2. **Answer questions clearly.** Put a concise, direct answer near the top of a section, then expand. Use descriptive headings that mirror real questions.
3. **Structure content.** Lists, steps, tables, definitions and summaries are easy to extract accurately.
4. **Offer original value.** First-hand experience, original data, expert opinion and unique examples give AI systems (and people) a reason to cite you rather than a generic page.
5. **Build entity clarity.** Consistent organisation and author information, About pages, structured data (such as Organization and Person) and consistent profiles across the web help systems understand who you are.
6. **Earn mentions.** Being referenced by reputable sites, publications and communities increases the chance that AI systems encounter and trust your brand.
7. **Keep content fresh and accurate.** Outdated prices, laws or product details can be repeated in AI answers and harm trust.

## Controlling AI crawlers

Many AI companies use crawlers with published user-agent names (for example OpenAI's GPTBot and OAI-SearchBot, Anthropic's ClaudeBot, PerplexityBot, and Google-Extended, which is a control token for whether content is used for Google's Gemini models rather than a separate crawler for Search). You can allow or disallow them in robots.txt. Consider the trade-off: blocking may protect content from training use but can reduce visibility in AI answers, depending on the system. Blocking Google-Extended does not remove you from Google Search — nor from AI Overviews or AI Mode, which draw on the normal Search index. Google's controls for those are snippet directives (such as `nosnippet`, which also affects classic snippets) and, since 2026, a **Search generative AI control** in Search Console that opts a site out of AI features in Search and Discover without affecting regular results. Check each provider's current documentation, because names and behaviours change.

## What is llms.txt?

**llms.txt** is a **proposed** convention (introduced in 2024) for a Markdown file at `example.com/llms.txt` that gives language models a concise overview of a site and links to its most useful content, sometimes with a companion `llms-full.txt` containing fuller text. A simple example:

```
# Studio Noor Interiors
> Dubai interior design studio specialising in apartments and villas.

## Guides
- [Interior design cost in Dubai](https://example.com/interior-design-cost-dubai/): pricing factors and ranges
- [Small apartment ideas](https://example.com/small-apartment-ideas/)

## Company
- [About and team](https://example.com/about/)
```

Important context:

- It is **not an official web standard**. Google's May 2026 guide to generative AI features states that Google Search does not use llms.txt or other special AI files or markup: creating one neither helps nor harms visibility in Google Search, including AI Overviews and AI Mode.
- Some AI tools and developer platforms do read it, especially for documentation sites.
- It is low-cost to create and maintain, so many sites add it as an optional extra. It does **not** replace robots.txt, sitemaps, good content or technical SEO.

## Measuring AI visibility

Measurement is still maturing:

- Search Console includes clicks and impressions from Google's AI features within overall Web search performance. The **Generative AI performance report**, rolled out to all sites by the end of August 2026, adds impressions for AI Overviews, AI Mode and AI features in Discover by page, country, device and date (Google says more metrics will follow — check the current help page).
- Analytics can show referrals from AI assistants (for example, sessions from chatgpt.com or perplexity.ai) – create a channel group or segment for them.
- Manually (or with emerging tools) test important questions in AI assistants and note whether and how your brand is cited.
- Track branded search growth and direct traffic as indirect signals.

## Worked example

A UK accountancy firm finds AI assistants often answer "how to register as self-employed in the UK" without citing it. It restructures its guide with a direct summary answer, step-by-step list and links to the official GOV.UK pages, adds an author bio for a named chartered accountant, updates dates and figures, adds Organization and Person structured data and publishes an llms.txt listing its key guides. It then tracks AI referrals in analytics and checks answer citations monthly.

## What Google's May 2026 guide says to do — and to ignore

Google Search Central's guide to optimising for generative AI features is the most authoritative source for Google's AI surfaces. In summary:

| Do | You can ignore (for Google Search) |
|---|---|
| Create valuable, **non-commodity** content with a unique point of view and first-hand insight | llms.txt and other special "AI files" or markup |
| Organise content so readers can use it; add high-quality images and video | "Chunking" content into tiny fragments — Google's systems understand multi-topic pages |
| Keep pages crawlable and indexable; follow Search Essentials | Rewriting content in an "AI style" or for every long-tail variation — AI features understand synonyms |
| Use structured data as part of normal SEO (no special schema is required for AI features) | Chasing inauthentic "mentions" across the web |

It also explains **query fan-out**: for complex questions, the model issues several related searches and combines the results, so a deep page can be cited for a sub-question even if it doesn't match the exact query. And Google's spam documentation (May 2026) states that techniques aimed at manipulating generative AI responses in Search are covered by its spam policies.

Other answer engines (ChatGPT search, Perplexity, Copilot, Claude with web search) have their own crawlers and retrieval systems and publish their own documentation; some developer tools do read llms.txt. The fundamentals — crawlable, clear, original, trustworthy content and genuine mentions — transfer across all of them. For an end-to-end programme, continue with **AI Search Optimization (ai-search-optimization-geo)**.

## Hands-on: a GA4 channel for AI assistants

In GA4 (Admin → Data display → Channel groups), copy the default group and add a channel **AI assistants** above *Referral*, with *Session source* matching the regex:

```text
(^|\.)(chatgpt\.com|chat\.openai\.com|perplexity\.ai|copilot\.microsoft\.com|gemini\.google\.com|claude\.ai)$
```

Clicks from Google's AI Overviews and AI Mode still arrive as `google / organic`; use Search Console for those.

## Common mistakes

- Treating llms.txt as a ranking factor or a substitute for SEO.
- Blocking all crawlers without considering visibility trade-offs.
- Publishing generic content that offers nothing an AI summary would need to cite.
- Ignoring accuracy, which AI answers can amplify.

## Video lecture: AI search, answer engines and llms.txt: what actually matters

Lecture coming soon · 12 chapters · about 8 minutes. Read the full transcript below.

1. AI search: what actually matters
2. Why it matters
3. The key idea
4. Google says do
5. Google says ignore
6. llms.txt, honestly
7. Crawler controls
8. Example 1 (simple)
9. Example 2 (illustrative)
10. Watch me do it: answer the AI question with evidence
11. Measure AI visibility
12. Mistakes + recap

## Lecture transcript

### AI search: what actually matters

Every week there's a new acronym. GEO, AEO, LLMO. And every week someone sells a new trick: add an llms dot txt file, chunk your content into tiny pieces, rewrite everything in AI style. So what actually matters for AI search? In this lecture you'll learn how AI features and answer engines use web content, what Google's own guide says to do and to ignore, the truth about llms dot txt and AI crawler controls, and how to measure AI visibility.

### Why it matters

Why does this matter? Because people now get answers from AI features inside search engines, like Google's AI Overviews and AI Mode and Microsoft Copilot in Bing, and from assistants with web search like ChatGPT, Perplexity, Gemini and Claude. Some simple informational queries are answered directly, which can reduce clicks. But being cited can drive highly qualified visits and brand awareness. Getting this right protects budgets from snake oil and puts effort where it pays.

### The key idea

Here's the key idea. For Google, AI Overviews and AI Mode are built on the same crawling, indexing and ranking systems as Search. Google's guide, published in May twenty twenty-six, describes AEO and GEO as still SEO. Think of AI answers as a skilled editor writing a summary from the best sources in a library. The editor can only quote books that are on the shelves, clearly written and trustworthy. Your job is to be one of those books.

### Google says do

What does Google's guide tell you to do? Create valuable, non commodity content with a unique point of view and first hand insight. Organise it so readers can use it, with good images and video. Keep pages crawlable and indexable. And use structured data as part of normal SEO. There's no special schema for AI features. It also explains query fan-out: for complex questions, the model runs several related searches and combines them, so a deep page can be cited for one sub question.

### Google says ignore

And what does it say you can ignore for Google Search? llms dot txt and other special AI files or markup. Chunking content into tiny fragments, because Google's systems understand multi topic pages. Rewriting in an AI style or for every long tail variation, because AI features understand synonyms. And chasing inauthentic mentions across the web. In fact, Google's spam documentation, also updated in May twenty twenty-six, says attempts to manipulate generative AI responses in Search are covered by its spam policies.

### llms.txt, honestly

So what is llms dot txt? It's a proposed convention from twenty twenty-four: a Markdown file at your site root that gives language models a concise overview and links to key content. It's not an official web standard, and Google says Google Search doesn't use it. Creating one neither helps nor harms visibility in Google, including AI Overviews. Some developer tools and AI products do read it, especially for documentation sites. So treat it as an optional, low cost extra for specific audiences, never as an SEO lever.

### Crawler controls

Now crawler controls, where many teams make an expensive mistake. AI companies publish crawler names, like GPTBot and OAI SearchBot from OpenAI, ClaudeBot from Anthropic, and PerplexityBot. You can allow or block them in robots.txt. For Google, there's the Google Extended token, which controls use for Gemini training and grounding in some Gemini products. But blocking it does not remove you from AI Overviews or AI Mode. For those, Google offers snippet controls, and since twenty twenty-six, a Search generative AI control in Search Console.

### Example 1 (simple)

Worked example one, simple. A small UK recipe site panics about AI and adds an llms dot txt file, then blocks Google Extended, expecting to protect its recipes from AI Overviews. Neither does what they expected. Google Search ignores the file, and Google Extended doesn't control AI Overviews. What actually helps them is making recipes genuinely non commodity: tested variations, step photos, and the reasons behind each step. If they truly want out of Google's AI features, the Search Console control is the tool, with the trade off of losing citations.

### Example 2 (illustrative)

Worked example two, realistic and illustrative. A UK accountancy firm finds AI assistants answer how to register as self employed without citing it. It restructures its guide with a direct summary answer, a step by step list, and links to the official GOV dot UK pages. It adds an author bio for a named chartered accountant, updates dates and figures, and adds Organization and Person structured data. It then tracks AI assistant referrals in GA4, checks the Generative AI performance report in Search Console, and runs a monthly prompt panel.

### Watch me do it: answer the AI question with evidence

Watch me do it. A UK accountancy firm asks me: should we add llms dot txt, block AI crawlers, and rewrite for AI? I'll answer with evidence in half an hour. First, I check how they appear today. I test five real client questions in AI Mode and two assistants, and record which sources are cited. Their guide on registering as self employed isn't cited; GOV dot UK and two competitors are. Next, the controls. I open their robots dot txt. Someone has blocked Google Extended, believing it removes them from AI Overviews. I explain that it controls Gemini training and grounding, not AI Overviews, and that the Search Console generative AI control is the tool if they ever want to opt out, with the trade off of losing citations. They decide to stay in. Then llms dot txt. I explain Google Search doesn't use it, but their developer tools partner might read one, so it's optional and low priority. Then the page itself. I restructure the guide with an answer first summary, a step list, links to the official pages, a named chartered accountant, and updated figures. Finally, I set up measurement: the GA4 AI assistants channel, the Generative AI performance report in Search Console, and a monthly prompt panel.

### Measure AI visibility

How do you measure AI visibility? Search Console includes AI feature clicks and impressions within normal web search totals, and the Generative AI performance report, rolled out to all sites by the end of August twenty twenty-six, shows impressions for AI Overviews, AI Mode and AI features in Discover by page, country and device. In GA4, create a channel for AI assistants using a regex on session source. And keep a monthly prompt panel of real questions. Note that clicks from Google's own AI features still show as Google organic.

### Mistakes + recap

Common mistakes. Treating llms dot txt as a ranking factor. Blocking Google Extended and thinking you've left AI Overviews. Blocking all crawlers without weighing visibility. Publishing generic content with nothing an AI summary would need to cite. And ignoring accuracy, which AI answers can amplify. Recap: AI features in Google run on Search foundations, so create non commodity, crawlable, trustworthy content. Ignore the gimmicks, understand the controls, and measure with first party data. Try this now. Test three real questions in two assistants and note which sources are cited. For the deep dive, take AI Search Optimization next.

## Video transcript

Search is changing. More and more people get answers from AI: Google's AI Overviews and AI Mode, ChatGPT, Perplexity, Gemini and Claude. Instead of ten blue links, they see a summary, sometimes with a few sources cited. So how do you become one of those sources? The good news is that it starts with solid SEO. If search engines can't crawl and index your page, AI features that rely on the index are unlikely to use it. Then, make your content easy to use as a source. Put a clear, direct answer near the top of each section. Use headings that match real questions. Add steps, lists and tables. Most importantly, offer something original: first-hand experience, your own data, a named expert. Generic content gives an AI system no reason to cite you. You may have heard of llms dot t x t. It's a proposed file that gives language models a short overview of your site and links to your best pages. It's cheap to create and some tools read it, but it isn't an official standard, and Google has said it isn't used for Google Search. Treat it as a small extra, never a replacement for good content. Finally, measure what you can. Track referrals from AI assistants in your analytics, test key questions regularly and watch your branded searches grow. The fundamentals haven't changed: be useful, be clear and be trustworthy.

## Key takeaways

- AI Overviews, AI Mode and answer engines summarise sources; being cited requires crawlable, clear, original, trustworthy content.
- Structure answers directly, build entity clarity and earn mentions from reputable sources.
- AI crawlers can be managed in robots.txt; weigh visibility against content-use concerns and check current documentation.
- llms.txt is an optional, proposed convention, not an official standard or ranking factor; it complements rather than replaces SEO.

## Try it

Pick three important questions your audience asks. Test them in two AI assistants and note which sources are cited. Rewrite one of your pages to answer the question more directly and draft an llms.txt for your site.

- [Previous: Content pruning and consolidation without losing value](https://optimizeall.com/learn/seo-and-content-strategy/content-pruning-and-consolidation)
- [Next: Measuring SEO with Google Search Console](https://optimizeall.com/learn/seo-and-content-strategy/search-console-measurement)
- [All lessons of SEO & Content Strategy](https://optimizeall.com/learn/seo-and-content-strategy)