llms.txt, snippet controls and choosing your AI access policy

0:00 llms.txt and your AI access policy

Every few months a new file or setting promises to control how AI uses your website. The best known is llms.txt. Clients ask about it constantly. Should we add it? Will it get us into ChatGPT? In this lecture you'll get an honest status report on llms.txt as of twenty twenty-six, a map of every control you actually have, and a decision framework for writing an AI access policy your whole team can follow.

0:32 What llms.txt is

What is llms.txt? It's a proposal, published in twenty twenty-four by Jeremy Howard of Answer dot AI, for a Markdown file at the root of a website. It gives language models a concise, curated overview of a site: a summary and links to the most useful pages, often with Markdown versions of documentation. The idea is sensible. Help models and agents find the right content without wading through complex HTML. Think of it as a hand-written guide left at a hotel reception for visitors who don't have time to explore.

1:11 Status in 2026

Now the honest status. It's not an official standard. No major search or answer engine has publicly committed to using it for rankings, retrieval or citations. Google is explicit: its May twenty twenty-six guide for generative AI features lists llms.txt among things you don't need, and a June twenty twenty-six clarification says Google Search ignores it, so it neither helps nor harms. Where it does have a niche is developer documentation, where people point coding assistants and agents at it directly. So it's cheap and harmless, but an experiment, not a strategy.

1:51 The real control panel

So what controls do you actually have? Robots.txt per bot, which stops compliant crawlers fetching content. Content-Signal lines, which state your use preferences for search, AI input and training. Google's Search generative AI control, which removes you from AI Overviews, AI Mode and AI in Discover. Snippet controls like nosnippet and max-snippet, which also cut normal snippets. Noindex, which removes pages from search entirely. Authentication and paywalls, which are real enforcement. And CDN bot settings, which must stay in sync with everything else. Each one has a trade-off.

2:29 Decision framework

How do you choose? Start from the business model, not from fear or hype. Question one: what do you gain from AI visibility? Lead generation businesses, SaaS, local services and e-commerce usually gain a lot, because customers ask assistants for recommendations. Publishers and paid content businesses face a harder trade, because answers can substitute for visits. Question two: what are you trying to prevent? Training use, appearance in answers, or server load? Each has a different lever. Question three: which content is sensitive? Premium research deserves stricter controls than marketing pages.

3:09 Typical policy profiles

Here are common policy profiles. A local service business, agency or SaaS company typically allows search and user-initiated bots, and makes a conscious decision on training bots. A subscription publisher allows Googlebot, weighs each AI bot carefully, publishes public summaries, and may pursue licensing. An e-commerce store allows search bots so products can be recommended, and protects account and checkout paths. A research firm with paid reports allows public summaries, gates full reports, and blocks training crawlers.

3:42 Example 1: Lahore restaurant

Worked example one, simple. A family-run restaurant in Lahore asks whether it needs llms.txt. We look at what matters: its menu and opening hours aren't in plain HTML, and its Business Profile has old hours. So the answer is no, not now. Fix the menu page and the Business Profile first, allow the search bots, and leave training bots as a conscious choice. llms.txt would change nothing for a restaurant's visibility today.

4:13 Example 2: Islamabad research firm (illustrative)

Worked example two, with illustrative details. A policy research firm in Islamabad sells detailed reports but wants its public briefings cited. Full reports sit behind a login. Public two-page summaries show key findings and methodology. Search and user-initiated bots are allowed on the summaries. Training crawlers are blocked, and the Content-Signal line says AI train no. The Search Console AI control stays off, because AI citations of summaries drive report enquiries. And it adds llms.txt only for its open data documentation, where developers genuinely use it.

4:50 Hands-on: the one-page policy

The hands-on for this lesson is a one-page AI access policy. The lesson text has a template: goals, which search and answer engines are allowed, how user-initiated fetchers are handled, which training crawlers are blocked, the Content-Signal line, your Search Console AI control decision with a name and date, what's gated, how the CDN is aligned, the status of llms.txt, and how you'll monitor. Keep it in version control next to robots.txt, and review it every six months or whenever a provider changes its documentation.

5:27 Common mistakes

Common mistakes. Selling llms.txt as a guaranteed AI ranking factor. Blanket-blocking every AI bot, then wondering why the brand never appears in assistants. Using nosnippet sitewide to stop AI and losing normal search snippets. And having no written policy, so marketing, IT and the CDN contractor all change settings independently. The written policy is the cure for all four.

5:53 Watch me do it: a policy in one meeting

Watch me do it. I'll write a real AI access policy with a client in one meeting. Step one: goals. I ask the founder of a Dubai interior design studio, what do you want from AI assistants? She says: be recommended when people ask for designers in Dubai, and don't let my portfolio text train models. I write both lines. Step two: I fill the allow list: Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot, and the user-initiated fetchers. Step three: the block list for training: GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended. Step four: Content-Signal: search yes, AI input yes, AI train no. Step five: the Google decision. I explain the Search generative AI control. She wants to stay in AI Overviews, so the control stays off, and I write her name and today's date next to it. Step six: gated content: none. Step seven: CDN. I open her Cloudflare dashboard with her, find the AI crawler setting, and align it with the lists. Step eight: llms.txt. We skip it; she has no documentation site. Step nine: monitoring, a monthly log check. She approves, and I commit the policy next to robots.txt.

7:15 Who owns which lever?

One more practical point: where these decisions actually live in a company. robots.txt is usually owned by developers. The CDN bot settings are often owned by IT or an outside contractor. The Search Console AI control sits with whoever has owner access, often marketing. And llms.txt might be added by the documentation team. That's four owners for one policy, which is exactly how contradictions creep in. So name one accountable owner for the policy, list who controls each lever, and make changes only through that owner. It sounds bureaucratic, but it takes ten minutes to set up and prevents months of invisible damage.

8:00 Recap and try this now

Recap. llms.txt is a cheap experiment with a real niche in developer docs, but Google Search ignores it and no engine has committed to using it. Your real controls are robots.txt, Content-Signal, the Search Console AI control, snippet controls, authentication and your CDN. Choose from the business model. Try this now. Fill in the one-page policy template for your site, get one decision-maker to approve it, and store it next to robots.txt.