robots.txt, meta robots and X-Robots-Tag

0:00 robots.txt, meta robots, X-Robots-Tag

robots.txt, meta robots and the X-Robots-Tag header look simple. They're also behind some of the most expensive mistakes in SEO, including the one-line production Disallow that removes a whole site from crawling. In this lecture you'll learn which tool does which job, the rules that trip up experts, the classic blocked-plus-noindex trap, and how to put robots.txt into automated tests so a bad deploy can't slip through.

0:29 Right tool, right job

Start with the job, not the tool. To stop crawling of a path, use robots.txt Disallow, but it doesn't guarantee de-indexing. To keep a crawlable page out of the index, use a meta robots noindex, and don't block it in robots.txt. For non-HTML files like PDFs, use the X-Robots-Tag HTTP header. To consolidate duplicates, use canonicals or redirects. For an urgent temporary removal, use Search Console's Removals tool, which lasts about six months. And to protect private content, use authentication, because robots.txt is public and advertises your paths.

1:07 robots.txt fundamentals

Now robots.txt fundamentals. It lives at the root of each host and protocol, so www and a shop subdomain each need their own. It follows RFC ninety-three-oh-nine. Four rules matter most. Grouping: a crawler obeys only the most specific user-agent group that matches it, so a Googlebot group means Googlebot ignores the star group entirely. Precedence: for Google, the longest matching rule wins, and if Allow and Disallow tie, Allow wins. Wildcards: star matches any sequence and dollar anchors the end. And paths are case-sensitive.

1:44 Facts people forget

A few more facts people forget. Google processes only the first five hundred kibibytes of robots.txt. Google doesn't support noindex, crawl-delay or nofollow lines in robots.txt, although Bing does support crawl-delay. And error handling matters. If robots.txt returns a four-hundred-level error, Google treats it as no restrictions. If it returns five-hundred-level errors for an extended period, Google may stop crawling the site. So a broken robots.txt server can quietly halt crawling.

2:15 Index directives

Meta robots and X-Robots-Tag carry the index directives. Common values: noindex, nofollow, none, nosnippet, max-snippet, max-image-preview, max-video-preview, noimageindex and unavailable after. There's also the data-nosnippet attribute to exclude a section from snippets. Snippet controls also apply to how Google may show your content in AI features, but they're blunt, because you lose normal snippets too. Since twenty twenty-six, Google also has a Search Console setting that excludes a site from AI Overviews and AI Mode without touching snippets. For PDFs, the lesson text has Apache and Nginx examples for an X-Robots-Tag noindex header.

2:55 The blocked + noindex trap

Now the classic trap. You Disallow a path in robots.txt and add noindex to the pages. Google can't crawl them, so it never sees the noindex, and it may keep the URLs indexed from links. The correct sequence is: allow crawling, serve noindex, wait until the URLs drop out, confirmed in URL Inspection or the Page indexing report, and only then consider a Disallow if you also want to stop crawling. Order matters.

3:27 Staging sites

Staging sites deserve their own warning. They get indexed constantly because someone links to them or a sitemap leaks. The robust approach is HTTP authentication or IP allow-listing, with a global X-Robots-Tag noindex header as a second layer. Don't rely on Disallow slash alone. And make sure deploy scripts can't copy staging robots rules to production, because a production Disallow slash is one of the most damaging one-line mistakes in SEO.

3:58 AI groups + hands-on tests

Robots.txt now carries AI crawler decisions too. Training bots like GPTBot and ClaudeBot, control tokens like Google-Extended and Applebot-Extended, and AI search bots like OAI-SearchBot and PerplexityBot. Technically, treat each AI group like any other: it ignores the star group, so repeat shared disallows, and include it in your tests. The policy side is covered in the AI Search Optimization course. Now the hands-on. The lesson text includes a pytest file that fetches robots.txt, fails the build if a sitewide Disallow appears, and asserts which agents can and can't fetch key paths.

4:38 Example 1: blocked CSS and JS

Worked example one, simple. A Birmingham law firm's developer blocks slash wp-content to hide uploads. That also blocks the theme's CSS and JavaScript, and URL Inspection shows an unstyled, broken render. The fix is removing that rule and, if some files must stay private, putting them behind authentication instead. Five minutes, and the page renders properly again.

5:03 Three layers of robots testing

Let's make testing concrete with Search Console. The robots.txt report shows which robots.txt files Google found for the top hosts in your property, when each was last fetched, and any fetch errors or parsing warnings. After any robots.txt change, check that Google has fetched the new version before you expect crawl behaviour to change. For individual URLs, URL Inspection tells you whether crawling is allowed. Combine those with your own automated tests and you have three layers: your CI test before deploy, Google's fetched version after deploy, and log evidence of how Googlebot actually behaves.

5:44 Watch me do it: robots regression tests

Watch me do it. I'll add robots regression tests to a real project. Step one: I list what must always be allowed for Googlebot: the home page, one product, one category, and the main JavaScript bundle. And what must be blocked: the cart and internal search. Step two: I copy the pytest file from the lesson into the tests folder and fill in those paths. Step three: I run pytest against production. Seven pass. One fails: Googlebot blocked from slash static slash app dot js. I open robots.txt and find a Disallow for slash static, added last year to hide design files. That rule is also blocking the JavaScript needed for rendering. Step four: I check URL Inspection for a product page, test live, and open the page resources. The app bundle shows as blocked, and the screenshot is half-rendered. Confirmed. Step five: I change the rule to disallow only the design subfolder, run the tests again, and everything passes. Step six: after deploy, I open Search Console's robots.txt report and confirm Google fetched the new version. And now the test runs in CI on every deploy, so the next accidental block fails the build instead of reaching production.

7:11 Example 2: noindex that never worked (illustrative)

Worked example two, with illustrative details. A Karachi e-learning site added noindex to eight thousand thin tag pages and, in the same release, disallowed the tag folder in robots.txt to be thorough. Three months later, the pages were still indexed. Googlebot couldn't fetch them, so it never saw the noindex. The fix followed the correct sequence: remove the Disallow, keep noindex, confirm the URLs dropping out in Page indexing, and only then consider re-adding the Disallow.

7:44 Mistakes, recap, try this now

Common mistakes: blocking CSS and JavaScript to save budget, which breaks rendering. Assuming robots.txt hides confidential URLs, when it advertises them. Leaving noindex on templates after launch. And forgetting that a specific Googlebot group overrides the star group. Recap: use each tool for its job, respect grouping and precedence, never combine a block with a noindex you need Google to see, and protect staging properly. Try this now. Add the pytest file to your project, fill in five must-allow and three must-block paths, and run it against production today.