Crawling, indexing, sitemaps and structured data

0:00 Crawling, indexing and structured data

You can write the best page on the internet, and if search engines can't crawl or index it, it doesn't exist. Not in the blue links, and not in AI answers either. In this lecture you'll learn how search engines process your site, the controls that decide what gets crawled and indexed, how structured data works, and what's changed with AI crawlers, including a control that trips up almost everyone.

0:30 Why it matters

Why does this matter for content strategists, not just developers? Because a single technical mistake can erase months of content work overnight. A staging site's disallow rule shipped to production. Thousands of filter URLs wasting crawl capacity so new articles aren't found. A canonical pointing to the wrong page. Knowing the basics lets you spot these problems early and brief developers clearly.

0:57 The pipeline

Here's the pipeline. Crawling: bots like Googlebot and Bingbot discover URLs by following links and reading sitemaps. Rendering: they load the page, including JavaScript, to see it as a user would. Indexing: they analyse and store eligible pages. Ranking and serving: when someone searches, they select and order indexed pages. Think of it like a library. Finding the book, reading it, cataloguing it, and recommending it. A failure at any step means no recommendation.

1:29 The controls

Now the controls. robots.txt sits at the root and tells crawlers which paths they may crawl. Here's the key idea: robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other pages link to it. To keep a page out of the index, allow crawling and use a noindex tag. Never block CSS or JavaScript needed for rendering. XML sitemaps list the canonical, indexable URLs you want found. And canonical tags tell search engines which duplicate version you prefer, supported by redirects and consistent internal links.

2:08 Structured data

Structured data, usually JSON-LD using the schema dot org vocabulary, describes page content in a machine readable way. It can make pages eligible for rich results like product prices, breadcrumbs, events, recipes, videos and organisation details. The rules: mark up only what's visible on the page, keep it accurate, and test it with the Rich Results Test and the Schema Markup Validator. Rich results are never guaranteed, and Google has reduced or retired several types over time. Misleading markup, like fake review stars, can lead to manual actions.

2:46 AI crawlers: the confusing bit

Now the AI part, and the control that confuses everyone. Google's AI Overviews and AI Mode draw on the normal search index, crawled by Googlebot. There's a robots token called Google Extended. Blocking it controls whether your content is used to train future Gemini models and to ground some Gemini products. But blocking it does not remove your pages from AI Overviews or AI Mode. Google's controls for those are snippet controls, like nosnippet, which also affect classic snippets, and, since twenty twenty-six, a Search generative AI control in Search Console.

3:26 A deliberate decision

That Search Console control lets a site opt out of AI features in Search and Discover without affecting normal results. It's a genuine business decision. Opting out may protect content from being summarised, but it also removes you as a potential cited source. Weigh visibility against your content use concerns, check the current Search Console help, and decide deliberately rather than by accident. Other AI companies publish their own crawler names, like GPTBot, OAI SearchBot, ClaudeBot and PerplexityBot, each with its own documentation.

4:02 Example 1 (simple)

Worked example one, simple. A small UK charity relaunches its website and traffic falls to almost nothing within two weeks. Someone checks robots.txt: it says disallow everything, copied from the staging site. They also find a site wide noindex tag in the theme settings. Both are fixed, the sitemap is resubmitted, and key pages are inspected in Search Console. Over the following weeks, pages return to the index. The lesson: add a robots and noindex check to every launch checklist.

4:37 Example 2 (illustrative)

Worked example two, realistic and illustrative. A Karachi e-commerce site sees many product pages marked discovered, currently not indexed. Investigation shows thousands of filter URLs being crawled, wasting crawl capacity, and missing canonicals. Fixes: canonical tags on filter pages, blocking crawl of infinite filter combinations where appropriate, cleaner internal links, and an accurate product sitemap. Over the following weeks more product pages are indexed. And the new pages start appearing as sources in some AI answers too, because they're finally in the index.

5:13 Five-minute check

Here's a five minute indexability check you can run today. Open your robots.txt: any disallow everything, or rules blocking CSS and JavaScript? Is the sitemap listed? Inspect three important URLs in Search Console: indexed, Google selected canonical matching yours, last crawl date, and mobile rendering. Check the page indexing report for reasons affecting pages you want indexed. And compare submitted and indexed counts in the sitemaps report.

5:42 Watch me do it: the 5-minute check

Watch me do it. I'll run the five minute indexability check on a charity website that just relaunched. First, I type the domain followed by slash robots dot txt. There it is: user agent star, disallow slash. The whole site is blocked, left over from staging. That's the emergency, and I message the developer immediately. Next, while that gets fixed, I open Search Console and inspect the donate page with URL Inspection. It says the page isn't indexed, and the live test shows a noindex tag. So there's a second problem: a site wide noindex setting in the theme. I add that to the developer message. Then I open the Pages report and check the reasons affecting pages the charity wants indexed. Most say blocked by robots dot txt, which matches. Then the Sitemaps report: the new sitemap was never submitted, so I submit it. Once the developer confirms both fixes are live, I run URL Inspection again on three key pages and request indexing. Finally, I test the charity's Organization structured data in the Rich Results Test, and it's valid. I add robots and noindex checks to their launch checklist, so this never happens again.

7:07 Common mistakes

Common mistakes. Using robots.txt to try to remove pages from the index. Shipping staging noindex or disallow rules to production. Sitemaps full of redirected or noindexed URLs. Structured data that doesn't match visible content. And assuming blocking Google Extended removes you from AI Overviews. It doesn't.

7:27 Recap

Recap. Search engines crawl, render, index and rank, and AI answers in Google draw on the same index. robots.txt controls crawling, noindex controls indexing, sitemaps and canonicals keep things clean, and structured data must match what's visible. Google Extended is about Gemini training and grounding, not AI Overviews. Try this now. Run the five minute indexability check on your site. For the deep dive, take Technical SEO Mastery next.