Configuring the crawl like Googlebot

0:00 Configuring the crawl

Before you look at a single finding, look at your crawler settings. They decide what you'll find. The same tool gives completely different results depending on configuration. A text-only crawl of a JavaScript storefront may report no content. A crawl blocked by bot protection reports a site full of four-oh-threes. In this lecture you'll configure a crawl for Kiran Home that approximates how Googlebot experiences the site, run it in a repeatable way, and record every setting so you can re-crawl identically after fixes.

0:37 The configuration sheet

Here's the configuration sheet for Kiran Home. Start at the canonical host. Seed the crawl with all XML sitemaps and the list of old URLs from the launch redirect map, so you find orphans and test redirects. Use a Googlebot smartphone user agent, because of mobile-first indexing. Turn JavaScript rendering on, because it's a headless React storefront. Respect robots.txt, and optionally run a second crawl ignoring it to see what's hidden. No depth limit, but cap the URL count and review patterns. Modest speed, agreed with developers. Crawl all parameters initially.

1:17 Extraction + safeguards

Extraction and integrations. Extract canonical, meta robots, X-Robots-Tag, hreflang, JSON-LD, H1, word count and the price element. Connect the Search Console API and GA4, so traffic and index data join the crawl by URL. Then two safeguards. First, get your crawler allow-listed with the CDN or bot protection for the audit window, otherwise you'll crawl a wall of challenge pages. Second, crawl staging separately with authentication, and check that it's noindexed, not the other way round.

1:50 Custom extraction

Custom extraction checks what matters to this business. For Kiran Home, we want to know whether product prices and add to basket buttons exist in the rendered HTML, and whether the JSON-LD price matches the visible price. So we extract the visible price with a CSS selector, all JSON-LD blocks with XPath, the add to basket button, and breadcrumb links. Default reports catch generic issues. Custom extraction answers the client's questions.

2:21 Segment first

Segment before you crawl. Configure segments by URL pattern: product, category, blog, facet and search, using regular expressions like the ones in the lesson text. Here's why that matters. Without segments, you report three thousand two hundred duplicate titles. With segments, you report that all three thousand one hundred duplicate titles are facet URLs, which shouldn't be crawlable anyway. Same data, completely different conclusion.

2:49 Four crawls

Two crawls are better than one. The rendered crawl, JavaScript on, respecting robots.txt, with a Googlebot smartphone agent, is your primary dataset. The raw crawl, JavaScript off, shows what depends on rendering: links, canonicals, content and structured data. Add a list crawl of old URLs from the pre-relaunch site to test redirects, and a sitemap-only crawl to test sitemap hygiene. Four views of the same site, each answering a different question.

3:20 Hands-on: headless, versioned crawls

Hands-on. Crawlers can run headless from the command line, which makes crawls repeatable and schedulable. The lesson text shows a Screaming Frog example with a saved configuration file, an output folder by date, selected exports and timestamped output. Check the help for your version's exact flags, and other crawlers offer equivalent scheduled or cloud crawls. Store the config file in version control with the audit. There's also a quick curl loop to check key URLs aren't being blocked before you start.

3:55 Watch the first minutes

Watch the first minutes of the crawl and stop to fix the config if you see these signs. All URLs returning four-oh-three or a challenge page means bot protection. The crawl exploding into parameter URLs within minutes means a facet trap, so cap it, note it as a finding, and exclude it from the main crawl. Rendered pages with no content means rendering timeouts, so increase the timeout or check blocked resources. And redirects to a login or geo page means IP-based geo-redirects, which is itself a finding for an international site.

4:35 Example 1: Kiran Home kick-off (illustrative)

Worked example one, simple, and illustrative. On the Kiran Home kick-off crawl, the first two hundred URLs all return five-oh-three. The CDN's bot fight mode is challenging the crawler. The developers add a temporary allow rule for the audit crawler's IP. But note the second-order question for the audit: does the same rule set challenge Bingbot or AI search bots? That becomes evidence for hypothesis five.

5:04 Example 2: the too-short timeout (illustrative)

Worked example two, also illustrative. A Doha fashion retailer's crawl finds only seven hundred URLs on a site with six thousand products. The rendered crawl used a two-second timeout, and the category grid loads after three. Increasing the timeout and checking resources reveals the full catalogue, and a genuine finding: the category API is slow, which also hurts real users and Google's rendering. Configuration problems often hide real problems.

5:34 Watch me do it: launching the crawl

Watch me do it. I'll set up and launch the Kiran Home crawl. Step one: in the crawler, I set the user agent to Googlebot Smartphone. Step two: I switch rendering to JavaScript, and raise the render timeout because the category grid loads slowly. Step three: under robots settings, I leave respect robots.txt on. Step four: I add custom extraction rules: the visible price by CSS selector, JSON-LD blocks by XPath, the add to basket button, and breadcrumb links. Step five: I add segments with the regex patterns for product, category, blog, facet and search. Step six: I connect the Search Console and GA4 APIs. Step seven: I load the seeds: the home page, the sitemap index, and a list of old URLs from the launch redirect map. Step eight: before the full run, I use the curl loop to test three URLs with the same user agent. All two hundreds, so the developers' temporary allow rule is working. Step nine: I start the crawl and watch the first five minutes. After two thousand URLs, the facet segment is growing fast, so I cap facet URLs, note the trap as a finding, and continue. Step ten: I save the configuration as version one point two and record the settings in the workbook.

7:06 Record it; avoid mistakes

Recording the configuration. Save the crawler's configuration file, and note the crawler version, date, settings, seeds and exclusions in your findings workbook. When you re-crawl for verification later in this workshop, you'll use the identical configuration, so any difference comes from fixes and not from settings. Common mistakes: crawling with a desktop user agent, leaving rendering off on a JavaScript site, crawling from a blocked IP and reporting site returns four-oh-three as a finding, and not seeding with old URLs.

7:41 Recap and try this now

Recap. Configure the crawl to approximate Googlebot: smartphone agent, rendering on, robots respected, seeded with sitemaps and old URLs, segmented, and allow-listed. Run rendered and raw crawls, plus list crawls. Record everything and re-use it. Try this now. Write a configuration sheet for a site you know, with a reason for every setting, and run a quick curl check on three key URLs with a Googlebot smartphone user agent to see if anything blocks you.