Technical SEO Audit WorkshopRunning the crawl and gathering evidence · Lesson 4 of 14
Search Console, logs and performance evidence
Video lecture
Search Console, logs and performance evidence
The narrated lecture is in production
Every chapter is scripted and ready. Browse the chapters and read the full transcript now — the video will appear here when it’s published.
Chapters
Transcript of the narration, chapter by chapter.
0:00 Search Console, logs and performance evidence
A crawler simulates. It shows you what could happen. Search Console shows what Google did with your URLs. Logs show what Googlebot actually requested. And field performance data shows what real users experienced. A strong audit joins all four. In this lecture, you'll gather that evidence for Kiran Home, segment it by market and template, add the newer twenty twenty-six sources, and build an evidence log that makes every claim in your report traceable.
0:32 Search Console questions
Start with Search Console, question by question. When did the decline start, and where? Use Performance, compare periods, and filter by page segment and country. For Kiran Home, the illustrative drop starts the week of relaunch, concentrated in UAE and Pakistan traffic to product pages. Which URLs aren't indexed, and why? Page indexing filtered by sitemap shows many product URLs as alternate page with proper canonical tag. What canonical did Google pick? URL Inspection on samples shows the UK product for UAE pages. And are rich results intact? Product items fell, with offers missing on some templates.
1:14 Faster, fairer comparisons
Use Search Console's newer features to make this faster. The branded queries filter, where it's available for the property, splits brand from non-brand, or you can use your own regex. Custom annotations let you mark the relaunch date directly on the chart. And always compare twenty-eight days before and after the relaunch, and the same windows a year earlier, to control for seasonality, for example if Ramadan fell in the comparison window one year and not the other.
1:48 Logs: what Googlebot did
Now logs. Cloudflare logs for four weeks, filtered to verified Googlebot and classified by segment, tell a clear story for Kiran Home. The large majority of requests go to facet parameters. A noticeable share goes to internal search URLs, linked from a popular searches widget. Only a small share goes to products, and many products weren't requested at all in four weeks. And Googlebot still requests pre-relaunch URLs heavily. Two conclusions: crawl attention is being spent on URLs with no value, and legacy URLs still matter to Google, so redirect gaps matter.
2:28 The unarguable join
Here's a join that makes the point unarguable. Take products with zero verified Googlebot hits in four weeks, and cross-reference them with revenue from the pre-relaunch period. If many former best-sellers haven't been crawled at all, that's a headline finding for the stakeholder report. Founders don't care about crawl budget. They care that their best-selling brass lanterns haven't been visited by Google in a month.
2:56 Performance + rendering evidence
Performance evidence. Run PageSpeed Insights for each template and read the field data first. For Kiran Home, illustratively, category pages have poor LCP because the hero carousel image is lazy-loaded, and product pages have poor INP and CLS because a reviews widget shifts layout and blocks the main thread. Where CrUX lacks data for a URL, use origin data and Search Console's Core Web Vitals groups, then use a lab trace to find the cause. Rendering evidence too: URL Inspection shows customers also bought links only load after interaction, so they add no discovery value.
3:37 New evidence sources
And the newer sources for twenty twenty-six. Search Console's generative AI report shows whether Kiran Home pages appear in AI Overviews or AI Mode, by page and country, so you can see if AI exposure fell too. Bing Webmaster Tools shows whether Bing indexing and Copilot citations dropped, and a second engine's view helps separate site problems from Google-specific changes. And CDN firewall events show whether real crawlers, including Bingbot and AI search bots, are being challenged since the relaunch. That's evidence for hypothesis five.
4:14 Hands-on: before/after by segment
Hands-on. The lesson text has a short script using the Search Console API that pulls clicks by page and country for twenty-eight days before and after the relaunch, assigns a segment from the URL, and pivots before versus after with a percentage change. It reuses the authenticated client from the Technical SEO Mastery course. For bigger sites, paginate with start row. This single table usually tells you where the drop is concentrated within minutes.
4:46 Example 2: seasonality, not SEO (illustrative)
Worked example two, a different client, illustrative, where the evidence exonerates the relaunch. A UK garden furniture retailer's segmentation shows the decline spread evenly across all templates and markets, starting two weeks before the relaunch, and matching the end of the spring buying season in last year's data too. Logs show healthy crawling, and indexing is stable. The evidence points to seasonality, not a technical regression. The audit says so, and saves the client from an unnecessary emergency rebuild. Good auditing sometimes means saying, it isn't technical.
5:24 Watch me do it: where did it drop?
Watch me do it. I'll build the evidence for where Kiran Home's drop happened. Step one: in Search Console Performance, I add a custom annotation on the relaunch date so every chart shows it. Step two: I apply the branded queries filter and choose non-branded, then compare twenty-eight days after the relaunch with twenty-eight days before. Step three: I add a page filter with the regex for UAE product URLs. Non-brand clicks are down sharply. I switch the filter to UK product URLs. Roughly stable. That contrast is the story. Step four: I repeat the comparison with the same dates last year to check seasonality. Last year's pattern was flat across those weeks, so seasonality doesn't explain it. Step five: I run the before-and-after API script by segment and country, and paste the pivot into the evidence log as E two. Step six: Page indexing, filtered by the UAE product sitemap: most URLs are alternate page with proper canonical tag. That's E eight. Step seven: I open the generative AI report and see UAE product pages also lost AI impressions, and in Bing Webmaster Tools the indexed product count dropped. Both go into the log. Each row has a source, dates, filter and file.
6:53 Check the calendar
Here's a quick mistake check before you trust your comparison. Look at the date ranges. If your before period includes the Ramadan and Eid weeks for UAE shoppers and your after period doesn't, or the other way round, the comparison is contaminated. The same goes for Black Friday in the UK. So for Kiran Home, I write both date ranges on the evidence log row, and next to them the holidays that fall inside each. If they don't match, I shift the windows or add a year-on-year comparison. It takes two minutes, and it stops the most common false conclusion in post-launch audits: blaming the relaunch for what was really a calendar effect.
7:42 The evidence log
Build the evidence log. Every claim in your final report must trace back to evidence: an ID, the source, the date range, the filter, the finding, and the file or screenshot. For Kiran Home: E seven, Cloudflare logs for March, verified Googlebot, facets dominate requests. E eight, Search Console Page indexing for the UAE product sitemap, most URLs alternate page with proper canonical tag. This protects you when a stakeholder challenges a recommendation, and it makes verification later straightforward.
8:16 Mistakes, recap, try this now
Common mistakes. Reading Performance data without segmentation, so a market-specific drop gets diluted. Comparing only with the previous twenty-eight days and ignoring seasonality. Using unverified Googlebot log lines. And treating Lighthouse lab scores as the verdict on Core Web Vitals. Recap: join Search Console, logs, field data and rendering evidence; segment everything; use the new AI and Bing sources; log every piece of evidence. Try this now: run a before-and-after comparison by segment and country for your own site's last major change.
Crawl data shows what could happen; other sources show what did
A crawler simulates. Search Console shows what Google did with your URLs, logs show what Googlebot actually requested, and field performance data shows what real users experienced. A strong audit joins all four.
Search Console: questions and where to answer them
| Question | Report | Kiran Home finding (illustrative) |
|---|---|---|
| When did the decline start and where? | Performance, compare periods, filter by page segment via regex | Drop starts the week of relaunch; concentrated in UAE and Pakistan traffic to product pages |
| Which URLs are not indexed and why? | Page indexing, filtered by sitemap | Many product URLs under "Alternate page with proper canonical tag" and "Duplicate, Google chose different canonical" |
| What canonical did Google pick? | URL Inspection on samples | Google-selected canonical = UK product for UAE pages |
| Are rich results intact? | Enhancements (Products, Breadcrumbs) | Product items fell after relaunch; missing offers in JSON-LD on some templates |
| Is the server healthy for Googlebot? | Crawl stats | Response time increased after relaunch; some 5xx spikes |
Useful filters in the Performance report — the built-in branded queries filter (where available for the property) or your own regex, plus custom annotations to mark the relaunch date on charts:
Page contains regex: /en-ae/products/|/ar-ae/products/
Query does not match regex: kiran ?home|kiranhomeCompare 28 days before relaunch with 28 days after, and also year-on-year to control for seasonality (for example, if the relaunch coincided with a Ramadan period one year and not the other).
Logs: what Googlebot actually did
Cloudflare logs for four weeks, filtered to verified Googlebot, then classified by segment:
| Segment | Share of Googlebot requests | Observation |
|---|---|---|
| Facet parameters | Large majority | Crawl trapped in filter combinations |
| Internal search | Noticeable share | /search?q= URLs linked from "popular searches" widget |
| Products | Small share | Many products not requested at all in four weeks |
| Old URLs (redirects/404s) | Moderate | Googlebot still requests pre-relaunch URLs heavily |
Shares described qualitatively; your real audit would quantify them.
Two conclusions for the report: Googlebot's attention is being spent on URLs with no value, and legacy URLs are still important to Google, so redirect gaps matter.
A simple join makes the point unarguable: products with zero Googlebot hits in four weeks cross-referenced with revenue from the pre-relaunch period. If many former best-sellers have not been crawled, that is a headline finding for the stakeholder report.
Performance evidence
Run PageSpeed Insights for each template and read the field data (CrUX) first:
| Template (mobile) | LCP | INP | CLS | Status |
|---|---|---|---|---|
| Home | Needs improvement | Good | Good | — |
| Category | Poor | Needs improvement | Good | Hero carousel image lazy-loaded |
| Product | Needs improvement | Poor | Poor | Reviews widget shifts layout and blocks main thread |
| Blog | Good | Good | Good | — |
Illustrative statuses. Where CrUX has insufficient data for a URL, use the origin-level data and the Search Console Core Web Vitals report groups. Then use a lab trace to identify the cause.
Rendering evidence
For three products, use URL Inspection's live test and view the rendered HTML. Search for a unique sentence from the product description and the price. On Kiran Home, the description renders, but "Customers also bought" links are loaded after interaction and do not appear — so they provide no discovery value.
Building the evidence log
Every claim in the final report must trace back to evidence. Keep an evidence log:
E-07 | Source: Cloudflare logs 1–28 Mar | Filter: verified Googlebot | Finding: facet URLs dominate requests | File: logs_segment_pivot.xlsx
E-08 | Source: GSC Page indexing, sitemap products-en-ae.xml | Finding: most URLs 'Alternate page with proper canonical tag' | Screenshot: e08.pngThis protects you when a stakeholder challenges a recommendation, and it makes verification later straightforward.
Newer evidence sources (2026)
| Source | What it adds for this audit |
|---|---|
| Search Console generative AI report | Whether Kiran Home pages appear in AI Overviews/AI Mode, by page and country (impressions only) — useful to see if AI exposure also fell after relaunch |
| Bing Webmaster Tools (Site Explorer, AI Performance) | Whether Bing indexing and Copilot citations dropped too — a second engine's view helps separate site problems from Google-specific changes |
| CDN firewall/bot events | Whether real crawlers (Googlebot, Bingbot, AI search bots) are being challenged since the relaunch (hypothesis H5) |
Hands-on: before/after by segment with the Search Console API
# Reuses the authenticated `gsc` client from the Technical SEO Mastery course (Lesson 7.3)
import pandas as pd
def pull(start, end):
body = {"startDate": start, "endDate": end, "dimensions": ["page", "country"],
"rowLimit": 25000, "dataState": "final"}
rows = gsc.searchanalytics().query(siteUrl="sc-domain:kiranhome.example", body=body).execute().get("rows", [])
return pd.DataFrame([{"page": r["keys"][0], "country": r["keys"][1], "clicks": r["clicks"]} for r in rows])
before, after = pull("2026-05-04", "2026-05-31"), pull("2026-06-08", "2026-07-05")
for d, label in ((before, "before"), (after, "after")):
d["segment"] = d.page.str.extract(r"/(products|collections|journal)/")[0].fillna("other")
d["period"] = label
both = pd.concat([before, after])
print(both.pivot_table(index=["segment", "country"], columns="period", values="clicks", aggfunc="sum").fillna(0)
.assign(change=lambda t: (t.after - t.before) / t.before.replace(0, pd.NA)).sort_values("before", ascending=False).head(15))(Dates are illustrative; use 28-day windows either side of the relaunch plus the same windows a year earlier. Paginate with startRow for larger sites.)
Worked example 2: when the evidence exonerates the relaunch
For a different client, a UK garden-furniture retailer (illustrative), the before/after segmentation shows the decline spread evenly across all templates and markets, starting two weeks before the relaunch and matching the end of the spring buying season in last year's data too. Logs show healthy crawling; indexing is stable. The evidence points to seasonality, not a technical regression — and the audit says so, saving the client from an unnecessary emergency rebuild.
Common mistakes
- Reading Performance data without regex segmentation, so a market-specific drop is diluted.
- Comparing with the previous 28 days only, ignoring seasonality.
- Using unverified "Googlebot" log lines.
- Treating Lighthouse lab scores as the verdict on Core Web Vitals.
Key takeaways
- Join crawl data with Search Console, logs and field performance data for evidence-backed findings.
- Segment Search Console Performance with regex and compare both period-over-period and year-over-year.
- Log analysis shows where Googlebot's attention actually goes; join it with revenue to show business impact.
- Keep an evidence log so every recommendation traces to a source.
Check your understanding
Quick questions to lock in the lesson. They don’t count towards your certificate.
Put it into practice
Create an evidence log template and fill in five entries from a real site's Search Console, logs or PageSpeed Insights.
Enrol for free to save your progress
Reading is always free. Enrol to keep your place, take the final assessment and earn a verifiable certificate.