Data sourcing, licensing and page uniqueness

0:00 Data sourcing and uniqueness

In programmatic SEO, the data is the product. It's the only thing that makes page one thousand two hundred and thirty-seven genuinely different from page one thousand two hundred and thirty-eight. In this lecture you'll learn the types of data sources and their risks, how to build a value-add layer on top of raw data, how to measure whether your pages are near-duplicates, how to keep data fresh and accurate, and the legal checks you can't skip.

0:33 Why it matters

Why does this matter? Because data is the only thing that makes one programmatic page genuinely different from its neighbor. Here's an analogy. Picture a street of houses built from the same blueprint. From the outside they look identical. What makes each one worth visiting is what's inside: the people, the furniture, the stories. Your template is the blueprint. Your data is what's inside. If every house is empty, nobody will visit, and search engines won't either.

1:06 Six source types

There are six broad source types. Proprietary data, like your own listings, bookings, prices and reviews, which is unique and defensible. Partner and licensed feeds, which are rich but come with license terms. Open public data, like national statistics offices in the UK, Saudi Arabia, Pakistan and the US, which is credible but available to everyone. User-generated content, which is fresh but needs moderation. Derived data you compute. And scraped data, which is high risk and adds nothing unless genuinely transformed.

1:41 The value-add layer

Here's the rule of thumb. Open data alone rarely makes a competitive page, because everyone can publish it. The winning combination is open data, plus proprietary data, plus computed insight. So for each page, write down what it gives that a user couldn't get from the raw source. Combinations, like rent plus commute plus schools. Computations, like medians, trends and calculators. Curation. Freshness. Expert context. And moderated reviews.

2:11 Measure similarity

Near-duplicate pages are the classic programmatic failure, so measure them. The lesson includes a Python snippet that takes the main content text of each generated page, builds a TF-IDF representation, computes cosine similarity between pages, and prints each page's closest neighbor with a score. Calibrate your threshold on pages you already consider good. Pages above it need more distinct data, or should be merged, kept out of the index, or not published.

2:42 Simple example: Dubai school directory (illustrative)

Here's a simple worked example. A Dubai school directory builds pages for British curriculum schools in each area. Open data gives them school names, curricula and official inspection ratings. That alone is what every competitor has. So they add their own layer: fee ranges collected from each school's published fee schedule, bus route coverage from school websites, and computed insights like average fees in this area compared with the city. They also add a parent Q and A, moderated. Now each area page answers the real question parents have, which school fits my budget and my commute.

3:24 Freshness and accuracy

Freshness and accuracy need design, not hope. Define update frequencies per field: exchange rates hourly, listings daily, statistics yearly. Show last updated dates truthfully, and only change them when content materially changes, because fake freshness is exactly what Google's guidance warns about. Add validation: ranges, missing values, outliers, and reconciliation against sources. And give users and businesses a way to report errors.

3:51 Legal checks

Now the legal checks. Read data licenses for commercial and SEO use, attribution and redistribution. Respect terms of service and robots rules; scraping sites that prohibit it is a legal and reputational risk. Never publish personal data without a lawful basis; publish aggregates like median tutor rates, not individual earnings. And follow fake review rules, like the FTC's rule in the US and the UK's consumer protection law on fake reviews.

4:22 Example: UK used-car marketplace (illustrative)

Here's an illustrative example. A UK used-car marketplace builds make, model and city pages. The data combines its own live listings, licensed open vehicle data in aggregate, computed price ranges and depreciation by year, and in-house mechanics' notes on what to check for each model. Pages need at least five live listings. The similarity check flagged small-city pages that looked identical, so those were merged into regional pages. Fewer pages, each much stronger.

4:54 Pitfalls and success

Common pitfalls. Relying on a single public dataset everyone uses. Displaying partner data in ways the license forbids. Changing dates without changing content. And publishing pages whose only difference is a city name in one sentence. Measure success with a documented data inventory, similarity scores within threshold, error reports fixed quickly, and pages that others cite or link to as references.

5:21 Mistakes + try this now

Common data mistakes. Using the same public dataset everyone else uses, with nothing added. Displaying partner data in ways the license doesn't allow. Updating the last updated date without changing anything. Publishing personal data instead of aggregates. And scraping sites that prohibit it. Try this now: pick one page type and write down, in one sentence, the thing each page will show that a user can't get anywhere else. If you can't finish that sentence, you don't have a data advantage yet.

5:57 Quick self-check

Quick self-check. Your competitor copies your city pages' structure exactly: same headings, same order, same URL pattern. Should you panic? Pause. Not if your advantage is in the data. They can copy the template in a day. They can't copy your verified listings, your booking-based price ranges or your moderated reviews. That's the point of this lesson. Invest in the layer competitors can't copy, and treat the template as the easy part.

6:28 Watch me do it: similarity check (illustrative)

Watch me do it. Let's run the similarity check on an illustrative pilot of sixty pages for a UK wedding photographer directory. Step one, I render every page and extract the main content text, excluding the header, footer and navigation. Step two, I run the TF-IDF script. Step three, results: most pages' nearest neighbor scores sit between point four and point six, which matches pages we already consider good. But twelve pages score above point nine. Step four, I open a pair: wedding photographers in Hexham and wedding photographers in Corbridge. They list the same four photographers, because both small towns are served by the same people, and the guidance text is identical. Step five, the decision: these are one need, not two. We create a regional page for Northumberland with all photographers covering the area, travel fees and venue lists, and 301 the small-town pages to it. Step six, I rerun the check: no pairs above threshold, and the new regional page is richer than either original. Similarity scores didn't just flag a risk; they pointed to a better page design.

7:47 Recap and next step

Recap. Data is the product. Combine open, proprietary and computed data. Measure similarity before you publish. Design freshness and accuracy honestly. And do the legal checks. Your next step: create a data inventory for one page type, listing sources, licenses, fields, update frequency, owner, and the value-add layer every page will carry.