Asking better analytical questions

0:00 Asking analytical questions

What's interesting in this data? It's the most common question people ask AI about data, and it's almost always the wrong one. You'll get a long list of observations and no decision. Now try this instead: which of our three paid social campaigns should get the extra budget next month, based on cost per qualified lead? Same data, completely different answer. In this lecture you'll learn to turn vague questions into decision-focused ones, to define metrics before measuring, and to use a glossary that keeps AI honest.

0:38 Why it matters

Why does this matter? Because AI can answer almost any question you ask of data, which makes the quality of the question the limiting factor. Vague questions produce long lists. Decision-focused questions produce useful answers, better methods, and relevant limitations, because the model knows what's at stake. And many arguments about analysis are really arguments about definitions. Fix the question and the definitions first, and half the work is done.

1:08 The framing template

Here's the template for framing. Decision: what will we decide? Options: A, B, C. The metric that decides it, like cost per qualified lead. The threshold: what difference would change our mind? Time frame and population: who and when. And known confounders: seasonality, a price change, a shift in channel mix. Think of it like a flight plan. A pilot doesn't just say fly somewhere nice. They file the destination, route, fuel and alternates before take-off. Share the frame with the AI before asking for analysis.

1:45 Four question types

Know which type of question you're asking. Descriptive: what happened, like sales by month. Diagnostic: why did it happen, like drivers of a drop. Predictive: what's likely to happen, which we'll cover with forecasting. And prescriptive: what should we do. Descriptive questions are safest to answer from observational data. Diagnostic and prescriptive questions need extra care, because correlation in historical data isn't proof of cause. AI assistants often slide from these moved together to this caused that. Watch for it.

2:20 Define before measuring

Define metrics before you measure. What counts as an active user: logged in, or performed a key action? Is revenue gross or net of refunds and discounts, before or after VAT? Is conversion per session, per visitor or per lead? Write definitions down, give them to the AI, and ask it to restate them at the top of its analysis. Misunderstandings then surface in the first paragraph instead of in the board meeting.

2:52 Leading and lagging

Leading and lagging indicators help too. Lagging indicators, like revenue and churn, confirm results after the fact. Leading indicators, like trial activations, demo bookings and onboarding completion, give earlier signals. When exploring, ask the AI to test whether a candidate leading indicator actually relates to the lagging outcome in your history, while remembering that relationships can change. A leading indicator that doesn't predict anything is just another vanity metric.

3:22 Thresholds first

And set the threshold before you look. It's tempting to analyze first and decide afterward what difference counts as meaningful. But then any difference you find will feel meaningful, because you've already seen it. So ask: what's the smallest difference that would change our decision? Maybe it's a twenty percent lower cost per qualified lead, or two percentage points of conversion. Write it down. It protects you from over-reacting to noise, and it helps the AI choose the right method and sample size.

3:58 Example 1: 'How are campaigns doing?'

First example, a simple one. A manager asks how are our campaigns doing? The analyst reframes it. Decision: which of three paid social campaigns gets the extra budget next month. Metric: cost per qualified lead, where qualified means accepted by sales within seven days. Threshold: move budget only if one campaign is at least twenty percent cheaper per qualified lead, illustratively. Population: leads from the last sixty days. Confounder: one campaign ran during a holiday week. Now the analysis almost writes itself.

4:34 Example 2: Lahore ed-tech

Second example, a business case. At a Lahore education technology company, two teams disagreed about whether engagement was rising. Product counted active users as anyone who logged in. Marketing counted anyone who watched a lesson for five minutes. The analyst wrote both into the glossary with separate names, logged in users and learning users, and reran the analysis. Logins were rising. Learning was flat. The debate ended in ten minutes, and the next question became how to turn logins into learning.

5:09 Watch me do it, part 1

Watch me set up a glossary file. It's a short YAML file with an owner and a review date. Active customer: placed at least one completed order in the last ninety days. Revenue: completed orders, net of refunds and discounts, excluding VAT, converted to pounds at the monthly average rate. Conversion rate: orders divided by sessions, not per visitor. Qualified lead: accepted by sales within seven days. I attach it to every analysis prompt and add one instruction: restate the definitions you're using, and ask me before defining any metric that isn't in the glossary.

5:50 Watch me do it, part 2

Then I let AI help with framing itself. I paste the vague request, how are our campaigns doing, and ask the assistant to rewrite it as a decision-focused frame with decision, options, deciding metric from the glossary, threshold, population, time frame and confounders, plus three questions I should ask my manager to confirm the frame. It suggests asking which budget decision is coming up, whether lead quality or volume matters more, and when the decision is due. I send those three questions before I touch the data.

6:28 Questions for the AI

And ask the AI about its own analysis. What assumptions did you make? What would change your conclusion? Which numbers are computed and which are estimates? What's the simplest alternative explanation? Those four questions take thirty seconds and routinely surface assumptions that would otherwise ride silently into your report.

6:49 Evidence fit

Finally, match the question type to the evidence you have. If your question is prescriptive, like should we raise prices, historical data alone is weak evidence, because you've probably never tested a price change. The frame should say so, and propose a better source of evidence: a controlled test in one region, a survey, or a staged rollout. Good analysts don't just answer the question they're given. They tell you what evidence would actually answer it.

7:22 Common mistakes

Common mistakes. Starting with what's interesting. No threshold, so any difference seems meaningful. Undefined metrics that mean different things to different teams. Treating diagnostic correlations as prescriptive answers. And skipping the step of confirming the frame with the decision-maker. Each of these is easy to prevent with the checks you've just seen, and each one, left unchecked, eventually reaches a decision-maker.

7:49 Recap

Recap. Decision-focused questions produce useful analysis. Frame the decision, options, deciding metric, threshold, population and confounders first. Know which type of question you're asking and watch the slide into causation. Define metrics in a shared glossary the AI reads first. And confirm the frame with the decision-maker before analyzing.

8:10 Try this now

Try this now. Rewrite one vague question your team asks into the framing template, including metric definitions and known confounders. Start a glossary file with at least four definitions, and use it in your next AI analysis prompt. See whether the answer changes.