AI for Data Analysis & Decision Making · Communicating uncertainty and making decisions · lesson 15 of 16 · 10 min

Communicating uncertainty clearly

Why uncertainty is hard to communicate

Stakeholders often want a single number and a firm answer. Analysts worry that caveats will make them sound unsure. AI drafts tend to either overstate confidence or drown findings in hedges. Communicating uncertainty well means being clear about what you know, what you don't, and how much it matters for the decision.

Sources of uncertainty

• Sampling: results from a sample (survey, test group) differ from the whole population.
• Measurement: data errors, tracking gaps, definitions.
• Model: forecasting or statistical model assumptions.
• Causal: whether an observed relationship is causal.
• Future: events that haven't happened yet.

Naming the main source helps readers understand what could change.

Techniques

1. Ranges instead of points. "Conversion rose by between 1 and 3 percentage points" is more honest than "rose 2 points", and more useful.

2. Consistent confidence language. Agree team-wide on what phrases mean. For example:

Almost certain   - we would be very surprised to be wrong
Likely           - more likely than not, with meaningful doubt
Uncertain        - evidence is mixed or thin
Unlikely         - evidence points the other way

Some organizations attach rough probability ranges to such terms; if you do, define them once and use them consistently.

3. Natural frequencies. "About 3 in 10 customers" is often understood better than "30%", especially for risks.

4. Visuals. Error bars, shaded forecast intervals, and scenario fans show uncertainty at a glance.

5. Decision relevance. Say whether the uncertainty changes the decision. "Even at the low end of the range, the campaign pays back within six months" is powerful. So is "The result could go either way; a two-week test would resolve it."

6. What would change our mind. State the evidence that would reverse the recommendation.

Prompting AI to express uncertainty well

Rewrite this summary for the leadership team. For each key claim:
- give a range where the data supports one
- label confidence as Almost certain / Likely / Uncertain / Unlikely
- say in one sentence whether the uncertainty changes the recommendation
Do not add hedges that don't change the decision.

The last line prevents the opposite failure: so many "may" and "might" words that nothing is said.

Worked example: campaign results

Original AI draft: "The campaign increased sales by 14%, proving its effectiveness."

Revised: "Sales in campaign regions rose about 8-18% versus comparable regions (likely a real effect; the comparison regions weren't perfectly matched). Even at the low end, the campaign covered its cost. Recommendation: repeat it in two more regions with matched controls to firm up the estimate."

The revised version is more useful to a decision-maker, and more defensible if challenged.

Avoiding false precision

Reporting "18.73%" from a sample of 200 implies precision that doesn't exist. Round sensibly and show the range. AI outputs frequently carry spurious decimals from computations; tidy them before presenting.

Handling pushback

When a stakeholder says "Just give me a number", give one, with its range attached: "Plan for about 4,200 units; we'd be surprised by fewer than 3,700 or more than 4,800." This respects their need for a planning figure while keeping the uncertainty visible. If they ask why the range is wide, explain the main source of uncertainty in one sentence and what would narrow it (more data, a test, time). Uncertainty presented this way reads as expertise, not indecision.

Common failure modes

• Burying the key caveat in a footnote nobody reads.
• Listing every possible limitation equally, so the important one is lost.
• Presenting a forecast point without a range.
• Using "significant" loosely, where readers may hear "statistically significant" or "large".

Hands-on: turn a result into a range with a bootstrap

When the metric is not a simple proportion (for example average order value or revenue per visitor), a bootstrap gives an honest range without heavy theory:

import numpy as np
import pandas as pd

orders = pd.read_csv("campaign_orders.csv")            # one row per order: group, value
rng = np.random.default_rng(42)

def boot_diff(df, n=5000):
    a = df[df.group == "campaign"]["value"].to_numpy()
    b = df[df.group == "control"]["value"].to_numpy()
    diffs = [rng.choice(a, a.size).mean() - rng.choice(b, b.size).mean() for _ in range(n)]
    return np.percentile(diffs, [2.5, 50, 97.5])

low, mid, high = boot_diff(orders)
print(f"AOV difference: {mid:.1f} (95% range {low:.1f} to {high:.1f})")

Then write it the way a decision-maker can use: "Average order value was about 6 AED higher in campaign regions (likely between 2 and 10 AED). Even at the low end the campaign covers its cost."

A confidence-language card for your team

Almost certain   we would be very surprised to be wrong      (roughly 90%+)
Likely           more likely than not, meaningful doubt       (roughly 60-90%)
Uncertain        evidence mixed or thin                       (roughly 40-60%)
Unlikely         evidence points the other way                (roughly 10-40%)
Always add:      the main source of uncertainty + what would change our mind

The probability bands are illustrative; agree your own once and use them consistently.

Second worked example: a UK charity's fundraising forecast

A charity's board asks for "the number" for next year's income. The analyst gives a planning figure with a range, names the main uncertainty (one large grant renewal), and shows two scenarios: renewal and no renewal. The board approves a budget based on the no-renewal scenario with a trigger to release extra spending if the grant is confirmed. Uncertainty became a plan, not an excuse.

Going further

Create a one-page uncertainty guide for your team: confidence terms, rounding rules, required ranges for key metrics, and a template "What would change our mind" line. Include it in AI prompts for reports so drafts follow your standard.

Video lecture: Communicating uncertainty clearly

Lecture coming soon · 16 chapters · about 8 minutes. Read the full transcript below.

1. Communicating uncertainty
2. Why it matters
3. Five sources
4. Techniques
5. Two powerful sentences
6. False precision
7. Example 1: the campaign summary
8. Example 2: charity income
9. Watch me do it, part 1
10. Watch me do it, part 2
11. Prompting for uncertainty
12. Handling pushback
13. A team uncertainty guide
14. Signs it's working
15. Common mistakes
16. Recap and try this now

Lecture transcript

Communicating uncertainty

Just give me a number. Every analyst has heard it. And every analyst has faced the choice: give a single confident figure and hope, or bury it in so many caveats that nobody can act. There's a better way. In this lecture you'll learn to name the sources of uncertainty, to express it with ranges, consistent confidence language, natural frequencies and visuals, to say whether the uncertainty changes the decision, to compute an honest range with a bootstrap, and to handle pushback gracefully.

Why it matters

Why does this matter? Because stakeholders make better decisions when they know what's solid and what's shaky. Overconfidence leads to over-commitment: too much stock, too much budget, a campaign repeated because of a lucky result. Endless hedging leads to paralysis. AI drafts tend toward one extreme or the other. Communicating uncertainty well means being clear about what you know, what you don't, and how much it matters for the decision in front of you.

Five sources

Start by naming the source. Sampling: results from a survey or a test group differ from the whole population. Measurement: data errors, tracking gaps, definitions. Model: forecasting or statistical assumptions. Causal: whether an observed relationship is causal. And future: events that haven't happened yet. Naming the main source tells readers what could change, and what would reduce the uncertainty. Think of a weather forecaster who says rain is likely because a front is approaching. The reason makes the forecast usable.

Techniques

Now the techniques. Ranges instead of points: conversion rose by between one and three percentage points is more honest, and more useful, than rose two points. Consistent confidence language, agreed team-wide: almost certain, likely, uncertain, unlikely, optionally with rough probability bands. Natural frequencies: about three in ten customers is often understood better than thirty percent, especially for risks. And visuals: error bars, shaded forecast intervals and scenario fans show uncertainty at a glance.

Two powerful sentences

And the two most powerful sentences in uncertainty communication. First, decision relevance: say whether the uncertainty changes the decision. Even at the low end of the range, the campaign pays back within six months, is powerful. So is: the result could go either way; a two-week test would resolve it. Second, what would change our mind: state the evidence that would reverse your recommendation. Those two sentences turn uncertainty from a disclaimer into guidance.

False precision

Avoid false precision too. Reporting eighteen point seven three percent from a sample of two hundred implies precision that doesn't exist. Round sensibly and show the range. AI outputs frequently carry spurious decimals straight from computations, so tidy them before presenting. And watch the word significant: readers may hear statistically significant, or large, or important. Say which you mean.

Example 1: the campaign summary

First example, a simple one. The original AI draft says: the campaign increased sales by fourteen percent, proving its effectiveness. The revised version: sales in campaign regions rose about eight to eighteen percent versus comparable regions; likely a real effect, although the comparison regions weren't perfectly matched. Even at the low end, the campaign covered its cost. Recommendation: repeat it in two more regions with matched controls to firm up the estimate. More useful, and far more defensible if challenged.

Example 2: charity income

Second example, a business case. A UK charity's board asks for the number for next year's income. The analyst gives a planning figure with a range, names the main uncertainty, one large grant renewal, and shows two scenarios: renewal and no renewal. The board approves a budget based on the no-renewal scenario, with a trigger to release extra spending if the grant is confirmed. Uncertainty became a plan, not an excuse.

Watch me do it, part 1

Watch me compute an honest range with a bootstrap, for a metric that isn't a simple proportion, like average order value. The idea: resample the orders with replacement thousands of times, recompute the difference between campaign and control each time, and take the middle ninety-five percent of those differences. In the lesson's code, I load one row per order with its group, run five thousand resamples, and print the median difference and the two point five and ninety-seven point five percentiles.

Watch me do it, part 2

It prints: average order value about six dirhams higher in campaign regions, with a ninety-five percent range of roughly two to ten. Now I write it for the decision-maker. Average order value was about six dirhams higher in campaign regions, likely between two and ten. Even at the low end, the campaign covers its cost. The main uncertainty is that the regions weren't perfectly matched. A matched test in two more regions would narrow the range. That's the whole story in four sentences.

Prompting for uncertainty

Now prompting AI to express uncertainty well. Rewrite this summary for the leadership team. For each key claim, give a range where the data supports one, label confidence as almost certain, likely, uncertain or unlikely, and say in one sentence whether the uncertainty changes the recommendation. Then add: do not add hedges that don't change the decision. That last line prevents the opposite failure, so many mays and mights that nothing is said.

Handling pushback

And handling pushback. When a stakeholder says just give me a number, give one, with its range attached: plan for about forty-two hundred units; we'd be surprised by fewer than thirty-seven hundred or more than forty-eight hundred. That respects their need for a planning figure while keeping uncertainty visible. If they ask why the range is wide, name the main source in one sentence and what would narrow it: more data, a test, or time. Uncertainty presented this way reads as expertise, not indecision.

A team uncertainty guide

Create a one-page uncertainty guide for your team. Confidence terms with the rough probability bands you've agreed. Rounding rules, like whole percentages for most business metrics. Which key metrics always need a range. And a template line: what would change our mind. Include the guide in your AI prompts for reports, so drafts follow your standard automatically. Consistency matters as much as accuracy here: if likely means sixty percent to one person and ninety to another, the word is useless.

Signs it's working

How do you know it's working? Stakeholders start asking for ranges themselves. Decisions include triggers and scenarios rather than single bets. Fewer surprises reach leadership unannounced, because the main uncertainty was named in advance. And when a result falls at the edge of your range, nobody accuses you of being wrong, because you said it could.

Common mistakes

Common mistakes. Burying the key caveat in a footnote nobody reads. Listing every possible limitation equally, so the important one is lost. Presenting forecasts without a range. Using significant loosely. And the opposite: so much hedging that no decision is possible.

Recap and try this now

Recap. Name the main sources of uncertainty: sampling, measurement, model, causal and future. Use ranges, consistent confidence language, natural frequencies and visuals. Say whether uncertainty changes the decision and what would change your mind. Avoid both overconfidence and empty hedging, and round away false precision. Try this now: rewrite a recent result summary with a range, computed with a bootstrap if needed, a confidence label, a decision-relevance line, and a what would change our mind line.

Key takeaways

• Name the main sources of uncertainty: sampling, measurement, model, causal, future.
• Use ranges, consistent confidence language, natural frequencies and visuals.
• Say whether uncertainty changes the decision and what would change your mind.
• Avoid both overconfidence and empty hedging; round away false precision.

Try it

Rewrite a recent result summary using ranges, a confidence label and a 'what would change our mind' line.

• Previous: Forecasting with AI: useful, with caveats
• Next: Making decisions with AI support
• All lessons of AI for Data Analysis & Decision Making