An emerging-tech evaluation scorecard

0:00 An evaluation scorecard

When a new technology hits the leadership agenda, the conversation usually swings between excitement and scepticism depending on who spoke last. In this lesson you'll get a scorecard that asks the same eight questions every time, a research prompt that uses AI to speed up the homework, and a way to run the scoring session so the loudest voice doesn't win.

0:27 Eight dimensions

Here are the eight dimensions, each scored one to five with a short justification and a source. Problem fit: does it solve a costly, frequent problem we already have? Technical maturity: is it proven in contexts like ours? Ecosystem readiness: vendors, standards, talent. Economics: a clear cost per outcome at our scale. Risk and regulation. Reversibility: can we pilot and exit easily? Strategic value. And internal readiness: do we have the data, the skills and an accountable owner?

1:01 Why it matters

Why does this matter? Because technology decisions are increasingly made by groups under time pressure, often after a persuasive demo. Without a shared structure, the conversation follows whoever is most senior, most excited or most sceptical. Good ideas get killed for the wrong reasons, and weak ideas get funded because they sounded exciting. A scorecard gives everyone the same questions, makes disagreements visible, and leaves a record you can learn from. It also protects you later, when someone asks why did we invest in this? You'll have the answer, with sources.

1:41 The pre-flight checklist

Here's an analogy. A scorecard is like a pilot's pre-flight checklist. Experienced pilots don't skip it because they're confident. They use it because confidence is exactly when people miss things. The checklist doesn't fly the plane, and your scorecard doesn't make the decision. It makes sure the same critical questions get asked every time, whoever happens to be in the room.

2:08 Simple example: meeting transcription

A simple example with just three dimensions. You're considering AI meeting transcription for a ten-person agency. Problem fit: five, because everyone hates writing notes. Maturity: five, because it's widely used in production. Risk: three, because client calls need consent and some clients are in regulated industries. The pattern says pilot, but with a consent process and an opt-out for sensitive clients. Three scores and a pattern turned a vague yes into a specific plan.

2:40 Two unscored fields

Add two fields that aren't scored but matter just as much. First, what would change our mind? Write down the specific evidence that would move a score up or down, like two peer companies in production, or official guidance from the regulator. Second, what's the cheapest next experiment? These two lines turn a debate into a plan.

3:05 Read patterns, not totals

Now, how do you read it? Don't add up the numbers into one total, because totals hide trade-offs. Read patterns instead. High strategic value and low maturity: watch closely and run cheap experiments. High maturity, strong problem fit and easy to reverse: pilot now. Low problem fit: park it, however exciting it looks. High regulatory uncertainty: bring legal and compliance in early, and consider waiting for guidance.

3:34 AI-assisted research

AI research tools can compress days of desk research into hours. The lesson text gives you a prompt that asks for evidence on each dimension, with sources and dates, prefers primary sources like regulators and standards bodies, labels every item as fact, claim or opinion, and says no reliable evidence found rather than guessing. Notice what it doesn't ask for: a recommendation. The AI gathers. You judge.

4:03 Verify the sources

And then verify. AI research tools can misattribute sources or invent citations that look perfectly plausible. Open the key sources yourself. Check the date. Confirm the claim actually appears as described. Your scorecard is only as credible as its weakest citation, and a single fabricated source in front of a board can undo months of trust.

4:27 Running the session

Run the session in five steps. Circulate the research pack a few days early. Everyone scores independently first, so nobody anchors on the most senior voice. Discuss only the dimensions where scores differ most. Agree the scores, the what-would-change-our-mind evidence, and the next experiment with an owner and date. Then store it, and revisit in three to six months.

4:53 Worked example: AI contract review

Here's how it played out at a two-hundred-person UK law firm looking at AI contract review. Problem fit scored five, because of high volumes of NDAs and leases. Maturity four. Economics three, with unclear pricing. Risk three, because of confidentiality and professional duties. The weakest score was internal readiness: no clean clause library. So before piloting, the firm spent six weeks building that library, which improved the pilot and turned out to be valuable on its own.

5:26 Economics, carefully

Economics deserve extra care, because they're where excitement hides the most. Ask for the cost per outcome, not per licence: per document reviewed, per lead qualified, per defect caught. Include integration, training, change management and ongoing oversight. And compare against the real alternative, which is often a simpler tool or a process fix. If nobody can express the economics in one sentence, score it low until they can.

5:56 Revisit and learn

One more habit: store every scorecard and revisit it. In six months, open it again. Did the evidence you said would change your mind arrive? Did the experiment happen? Was your stage call early, right or late? That review turns scoring from a meeting ritual into a learning system, and over time your organisation gets measurably better at timing technology bets.

6:23 Three mistakes

Three common mistakes. First, summing the scores into one number, which hides the trade-off that matters, like a brilliant idea with a legal red flag. Second, letting the vendor fill in the scorecard, which is like asking a car salesperson to do your inspection. Third, scoring once and never revisiting, so you never learn whether your judgement was early, right or late.

6:50 Try this now

Try this now. Take a technology your organisation is currently debating, and fill in just four of the eight dimensions: problem fit, maturity, risk and reversibility. For each, write a score from one to five, one sentence of justification, and a source link. Then write what would change your mind. If you can't find a source for a score, mark it unknown rather than guessing. Share the half-finished scorecard with two colleagues and ask them to score independently before you compare. You'll learn more from the differences than from the numbers themselves.

7:30 Recap

To recap. Score eight dimensions with sources, add what would change your mind and the cheapest next experiment, and read patterns rather than totals. Use AI to gather and label evidence, then verify it yourself. Score independently before discussing, and revisit regularly. Your next step: run the research prompt on one technology, verify three sources, and complete the scorecard with your team. Next module: the frontier of AI itself.