Can you defend the analysis your AI produced?

The eight tests

AI systems can now search, read, calculate and draft analysis that reaches boards, regulators and funders. The hard question is not whether the output looks expert. It is whether you could defend it if someone challenged it. These eight tests are what an expert reviewer would ask.

1. Baselines

Ask: Has the AI been compared with a qualified expert and with a simpler method, on the same tasks?

An impressive output means little until you know what it beats. If a simpler method or a junior analyst does as well, the system adds cost and risk without adding quality.

What a Yes looks like: Results on cases with known answers, side by side with an expert and a simpler approach.

2. Verification separate from generation

Ask: Is the work checked by something other than the system that produced it?

A system that grades its own work tends to share its own blind spots. The check that matters most is for what is missing, not only for what is wrong.

What a Yes looks like: An independent check, by a different method or a person, that confirms required content is present and correct.

3. Executed, not generated, numbers

Ask: Does every number that matters come from a calculation that was actually run?

Language models can write plausible figures that no calculation produced. A number that cannot be traced to a run cannot be defended.

What a Yes looks like: Each figure links to the calculation that produced it, and rerunning it gives the same value.

4. Specification uncertainty

Ask: Has anyone tested whether other reasonable choices of data, sample, definitions or model change the answer?

Many analytical answers depend on choices that experts could make differently. If the conclusion flips under a reasonable alternative, the decision-maker needs to know.

What a Yes looks like: Results shown across the plausible alternatives, with the choices that drive the answer named.

5. Causal-claim checks

Ask: Where the output says one thing caused another, does it state the comparison and the assumptions behind the claim?

Correlation presented as effect is the most expensive error in decision analysis. AI-written text slips between the two easily.

What a Yes looks like: Each causal claim names what was compared with what, and the assumption that makes the comparison fair.

6. Provenance

Ask: Can every claim and figure be traced to a real, dated source that someone opened?

Invented or misread citations are a known failure. A claim whose source nobody checked is a liability in front of a regulator or board.

What a Yes looks like: Every source link works, was checked by a person or an independent process, and carries the date it was accessed.

7. Human gates for assumptions and values

Ask: Do named people decide the assumptions, value judgments and final recommendation?

Discount rates, thresholds, which outcomes count and who bears the risk are choices, not facts. They belong to accountable people.

What a Yes looks like: Defined sign-off points where a named person approves assumptions and the recommendation, with a record of what they approved.

8. Reproducibility

Ask: Could someone else rerun the analysis and get the same answer?

If a result cannot be reproduced, it cannot be audited, corrected or updated when the data changes.

What a Yes looks like: Inputs, versions, settings and steps are kept, and a rerun reproduces the result.

How to use the score

• 14 to 16, with no red flags. Evidence-gated. The output can support decisions with routine expert review.
• 9 to 13. Usable with caution. Treat the output as a strong draft and close the gaps before a decision relies on it.
• 8 or below, or any red flag. Not yet defensible. Use the output as a starting point for expert work, not as the evidence.

Score honestly. "Partly" usually means the practice exists but is not applied every time, or nobody has checked that it works. A self-score is a starting point. An independent test of the system against expert baselines is what turns it into evidence.

Related: Agentic AI for research and reporting: where it helps, and where it quietly fails, What is an estimand? and our AI Expert Audit.