What is an estimand, and why does your program evaluation need one?
Most disappointing evaluations fail before any data is analysed, because nobody wrote down exactly what was being measured. The estimand fixes that.
A ministry funds a return-to-work program for injured workers. Two years later it commissions an evaluation, which reports that participants returned to work 18 days sooner than non-participants. The program is declared a success.
Then the questions start. Were participants healthier to begin with? Does “returned to work” mean any work, or the same job? What happened to people who started the program and left? Is 18 days the effect for everyone who was eligible, or only for the people who chose to enrol? Each question has a reasonable answer, and each answer gives a different number. The evaluation never said which number it was estimating.
That missing statement has a name: the estimand.
What an estimand is
An estimand is a precise description of the quantity a study is trying to measure. The term comes from statistics, and since 2019 it has been central to how clinical trials are designed, through the international ICH E9(R1) guideline on estimands and sensitivity analysis. The idea applies just as well to program and policy evaluation, where it is still rarely used.
A complete estimand answers five questions:
- Treatment. What intervention, and compared with what alternative? “The program” versus “no program” is rarely precise enough. Compared with the services people would otherwise have received?
- Population. Who is the effect for? Everyone eligible, everyone who enrolled, or a particular group?
- Outcome. What exactly is measured, and when? Days to any paid work, or sustained return to the pre-injury job at six months?
- Intervening events. What happens to people whose path is interrupted, for example by dropping out, re-injury or moving away? Each choice here changes the question.
- Summary measure. The difference in average days, the difference in the proportion back at work, or a ratio?
Write those five answers down and you have stated your question. Leave them implicit and the method, the dataset or the analyst’s habits will answer them for you, usually without anyone noticing.
Why the choice matters
Take just the population question. The effect of a program on everyone eligible tells a ministry what to expect if it rolls the program out. The effect on those who chose to enrol tells it how well the program works for willing participants. Both are legitimate, and they can differ by a factor of two or more, because the people who volunteer are seldom typical.
Or take intervening events. If people who drop out are excluded from the analysis, the program is judged only on those it suited. If they are kept in and counted as not returning to work, the estimate reflects the program as it is really delivered. A decision about funding usually needs the second number. Many evaluations quietly report the first.
None of these choices is a technical detail to be settled after the data arrives. They are decisions about what the organization needs to know, and they belong to the people commissioning the work.
What changes when you start with the estimand
The right comparison becomes obvious. Once you know whose effect you are estimating, you know who the comparison group must resemble. Many evaluation designs, such as difference-in-differences, matching and regression discontinuity, are simply different ways of building that comparison. The estimand tells you which ones can work with your data.
Data gaps show up early. If the estimand needs outcomes at six months and the administrative data stops at three, you find out at the design stage, not in the final report.
Disagreements happen before the money is spent. Boards and program staff often hold different questions in mind. Writing the estimand surfaces that difference while it can still be resolved.
Results become interpretable. “Participants returned to work 18 days sooner” becomes “among workers eligible for the program in 2024, offering it reduced average time to sustained return to the pre-injury employer by an estimated 11 days (95% interval 4 to 18), compared with standard case management.” That second sentence can support a decision. The first cannot.
A short checklist for your next evaluation
Before commissioning or starting an evaluation, ask for one page that answers:
- What intervention is being evaluated, and what is the alternative it is compared with?
- Whose effect are we estimating, and why is that the population the decision is about?
- What is the outcome, how is it measured, and at what time point?
- How will dropouts, deaths, re-injury and other interruptions be treated?
- What summary will be reported, and with what measure of uncertainty?
- Can the available data support this estimand? If not, which estimand can it support?
If the evaluation team cannot answer these clearly, the analysis is not ready to begin.
The bottom line
The estimand is not jargon for its own sake. It is the written agreement between the people who need an answer and the people producing it about what the answer will mean. It costs a few hours at the start of a project, and it is the most reliable way we know to avoid an evaluation that answers a question nobody asked.