TC1-12 Process evaluation and impact evaluation
Examines how an intervention was delivered and estimates what difference it made, and produces findings on delivery quality or an attributed effect estimate.
What it does
Process evaluation examines how an intervention was delivered; impact evaluation estimates what difference it made. Artefact: findings on delivery quality, or an attributed effect estimate; read process findings to decide whether a shortfall comes from design or delivery.
When to use it
Use when you must separate whether a program was delivered as designed from whether it changed outcomes.
When it does not fit
A randomised controlled trial lacks sample size or feasible randomisation, or the control group is affected. Size of impact rarely explains why it occurred.
Result format
The result is in two parts: process findings on how well the intervention was delivered, and an attributed effect estimate of what difference it made. It ends by stating whether any shortfall comes from design or from delivery.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Content type | Always applies | requires=text; requires=tabular |
| Unit of analysis | Always applies | requires=topic; requires=dataset |
| Level of detail available | Always applies | requires=aggregate tables |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | Delivery data for a process evaluation. For an impact evaluation, outcome data for treated and comparison units under an experimental, quasi-experimental or theory-based design. |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Wrong choice when a randomised trial lacks sample size or feasible randomisation, or when the control group is affected. The size of an impact rarely explains why it occurred. This record joins two designs and the impact half needs one of the estimation techniques. |
Choosing and ordering
| Kind of technique | Primary technique, runs in step 5.3 |
|---|---|
| Phase tag | analyse |
| Other phase tags | validate |
| Kind of finding | cause; gap |
| Question answered | how much; why |
| Strongest statement it supports | estimate the effect of an action |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to gather |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | assumption and limitation statement | none | no |
| decision-informing | assumption and limitation statement | none | no |
| decision-informing | alternative explanations check | none | no |
| decision-informing | sensitivity analysis | none | no |
| high-stakes | assumption and limitation statement | none | no |
| high-stakes | alternative explanations check | none | no |
| high-stakes | sensitivity analysis | none | no |
| high-stakes | second technique with different assumptions | none | conditional |
| high-stakes | expert review | needed | no |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | human judgement required |
|---|---|
| Typical model failure | Reports an impact estimate from a design with no valid comparison group, or reads a delivery shortfall as a design failure. |
| Error sources | input data; statistical assumptions; source selection |
| Certainty limits | Certainty drops when the control group is affected, sample size is small, randomisation was not feasible or delivery data are incomplete. Theory-based designs give no precise effect size. |
Related techniques
- should follow TC1-09 Logic model and theory of change: the logic model names the delivery steps and outcomes to examine
- pairs with TC1-06 Difference-in-differences Also serves: Compare.: a quasi-experimental estimate supplies the impact half when no trial was run
Evidence and tag basis
Documented. Described in government guidance; none gives precise effect sizes from theory-based techniques. (Magenta Book, central government guidance on evaluation; Magenta Book, Annex A, analytical methods for use within an evaluation)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.