startthat.work home

TC1-12 Process evaluation and impact evaluation

Examines how an intervention was delivered and estimates what difference it made, and produces findings on delivery quality or an attributed effect estimate.

Trace cause and test claimsEvidence: DocumentedPrimary technique, runs in step 5.3Strongest statement: estimate the effect of an action

What it does

Process evaluation examines how an intervention was delivered; impact evaluation estimates what difference it made. Artefact: findings on delivery quality, or an attributed effect estimate; read process findings to decide whether a shortfall comes from design or delivery.

When to use it

Use when you must separate whether a program was delivered as designed from whether it changed outcomes.

When it does not fit

A randomised controlled trial lacks sample size or feasible randomisation, or the control group is affected. Size of impact rarely explains why it occurred.

Result format

The result is in two parts: process findings on how well the intervention was delivered, and an attributed effect estimate of what difference it made. It ends by stating whether any shortfall comes from design or from delivery.

Requirements on the material

PropertyRequirement typeRequirement
Content typeAlways appliesrequires=text; requires=tabular
Unit of analysisAlways appliesrequires=topic; requires=dataset
Level of detail availableAlways appliesrequires=aggregate tables
Appraises source credibilityApplies in some casesno

Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialDelivery data for a process evaluation. For an impact evaluation, outcome data for treated and comparison units under an experimental, quasi-experimental or theory-based design.
Additional sources needednone
Not for (rules)none
Not for (text)Wrong choice when a randomised trial lacks sample size or feasible randomisation, or when the control group is affected. The size of an impact rarely explains why it occurred. This record joins two designs and the impact half needs one of the estimation techniques.

Choosing and ordering

Kind of techniquePrimary technique, runs in step 5.3
Phase taganalyse
Other phase tagsvalidate
Kind of findingcause; gap
Question answeredhow much; why
Strongest statement it supportsestimate the effect of an action
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to gather

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.

StakesCheckPerson involvedCounts as a different method
exploratoryassumption and limitation statementnoneno
decision-informingassumption and limitation statementnoneno
decision-informingalternative explanations checknoneno
decision-informingsensitivity analysisnoneno
high-stakesassumption and limitation statementnoneno
high-stakesalternative explanations checknoneno
high-stakessensitivity analysisnoneno
high-stakessecond technique with different assumptionsnoneconditional
high-stakesexpert reviewneededno
high-stakesindependent reviewneededno

Failure modes and limits

Model suitabilityhuman judgement required
Typical model failureReports an impact estimate from a design with no valid comparison group, or reads a delivery shortfall as a design failure.
Error sourcesinput data; statistical assumptions; source selection
Certainty limitsCertainty drops when the control group is affected, sample size is small, randomisation was not feasible or delivery data are incomplete. Theory-based designs give no precise effect size.

Evidence and tag basis

Documented. Described in government guidance; none gives precise effect sizes from theory-based techniques. (Magenta Book, central government guidance on evaluation; Magenta Book, Annex A, analytical methods for use within an evaluation)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.