TC1-11 Realist evaluation (context, mechanism, outcome configurations)
Tests what works for whom and in what circumstances through context, mechanism and outcome configurations and produces a refined program theory.
What it does
Tests what works, for whom and in what circumstances. Artefact: a refined program theory; suited to new or scaled programs and programs with mixed outcomes.
When to use it
Use when average results hide differences and you need to know why a program works in some settings and not others.
When it does not fit
Only the average net effect is needed, outcome data are unavailable, the program is simple, or resources are lacking. It is time consuming and mechanisms are inferred.
Result format
The result is a set of context, mechanism and outcome configurations stating what works, for whom and in what circumstances. It ends with a refined program theory that accounts for the mixed outcomes.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Content type | Always applies | requires=text; requires=tabular |
| Unit of analysis | Always applies | requires=topic |
| Level of detail available | Always applies | requires=aggregate tables |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | Evaluation questions, hypotheses stated as context, mechanism and outcome configurations, and outcome data for different groups or settings. |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Wrong choice when only the average net effect is needed, outcome data are unavailable, the program is simple, or time and resources are short. It is time consuming and mechanisms are inferred. |
Choosing and ordering
| Kind of technique | Primary technique, runs in step 5.3 |
|---|---|
| Phase tag | analyse |
| Other phase tags | none |
| Kind of finding | cause; difference |
| Question answered | why; what |
| Strongest statement it supports | explain how or why |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to gather |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | assumption and limitation statement | none | no |
| decision-informing | assumption and limitation statement | none | no |
| decision-informing | alternative explanations check | none | no |
| decision-informing | source-span citation | none | no |
| high-stakes | assumption and limitation statement | none | no |
| high-stakes | alternative explanations check | none | no |
| high-stakes | source-span citation | none | no |
| high-stakes | second technique with different assumptions | none | conditional |
| high-stakes | expert review | needed | no |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | human judgement required |
|---|---|
| Typical model failure | Labels factors as mechanisms without evidence that they operate, and reports configurations that fit the data by construction. |
| Error sources | input data; source selection; coder or rater judgement |
| Certainty limits | Mechanisms are inferred, so certainty drops when no data describe the mechanism itself, when subgroups are small, or when configurations are not tested on a second dataset (imprecision and indirectness). |
Related techniques
- should follow TC1-09 Logic model and theory of change: the program theory supplies the initial configurations to test
- pairs with TC1-12 Process evaluation and impact evaluation: an impact estimate gives the average effect that realist evaluation then explains
Evidence and tag basis
Documented. Described in government guidance. (Magenta Book, realist evaluation supplementary guide; Magenta Book, Annex A, analytical methods for use within an evaluation)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.