TC1-10 Contribution analysis
Builds a reasoned line of evidence that a program contributed to an outcome and produces an evidenced contribution story.
What it does
Builds a reasoned line of evidence that a program contributed to an outcome when an experiment is not possible. Artefact: an evidenced contribution story; it is not definitive proof.
When to use it
Use when you must judge whether a program helped cause an outcome and an experiment or comparison group is not possible.
When it does not fit
The theory of change is poor, because the quality of the analysis depends on it. No precise effect size results.
Result format
The result is a contribution story that runs from the program to the outcome, with the evidence for each step in the story. It states what the evidence does not establish, since the story is not definitive proof, and ends by naming the remaining alternative explanations.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Content type | Always applies | requires=text |
| Unit of analysis | Always applies | requires=topic; requires=claim |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | A stated theory of change, evidence for each link, and a list of other influences on the outcome. |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Wrong choice when the theory of change is poor, because the quality of the analysis depends on it. No precise effect size results, and the story is not proof. |
Choosing and ordering
| Kind of technique | Primary technique, runs in step 5.3 |
|---|---|
| Phase tag | analyse |
| Other phase tags | conclude |
| Kind of finding | cause; gap |
| Question answered | why |
| Strongest statement it supports | explain how or why |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to frame |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | assumption and limitation statement | none | no |
| decision-informing | assumption and limitation statement | none | no |
| decision-informing | alternative explanations check | none | no |
| decision-informing | source-span citation | none | no |
| high-stakes | assumption and limitation statement | none | no |
| high-stakes | alternative explanations check | none | no |
| high-stakes | source-span citation | none | no |
| high-stakes | second technique with different assumptions | none | conditional |
| high-stakes | expert review | needed | no |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | human judgement required |
|---|---|
| Typical model failure | Writes a coherent contribution story that leaves out rival explanations and weak links and presents it as proof. |
| Error sources | input data; source selection; expert opinion |
| Certainty limits | Certainty drops when the theory of change is poorly specified, when evidence for a link is thin, or when other influences are not ruled out (risk of bias and indirectness). |
Related techniques
- should follow TC1-09 Logic model and theory of change: the theory of change and its assumptions supply the links to be tested
- substitutes for TC1-06 Difference-in-differences Also serves: Compare.: when no comparison group or outcome tables exist (access_depth=narrative only)
Evidence and tag basis
Documented. Described in government evaluation guidance. (Magenta Book, Annex A, analytical methods for use within an evaluation)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.