TC1-04 Process tracing (diagnostic tests: straw in the wind, hoop, smoking gun, doubly decisive)
Tests what each rival hypothesis predicts against diagnostic evidence within one case and produces a test result per hypothesis in a hypotheses matrix.
What it does
Examines diagnostic evidence within one case and compares what each rival hypothesis predicts with what is found. Artefact: a test result per hypothesis; failing a hoop test eliminates a hypothesis and passing a smoking gun test supports it strongly.
When to use it
Use when you need to decide which of several explanations fits the detailed evidence in a single case and no comparison group exists.
When it does not fit
The evidence is not diagnostic given prior knowledge, or the rival set is incomplete. It is often unclear which test applies, and doubly decisive tests are rarely achieved in social science.
Result format
The result is a table of diagnostic evidence items against each rival hypothesis, giving the test type (straw in the wind, hoop, smoking gun) and the outcome of the test for each hypothesis. It names the hypotheses eliminated by a failed hoop test and the hypotheses strongly supported by a passed smoking gun test.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Content type | Always applies | requires=text |
| Unit of analysis | Always applies | requires=claim; requires=topic |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | Detailed evidence from one case and at least two rival hypotheses, each with predictions written before the evidence is read. |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Wrong choice when the evidence is not diagnostic given prior knowledge or the rival set is incomplete. It is often unclear which test applies, and doubly decisive tests are rare in social science. |
Choosing and ordering
| Kind of technique | Primary technique, runs in step 5.3 |
|---|---|
| Phase tag | analyse |
| Other phase tags | validate |
| Kind of finding | cause; contradiction |
| Question answered | why |
| Strongest statement it supports | explain how or why |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to gather |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | assumption and limitation statement | none | no |
| decision-informing | assumption and limitation statement | none | no |
| decision-informing | alternative explanations check | none | no |
| decision-informing | source-span citation | none | no |
| high-stakes | assumption and limitation statement | none | no |
| high-stakes | alternative explanations check | none | no |
| high-stakes | source-span citation | none | no |
| high-stakes | second technique with different assumptions | none | conditional |
| high-stakes | expert review | needed | no |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | human judgement required |
|---|---|
| Typical model failure | Classifies a piece of evidence as a smoking gun after seeing it and not before, and ignores rival hypotheses that were never stated. |
| Error sources | input data; source selection; coder or rater judgement |
| Certainty limits | Certainty drops when the rival set is incomplete, when test strength is judged after the evidence is seen, or when the documents come from interested parties (risk of bias). |
Related techniques
- pairs with TC1-13 Analysis of competing hypotheses (ACH): analysis of competing hypotheses compares all evidence across all hypotheses while process tracing sets the strength of each test
- pairs with CM1-05 Comparative case study (structured focused comparison, small-N comparison): comparative cases show whether the traced mechanism recurs
Evidence and tag basis
Documented. A critic disputes how the tests map to necessity and sufficiency. The Magenta Book suggests it for single cases with no counterfactual. (Collier, Understanding Process Tracing; BetterEvaluation commentary on process tracing tests; Magenta Book, Annex A, analytical methods for use within an evaluation)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.