startthat.work home

TC1-04 Process tracing (diagnostic tests: straw in the wind, hoop, smoking gun, doubly decisive)

Tests what each rival hypothesis predicts against diagnostic evidence within one case and produces a test result per hypothesis in a hypotheses matrix.

Trace cause and test claimsEvidence: DocumentedPrimary technique, runs in step 5.3Strongest statement: explain how or why

What it does

Examines diagnostic evidence within one case and compares what each rival hypothesis predicts with what is found. Artefact: a test result per hypothesis; failing a hoop test eliminates a hypothesis and passing a smoking gun test supports it strongly.

When to use it

Use when you need to decide which of several explanations fits the detailed evidence in a single case and no comparison group exists.

When it does not fit

The evidence is not diagnostic given prior knowledge, or the rival set is incomplete. It is often unclear which test applies, and doubly decisive tests are rarely achieved in social science.

Result format

The result is a table of diagnostic evidence items against each rival hypothesis, giving the test type (straw in the wind, hoop, smoking gun) and the outcome of the test for each hypothesis. It names the hypotheses eliminated by a failed hoop test and the hypotheses strongly supported by a passed smoking gun test.

Requirements on the material

PropertyRequirement typeRequirement
Content typeAlways appliesrequires=text
Unit of analysisAlways appliesrequires=claim; requires=topic
Appraises source credibilityApplies in some casesno

Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialDetailed evidence from one case and at least two rival hypotheses, each with predictions written before the evidence is read.
Additional sources needednone
Not for (rules)none
Not for (text)Wrong choice when the evidence is not diagnostic given prior knowledge or the rival set is incomplete. It is often unclear which test applies, and doubly decisive tests are rare in social science.

Choosing and ordering

Kind of techniquePrimary technique, runs in step 5.3
Phase taganalyse
Other phase tagsvalidate
Kind of findingcause; contradiction
Question answeredwhy
Strongest statement it supportsexplain how or why
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to gather

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.

StakesCheckPerson involvedCounts as a different method
exploratoryassumption and limitation statementnoneno
decision-informingassumption and limitation statementnoneno
decision-informingalternative explanations checknoneno
decision-informingsource-span citationnoneno
high-stakesassumption and limitation statementnoneno
high-stakesalternative explanations checknoneno
high-stakessource-span citationnoneno
high-stakessecond technique with different assumptionsnoneconditional
high-stakesexpert reviewneededno
high-stakesindependent reviewneededno

Failure modes and limits

Model suitabilityhuman judgement required
Typical model failureClassifies a piece of evidence as a smoking gun after seeing it and not before, and ignores rival hypotheses that were never stated.
Error sourcesinput data; source selection; coder or rater judgement
Certainty limitsCertainty drops when the rival set is incomplete, when test strength is judged after the evidence is seen, or when the documents come from interested parties (risk of bias).

Evidence and tag basis

Documented. A critic disputes how the tests map to necessity and sufficiency. The Magenta Book suggests it for single cases with no counterfactual. (Collier, Understanding Process Tracing; BetterEvaluation commentary on process tracing tests; Magenta Book, Annex A, analytical methods for use within an evaluation)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.