startthat.work home

TC1-05 Counterfactual analysis (potential outcomes, minimal rewrite of history) Also serves: Compare.

Sets the actual outcome against an argued account of what would have happened had one factor differed and produces a counterfactual scenario.

Trace cause and test claimsEvidence: DocumentedPrimary technique, runs in step 5.3Strongest statement: explain how or why

What it does

Sets the actual outcome against an argued account of what would have happened had one factor differed. Artefact: a counterfactual scenario; read divergence from the actual outcome as the causal effect of the factor.

When to use it

Use when you must judge whether one factor caused an outcome in a single case and no comparison data exist.

When it does not fit

Varying the factor requires changing other factors that would also matter (low cotenability). Fewer required changes make the inference easier to defend; a 'but for' condition is not automatically a cause.

Result format

The result states the actual outcome, the one factor that is changed, and a counterfactual scenario describing what would have happened had that factor differed. It ends with the divergence between the scenario and the actual outcome, read as the causal effect of the factor.

Requirements on the material

PropertyRequirement typeRequirement
Content typeAlways appliesrequires=text
Unit of analysisAlways appliesrequires=claim
Appraises source credibilityApplies in some casesno

Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialOne specified change to a single factor, the relevant historical facts, and a stated theory or regularity linking the factor to the outcome.
Additional sources needednone
Not for (rules)none
Not for (text)Wrong choice when varying the factor requires changing other factors that would also matter (low cotenability). A but-for condition is not automatically a cause.

Choosing and ordering

Kind of techniquePrimary technique, runs in step 5.3
Phase taganalyse
Other phase tagsnone
Kind of findingcause; difference
Question answeredwhy
Strongest statement it supportsexplain how or why
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to analyse

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.

StakesCheckPerson involvedCounts as a different method
exploratoryassumption and limitation statementnoneno
decision-informingassumption and limitation statementnoneno
decision-informingalternative explanations checknoneno
high-stakesassumption and limitation statementnoneno
high-stakesalternative explanations checknoneno
high-stakessecond technique with different assumptionsnoneconditional
high-stakesexpert reviewneededno
high-stakesindependent reviewneededno

Failure modes and limits

Model suitabilityhuman judgement required
Typical model failureRewrites history so that several other factors change at once, then credits the divergence to the one factor named.
Error sourcesinput data; coder or rater judgement; expert opinion
Certainty limitsCertainty drops when the change needs other changes, when the theory linking factor to outcome is weak, or when the argued outcome cannot be checked against any evidence (indirectness).

Evidence and tag basis

Documented. Fearon argues counterfactuals are unavoidable in causal testing with few cases; the Stanford Encyclopedia lists vagueness of similarity and counterfactual scepticism as limits. (Fearon, Counterfactuals and Hypothesis Testing in Political Science; Stanford Encyclopedia of Philosophy, Counterfactuals)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.