startthat.work home

TC1-07 Quasi-experimental designs (natural experiment, instrumental variables, regression discontinuity)

Estimates an effect from observational data in which exposure was assigned as if at random (natural experiment, instrument or cutoff) and produces an effect estimate under stated assumptions.

Trace cause and test claimsEvidence: UnconfirmedPrimary technique, runs in step 5.3Strongest statement: estimate the effect of an action

What it does

Each uses existing data in which exposure is assigned as if at random: by an event or rule (natural experiment), by a variable that changes exposure but affects the outcome only through it (instrument), or by a cutoff (discontinuity). Artefact: an effect estimate under stated assumptions.

When to use it

Use when a randomised trial is impossible but an event, rule, instrument or threshold gave some units exposure for reasons unrelated to the outcome.

When it does not fit

Assignment is related to outcomes, the instrument is weak or has a direct effect, or units can manipulate their position relative to the cutoff.

Result format

The result names the design used (natural experiment, instrument or discontinuity) and describes the event, rule, variable or cutoff that assigns exposure as if at random. It gives an effect estimate and lists the assumptions under which the estimate holds.

Requirements on the material

PropertyRequirement typeRequirement
Content typeAlways appliesrequires=tabular
Unit of analysisAlways appliesrequires=record
Variable typesRaises a warningprefers=interval; prefers=ratio; prefers=counts or fractions
Level of detail availableAlways appliesrequires=record-level
Appraises source credibilityApplies in some casesno

Not restricted on: Structure, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialRecord-level data with exposure and outcome, plus a clear assignment event, a valid instrument or a running variable with a cutoff. Discontinuity designs need enough records near the cutoff.
Additional sources needednone
Not for (rules)none
Not for (text)Wrong choice when assignment is related to outcomes, the instrument is weak or affects the outcome directly, or units can manipulate their position relative to the cutoff.

Choosing and ordering

Kind of techniquePrimary technique, runs in step 5.3
Phase taganalyse
Other phase tagsnone
Kind of findingcause
Question answeredhow much; why
Strongest statement it supportsestimate the effect of an action
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to gather

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: deterministic.

StakesCheckPerson involvedCounts as a different method
exploratoryassumption and limitation statementnoneno
exploratoryinput validationnoneno
decision-informingassumption and limitation statementnoneno
decision-informinginput validationnoneno
decision-informingalternative explanations checknoneno
decision-informingsensitivity analysisnoneno
decision-informingcode execution for arithmeticnoneno
high-stakesassumption and limitation statementnoneno
high-stakesinput validationnoneno
high-stakesalternative explanations checknoneno
high-stakessensitivity analysisnoneno
high-stakescode execution for arithmeticnoneno
high-stakessecond technique with different assumptionsnoneconditional
high-stakesexpert reviewneededno

Failure modes and limits

Model suitabilityhuman judgement required
Typical model failureAccepts an instrument or cutoff without testing its assumptions and reports a precise estimate that depends on an invalid design.
Error sourcesinput data; statistical assumptions; data transformation
Certainty limitsCertainty drops when the instrument is weak or has a direct effect, when units manipulate the cutoff, or when the estimate applies only to a narrow group of units (risk of bias and indirectness).

Evidence and tag basis

Unconfirmed. The Hernán and Robins book lists instrumental variables among its topics, but the assumptions here come from general knowledge and no source on natural experiments or regression discontinuity was opened. (Hernán and Robins, Causal Inference: What If)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.