TC1-07 Quasi-experimental designs (natural experiment, instrumental variables, regression discontinuity)
Estimates an effect from observational data in which exposure was assigned as if at random (natural experiment, instrument or cutoff) and produces an effect estimate under stated assumptions.
What it does
Each uses existing data in which exposure is assigned as if at random: by an event or rule (natural experiment), by a variable that changes exposure but affects the outcome only through it (instrument), or by a cutoff (discontinuity). Artefact: an effect estimate under stated assumptions.
When to use it
Use when a randomised trial is impossible but an event, rule, instrument or threshold gave some units exposure for reasons unrelated to the outcome.
When it does not fit
Assignment is related to outcomes, the instrument is weak or has a direct effect, or units can manipulate their position relative to the cutoff.
Result format
The result names the design used (natural experiment, instrument or discontinuity) and describes the event, rule, variable or cutoff that assigns exposure as if at random. It gives an effect estimate and lists the assumptions under which the estimate holds.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Content type | Always applies | requires=tabular |
| Unit of analysis | Always applies | requires=record |
| Variable types | Raises a warning | prefers=interval; prefers=ratio; prefers=counts or fractions |
| Level of detail available | Always applies | requires=record-level |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | Record-level data with exposure and outcome, plus a clear assignment event, a valid instrument or a running variable with a cutoff. Discontinuity designs need enough records near the cutoff. |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Wrong choice when assignment is related to outcomes, the instrument is weak or affects the outcome directly, or units can manipulate their position relative to the cutoff. |
Choosing and ordering
| Kind of technique | Primary technique, runs in step 5.3 |
|---|---|
| Phase tag | analyse |
| Other phase tags | none |
| Kind of finding | cause |
| Question answered | how much; why |
| Strongest statement it supports | estimate the effect of an action |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to gather |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: deterministic.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | assumption and limitation statement | none | no |
| exploratory | input validation | none | no |
| decision-informing | assumption and limitation statement | none | no |
| decision-informing | input validation | none | no |
| decision-informing | alternative explanations check | none | no |
| decision-informing | sensitivity analysis | none | no |
| decision-informing | code execution for arithmetic | none | no |
| high-stakes | assumption and limitation statement | none | no |
| high-stakes | input validation | none | no |
| high-stakes | alternative explanations check | none | no |
| high-stakes | sensitivity analysis | none | no |
| high-stakes | code execution for arithmetic | none | no |
| high-stakes | second technique with different assumptions | none | conditional |
| high-stakes | expert review | needed | no |
Failure modes and limits
| Model suitability | human judgement required |
|---|---|
| Typical model failure | Accepts an instrument or cutoff without testing its assumptions and reports a precise estimate that depends on an invalid design. |
| Error sources | input data; statistical assumptions; data transformation |
| Certainty limits | Certainty drops when the instrument is weak or has a direct effect, when units manipulate the cutoff, or when the estimate applies only to a narrow group of units (risk of bias and indirectness). |
Related techniques
- should follow TC1-08 Confounder identification with directed acyclic graphs (adjustment sets, backdoor criterion): the graph states which paths the instrument or cutoff must not open
- substitutes for TC1-06 Difference-in-differences Also serves: Compare.: when no untreated comparison group with pre-treatment data exists (time_structure=cross-section) but an instrument or cutoff does
- pairs with TC1-17 Sensitivity analysis (robustness check, leave-one-out): sensitivity checks vary the bandwidth, instrument and controls
Evidence and tag basis
Unconfirmed. The Hernán and Robins book lists instrumental variables among its topics, but the assumptions here come from general knowledge and no source on natural experiments or regression discontinuity was opened. (Hernán and Robins, Causal Inference: What If)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.