TC1-06 Difference-in-differences Also serves: Compare.
Compares the change over time in a treated group with the change in a control group and produces an interaction coefficient read as the treatment effect.
What it does
Compares the change over time in a treated group with the change over time in a control group. Artefact: the interaction coefficient (treatment group by post period), read as the treatment effect.
When to use it
Use when a policy or event affected one group and not another and you want to know how much it changed an outcome.
When it does not fit
The groups would not have changed by the same amount without treatment (parallel trends fails), a concurrent shock affected one group, or group composition changed.
Result format
The result gives the outcome for the treated and control groups before and after the change, and the change over time in each group. It ends with the interaction coefficient (treatment group by post period) and states it as the treatment effect.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Content type | Always applies | requires=tabular |
| Unit of analysis | Always applies | requires=record; requires=dataset |
| Variable types | Raises a warning | prefers=interval; prefers=ratio; prefers=counts or fractions |
| Time structure | Applies in some cases | requires=panel; requires=repeated cross-section; requires=aggregate time series |
| Level of detail available | Always applies | requires=aggregate tables |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | At least one treated and one comparison group, with at least one period before and one after treatment. Several pre-treatment periods are needed to check parallel trends. |
|---|---|
| Additional sources needed | none |
| Not for (rules) | time_structure=cross-section :: the design needs observations before and after treatment |
| Not for (text) | Wrong choice when the groups would not have changed by the same amount without treatment (parallel trends fails), a concurrent shock affected one group, or group composition changed. |
Choosing and ordering
| Kind of technique | Primary technique, runs in step 5.3 |
|---|---|
| Phase tag | analyse |
| Other phase tags | none |
| Kind of finding | cause; difference |
| Question answered | how much; why |
| Strongest statement it supports | estimate the effect of an action |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to gather |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: deterministic.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | assumption and limitation statement | none | no |
| exploratory | input validation | none | no |
| decision-informing | assumption and limitation statement | none | no |
| decision-informing | input validation | none | no |
| decision-informing | alternative explanations check | none | no |
| decision-informing | sensitivity analysis | none | no |
| decision-informing | code execution for arithmetic | none | no |
| high-stakes | assumption and limitation statement | none | no |
| high-stakes | input validation | none | no |
| high-stakes | alternative explanations check | none | no |
| high-stakes | sensitivity analysis | none | no |
| high-stakes | code execution for arithmetic | none | no |
| high-stakes | second technique with different assumptions | none | conditional |
| high-stakes | expert review | needed | no |
Failure modes and limits
| Model suitability | human judgement required |
|---|---|
| Typical model failure | Runs the interaction regression without checking pre-treatment trends, reads the coefficient as causal, and reports standard errors that ignore clustering. |
| Error sources | input data; statistical assumptions; data transformation |
| Certainty limits | Certainty drops when pre-treatment trends differ, a concurrent shock hit one group, group composition changed, or there are few clusters or periods (risk of bias and imprecision). |
Related techniques
- should follow TC1-08 Confounder identification with directed acyclic graphs (adjustment sets, backdoor criterion): the graph shows which covariates to include and which to leave out
- pairs with TC1-17 Sensitivity analysis (robustness check, leave-one-out): varying pre-periods, controls and group definitions tests the parallel trends assumption
- substitutes for CM1-11 Before and after comparison (pre-post design): when a comparison group exists and time_structure=panel|repeated cross-section|aggregate time series
Evidence and tag basis
Documented. Described on an encyclopaedia page. (Wikipedia, Difference in differences)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.