startthat.work home

TC1-06 Difference-in-differences Also serves: Compare.

Compares the change over time in a treated group with the change in a control group and produces an interaction coefficient read as the treatment effect.

Trace cause and test claimsEvidence: DocumentedPrimary technique, runs in step 5.3Strongest statement: estimate the effect of an action

What it does

Compares the change over time in a treated group with the change over time in a control group. Artefact: the interaction coefficient (treatment group by post period), read as the treatment effect.

When to use it

Use when a policy or event affected one group and not another and you want to know how much it changed an outcome.

When it does not fit

The groups would not have changed by the same amount without treatment (parallel trends fails), a concurrent shock affected one group, or group composition changed.

Result format

The result gives the outcome for the treated and control groups before and after the change, and the change over time in each group. It ends with the interaction coefficient (treatment group by post period) and states it as the treatment effect.

Requirements on the material

PropertyRequirement typeRequirement
Content typeAlways appliesrequires=tabular
Unit of analysisAlways appliesrequires=record; requires=dataset
Variable typesRaises a warningprefers=interval; prefers=ratio; prefers=counts or fractions
Time structureApplies in some casesrequires=panel; requires=repeated cross-section; requires=aggregate time series
Level of detail availableAlways appliesrequires=aggregate tables
Appraises source credibilityApplies in some casesno

Not restricted on: Structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialAt least one treated and one comparison group, with at least one period before and one after treatment. Several pre-treatment periods are needed to check parallel trends.
Additional sources needednone
Not for (rules)time_structure=cross-section :: the design needs observations before and after treatment
Not for (text)Wrong choice when the groups would not have changed by the same amount without treatment (parallel trends fails), a concurrent shock affected one group, or group composition changed.

Choosing and ordering

Kind of techniquePrimary technique, runs in step 5.3
Phase taganalyse
Other phase tagsnone
Kind of findingcause; difference
Question answeredhow much; why
Strongest statement it supportsestimate the effect of an action
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to gather

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: deterministic.

StakesCheckPerson involvedCounts as a different method
exploratoryassumption and limitation statementnoneno
exploratoryinput validationnoneno
decision-informingassumption and limitation statementnoneno
decision-informinginput validationnoneno
decision-informingalternative explanations checknoneno
decision-informingsensitivity analysisnoneno
decision-informingcode execution for arithmeticnoneno
high-stakesassumption and limitation statementnoneno
high-stakesinput validationnoneno
high-stakesalternative explanations checknoneno
high-stakessensitivity analysisnoneno
high-stakescode execution for arithmeticnoneno
high-stakessecond technique with different assumptionsnoneconditional
high-stakesexpert reviewneededno

Failure modes and limits

Model suitabilityhuman judgement required
Typical model failureRuns the interaction regression without checking pre-treatment trends, reads the coefficient as causal, and reports standard errors that ignore clustering.
Error sourcesinput data; statistical assumptions; data transformation
Certainty limitsCertainty drops when pre-treatment trends differ, a concurrent shock hit one group, group composition changed, or there are few clusters or periods (risk of bias and imprecision).

Evidence and tag basis

Documented. Described on an encyclopaedia page. (Wikipedia, Difference in differences)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.