CM1-11 Before and after comparison (pre-post design)
Compares an outcome for the same unit before and after an event, producing a change estimate.
What it does
Compares an outcome for the same unit before and after an event. Artefact: a change estimate.
When to use it
Use when a user asks what changed after a program, law or event and has measurements from before and after it.
When it does not fit
Anything else changed at the same time. This limit is inferred from the difference-in-differences source.
Result format
The result gives the outcome for the same unit before and after the named event, and a change estimate stating the size and direction of the difference.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Content type | Always applies | requires=tabular; requires=time series |
| Unit of analysis | Always applies | requires=record; requires=variable |
| Variable types | Raises a warning | prefers=interval; prefers=ratio; prefers=counts or fractions |
| Time structure | Applies in some cases | requires=panel; requires=aggregate time series; requires=repeated cross-section; requires=cohort |
| Level of detail available | Always applies | requires=aggregate tables |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | Measurements for the same unit at least once before and once after the event; several earlier points are needed to judge any existing trend. |
|---|---|
| Additional sources needed | earlier period |
| Not for (rules) | none |
| Not for (text) | Wrong choice when anything else changed at the same time as the event. This limit is inferred from the difference-in-differences source. |
Choosing and ordering
| Kind of technique | Primary technique, runs in step 5.3 |
|---|---|
| Phase tag | analyse |
| Other phase tags | none |
| Kind of finding | difference; trend; cause |
| Question answered | how much; why |
| Strongest statement it supports | show a link between things |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to analyse |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: deterministic.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | input validation | none | no |
| decision-informing | input validation | none | no |
| decision-informing | code execution for arithmetic | none | no |
| decision-informing | alternative explanations check | none | no |
| high-stakes | input validation | none | no |
| high-stakes | code execution for arithmetic | none | no |
| high-stakes | alternative explanations check | none | no |
| high-stakes | sensitivity analysis | none | no |
| high-stakes | second technique with different assumptions | none | conditional |
| high-stakes | expert review | needed | no |
Failure modes and limits
| Model suitability | model plus code |
|---|---|
| Typical model failure | Credits the whole change to the event and ignores concurrent changes, a pre-existing trend and regression to the mean. |
| Error sources | input data; statistical assumptions |
| Certainty limits | Certainty drops when other events coincided with the intervention (risk of bias), when the series before the event is short (imprecision), or when the measure changed definition (inconsistency). |
Related techniques
- pairs with TC1-06 Difference-in-differences Also serves: Compare.: a control group removes changes that affected both groups
- pairs with TC1-17 Sensitivity analysis (robustness check, leave-one-out): re-running with other baselines shows whether the change estimate reverses
Evidence and tag basis
Unconfirmed. No separate source on simple before and after designs was retrieved. (Wikipedia, Difference in differences)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.