startthat.work home

CM1-11 Before and after comparison (pre-post design)

Compares an outcome for the same unit before and after an event, producing a change estimate.

CompareEvidence: UnconfirmedPrimary technique, runs in step 5.3Strongest statement: show a link between things

What it does

Compares an outcome for the same unit before and after an event. Artefact: a change estimate.

When to use it

Use when a user asks what changed after a program, law or event and has measurements from before and after it.

When it does not fit

Anything else changed at the same time. This limit is inferred from the difference-in-differences source.

Result format

The result gives the outcome for the same unit before and after the named event, and a change estimate stating the size and direction of the difference.

Requirements on the material

PropertyRequirement typeRequirement
Content typeAlways appliesrequires=tabular; requires=time series
Unit of analysisAlways appliesrequires=record; requires=variable
Variable typesRaises a warningprefers=interval; prefers=ratio; prefers=counts or fractions
Time structureApplies in some casesrequires=panel; requires=aggregate time series; requires=repeated cross-section; requires=cohort
Level of detail availableAlways appliesrequires=aggregate tables
Appraises source credibilityApplies in some casesno

Not restricted on: Structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialMeasurements for the same unit at least once before and once after the event; several earlier points are needed to judge any existing trend.
Additional sources neededearlier period
Not for (rules)none
Not for (text)Wrong choice when anything else changed at the same time as the event. This limit is inferred from the difference-in-differences source.

Choosing and ordering

Kind of techniquePrimary technique, runs in step 5.3
Phase taganalyse
Other phase tagsnone
Kind of findingdifference; trend; cause
Question answeredhow much; why
Strongest statement it supportsshow a link between things
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to analyse

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: deterministic.

StakesCheckPerson involvedCounts as a different method
exploratoryinput validationnoneno
decision-informinginput validationnoneno
decision-informingcode execution for arithmeticnoneno
decision-informingalternative explanations checknoneno
high-stakesinput validationnoneno
high-stakescode execution for arithmeticnoneno
high-stakesalternative explanations checknoneno
high-stakessensitivity analysisnoneno
high-stakessecond technique with different assumptionsnoneconditional
high-stakesexpert reviewneededno

Failure modes and limits

Model suitabilitymodel plus code
Typical model failureCredits the whole change to the event and ignores concurrent changes, a pre-existing trend and regression to the mean.
Error sourcesinput data; statistical assumptions
Certainty limitsCertainty drops when other events coincided with the intervention (risk of bias), when the series before the event is short (imprecision), or when the measure changed definition (inconsistency).

Evidence and tag basis

Unconfirmed. No separate source on simple before and after designs was retrieved. (Wikipedia, Difference in differences)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.