TC-05 Subgroup and Simpson's paradox check (stratified reanalysis; confounding check; collapsibility check)
Recomputes an association within subgroups to see whether the pooled direction reverses or vanishes and produces pooled and stratified estimates side by side.
What it does
Recomputes an association within subgroups to see whether the pooled direction reverses or vanishes. Artefact: pooled and stratified estimates side by side. Read for a change of sign. In the encyclopaedia example, recovery was 50 per cent in treated and untreated groups overall, yet 61 against 57 per cent in men and 44 against 40 per cent in women.
When to use it
Use when aggregate rates are compared across groups of unequal composition and you want to see whether the pattern holds inside each group.
When it does not fit
Run whenever aggregate rates are compared across groups of unequal composition. Which partition to trust depends on causal structure, not on the numbers, and conditioning on a mediator distorts the effect.
Result format
The result is a table giving the pooled estimate and the estimate within each subgroup, side by side. It states whether the direction changes sign or the association vanishes in the subgroups.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Content type | Always applies | requires=tabular |
| Unit of analysis | Always applies | requires=dataset; requires=record |
| Variable types | Raises a warning | prefers=nominal; prefers=counts or fractions |
| Level of detail available | Always applies | requires=aggregate tables |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | A pooled comparison, a grouping variable plausibly related to both exposure and outcome, and enough records in each subgroup to estimate the association. |
|---|---|
| Additional sources needed | reference structure |
| Not for (rules) | none |
| Not for (text) | Misused when the stratified result is trusted without asking which partition fits the causal structure. Conditioning on a mediator distorts the effect. |
Choosing and ordering
| Kind of technique | Verification technique, runs in step 6 |
|---|---|
| Phase tag | validate |
| Other phase tags | validate |
| Kind of finding | contradiction; difference |
| Question answered | why; how much |
| Strongest statement it supports | show a link between things |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to analyse |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | input validation | none | no |
| decision-informing | input validation | none | no |
| decision-informing | code execution for arithmetic | none | no |
| decision-informing | sensitivity analysis | none | no |
| high-stakes | input validation | none | no |
| high-stakes | code execution for arithmetic | none | no |
| high-stakes | sensitivity analysis | none | no |
| high-stakes | second technique with different assumptions | none | conditional |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | human judgement required |
|---|---|
| Typical model failure | Reports the stratified result as the true effect without asking which partition fits the causal structure, or stratifies on a mediator. |
| Error sources | input data; statistical assumptions; data transformation |
| Certainty limits | The correct partition depends on causal structure outside the data. Certainty drops when strata are small or the grouping variable is a mediator or collider (risk of bias and imprecision). |
Related techniques
- should follow TC1-08 Confounder identification with directed acyclic graphs (adjustment sets, backdoor criterion): the graph decides which partition to trust
- pairs with TC-06 Ecological fallacy check (aggregation bias check): the ecological check asks whether group-level results apply to individuals
Evidence and tag basis
Documented. Different partitions of the same data can support incompatible causal conclusions. (Stanford Encyclopedia of Philosophy: Simpson's paradox; Stanford Encyclopedia: Simpson's paradox (2006 archive))
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.