CK-13 Multiple-testing correction (Benjamini-Hochberg false discovery rate control; Bonferroni; Holm)
Adjust the significance threshold when many comparisons are made, and produce adjusted p-values or a list of retained results.
What it does
Adjusts the significance threshold when many comparisons are made. False discovery rate control keeps the expected share of false discoveries among reported results at or below a chosen level, and family-wise control limits the chance of any false positive. Artefact: adjusted p-values or a list of retained results. Read how many tests were run, not only how many passed.
When to use it
Use when dozens or hundreds of subgroup, theme or variable comparisons were run and you need to know which results survive.
When it does not fit
Fits dozens or hundreds of subgroup or theme comparisons from exploratory work. Misfit: a single pre-specified hypothesis. Family-wise methods are conservative and lose power with many tests, and false discovery rate control allows some false discoveries by design.
Result format
The result gives the number of tests run and the number that passed, then lists each test with its original and adjusted p-value and whether it is retained. It names the correction method used and the level it controls.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Structure | Applies in some cases | requires=structured |
| Content type | Always applies | requires=tabular |
| Unit of analysis | Always applies | requires=dataset; requires=claim |
| Level of detail available | Always applies | requires=record-level |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | the full list of tests run, with p-values, and a family of tests defined before looking at results |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Wrong choice for a single pre-specified hypothesis. Family-wise methods lose power with many tests, and false discovery rate control allows some false discoveries by design. |
Choosing and ordering
| Kind of technique | Verification technique, runs in step 6 |
|---|---|
| Phase tag | validate |
| Other phase tags | analyse |
| Kind of finding | difference; pattern |
| Question answered | how much; which option |
| Strongest statement it supports | show a link between things |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | proceed with limit stated |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: deterministic.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | input validation | none | no |
| decision-informing | input validation | none | no |
| decision-informing | code execution for arithmetic | none | no |
| high-stakes | input validation | none | no |
| high-stakes | code execution for arithmetic | none | no |
| high-stakes | sensitivity analysis | none | no |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | model plus code |
|---|---|
| Typical model failure | Runs many comparisons and reports the significant ones without counting them, or defines the family of tests after the fact. |
| Error sources | input data; statistical assumptions |
| Certainty limits | Certainty drops when the number of tests run is not known (risk of bias) and when the family was defined after the results (reporting bias). |
Related techniques
- pairs with MM-01 Exploratory data analysis with plots and distribution checks (EDA (exploratory data analysis); summary statistics; Q-Q plot; scatterplot matrix): exploratory findings from many plots and subgroups are the usual input
- pairs with CK-12 Specification sweep across models and prompts (LLM hacking test; multiverse analysis; specification curve): specification sweeps multiply tests
Evidence and tag basis
Documented. Described only on an encyclopaedia page; the original papers were not opened. (Wikipedia: False discovery rate)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.