startthat.work home

CK-13 Multiple-testing correction (Benjamini-Hochberg false discovery rate control; Bonferroni; Holm)

Adjust the significance threshold when many comparisons are made, and produce adjusted p-values or a list of retained results.

Check extracted resultsEvidence: DocumentedVerification technique, runs in step 6Strongest statement: show a link between things

What it does

Adjusts the significance threshold when many comparisons are made. False discovery rate control keeps the expected share of false discoveries among reported results at or below a chosen level, and family-wise control limits the chance of any false positive. Artefact: adjusted p-values or a list of retained results. Read how many tests were run, not only how many passed.

When to use it

Use when dozens or hundreds of subgroup, theme or variable comparisons were run and you need to know which results survive.

When it does not fit

Fits dozens or hundreds of subgroup or theme comparisons from exploratory work. Misfit: a single pre-specified hypothesis. Family-wise methods are conservative and lose power with many tests, and false discovery rate control allows some false discoveries by design.

Result format

The result gives the number of tests run and the number that passed, then lists each test with its original and adjusted p-value and whether it is retained. It names the correction method used and the level it controls.

Requirements on the material

PropertyRequirement typeRequirement
StructureApplies in some casesrequires=structured
Content typeAlways appliesrequires=tabular
Unit of analysisAlways appliesrequires=dataset; requires=claim
Level of detail availableAlways appliesrequires=record-level
Appraises source credibilityApplies in some casesno

Not restricted on: Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialthe full list of tests run, with p-values, and a family of tests defined before looking at results
Additional sources needednone
Not for (rules)none
Not for (text)Wrong choice for a single pre-specified hypothesis. Family-wise methods lose power with many tests, and false discovery rate control allows some false discoveries by design.

Choosing and ordering

Kind of techniqueVerification technique, runs in step 6
Phase tagvalidate
Other phase tagsanalyse
Kind of findingdifference; pattern
Question answeredhow much; which option
Strongest statement it supportsshow a link between things
Needs as input
Makes as output
Must exist firstNone
Loop targetproceed with limit stated

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: deterministic.

StakesCheckPerson involvedCounts as a different method
exploratoryinput validationnoneno
decision-informinginput validationnoneno
decision-informingcode execution for arithmeticnoneno
high-stakesinput validationnoneno
high-stakescode execution for arithmeticnoneno
high-stakessensitivity analysisnoneno
high-stakesindependent reviewneededno

Failure modes and limits

Model suitabilitymodel plus code
Typical model failureRuns many comparisons and reports the significant ones without counting them, or defines the family of tests after the fact.
Error sourcesinput data; statistical assumptions
Certainty limitsCertainty drops when the number of tests run is not known (risk of bias) and when the family was defined after the results (reporting bias).

Evidence and tag basis

Documented. Described only on an encyclopaedia page; the original papers were not opened. (Wikipedia: False discovery rate)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.