startthat.work home

CK-06 Bias correction of model labels with a human-coded subsample (design-based supervised learning (DSL); prediction-powered inference (PPI); regression estimator correction)

Combine model labels for all items with human labels for a random subsample so that downstream estimates and intervals stay valid, and produce a corrected estimate with an interval.

Check extracted resultsEvidence: TestedVerification technique, runs in step 6Strongest statement: show a link between things

What it does

Combines model labels for all items with human labels for a random subsample, so that downstream estimates and intervals stay valid. Artefact: a corrected estimate with an interval. Read whether the conclusion survives the correction.

When to use it

Use when model-generated labels are used in a numeric claim or a regression and you need estimates that are not biased by label errors.

When it does not fit

Fits when model labels are used in a numeric claim or comparison. Misfit: no random human-coded subsample with known sampling probabilities. Correction trades Type I against Type II error.

Result format

The result gives a corrected estimate with its interval, set beside the uncorrected estimate from the model labels alone. It states the size of the human-coded subsample and whether the conclusion survives the correction.

Requirements on the material

PropertyRequirement typeRequirement
Unit of analysisAlways appliesrequires=document; requires=record
Level of detail availableAlways appliesrequires=record-level
Appraises source credibilityApplies in some casesno

Not restricted on: Structure, Content type, Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materiala random human-coded subsample with known sampling probabilities, usually a few hundred items
Additional sources needednone
Not for (rules)none
Not for (text)Wrong choice when there is no random human-coded subsample with known probabilities. The correction trades Type I against Type II error and does not repair a badly defined category.

Choosing and ordering

Kind of techniqueVerification technique, runs in step 6
Phase tagvalidate
Other phase tagsanalyse
Kind of findingdifference; trend
Question answeredhow much
Strongest statement it supportsshow a link between things
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to gather

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: repeatable with fixed settings.

StakesCheckPerson involvedCounts as a different method
exploratoryinput validationnoneno
decision-informinginput validationnoneno
decision-informingsensitivity analysisnoneno
decision-informingcode execution for arithmeticnoneno
high-stakesinput validationnoneno
high-stakessensitivity analysisnoneno
high-stakescode execution for arithmeticnoneno
high-stakessecond technique with different assumptionsnoneconditional
high-stakesindependent reviewneededno

Failure modes and limits

Model suitabilitymodel plus code
Typical model failureCounts model labels as truth in a regression, or reports uncorrected label counts as population values.
Error sourcesinput data; language model; statistical assumptions; coder or rater judgement
Certainty limitsCertainty drops when the human subsample is not random (risk of bias), when it is small (imprecision) and when human labels are unreliable (indirectness).

Evidence and tag basis

Tested. Using surrogate labels directly gave substantial bias and invalid confidence intervals even when labels were 80 to 90 per cent accurate (2023). Prediction-powered inference gives valid intervals with no assumption about the prediction model (2023, abstract). (Egami et al. 2023, arXiv 2306.04746; Angelopoulos et al. 2023, arXiv 2301.09633; Baumann et al. 2025, arXiv 2509.08825)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.