CK-06 Bias correction of model labels with a human-coded subsample (design-based supervised learning (DSL); prediction-powered inference (PPI); regression estimator correction)
Combine model labels for all items with human labels for a random subsample so that downstream estimates and intervals stay valid, and produce a corrected estimate with an interval.
What it does
Combines model labels for all items with human labels for a random subsample, so that downstream estimates and intervals stay valid. Artefact: a corrected estimate with an interval. Read whether the conclusion survives the correction.
When to use it
Use when model-generated labels are used in a numeric claim or a regression and you need estimates that are not biased by label errors.
When it does not fit
Fits when model labels are used in a numeric claim or comparison. Misfit: no random human-coded subsample with known sampling probabilities. Correction trades Type I against Type II error.
Result format
The result gives a corrected estimate with its interval, set beside the uncorrected estimate from the model labels alone. It states the size of the human-coded subsample and whether the conclusion survives the correction.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Unit of analysis | Always applies | requires=document; requires=record |
| Level of detail available | Always applies | requires=record-level |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Content type, Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | a random human-coded subsample with known sampling probabilities, usually a few hundred items |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Wrong choice when there is no random human-coded subsample with known probabilities. The correction trades Type I against Type II error and does not repair a badly defined category. |
Choosing and ordering
| Kind of technique | Verification technique, runs in step 6 |
|---|---|
| Phase tag | validate |
| Other phase tags | analyse |
| Kind of finding | difference; trend |
| Question answered | how much |
| Strongest statement it supports | show a link between things |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to gather |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: repeatable with fixed settings.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | input validation | none | no |
| decision-informing | input validation | none | no |
| decision-informing | sensitivity analysis | none | no |
| decision-informing | code execution for arithmetic | none | no |
| high-stakes | input validation | none | no |
| high-stakes | sensitivity analysis | none | no |
| high-stakes | code execution for arithmetic | none | no |
| high-stakes | second technique with different assumptions | none | conditional |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | model plus code |
|---|---|
| Typical model failure | Counts model labels as truth in a regression, or reports uncorrected label counts as population values. |
| Error sources | input data; language model; statistical assumptions; coder or rater judgement |
| Certainty limits | Certainty drops when the human subsample is not random (risk of bias), when it is small (imprecision) and when human labels are unreliable (indirectness). |
Related techniques
- should follow MM-08 LLM coding and zero-shot classification of text (LLM annotation; LLM-assisted deductive coding; few-shot classification; natural language inference classification): model labels for all items are the input
- should follow CK-04 Gold-standard validation sample (ground truth set; human-coded validation set): the random human-coded subsample is the correction set
- pairs with CK-12 Specification sweep across models and prompts (LLM hacking test; multiverse analysis; specification curve): a specification sweep shows how far conclusions depend on label choices
Evidence and tag basis
Tested. Using surrogate labels directly gave substantial bias and invalid confidence intervals even when labels were 80 to 90 per cent accurate (2023). Prediction-powered inference gives valid intervals with no assumption about the prediction model (2023, abstract). (Egami et al. 2023, arXiv 2306.04746; Angelopoulos et al. 2023, arXiv 2301.09633; Baumann et al. 2025, arXiv 2509.08825)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.