startthat.work home

CK-04 Gold-standard validation sample (ground truth set; human-coded validation set)

Build a random or stratified human-coded subset against which model output is scored and produce a labelled set.

Check extracted resultsEvidence: DocumentedVerification technique, runs in step 6Strongest statement: describe

What it does

Builds a human-coded random or stratified subset against which model output is scored. Artefact: a labelled set. Read agreement among the human coders first.

When to use it

Use when you need human-made labels to test or tune a model or codebook and have none yet.

When it does not fit

Needs categories that humans can apply. Misfit: humans cannot produce the measure reliably. Revise the codebook after ambiguities, and recheck the set with agreement statistics.

Result format

The result is a labelled set of items drawn at random or by stratum, with one row per item showing the item and the code each human coder gave it. It reports agreement among the human coders and lists the items on which they disagreed.

Requirements on the material

PropertyRequirement typeRequirement
Content typeAlways appliesrequires=text; requires=tabular
Unit of analysisAlways appliesrequires=document; requires=record; requires=claim
Appraises source credibilityApplies in some casesno

Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialabout 100 items at the least, and about 500 documents where categories are many or rare, from Grimmer and Stewart
Additional sources needednone
Not for (rules)none
Not for (text)Wrong choice when humans cannot apply the categories reliably. A model must not be used to build the gold set, or the check will share the output's errors.

Choosing and ordering

Kind of techniqueVerification technique, runs in step 6
Phase tagvalidate
Other phase tagsverify and prepare
Kind of findingrisk
Question answeredhow much
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to verify and prepare

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.

StakesCheckPerson involvedCounts as a different method
exploratoryspot checkoptionalno
decision-informingspot checkoptionalno
decision-informinghuman review sampleneeded; left out below high stakesno
decision-informinggold standard agreementconditional; left out below high stakesno
high-stakesspot checkoptionalno
high-stakeshuman review sampleneededno
high-stakesgold standard agreementconditionalno
high-stakesindependent reviewneededno

Failure modes and limits

Model suitabilityhuman judgement required
Typical model failureUses a model to build the gold set, so the errors in the check match the errors in the output.
Error sourcesinput data; coder or rater judgement; source selection
Certainty limitsCertainty drops when coders disagree (risk of bias), when the sample is not random or stratified (indirectness) and when it is small (imprecision).

Evidence and tag basis

Documented. The source states that "no statistical model can repair a poorly constructed training set" and quotes about 500 documents as a rule of thumb, with 100 "probably being enough". (Grimmer and Stewart 2013, Political Analysis)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.