CK-04 Gold-standard validation sample (ground truth set; human-coded validation set)
Build a random or stratified human-coded subset against which model output is scored and produce a labelled set.
What it does
Builds a human-coded random or stratified subset against which model output is scored. Artefact: a labelled set. Read agreement among the human coders first.
When to use it
Use when you need human-made labels to test or tune a model or codebook and have none yet.
When it does not fit
Needs categories that humans can apply. Misfit: humans cannot produce the measure reliably. Revise the codebook after ambiguities, and recheck the set with agreement statistics.
Result format
The result is a labelled set of items drawn at random or by stratum, with one row per item showing the item and the code each human coder gave it. It reports agreement among the human coders and lists the items on which they disagreed.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Content type | Always applies | requires=text; requires=tabular |
| Unit of analysis | Always applies | requires=document; requires=record; requires=claim |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | about 100 items at the least, and about 500 documents where categories are many or rare, from Grimmer and Stewart |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Wrong choice when humans cannot apply the categories reliably. A model must not be used to build the gold set, or the check will share the output's errors. |
Choosing and ordering
| Kind of technique | Verification technique, runs in step 6 |
|---|---|
| Phase tag | validate |
| Other phase tags | verify and prepare |
| Kind of finding | risk |
| Question answered | how much |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to verify and prepare |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | spot check | optional | no |
| decision-informing | spot check | optional | no |
| decision-informing | human review sample | needed; left out below high stakes | no |
| decision-informing | gold standard agreement | conditional; left out below high stakes | no |
| high-stakes | spot check | optional | no |
| high-stakes | human review sample | needed | no |
| high-stakes | gold standard agreement | conditional | no |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | human judgement required |
|---|---|
| Typical model failure | Uses a model to build the gold set, so the errors in the check match the errors in the output. |
| Error sources | input data; coder or rater judgement; source selection |
| Certainty limits | Certainty drops when coders disagree (risk of bias), when the sample is not random or stratified (indirectness) and when it is small (imprecision). |
Related techniques
- pairs with CK-01 Inter-coder agreement statistics (percentage agreement; Cohen's kappa; weighted kappa; Scott's pi; Fleiss' kappa; Krippendorff's alpha; intercoder reliability testing): agreement statistics test whether the human coders agree
- pairs with CK-24 Review sampling design (random audit sample; stratified sample; risk-based sampling; acceptance sampling; rule of three): the sample design controls how items are drawn
Evidence and tag basis
Documented. The source states that "no statistical model can repair a poorly constructed training set" and quotes about 500 documents as a rule of thumb, with 100 "probably being enough". (Grimmer and Stewart 2013, Political Analysis)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.