CK-02 Model versus human coding agreement protocol (LLM-as-coder reliability test; intra-coder agreement across runs)
Score a model as one coder against a human coder or panel and against itself across repeated runs, and produce human-human, model-human and model-model agreement in a validation report.
What it does
Scores the model as one coder against a human coder or panel, and against itself across repeated runs. Artefact: human-human agreement (the upper limit), model-human agreement and model-model agreement. Read the model-human figure against the human-human figure, not against 1.
When to use it
Use when a language model has coded text and you need to know if it agrees with humans as well as humans agree with each other.
When it does not fit
Needs an acceptable human-human baseline. Misfit: the humans disagree with each other as much as with the model, so the task is under-specified. High self-agreement does not show accuracy.
Result format
The result reports three agreement figures: human-human agreement as the upper limit, model-human agreement and model-model agreement across repeated runs. It states the model-human figure as a comparison with the human-human figure and says whether the humans agree with each other well enough to serve as a baseline.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Content type | Always applies | requires=text |
| Unit of analysis | Always applies | requires=document; requires=record; requires=claim |
| Level of detail available | Always applies | requires=record-level |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | at least two human coders on a shared sample, a human-human baseline and at least three repeated model runs |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Wrong choice when humans disagree with each other as much as with the model, because the task is then under-specified. High self-agreement across runs does not show accuracy. |
Choosing and ordering
| Kind of technique | Verification technique, runs in step 6 |
|---|---|
| Phase tag | validate |
| Other phase tags | none |
| Kind of finding | risk |
| Question answered | how much |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to verify and prepare |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | spot check | optional | no |
| exploratory | repeat runs | none | no |
| decision-informing | spot check | optional | no |
| decision-informing | repeat runs | none | no |
| decision-informing | gold standard agreement | conditional; left out below high stakes | no |
| decision-informing | human review sample | needed; left out below high stakes | no |
| high-stakes | spot check | optional | no |
| high-stakes | repeat runs | none | no |
| high-stakes | gold standard agreement | conditional | no |
| high-stakes | human review sample | needed | no |
| high-stakes | prompt variation | none | no |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | model plus code |
|---|---|
| Typical model failure | Consistent but wrong labels pass a self-agreement test. |
| Error sources | input data; coder or rater judgement; language model |
| Certainty limits | Certainty drops when the human baseline is weak (risk of bias), when run-to-run agreement is read as accuracy (indirectness) and when the sample is small (imprecision). |
Related techniques
- should follow MM-08 LLM coding and zero-shot classification of text (LLM annotation; LLM-assisted deductive coding; few-shot classification; natural language inference classification): the model codes are what is tested
- should follow CK-04 Gold-standard validation sample (ground truth set; human-coded validation set): the human-coded sample gives the human baseline
- pairs with CK-01 Inter-coder agreement statistics (percentage agreement; Cohen's kappa; weighted kappa; Scott's pi; Fleiss' kappa; Krippendorff's alpha; intercoder reliability testing): agreement coefficients give the human-human baseline
- pairs with CK-10 Repeated-run consistency test (test-retest; intra-coder reliability; TARr@N and TARa@N (raw and parsed answer agreement); settings pinning): repeated-run tests cover self-agreement in more detail
Evidence and tag basis
Tested. ChatGPT agreed with itself across runs more than crowd workers and trained annotators did on all tasks (2023). Another study found consistency fell short of accepted reliability thresholds on a news classification task (2023). (Gilardi et al., arXiv 2303.15056; Reiss, arXiv 2304.11085)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.