startthat.work home

CK-02 Model versus human coding agreement protocol (LLM-as-coder reliability test; intra-coder agreement across runs)

Score a model as one coder against a human coder or panel and against itself across repeated runs, and produce human-human, model-human and model-model agreement in a validation report.

Check extracted resultsEvidence: TestedVerification technique, runs in step 6Strongest statement: describe

What it does

Scores the model as one coder against a human coder or panel, and against itself across repeated runs. Artefact: human-human agreement (the upper limit), model-human agreement and model-model agreement. Read the model-human figure against the human-human figure, not against 1.

When to use it

Use when a language model has coded text and you need to know if it agrees with humans as well as humans agree with each other.

When it does not fit

Needs an acceptable human-human baseline. Misfit: the humans disagree with each other as much as with the model, so the task is under-specified. High self-agreement does not show accuracy.

Result format

The result reports three agreement figures: human-human agreement as the upper limit, model-human agreement and model-model agreement across repeated runs. It states the model-human figure as a comparison with the human-human figure and says whether the humans agree with each other well enough to serve as a baseline.

Requirements on the material

PropertyRequirement typeRequirement
Content typeAlways appliesrequires=text
Unit of analysisAlways appliesrequires=document; requires=record; requires=claim
Level of detail availableAlways appliesrequires=record-level
Appraises source credibilityApplies in some casesno

Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialat least two human coders on a shared sample, a human-human baseline and at least three repeated model runs
Additional sources needednone
Not for (rules)none
Not for (text)Wrong choice when humans disagree with each other as much as with the model, because the task is then under-specified. High self-agreement across runs does not show accuracy.

Choosing and ordering

Kind of techniqueVerification technique, runs in step 6
Phase tagvalidate
Other phase tagsnone
Kind of findingrisk
Question answeredhow much
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to verify and prepare

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.

StakesCheckPerson involvedCounts as a different method
exploratoryspot checkoptionalno
exploratoryrepeat runsnoneno
decision-informingspot checkoptionalno
decision-informingrepeat runsnoneno
decision-informinggold standard agreementconditional; left out below high stakesno
decision-informinghuman review sampleneeded; left out below high stakesno
high-stakesspot checkoptionalno
high-stakesrepeat runsnoneno
high-stakesgold standard agreementconditionalno
high-stakeshuman review sampleneededno
high-stakesprompt variationnoneno
high-stakesindependent reviewneededno

Failure modes and limits

Model suitabilitymodel plus code
Typical model failureConsistent but wrong labels pass a self-agreement test.
Error sourcesinput data; coder or rater judgement; language model
Certainty limitsCertainty drops when the human baseline is weak (risk of bias), when run-to-run agreement is read as accuracy (indirectness) and when the sample is small (imprecision).

Evidence and tag basis

Tested. ChatGPT agreed with itself across runs more than crowd workers and trained annotators did on all tasks (2023). Another study found consistency fell short of accepted reliability thresholds on a news classification task (2023). (Gilardi et al., arXiv 2303.15056; Reiss, arXiv 2304.11085)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.