CK-01 Inter-coder agreement statistics (percentage agreement; Cohen's kappa; weighted kappa; Scott's pi; Fleiss' kappa; Krippendorff's alpha; intercoder reliability testing)
Measure how far coders, human or model, give the same code to the same items, corrected for chance, and produce an agreement coefficient per code with a bootstrap interval in a validation report.
What it does
Measures how far coders, human or model, give the same code to the same items, corrected for chance. Artefact: a coefficient per code, with a bootstrap interval. Read each code separately, not only the overall figure, and read the category distribution beside it.
When to use it
Use when two or more coders have coded the same items and you need to know whether the coding is reliable enough to use.
When it does not fit
Cohen's kappa suits exactly two coders. Scott's pi, Fleiss' kappa and Krippendorff's alpha extend to more coders, and alpha also handles missing codes and ordinal or numeric codes. Percentage agreement is an upper bound because it includes chance agreement. Skewed categories make kappa hard to interpret.
Result format
The result is a table with one row per code, giving the agreement coefficient, its bootstrap interval and the category distribution for that code. It also gives the overall figure and names the codes with the lowest agreement.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Unit of analysis | Always applies | requires=record; requires=document; requires=claim |
| Variable types | Raises a warning | prefers=nominal; prefers=ordinal |
| Level of detail available | Always applies | requires=record-level |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Content type, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | at least two coders on the same items, at least 50 items overall and enough items in each code to estimate agreement |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Wrong choice when the model is scored against itself and treated as another independent coder. Skewed category distributions make kappa hard to read, and percentage agreement overstates reliability. |
Choosing and ordering
| Kind of technique | Verification technique, runs in step 6 |
|---|---|
| Phase tag | validate |
| Other phase tags | verify and prepare |
| Kind of finding | risk |
| Question answered | how much |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to verify and prepare |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: deterministic.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | input validation | none | no |
| decision-informing | input validation | none | no |
| decision-informing | code execution for arithmetic | none | no |
| high-stakes | input validation | none | no |
| high-stakes | code execution for arithmetic | none | no |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | model plus code |
|---|---|
| Typical model failure | Scores the model against itself and not against humans, or treats the model as one more independent coder though its errors are correlated across items. |
| Error sources | input data; coder or rater judgement; statistical assumptions |
| Certainty limits | Certainty drops with few coders or items (imprecision), skewed categories (indirectness) and coders who were not independent (risk of bias). |
Related techniques
- should follow MM-08 LLM coding and zero-shot classification of text (LLM annotation; LLM-assisted deductive coding; few-shot classification; natural language inference classification): model codes are one of the coder sets compared
- should follow CK-04 Gold-standard validation sample (ground truth set; human-coded validation set): the human-coded set gives the human coders
- pairs with CK-03 Bootstrap confidence intervals for agreement and accuracy (bootstrap resampling of items): bootstrap intervals show the precision of each coefficient
Evidence and tag basis
Documented. Alpha has no universal cutoff. Krippendorff's conventions, reported secondhand, are 0.8 for serious purposes and 0.67 as a tentative lower bound. Artstein and Poesio advise at least three coders and bootstrap intervals. (Hayes and Krippendorff 2007; Artstein and Poesio 2008, Inter-coder agreement for computational linguistics; Krippendorff, Computing Krippendorff alpha-reliability (abstract))
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.