startthat.work home

CK-01 Inter-coder agreement statistics (percentage agreement; Cohen's kappa; weighted kappa; Scott's pi; Fleiss' kappa; Krippendorff's alpha; intercoder reliability testing)

Measure how far coders, human or model, give the same code to the same items, corrected for chance, and produce an agreement coefficient per code with a bootstrap interval in a validation report.

Check extracted resultsEvidence: DocumentedVerification technique, runs in step 6Strongest statement: describe

What it does

Measures how far coders, human or model, give the same code to the same items, corrected for chance. Artefact: a coefficient per code, with a bootstrap interval. Read each code separately, not only the overall figure, and read the category distribution beside it.

When to use it

Use when two or more coders have coded the same items and you need to know whether the coding is reliable enough to use.

When it does not fit

Cohen's kappa suits exactly two coders. Scott's pi, Fleiss' kappa and Krippendorff's alpha extend to more coders, and alpha also handles missing codes and ordinal or numeric codes. Percentage agreement is an upper bound because it includes chance agreement. Skewed categories make kappa hard to interpret.

Result format

The result is a table with one row per code, giving the agreement coefficient, its bootstrap interval and the category distribution for that code. It also gives the overall figure and names the codes with the lowest agreement.

Requirements on the material

PropertyRequirement typeRequirement
Unit of analysisAlways appliesrequires=record; requires=document; requires=claim
Variable typesRaises a warningprefers=nominal; prefers=ordinal
Level of detail availableAlways appliesrequires=record-level
Appraises source credibilityApplies in some casesno

Not restricted on: Structure, Content type, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialat least two coders on the same items, at least 50 items overall and enough items in each code to estimate agreement
Additional sources needednone
Not for (rules)none
Not for (text)Wrong choice when the model is scored against itself and treated as another independent coder. Skewed category distributions make kappa hard to read, and percentage agreement overstates reliability.

Choosing and ordering

Kind of techniqueVerification technique, runs in step 6
Phase tagvalidate
Other phase tagsverify and prepare
Kind of findingrisk
Question answeredhow much
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to verify and prepare

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: deterministic.

StakesCheckPerson involvedCounts as a different method
exploratoryinput validationnoneno
decision-informinginput validationnoneno
decision-informingcode execution for arithmeticnoneno
high-stakesinput validationnoneno
high-stakescode execution for arithmeticnoneno
high-stakesindependent reviewneededno

Failure modes and limits

Model suitabilitymodel plus code
Typical model failureScores the model against itself and not against humans, or treats the model as one more independent coder though its errors are correlated across items.
Error sourcesinput data; coder or rater judgement; statistical assumptions
Certainty limitsCertainty drops with few coders or items (imprecision), skewed categories (indirectness) and coders who were not independent (risk of bias).

Evidence and tag basis

Documented. Alpha has no universal cutoff. Krippendorff's conventions, reported secondhand, are 0.8 for serious purposes and 0.67 as a tentative lower bound. Artstein and Poesio advise at least three coders and bootstrap intervals. (Hayes and Krippendorff 2007; Artstein and Poesio 2008, Inter-coder agreement for computational linguistics; Krippendorff, Computing Krippendorff alpha-reliability (abstract))

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.