startthat.work home

MM-08 LLM coding and zero-shot classification of text (LLM annotation; LLM-assisted deductive coding; few-shot classification; natural language inference classification)

Apply a human-written codebook or category descriptions to text with a large language model and produce codes, rationales and per-code agreement with human coders.

Measure and modelEvidence: TestedPrimary technique, runs in step 5.3Strongest statement: describe

What it does

Applies a human-written codebook, or category descriptions, to text with a large language model (LLM). Artefact: codes, rationales and per-code agreement with human coders. Read the failing codes and the disagreement cases.

When to use it

Use when you have a written codebook and a body of text too large to code by hand and want a model to apply the codes.

When it does not fit

Pilot on a human-coded sample, and require per-code agreement, repeated-run stability and error analysis before scaling. Misfit: agreement checked only overall, one prompt tried, or error patterns by group not examined.

Result format

The result is a table with one row per document, giving the code assigned and the rationale, and a per-code agreement figure with human coders. It lists the codes that fail and the cases where the model and the human coders disagree.

Requirements on the material

PropertyRequirement typeRequirement
StructureApplies in some casesrequires=unstructured; requires=semi-structured
Content typeAlways appliesrequires=text
Unit of analysisAlways appliesrequires=document; requires=record
Amount of materialApplies in some casesprefers=needs automation
Appraises source credibilityApplies in some casesno

Not restricted on: Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materiala human-coded pilot sample with enough items per code to estimate agreement, usually 100 or more in total
Additional sources neededreference structure
Not for (rules)none
Not for (text)Wrong choice when agreement is checked only overall, only one prompt is tried or error patterns by group are not examined. A rationale that sounds right is not evidence that the label is right.

Choosing and ordering

Kind of techniquePrimary technique, runs in step 5.3
Phase taganalyse
Other phase tagsnone
Kind of findingpattern; difference
Question answeredhow much; what
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to verify and prepare

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.

StakesCheckPerson involvedCounts as a different method
exploratoryspot checkoptionalno
exploratoryrepeat runsnoneno
decision-informingspot checkoptionalno
decision-informingrepeat runsnoneno
decision-informinggold standard agreementconditional; left out below high stakesno
decision-informingprompt variationnoneno
decision-informinghuman review sampleneeded; left out below high stakesno
high-stakesspot checkoptionalno
high-stakesrepeat runsnoneno
high-stakesgold standard agreementconditionalno
high-stakesprompt variationnoneno
high-stakeshuman review sampleneededno
high-stakessensitivity analysisnoneno
high-stakesindependent reviewneededno

Failure modes and limits

Model suitabilitymodel plus code
Typical model failureCodes whole sets confidently without a validation sample, returns different labels on a rerun, and gives plausible rationales that do not justify the label.
Error sourcesinput data; language model; coder or rater judgement
Certainty limitsCertainty drops when agreement is low on any code (risk of bias), when labels change across runs or prompts (inconsistency) and when errors differ by subject group (indirectness).

Evidence and tag basis

Tested. Studies from 2023 to 2025 disagree by task. ChatGPT beat crowd workers on four of five tasks on 2,382 tweets. GPT-4 reached kappa of 0.79 or more on 3 of 9 codes and 0.6 or more on 8 of 9 in one case study. Across 13 million labels from 18 models, about 31 per cent of hypotheses reached incorrect conclusions. Errors on interview transcripts were not random with respect to subject characteristics. (Gilardi et al., arXiv 2303.15056; Dunivin, arXiv 2401.15170; Baumann et al. 2025, arXiv 2509.08825; Ashwin et al., arXiv 2309.17147; Ziems et al., arXiv 2305.03514; Pangakis et al., arXiv 2306.00176; Reiss, arXiv 2304.11085; Bano et al., arXiv 2306.13298; Tai et al., DOAJ record; Törnberg, arXiv 2304.06588)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.