MM-08 LLM coding and zero-shot classification of text (LLM annotation; LLM-assisted deductive coding; few-shot classification; natural language inference classification)
Apply a human-written codebook or category descriptions to text with a large language model and produce codes, rationales and per-code agreement with human coders.
What it does
Applies a human-written codebook, or category descriptions, to text with a large language model (LLM). Artefact: codes, rationales and per-code agreement with human coders. Read the failing codes and the disagreement cases.
When to use it
Use when you have a written codebook and a body of text too large to code by hand and want a model to apply the codes.
When it does not fit
Pilot on a human-coded sample, and require per-code agreement, repeated-run stability and error analysis before scaling. Misfit: agreement checked only overall, one prompt tried, or error patterns by group not examined.
Result format
The result is a table with one row per document, giving the code assigned and the rationale, and a per-code agreement figure with human coders. It lists the codes that fail and the cases where the model and the human coders disagree.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Structure | Applies in some cases | requires=unstructured; requires=semi-structured |
| Content type | Always applies | requires=text |
| Unit of analysis | Always applies | requires=document; requires=record |
| Amount of material | Applies in some cases | prefers=needs automation |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | a human-coded pilot sample with enough items per code to estimate agreement, usually 100 or more in total |
|---|---|
| Additional sources needed | reference structure |
| Not for (rules) | none |
| Not for (text) | Wrong choice when agreement is checked only overall, only one prompt is tried or error patterns by group are not examined. A rationale that sounds right is not evidence that the label is right. |
Choosing and ordering
| Kind of technique | Primary technique, runs in step 5.3 |
|---|---|
| Phase tag | analyse |
| Other phase tags | none |
| Kind of finding | pattern; difference |
| Question answered | how much; what |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to verify and prepare |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | spot check | optional | no |
| exploratory | repeat runs | none | no |
| decision-informing | spot check | optional | no |
| decision-informing | repeat runs | none | no |
| decision-informing | gold standard agreement | conditional; left out below high stakes | no |
| decision-informing | prompt variation | none | no |
| decision-informing | human review sample | needed; left out below high stakes | no |
| high-stakes | spot check | optional | no |
| high-stakes | repeat runs | none | no |
| high-stakes | gold standard agreement | conditional | no |
| high-stakes | prompt variation | none | no |
| high-stakes | human review sample | needed | no |
| high-stakes | sensitivity analysis | none | no |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | model plus code |
|---|---|
| Typical model failure | Codes whole sets confidently without a validation sample, returns different labels on a rerun, and gives plausible rationales that do not justify the label. |
| Error sources | input data; language model; coder or rater judgement |
| Certainty limits | Certainty drops when agreement is low on any code (risk of bias), when labels change across runs or prompts (inconsistency) and when errors differ by subject group (indirectness). |
Related techniques
- should follow CK-04 Gold-standard validation sample (ground truth set; human-coded validation set): a human-coded sample is needed for the pilot
- pairs with CK-02 Model versus human coding agreement protocol (LLM-as-coder reliability test; intra-coder agreement across runs): agreement against human coders and across runs
- pairs with CK-10 Repeated-run consistency test (test-retest; intra-coder reliability; TARr@N and TARa@N (raw and parsed answer agreement); settings pinning): repeated runs show stability
- pairs with CK-12 Specification sweep across models and prompts (LLM hacking test; multiverse analysis; specification curve): a specification sweep shows whether conclusions depend on prompt and model
- pairs with CK-06 Bias correction of model labels with a human-coded subsample (design-based supervised learning (DSL); prediction-powered inference (PPI); regression estimator correction): bias correction is needed when labels are used in a numeric claim
- substitutes for HE1-15 Content analysis (conventional, directed and summative): when a fixed codebook exists and the volume class is needs automation
Evidence and tag basis
Tested. Studies from 2023 to 2025 disagree by task. ChatGPT beat crowd workers on four of five tasks on 2,382 tweets. GPT-4 reached kappa of 0.79 or more on 3 of 9 codes and 0.6 or more on 8 of 9 in one case study. Across 13 million labels from 18 models, about 31 per cent of hypotheses reached incorrect conclusions. Errors on interview transcripts were not random with respect to subject characteristics. (Gilardi et al., arXiv 2303.15056; Dunivin, arXiv 2401.15170; Baumann et al. 2025, arXiv 2509.08825; Ashwin et al., arXiv 2309.17147; Ziems et al., arXiv 2305.03514; Pangakis et al., arXiv 2306.00176; Reiss, arXiv 2304.11085; Bano et al., arXiv 2306.13298; Tai et al., DOAJ record; Törnberg, arXiv 2304.06588)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.