MM-07 Supervised classification and dictionary methods (text classification; fine-tuned classifier; lexicon method)
Assign documents to predefined categories using human-labelled examples or word lists and produce coded documents with confidence and a confusion matrix.
What it does
Assigns documents to predefined categories using human-labelled examples or word lists. Artefact: labels with confidence and a confusion matrix against human labels not used in training. Read precision (the share of assigned labels that are correct) and recall (the share of true cases found) for each category.
When to use it
Use when you have fixed categories and labelled examples, or a validated word list, and need to label many documents.
When it does not fit
Fits fixed, defined categories that humans can code reliably. Misfit: imbalanced or overlapping categories, or training data from another period or source. Dictionaries need explicit validation because word lists fail across domains.
Result format
The result has a label and a confidence value for each document, and a confusion matrix against human labels that were not used in training. It reports precision and recall for each category.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Structure | Applies in some cases | requires=unstructured; requires=semi-structured |
| Content type | Always applies | requires=text |
| Unit of analysis | Always applies | requires=document |
| Level of detail available | Always applies | requires=record-level |
| Amount of material | Applies in some cases | prefers=needs automation |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | labelled examples for every category, usually several hundred in total, with enough examples of the rare categories |
|---|---|
| Additional sources needed | reference structure |
| Not for (rules) | none |
| Not for (text) | Wrong choice when categories overlap, are badly imbalanced or humans cannot code them reliably. Training data from another period or source also fails. Dictionaries need explicit validation in the domain. |
Choosing and ordering
| Kind of technique | Primary technique, runs in step 5.3 |
|---|---|
| Phase tag | analyse |
| Other phase tags | none |
| Kind of finding | pattern |
| Question answered | how much; what |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to gather |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: repeatable with fixed settings.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | input validation | none | no |
| exploratory | spot check | optional | no |
| decision-informing | input validation | none | no |
| decision-informing | spot check | optional | no |
| decision-informing | gold standard agreement | conditional; left out below high stakes | no |
| decision-informing | human review sample | needed; left out below high stakes | no |
| high-stakes | input validation | none | no |
| high-stakes | spot check | optional | no |
| high-stakes | gold standard agreement | conditional | no |
| high-stakes | human review sample | needed | no |
| high-stakes | leakage check | none | no |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | model plus code |
|---|---|
| Typical model failure | Reports overall accuracy and does not show poor performance on rare categories. |
| Error sources | input data; data transformation; statistical assumptions; coder or rater judgement |
| Certainty limits | Certainty drops when training and target documents differ in period or source (indirectness), when rare categories have few examples (imprecision) and when human labels are unreliable (risk of bias). |
Related techniques
- should follow CK-04 Gold-standard validation sample (ground truth set; human-coded validation set): human-coded labels are needed to train and score
- pairs with CK-05 Out-of-sample validation (train and test split; k-fold cross-validation; bootstrap; permutation test): out-of-sample testing gives an honest error estimate
- pairs with CK-14 Data leakage audit (train-test contamination check; model information sheet): a leakage audit checks that test items did not shape the classifier
- substitutes for MM-08 LLM coding and zero-shot classification of text (LLM annotation; LLM-assisted deductive coding; few-shot classification; natural language inference classification): when many labelled examples exist and the volume class is needs automation
Evidence and tag basis
Tested. Supervised learning outperformed dictionaries on a number of criteria (2021). Fine-tuning a natural language inference model on 500 texts matched classical models trained on about 5,000 (2024). Both are abstract-level findings. (Barberá et al. 2021 via EconPapers; Laurer et al. 2024 via EconPapers; Grimmer and Stewart 2013, Political Analysis)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.