startthat.work home

MM-07 Supervised classification and dictionary methods (text classification; fine-tuned classifier; lexicon method)

Assign documents to predefined categories using human-labelled examples or word lists and produce coded documents with confidence and a confusion matrix.

Measure and modelEvidence: TestedPrimary technique, runs in step 5.3Strongest statement: describe

What it does

Assigns documents to predefined categories using human-labelled examples or word lists. Artefact: labels with confidence and a confusion matrix against human labels not used in training. Read precision (the share of assigned labels that are correct) and recall (the share of true cases found) for each category.

When to use it

Use when you have fixed categories and labelled examples, or a validated word list, and need to label many documents.

When it does not fit

Fits fixed, defined categories that humans can code reliably. Misfit: imbalanced or overlapping categories, or training data from another period or source. Dictionaries need explicit validation because word lists fail across domains.

Result format

The result has a label and a confidence value for each document, and a confusion matrix against human labels that were not used in training. It reports precision and recall for each category.

Requirements on the material

PropertyRequirement typeRequirement
StructureApplies in some casesrequires=unstructured; requires=semi-structured
Content typeAlways appliesrequires=text
Unit of analysisAlways appliesrequires=document
Level of detail availableAlways appliesrequires=record-level
Amount of materialApplies in some casesprefers=needs automation
Appraises source credibilityApplies in some casesno

Not restricted on: Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materiallabelled examples for every category, usually several hundred in total, with enough examples of the rare categories
Additional sources neededreference structure
Not for (rules)none
Not for (text)Wrong choice when categories overlap, are badly imbalanced or humans cannot code them reliably. Training data from another period or source also fails. Dictionaries need explicit validation in the domain.

Choosing and ordering

Kind of techniquePrimary technique, runs in step 5.3
Phase taganalyse
Other phase tagsnone
Kind of findingpattern
Question answeredhow much; what
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to gather

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: repeatable with fixed settings.

StakesCheckPerson involvedCounts as a different method
exploratoryinput validationnoneno
exploratoryspot checkoptionalno
decision-informinginput validationnoneno
decision-informingspot checkoptionalno
decision-informinggold standard agreementconditional; left out below high stakesno
decision-informinghuman review sampleneeded; left out below high stakesno
high-stakesinput validationnoneno
high-stakesspot checkoptionalno
high-stakesgold standard agreementconditionalno
high-stakeshuman review sampleneededno
high-stakesleakage checknoneno
high-stakesindependent reviewneededno

Failure modes and limits

Model suitabilitymodel plus code
Typical model failureReports overall accuracy and does not show poor performance on rare categories.
Error sourcesinput data; data transformation; statistical assumptions; coder or rater judgement
Certainty limitsCertainty drops when training and target documents differ in period or source (indirectness), when rare categories have few examples (imprecision) and when human labels are unreliable (risk of bias).

Evidence and tag basis

Tested. Supervised learning outperformed dictionaries on a number of criteria (2021). Fine-tuning a natural language inference model on 500 texts matched classical models trained on about 5,000 (2024). Both are abstract-level findings. (Barberá et al. 2021 via EconPapers; Laurer et al. 2024 via EconPapers; Grimmer and Stewart 2013, Political Analysis)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.