startthat.work home

MM-14 Document clustering (computer-assisted clustering; k-means on document vectors)

Group similar documents without predefined categories using counts or embeddings and produce cluster assignments, centroids and exemplar documents.

Measure and modelEvidence: DocumentedPrimary technique, runs in step 5.3Strongest statement: describe

What it does

Groups similar documents without predefined categories, using counts or embeddings. Artefact: cluster assignments, centroids and exemplars. Read the exemplars and the documents nearest each boundary.

When to use it

Use when you want to sort a large set of documents into groups of similar content before deciding what the groups mean.

When it does not fit

Needs samples from each cluster that a person can inspect. Misfit: clusters reflect length, source style or language and not subject. Many valid partitions of the same documents exist, so choose by usefulness and validate (the note author's paraphrase of the source, to be checked against the full text).

Result format

The result has a cluster assignment for each document, the centroid of each cluster and exemplar documents for each cluster. It also lists the documents nearest each cluster boundary.

Requirements on the material

PropertyRequirement typeRequirement
StructureApplies in some casesrequires=unstructured; requires=semi-structured
Content typeAlways appliesrequires=text
Unit of analysisAlways appliesrequires=document
Level of detail availableAlways appliesrequires=record-level
Amount of materialApplies in some casesprefers=needs automation
Appraises source credibilityApplies in some casesno

Not restricted on: Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialenough documents for each cluster to have inspectable samples, usually several hundred
Additional sources needednone
Not for (rules)none
Not for (text)Wrong choice when clusters reflect length, source style or language and not subject. Many valid partitions of the same documents exist, so a partition should be chosen by usefulness and checked.

Choosing and ordering

Kind of techniquePrimary technique, runs in step 5.3
Phase taganalyse
Other phase tagsnone
Kind of findingpattern; difference
Question answeredwhat
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to analyse

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: repeatable with fixed settings.

StakesCheckPerson involvedCounts as a different method
exploratoryrepeat runsnoneno
exploratoryspot checkoptionalno
decision-informingrepeat runsnoneno
decision-informingspot checkoptionalno
decision-informingsensitivity analysisnoneno
high-stakesrepeat runsnoneno
high-stakesspot checkoptionalno
high-stakessensitivity analysisnoneno
high-stakessecond technique with different assumptionsnoneconditional
high-stakesindependent reviewneededno

Failure modes and limits

Model suitabilityhuman judgement required
Typical model failureSummarises each cluster instead of testing whether it is coherent.
Error sourcesinput data; data transformation; statistical assumptions
Certainty limitsCertainty drops when clusters follow style or length (indirectness), when partitions differ across settings (inconsistency) and when clusters are not inspected (risk of bias).

Evidence and tag basis

Documented. The source describes fully automated and computer-assisted clustering. (Grimmer and Stewart 2013, Political Analysis)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.