startthat.work home

MM-12 Probabilistic topic modelling (LDA (latent Dirichlet allocation); STM (structural topic model); mixed-membership model)

Model each document as a mixture of topics and each topic as a distribution over words, and produce topic word lists and document-topic proportions.

Measure and modelEvidence: TestedPrimary technique, runs in step 5.3Strongest statement: describe

What it does

Models each document as a mixture of topics and each topic as a distribution over words. Artefact: topic word lists and document-topic proportions. Read top words, exemplar documents, and proportions over time or by source. The label is the analyst's interpretation.

When to use it

Use when you have a large set of long texts and want to discover what they talk about, with no fixed categories.

When it does not fit

Fits long texts that contain several topics, when discovery and not fixed coding is wanted. Misfit: short texts, unstable runs, uninterpretable topics. Validation needs semantic, predictive and convergent evidence.

Result format

The result has a word list for each topic and the topic proportions for each document, with the proportions shown over time or by source. It gives exemplar documents for each topic and a label for each topic that is stated to be the analyst's interpretation.

Requirements on the material

PropertyRequirement typeRequirement
StructureApplies in some casesrequires=unstructured; requires=semi-structured
Content typeAlways appliesrequires=text
Unit of analysisAlways appliesrequires=document
Level of detail availableAlways appliesrequires=record-level
Amount of materialApplies in some casesrequires=needs automation
Appraises source credibilityApplies in some casesno

Not restricted on: Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialat least several hundred documents of reasonable length
Additional sources needednone
Not for (rules)none
Not for (text)Wrong choice for short texts, for runs that give different topics each time or for topics that cannot be interpreted. Topic labels are the analyst's interpretation and not a finding.

Choosing and ordering

Kind of techniquePrimary technique, runs in step 5.3
Phase taganalyse
Other phase tagsnone
Kind of findingpattern; trend
Question answeredwhat
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to analyse

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: repeatable with fixed settings.

StakesCheckPerson involvedCounts as a different method
exploratoryrepeat runsnoneno
exploratoryspot checkoptionalno
decision-informingrepeat runsnoneno
decision-informingspot checkoptionalno
decision-informingsensitivity analysisnoneno
decision-informinghuman review sampleneeded; left out below high stakesno
high-stakesrepeat runsnoneno
high-stakesspot checkoptionalno
high-stakessensitivity analysisnoneno
high-stakeshuman review sampleneededno
high-stakessecond technique with different assumptionsnoneconditional
high-stakesexpert reviewneededno

Failure modes and limits

Model suitabilityhuman judgement required
Typical model failureLabels word lists with confident themes and overstates a theme found in few documents.
Error sourcesinput data; data transformation; statistical assumptions; coder or rater judgement
Certainty limitsCertainty drops when topics change across runs or topic counts (inconsistency), when labels rest on top words only (risk of bias) and when topics have no external validation (indirectness).

Evidence and tag basis

Tested. Models that score better on held-out likelihood "may infer less semantically meaningful topics" in large user studies (2009). LDA assumes each document is a mixture over a shared topic set, and STM adds document covariates such as time. (Chang et al. 2009, NeurIPS; Blei et al. 2003, Latent Dirichlet allocation; Roberts, Stewart and Tingley, structural topic models; Grimmer and Stewart 2013, Political Analysis)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.