MM-12 Probabilistic topic modelling (LDA (latent Dirichlet allocation); STM (structural topic model); mixed-membership model)
Model each document as a mixture of topics and each topic as a distribution over words, and produce topic word lists and document-topic proportions.
What it does
Models each document as a mixture of topics and each topic as a distribution over words. Artefact: topic word lists and document-topic proportions. Read top words, exemplar documents, and proportions over time or by source. The label is the analyst's interpretation.
When to use it
Use when you have a large set of long texts and want to discover what they talk about, with no fixed categories.
When it does not fit
Fits long texts that contain several topics, when discovery and not fixed coding is wanted. Misfit: short texts, unstable runs, uninterpretable topics. Validation needs semantic, predictive and convergent evidence.
Result format
The result has a word list for each topic and the topic proportions for each document, with the proportions shown over time or by source. It gives exemplar documents for each topic and a label for each topic that is stated to be the analyst's interpretation.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Structure | Applies in some cases | requires=unstructured; requires=semi-structured |
| Content type | Always applies | requires=text |
| Unit of analysis | Always applies | requires=document |
| Level of detail available | Always applies | requires=record-level |
| Amount of material | Applies in some cases | requires=needs automation |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | at least several hundred documents of reasonable length |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Wrong choice for short texts, for runs that give different topics each time or for topics that cannot be interpreted. Topic labels are the analyst's interpretation and not a finding. |
Choosing and ordering
| Kind of technique | Primary technique, runs in step 5.3 |
|---|---|
| Phase tag | analyse |
| Other phase tags | none |
| Kind of finding | pattern; trend |
| Question answered | what |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to analyse |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: repeatable with fixed settings.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | repeat runs | none | no |
| exploratory | spot check | optional | no |
| decision-informing | repeat runs | none | no |
| decision-informing | spot check | optional | no |
| decision-informing | sensitivity analysis | none | no |
| decision-informing | human review sample | needed; left out below high stakes | no |
| high-stakes | repeat runs | none | no |
| high-stakes | spot check | optional | no |
| high-stakes | sensitivity analysis | none | no |
| high-stakes | human review sample | needed | no |
| high-stakes | second technique with different assumptions | none | conditional |
| high-stakes | expert review | needed | no |
Failure modes and limits
| Model suitability | human judgement required |
|---|---|
| Typical model failure | Labels word lists with confident themes and overstates a theme found in few documents. |
| Error sources | input data; data transformation; statistical assumptions; coder or rater judgement |
| Certainty limits | Certainty drops when topics change across runs or topic counts (inconsistency), when labels rest on top words only (risk of bias) and when topics have no external validation (indirectness). |
Related techniques
- should follow PE-02 Preprocessing sensitivity testing (text cleaning; tokenisation; stopword removal; stemming; boilerplate removal; preText): cleaning rules change topics, so sensitivity to them should be known
- pairs with CK-08 Semantic, predictive and convergent validity checks (construct validity checks): semantic, predictive and convergent checks validate topics
- substitutes for MM-13 Embedding-based topic modelling (BERTopic; Top2Vec; clustering-based topic models): when documents are long and multi-topic and the volume class is needs automation
Evidence and tag basis
Tested. Models that score better on held-out likelihood "may infer less semantically meaningful topics" in large user studies (2009). LDA assumes each document is a mixture over a shared topic set, and STM adds document covariates such as time. (Chang et al. 2009, NeurIPS; Blei et al. 2003, Latent Dirichlet allocation; Roberts, Stewart and Tingley, structural topic models; Grimmer and Stewart 2013, Political Analysis)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.