MM-13 Embedding-based topic modelling (BERTopic; Top2Vec; clustering-based topic models)
Embed documents with a neural model, cluster the embeddings and describe each cluster by distinctive words, and produce topics with representative words, representative documents and an outlier group.
What it does
Embeds documents with a neural model, clusters the embeddings and describes each cluster with distinctive words. Artefact: topics with representative words and documents, and an outlier group. Read the size of the outlier group and the exemplar documents.
When to use it
Use when you want to find themes in a varied or short-text corpus and a neural embedding model handles its language and domain.
When it does not fit
Needs a varied corpus and an embedding model that handles the language and domain. Misfit: a large outlier share, or one theme split into many topics.
Result format
The result is a table of topics, each with its representative words and representative documents, and an outlier group. It states the size of the outlier group.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Structure | Applies in some cases | requires=unstructured; requires=semi-structured |
| Content type | Always applies | requires=text |
| Unit of analysis | Always applies | requires=document |
| Level of detail available | Always applies | requires=record-level |
| Amount of material | Applies in some cases | requires=needs automation |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | at least several hundred varied documents and an embedding model suited to the language |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Wrong choice when a large share of documents fall in the outlier group or one theme is split into many topics. Automated coherence scores were not validated against human judgement. |
Choosing and ordering
| Kind of technique | Primary technique, runs in step 5.3 |
|---|---|
| Phase tag | analyse |
| Other phase tags | none |
| Kind of finding | pattern |
| Question answered | what |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to analyse |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: repeatable with fixed settings.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | repeat runs | none | no |
| exploratory | spot check | optional | no |
| decision-informing | repeat runs | none | no |
| decision-informing | spot check | optional | no |
| decision-informing | sensitivity analysis | none | no |
| decision-informing | human review sample | needed; left out below high stakes | no |
| high-stakes | repeat runs | none | no |
| high-stakes | spot check | optional | no |
| high-stakes | sensitivity analysis | none | no |
| high-stakes | human review sample | needed | no |
| high-stakes | second technique with different assumptions | none | conditional |
| high-stakes | expert review | needed | no |
Failure modes and limits
| Model suitability | human judgement required |
|---|---|
| Typical model failure | Reports the number of clusters as the number of real themes. |
| Error sources | input data; data transformation; statistical assumptions; language model; coder or rater judgement |
| Certainty limits | Certainty drops with a large outlier group (imprecision), results that change with embedding model or clustering settings (inconsistency) and benchmark claims that come only from the method authors (risk of bias). |
Related techniques
- pairs with CK-08 Semantic, predictive and convergent validity checks (construct validity checks): human checks of topic coherence are needed because automated scores are not validated
- substitutes for MM-12 Probabilistic topic modelling (LDA (latent Dirichlet allocation); STM (structural topic model); mixed-membership model): when texts are short or vocabulary is varied and the volume class is needs automation
Evidence and tag basis
Documented. The two papers report the authors' own benchmark claims. Automated coherence metrics were not validated against human judgement for neural topic models, and sometimes declared a winning model where human evaluation did not. (BERTopic, arXiv 2203.05794; Top2Vec, arXiv 2008.09470; Hoyle et al. 2021, arXiv 2107.02173)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.