startthat.work home

MM-13 Embedding-based topic modelling (BERTopic; Top2Vec; clustering-based topic models)

Embed documents with a neural model, cluster the embeddings and describe each cluster by distinctive words, and produce topics with representative words, representative documents and an outlier group.

Measure and modelEvidence: DocumentedPrimary technique, runs in step 5.3Strongest statement: describe

What it does

Embeds documents with a neural model, clusters the embeddings and describes each cluster with distinctive words. Artefact: topics with representative words and documents, and an outlier group. Read the size of the outlier group and the exemplar documents.

When to use it

Use when you want to find themes in a varied or short-text corpus and a neural embedding model handles its language and domain.

When it does not fit

Needs a varied corpus and an embedding model that handles the language and domain. Misfit: a large outlier share, or one theme split into many topics.

Result format

The result is a table of topics, each with its representative words and representative documents, and an outlier group. It states the size of the outlier group.

Requirements on the material

PropertyRequirement typeRequirement
StructureApplies in some casesrequires=unstructured; requires=semi-structured
Content typeAlways appliesrequires=text
Unit of analysisAlways appliesrequires=document
Level of detail availableAlways appliesrequires=record-level
Amount of materialApplies in some casesrequires=needs automation
Appraises source credibilityApplies in some casesno

Not restricted on: Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialat least several hundred varied documents and an embedding model suited to the language
Additional sources needednone
Not for (rules)none
Not for (text)Wrong choice when a large share of documents fall in the outlier group or one theme is split into many topics. Automated coherence scores were not validated against human judgement.

Choosing and ordering

Kind of techniquePrimary technique, runs in step 5.3
Phase taganalyse
Other phase tagsnone
Kind of findingpattern
Question answeredwhat
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to analyse

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: repeatable with fixed settings.

StakesCheckPerson involvedCounts as a different method
exploratoryrepeat runsnoneno
exploratoryspot checkoptionalno
decision-informingrepeat runsnoneno
decision-informingspot checkoptionalno
decision-informingsensitivity analysisnoneno
decision-informinghuman review sampleneeded; left out below high stakesno
high-stakesrepeat runsnoneno
high-stakesspot checkoptionalno
high-stakessensitivity analysisnoneno
high-stakeshuman review sampleneededno
high-stakessecond technique with different assumptionsnoneconditional
high-stakesexpert reviewneededno

Failure modes and limits

Model suitabilityhuman judgement required
Typical model failureReports the number of clusters as the number of real themes.
Error sourcesinput data; data transformation; statistical assumptions; language model; coder or rater judgement
Certainty limitsCertainty drops with a large outlier group (imprecision), results that change with embedding model or clustering settings (inconsistency) and benchmark claims that come only from the method authors (risk of bias).

Evidence and tag basis

Documented. The two papers report the authors' own benchmark claims. Automated coherence metrics were not validated against human judgement for neural topic models, and sometimes declared a winning model where human evaluation did not. (BERTopic, arXiv 2203.05794; Top2Vec, arXiv 2008.09470; Hoyle et al. 2021, arXiv 2107.02173)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.