startthat.work home

MM-15 Computer-assisted theme building (reading-assisted thematic analysis; human-guided topic labelling)

Have a person read exemplars from computer-generated groups, then merge, split and name them, and produce a named theme set with exemplar quotations and a count per theme.

Measure and modelEvidence: DocumentedPrimary technique, runs in step 5.3Strongest statement: describe

What it does

A person reads exemplars from computer-generated groups, then merges, splits and names them and records why. Artefact: a named theme set with exemplar quotations and a count per theme. Read the unplaced residual.

When to use it

Use when computer-generated groups of text need to be turned into named themes that a reader can trust.

When it does not fit

Check themes against documents not used to form them. Misfit: themes named from top words only.

Result format

The result is a named set of themes, each with exemplar quotations, a count of documents and a record of why the group was merged, split or named. It lists the residual documents that were not placed in any theme.

Requirements on the material

PropertyRequirement typeRequirement
StructureApplies in some casesrequires=unstructured; requires=semi-structured
Content typeAlways appliesrequires=text
Unit of analysisAlways appliesrequires=document
Level of detail availableAlways appliesrequires=record-level
Appraises source credibilityApplies in some casesno

Not restricted on: Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialcomputer-generated groups with at least a few exemplar documents each
Additional sources needednone
Not for (rules)none
Not for (text)Wrong choice when themes are named from top words alone, without reading documents. Themes should be checked against documents that were not used to form them.

Choosing and ordering

Kind of techniquePrimary technique, runs in step 5.3
Phase taganalyse
Other phase tagsconclude
Kind of findingpattern; gap
Question answeredwhat
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to analyse

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.

StakesCheckPerson involvedCounts as a different method
exploratoryspot checkoptionalno
decision-informingspot checkoptionalno
decision-informinghuman review sampleneeded; left out below high stakesno
high-stakesspot checkoptionalno
high-stakeshuman review sampleneededno
high-stakesindependent reviewneededno
high-stakesassumption and limitation statementnoneno

Failure modes and limits

Model suitabilityhuman judgement required
Typical model failureLabels clusters fluently without reading their contents.
Error sourcesinput data; coder or rater judgement
Certainty limitsCertainty drops when themes were not tested on unseen documents (risk of bias), when the unplaced residual is large (imprecision) and when one reader made all decisions (inconsistency).

Evidence and tag basis

Documented. Semantic validity means human rating of whether the groups are coherent. (Grimmer and Stewart 2013, Political Analysis)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.