MM-16 Word embeddings for concept measurement and semantic change (diachronic embeddings; concept direction)
Measure how terms relate in meaning within a corpus or across periods and produce nearest neighbours, similarity to seed words and change scores.
What it does
Measures how terms relate in meaning within a corpus or across periods. Artefact: nearest neighbours, similarity to seed words and change scores. Read the neighbours before the scores.
When to use it
Use when you want to know whether the meaning or use of a term has shifted across periods or groups, or how close two concepts are in a corpus.
When it does not fit
Needs enough text per group or period and a stable method. Small corpora give unstable vectors.
Result format
The result has the nearest neighbours of each term, the similarity of each term to the seed words, and a change score for each term across periods or groups. The neighbours are listed before the scores.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Structure | Applies in some cases | requires=unstructured; requires=semi-structured |
| Content type | Always applies | requires=text |
| Unit of analysis | Always applies | requires=dataset |
| Level of detail available | Always applies | requires=record-level |
| Amount of material | Applies in some cases | requires=needs automation |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | enough text in each period or group to train stable vectors, usually millions of words |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Wrong choice for small corpora, which give unstable vectors. Similarity scores are easy to over-read before the neighbours are read. |
Choosing and ordering
| Kind of technique | Primary technique, runs in step 5.3 |
|---|---|
| Phase tag | analyse |
| Other phase tags | none |
| Kind of finding | trend; difference |
| Question answered | what; how much |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to gather |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: repeatable with fixed settings.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | repeat runs | none | no |
| exploratory | spot check | optional | no |
| decision-informing | repeat runs | none | no |
| decision-informing | spot check | optional | no |
| decision-informing | sensitivity analysis | none | no |
| high-stakes | repeat runs | none | no |
| high-stakes | spot check | optional | no |
| high-stakes | sensitivity analysis | none | no |
| high-stakes | second technique with different assumptions | none | conditional |
| high-stakes | expert review | needed | no |
Failure modes and limits
| Model suitability | model plus code |
|---|---|
| Typical model failure | Reports semantic change from a model that has not seen the domain. |
| Error sources | input data; data transformation; statistical assumptions |
| Certainty limits | Certainty drops with small corpora (imprecision), vectors that change with training settings (inconsistency) and a model that has not seen the domain (indirectness). |
Related techniques
- should follow PE-02 Preprocessing sensitivity testing (text cleaning; tokenisation; stopword removal; stemming; boilerplate removal; preText): tokenisation and cleaning choices change the vectors
- pairs with PE-06 Keyword in context (KWIC; concordance): keyword in context shows how the term is used for each neighbour
- pairs with CK-08 Semantic, predictive and convergent validity checks (construct validity checks): predictive validity checks compare change scores with known events
Evidence and tag basis
Tested. Three embedding methods were tested against known historical shifts on six corpora, and less frequent and more polysemous words changed faster (2016, abstract). (Hamilton et al. 2016, arXiv 1605.09096)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.