PE-07 Collocation and n-gram extraction (phrase detection; association measures)
Find word sequences that occur together more often than chance, giving ranked phrases with association scores as a feature set.
What it does
Finds word sequences that occur together more often than chance. Artefact: ranked phrases with association scores. Read recurring named concepts, fixed expressions and repeated boilerplate.
When to use it
Use when you need to find multi-word terms and fixed expressions before dictionary or topic work.
When it does not fit
Use to find multi-word terms before dictionary or topic work. Boilerplate and repeated headers produce top-ranked phrases. A low default minimum count means rare sequences are scored.
Result format
The result is a ranked list of phrases with an association score for each. It names the recurring concepts, fixed expressions and repeated boilerplate among them.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Structure | Applies in some cases | requires=unstructured; requires=semi-structured |
| Content type | Always applies | requires=text |
| Unit of analysis | Always applies | requires=document |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | A minimum phrase count set above the default of 2 for the size of the corpus. |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Repeated headers and boilerplate have not been removed, so they would rank as the top phrases. |
Choosing and ordering
| Kind of technique | Preparation technique, runs in step 5.2 |
|---|---|
| Phase tag | verify and prepare |
| Other phase tags | analyse |
| Kind of finding | pattern |
| Question answered | what |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to verify and prepare |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: deterministic.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | input validation | none | no |
| decision-informing | input validation | none | no |
| decision-informing | spot check | optional | no |
| decision-informing | sensitivity analysis | none | no |
| high-stakes | input validation | none | no |
| high-stakes | spot check | optional | no |
| high-stakes | sensitivity analysis | none | no |
| high-stakes | human review sample | needed | no |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | model plus code |
|---|---|
| Typical model failure | Reports boilerplate and repeated headers as recurring concepts. |
| Error sources | input data; data transformation; statistical assumptions |
| Certainty limits | Certainty drops when the minimum count is low so rare sequences are scored (imprecision) or when boilerplate remains (risk of bias). |
Related techniques
- should follow PE-02 Preprocessing sensitivity testing (text cleaning; tokenisation; stopword removal; stemming; boilerplate removal; preText): cleaning and boilerplate removal stop repeated headers ranking as phrases
- pairs with PE-06 Keyword in context (KWIC; concordance): reading top phrases in context shows what they mean
Evidence and tag basis
Documented. In quanteda the score is a lambda statistic with a Wald z value, the default minimum count is 2 and the default size is 2. (quanteda textstat_collocations)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.