PE-02 Preprocessing sensitivity testing (text cleaning; tokenisation; stopword removal; stemming; boilerplate removal; preText)
Convert raw text into countable units and re-run the main result under several defensible cleaning regimes, giving cleaned text, a document-term matrix, a rule log and a sensitivity result.
What it does
Converts raw text into countable units and re-runs the main result under several defensible cleaning regimes. Artefact: cleaned text or a document-term matrix, a log of every rule, and a sensitivity result. Read whether themes or rankings change across regimes.
When to use it
Use when you are about to count words or run an unsupervised model on raw text and need to know whether cleaning choices change the result.
When it does not fit
Use before any unsupervised model. A finding that changes across regimes is unstable. Whether neural embedding methods are equally sensitive was not confirmed.
Result format
The result has the cleaned text or document-term matrix, a log listing every cleaning rule applied, and a sensitivity table showing the main result under each cleaning regime. It ends by stating whether the themes or rankings changed across regimes.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Structure | Applies in some cases | requires=unstructured; requires=semi-structured |
| Content type | Always applies | requires=text |
| Unit of analysis | Always applies | requires=document |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | A tokeniser and stopword list for the language of the text. |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Whether neural embedding methods are equally sensitive to cleaning was not confirmed, so the technique may add little before an embedding method. |
Choosing and ordering
| Kind of technique | Preparation technique, runs in step 5.2 |
|---|---|
| Phase tag | verify and prepare |
| Other phase tags | validate |
| Kind of finding | risk; pattern |
| Question answered | what; how much |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to verify and prepare |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: deterministic.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | input validation | none | no |
| decision-informing | input validation | none | no |
| decision-informing | provenance log | none | no |
| decision-informing | spot check | optional | no |
| high-stakes | input validation | none | no |
| high-stakes | provenance log | none | no |
| high-stakes | spot check | optional | no |
| high-stakes | human review sample | needed | no |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | model plus code |
|---|---|
| Typical model failure | Applies one cleaning rule set without recording it, so the result cannot be reproduced. |
| Error sources | input data; data transformation |
| Certainty limits | A finding that changes across cleaning regimes is unstable and its certainty drops (inconsistency); the regimes tried may not cover the defensible ones (reporting bias). |
Related techniques
- should follow PE-03 Near-duplicate detection (shingling; MinHash; locality-sensitive hashing; text reuse detection): removing near-duplicates first stops repeats distorting counts
- pairs with MM-12 Probabilistic topic modelling (LDA (latent Dirichlet allocation); STM (structural topic model); mixed-membership model): preprocessing sensitivity tests the stability of a topic model result
- pairs with MM-14 Document clustering (computer-assisted clustering; k-means on document vectors): document clustering results can change with the cleaning rules
Evidence and tag basis
Tested. Preprocessing choices have "profound effects on the results of real models for real data" (abstract); the authors propose preText to measure the sensitivity. (Denny and Spirling 2018, Political Analysis)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.