startthat.work home

PE-02 Preprocessing sensitivity testing (text cleaning; tokenisation; stopword removal; stemming; boilerplate removal; preText)

Convert raw text into countable units and re-run the main result under several defensible cleaning regimes, giving cleaned text, a document-term matrix, a rule log and a sensitivity result.

Prepare and extractEvidence: TestedPreparation technique, runs in step 5.2Strongest statement: describe

What it does

Converts raw text into countable units and re-runs the main result under several defensible cleaning regimes. Artefact: cleaned text or a document-term matrix, a log of every rule, and a sensitivity result. Read whether themes or rankings change across regimes.

When to use it

Use when you are about to count words or run an unsupervised model on raw text and need to know whether cleaning choices change the result.

When it does not fit

Use before any unsupervised model. A finding that changes across regimes is unstable. Whether neural embedding methods are equally sensitive was not confirmed.

Result format

The result has the cleaned text or document-term matrix, a log listing every cleaning rule applied, and a sensitivity table showing the main result under each cleaning regime. It ends by stating whether the themes or rankings changed across regimes.

Requirements on the material

PropertyRequirement typeRequirement
StructureApplies in some casesrequires=unstructured; requires=semi-structured
Content typeAlways appliesrequires=text
Unit of analysisAlways appliesrequires=document
Appraises source credibilityApplies in some casesno

Not restricted on: Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialA tokeniser and stopword list for the language of the text.
Additional sources needednone
Not for (rules)none
Not for (text)Whether neural embedding methods are equally sensitive to cleaning was not confirmed, so the technique may add little before an embedding method.

Choosing and ordering

Kind of techniquePreparation technique, runs in step 5.2
Phase tagverify and prepare
Other phase tagsvalidate
Kind of findingrisk; pattern
Question answeredwhat; how much
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to verify and prepare

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: deterministic.

StakesCheckPerson involvedCounts as a different method
exploratoryinput validationnoneno
decision-informinginput validationnoneno
decision-informingprovenance lognoneno
decision-informingspot checkoptionalno
high-stakesinput validationnoneno
high-stakesprovenance lognoneno
high-stakesspot checkoptionalno
high-stakeshuman review sampleneededno
high-stakesindependent reviewneededno

Failure modes and limits

Model suitabilitymodel plus code
Typical model failureApplies one cleaning rule set without recording it, so the result cannot be reproduced.
Error sourcesinput data; data transformation
Certainty limitsA finding that changes across cleaning regimes is unstable and its certainty drops (inconsistency); the regimes tried may not cover the defensible ones (reporting bias).

Evidence and tag basis

Tested. Preprocessing choices have "profound effects on the results of real models for real data" (abstract); the authors propose preText to measure the sensitivity. (Denny and Spirling 2018, Political Analysis)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.