MM-09 Sentiment analysis (polarity detection; opinion mining; affect lexicon)
Score text as positive, negative or neutral, or by emotion, and produce scores per document with aggregates by group or period.
What it does
Scores text as positive, negative or neutral, or by emotion. Artefact: scores per document and aggregates by group or period. Read the distribution and a sample of high and low scores.
When to use it
Use when tone or emotion in text is itself the quantity you want to compare across groups or periods.
When it does not fit
Fits when tone is the quantity of interest. Misfit: controlled language in policy documents and filings, or negation and sarcasm that reverse a score. Off-the-shelf lexicons need validation in the domain.
Result format
The result has a sentiment score for each document and aggregates by group or period. It shows the distribution of scores and a sample of high-scoring and low-scoring documents.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Structure | Applies in some cases | requires=unstructured; requires=semi-structured |
| Content type | Always applies | requires=text |
| Unit of analysis | Always applies | requires=document |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | enough documents per group or period to compare a distribution, usually 30 or more |
|---|---|
| Additional sources needed | none |
| Not for (rules) | document_genre=legal or regulatory text|government or agency report :: controlled language in policy documents and filings gives uninformative tone scores |
| Not for (text) | Wrong choice when negation or sarcasm reverses scores, or when tone is being read as position. Off-the-shelf word lists need validation in the domain. |
Choosing and ordering
| Kind of technique | Primary technique, runs in step 5.3 |
|---|---|
| Phase tag | analyse |
| Other phase tags | none |
| Kind of finding | difference; trend |
| Question answered | how much |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | proceed with limit stated |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: repeatable with fixed settings.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | spot check | optional | no |
| decision-informing | spot check | optional | no |
| decision-informing | gold standard agreement | conditional; left out below high stakes | no |
| high-stakes | spot check | optional | no |
| high-stakes | gold standard agreement | conditional | no |
| high-stakes | second technique with different assumptions | none | conditional |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | model plus code |
|---|---|
| Typical model failure | Reports tone as if it were position. |
| Error sources | input data; data transformation; statistical assumptions |
| Certainty limits | Certainty drops when the lexicon or model was built for another domain (indirectness) and when scores are not checked against human ratings (risk of bias). |
Related techniques
- should follow PE-02 Preprocessing sensitivity testing (text cleaning; tokenisation; stopword removal; stemming; boilerplate removal; preText): cleaning choices change scores
- pairs with CK-08 Semantic, predictive and convergent validity checks (construct validity checks): validity checks test whether scores behave as tone
- pairs with MM-10 Stance detection (position detection): tone is not stance, so stance detection answers the position question
Evidence and tag basis
Documented. A text can be negative in wording and still support its target. (Mohammad et al. 2016, arXiv 1605.01655; Grimmer and Stewart 2013, Political Analysis)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.