MM-10 Stance detection (position detection)
Label whether each text favours, opposes or is neutral toward a named target and produce a stance label per text and target.
What it does
Labels whether an author favours, opposes or is neutral toward a named target. Artefact: a stance label per text and target. Read stance by source and over time, and the share of "none" labels.
When to use it
Use when you need to know who supports or opposes a named policy, person or proposal and how that changes by source or period.
When it does not fit
The target must be explicit and defined. Misfit: the target is not mentioned in the text, or reported speech is mistaken for the author's stance.
Result format
The result is a table with one row per text and target, giving the stance label of favour, oppose, neutral or none. It reports stance by source and over time, and the share of none labels.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Structure | Applies in some cases | requires=unstructured; requires=semi-structured |
| Content type | Always applies | requires=text |
| Unit of analysis | Always applies | requires=document |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | a defined target and enough texts that mention it, usually 100 or more |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Wrong choice when the target is not mentioned in the text. Reported speech is easily mistaken for the author's own stance. |
Choosing and ordering
| Kind of technique | Primary technique, runs in step 5.3 |
|---|---|
| Phase tag | analyse |
| Other phase tags | none |
| Kind of finding | difference; trend |
| Question answered | who; how much |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to frame |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: repeatable with fixed settings.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | spot check | optional | no |
| exploratory | repeat runs | none | no |
| decision-informing | spot check | optional | no |
| decision-informing | repeat runs | none | no |
| decision-informing | gold standard agreement | conditional; left out below high stakes | no |
| decision-informing | human review sample | needed; left out below high stakes | no |
| high-stakes | spot check | optional | no |
| high-stakes | repeat runs | none | no |
| high-stakes | gold standard agreement | conditional | no |
| high-stakes | human review sample | needed | no |
| high-stakes | prompt variation | none | no |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | model plus code |
|---|---|
| Typical model failure | Attributes a quoted position to the quoting author. |
| Error sources | input data; language model; coder or rater judgement |
| Certainty limits | Certainty drops when evidence rests on a single dataset (indirectness), when the target is implicit (risk of bias) and when the none label is common (imprecision). |
Related techniques
- should follow PE-08 Named entity recognition (NER; entity extraction): entity tagging finds where the target is mentioned
- pairs with CK-02 Model versus human coding agreement protocol (LLM-as-coder reliability test; intra-coder agreement across runs): model labels need agreement checks against human coders
- pairs with MM-09 Sentiment analysis (polarity detection; opinion mining; affect lexicon): tone and stance are different and should be kept apart
Evidence and tag basis
Tested. ChatGPT beat crowd workers on stance and three other tasks on 2,382 tweets, and GPT-4 was reported more accurate than experts and crowd workers on one election dataset. Both are single-dataset results, and the second page gave no figures. (Gilardi et al., arXiv 2303.15056; Törnberg, arXiv 2304.06588; Mohammad et al. 2016, arXiv 1605.01655)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.