MM-11 Framing analysis at scale (frame detection; issue framing; media frames coding)
Identify which aspects of an issue each text emphasises using a defined frame scheme and produce frame labels with frame shares by source and period.
What it does
Identifies which aspects of an issue a text emphasises, using a defined frame scheme. Artefact: frame labels and frame shares by source and period. Read changes in the dominant frame and the frames that are absent.
When to use it
Use when you want to know how different sources or periods present the same issue, using a set of frames agreed beforehand.
When it does not fit
Needs a defined frame set that coders agree on. Misfit: frames invented after reading, so the scheme fits the finding.
Result format
The result has a frame label for each text and a table of frame shares by source and period. It names the changes in the dominant frame and the frames of the scheme that are absent.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Structure | Applies in some cases | requires=unstructured; requires=semi-structured |
| Content type | Always applies | requires=text |
| Unit of analysis | Always applies | requires=document |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | a fixed frame scheme that human coders can apply with acceptable agreement |
|---|---|
| Additional sources needed | reference structure |
| Not for (rules) | none |
| Not for (text) | Wrong choice when the frames were invented after reading the texts, because the scheme then fits the finding. Assigning a frame to every document, including those with none, is a misuse. |
Choosing and ordering
| Kind of technique | Primary technique, runs in step 5.3 |
|---|---|
| Phase tag | analyse |
| Other phase tags | none |
| Kind of finding | difference; trend; gap |
| Question answered | what; how much |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to frame |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | spot check | optional | no |
| decision-informing | spot check | optional | no |
| decision-informing | gold standard agreement | conditional; left out below high stakes | no |
| decision-informing | human review sample | needed; left out below high stakes | no |
| high-stakes | spot check | optional | no |
| high-stakes | gold standard agreement | conditional | no |
| high-stakes | human review sample | needed | no |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | human judgement required |
|---|---|
| Typical model failure | Assigns a frame to every document, including those that have none. |
| Error sources | input data; coder or rater judgement; language model |
| Certainty limits | Certainty drops when coders disagree on frames (risk of bias), when the frame set was fitted to the texts (indirectness) and when some frames are rare (imprecision). |
Related techniques
- pairs with CK-01 Inter-coder agreement statistics (percentage agreement; Cohen's kappa; weighted kappa; Scott's pi; Fleiss' kappa; Krippendorff's alpha; intercoder reliability testing): agreement statistics test whether coders apply the frame set the same way
- substitutes for HE1-15 Content analysis (conventional, directed and summative): when a fixed frame scheme exists and the volume class is needs automation
Evidence and tag basis
Tested. Frame detection was one of four tasks on which ChatGPT beat crowd workers in a 2,382-tweet study. The content of the Media Frames Corpus was not confirmed. (Gilardi et al., arXiv 2303.15056)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.