startthat.work home

MM-11 Framing analysis at scale (frame detection; issue framing; media frames coding)

Identify which aspects of an issue each text emphasises using a defined frame scheme and produce frame labels with frame shares by source and period.

Measure and modelEvidence: TestedPrimary technique, runs in step 5.3Strongest statement: describe

What it does

Identifies which aspects of an issue a text emphasises, using a defined frame scheme. Artefact: frame labels and frame shares by source and period. Read changes in the dominant frame and the frames that are absent.

When to use it

Use when you want to know how different sources or periods present the same issue, using a set of frames agreed beforehand.

When it does not fit

Needs a defined frame set that coders agree on. Misfit: frames invented after reading, so the scheme fits the finding.

Result format

The result has a frame label for each text and a table of frame shares by source and period. It names the changes in the dominant frame and the frames of the scheme that are absent.

Requirements on the material

PropertyRequirement typeRequirement
StructureApplies in some casesrequires=unstructured; requires=semi-structured
Content typeAlways appliesrequires=text
Unit of analysisAlways appliesrequires=document
Appraises source credibilityApplies in some casesno

Not restricted on: Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materiala fixed frame scheme that human coders can apply with acceptable agreement
Additional sources neededreference structure
Not for (rules)none
Not for (text)Wrong choice when the frames were invented after reading the texts, because the scheme then fits the finding. Assigning a frame to every document, including those with none, is a misuse.

Choosing and ordering

Kind of techniquePrimary technique, runs in step 5.3
Phase taganalyse
Other phase tagsnone
Kind of findingdifference; trend; gap
Question answeredwhat; how much
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to frame

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.

StakesCheckPerson involvedCounts as a different method
exploratoryspot checkoptionalno
decision-informingspot checkoptionalno
decision-informinggold standard agreementconditional; left out below high stakesno
decision-informinghuman review sampleneeded; left out below high stakesno
high-stakesspot checkoptionalno
high-stakesgold standard agreementconditionalno
high-stakeshuman review sampleneededno
high-stakesindependent reviewneededno

Failure modes and limits

Model suitabilityhuman judgement required
Typical model failureAssigns a frame to every document, including those that have none.
Error sourcesinput data; coder or rater judgement; language model
Certainty limitsCertainty drops when coders disagree on frames (risk of bias), when the frame set was fitted to the texts (indirectness) and when some frames are rare (imprecision).

Evidence and tag basis

Tested. Frame detection was one of four tasks on which ChatGPT beat crowd workers in a 2,382-tweet study. The content of the Media Frames Corpus was not confirmed. (Gilardi et al., arXiv 2303.15056)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.