startthat.work home

PE-09 Relation and event extraction (information extraction; open information extraction; temporal expression extraction)

Extract who did what to whom, with arguments and time, as structured rows, giving an event or relation table and a coded corpus.

Prepare and extractEvidence: TestedPreparation technique, runs in step 5.2Strongest statement: describe

What it does

Extracts who did what to whom, with arguments and time, as structured rows. Artefact: an event or relation table. Read counts by type and actor, and check a sample of rows against the source sentence.

When to use it

Use when you need counts of events or relations between actors and the source text reports them in sentences.

When it does not fit

The schema must be fixed before extraction. Misfit: the schema changes between documents, or one event reported by five outlets is counted five times.

Result format

The result is a table of events or relations, one row per extracted item, giving the actor, the action, the object and the time. It also gives counts by type and actor, and a sample of rows checked against the source sentence.

Requirements on the material

PropertyRequirement typeRequirement
StructureApplies in some casesrequires=unstructured; requires=semi-structured
Content typeAlways appliesrequires=text
Unit of analysisAlways appliesrequires=document
Appraises source credibilityApplies in some casesno
Personal informationAlways appliesrequires=no personal information; requires=personal information

Not restricted on: Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Security classification, Licence.

Minimum materialA fixed schema of event or relation types written before extraction.
Additional sources neededreference structure
Not for (rules)none
Not for (text)The schema changes between documents, or one event reported by several outlets would be counted as several events.

Choosing and ordering

Kind of techniquePreparation technique, runs in step 5.2
Phase tagverify and prepare
Other phase tagsanalyse
Kind of findingpattern
Question answeredwho; what
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to frame

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: repeatable with fixed settings.

StakesCheckPerson involvedCounts as a different method
exploratoryspot checkoptionalno
decision-informingspot checkoptionalno
decision-informinghuman review sampleneeded; left out below high stakesno
decision-informinggold standard agreementconditional; left out below high stakesno
decision-informingsource-span citationnoneno
high-stakesspot checkoptionalno
high-stakeshuman review sampleneededno
high-stakesgold standard agreementconditionalno
high-stakessource-span citationnoneno
high-stakesindependent reviewneededno
high-stakesprovenance lognoneno
high-stakesprompt variationnoneno

Failure modes and limits

Model suitabilitymodel plus code
Typical model failureFollows an event codebook poorly when run zero-shot and counts one event once per outlet.
Error sourcesinput data; data transformation; language model
Certainty limitsCertainty drops when the schema is not fixed in advance (risk of bias), when zero-shot accuracy has not been measured (indirectness), or when duplicate reports inflate counts (risk of bias).

Evidence and tag basis

Tested. Open-weight models of 7 to 12 billion parameters followed event codebooks poorly zero-shot on protest and political violence coding, and supervised instruction tuning helped substantially (abstract, 2024). (Halterman and Keith, arXiv 2407.10747; Jurafsky and Martin, SLP3 contents)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.