PE-09 Relation and event extraction (information extraction; open information extraction; temporal expression extraction)
Extract who did what to whom, with arguments and time, as structured rows, giving an event or relation table and a coded corpus.
What it does
Extracts who did what to whom, with arguments and time, as structured rows. Artefact: an event or relation table. Read counts by type and actor, and check a sample of rows against the source sentence.
When to use it
Use when you need counts of events or relations between actors and the source text reports them in sentences.
When it does not fit
The schema must be fixed before extraction. Misfit: the schema changes between documents, or one event reported by five outlets is counted five times.
Result format
The result is a table of events or relations, one row per extracted item, giving the actor, the action, the object and the time. It also gives counts by type and actor, and a sample of rows checked against the source sentence.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Structure | Applies in some cases | requires=unstructured; requires=semi-structured |
| Content type | Always applies | requires=text |
| Unit of analysis | Always applies | requires=document |
| Appraises source credibility | Applies in some cases | no |
| Personal information | Always applies | requires=no personal information; requires=personal information |
Not restricted on: Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Security classification, Licence.
| Minimum material | A fixed schema of event or relation types written before extraction. |
|---|---|
| Additional sources needed | reference structure |
| Not for (rules) | none |
| Not for (text) | The schema changes between documents, or one event reported by several outlets would be counted as several events. |
Choosing and ordering
| Kind of technique | Preparation technique, runs in step 5.2 |
|---|---|
| Phase tag | verify and prepare |
| Other phase tags | analyse |
| Kind of finding | pattern |
| Question answered | who; what |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to frame |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: repeatable with fixed settings.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | spot check | optional | no |
| decision-informing | spot check | optional | no |
| decision-informing | human review sample | needed; left out below high stakes | no |
| decision-informing | gold standard agreement | conditional; left out below high stakes | no |
| decision-informing | source-span citation | none | no |
| high-stakes | spot check | optional | no |
| high-stakes | human review sample | needed | no |
| high-stakes | gold standard agreement | conditional | no |
| high-stakes | source-span citation | none | no |
| high-stakes | independent review | needed | no |
| high-stakes | provenance log | none | no |
| high-stakes | prompt variation | none | no |
Failure modes and limits
| Model suitability | model plus code |
|---|---|
| Typical model failure | Follows an event codebook poorly when run zero-shot and counts one event once per outlet. |
| Error sources | input data; data transformation; language model |
| Certainty limits | Certainty drops when the schema is not fixed in advance (risk of bias), when zero-shot accuracy has not been measured (indirectness), or when duplicate reports inflate counts (risk of bias). |
Related techniques
- should follow PE-03 Near-duplicate detection (shingling; MinHash; locality-sensitive hashing; text reuse detection): removing repeated reports stops one event counting several times
- pairs with CK-02 Model versus human coding agreement protocol (LLM-as-coder reliability test; intra-coder agreement across runs): a model versus human agreement protocol measures extraction quality
- pairs with IS-01 Entity resolution and linking (record linkage; entity matching; entity linking; coreference resolution; blocking): actors named differently across reports need resolving
Evidence and tag basis
Tested. Open-weight models of 7 to 12 billion parameters followed event codebooks poorly zero-shot on protest and political violence coding, and supervised instruction tuning helped substantially (abstract, 2024). (Halterman and Keith, arXiv 2407.10747; Jurafsky and Martin, SLP3 contents)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.