startthat.work home

PE-10 Schema-based structured extraction with an LLM (LLM (large language model) information extraction; form filling from documents)

Fill a fixed set of fields, such as amount, date, party, stated risk or decision, for each document, giving one record per document and the share of missing or not stated values.

Prepare and extractEvidence: TestedPreparation technique, runs in step 5.2Strongest statement: describe

What it does

Fills a fixed set of fields, such as amount, date, party, stated risk or decision, for each document. Artefact: one record per document. Read the missing and "not stated" rates and compare a sample with the source.

When to use it

Use when you need the same facts pulled from many reports or filings into one table.

When it does not fit

Fields must be explicit in the text, and inferred fields should be flagged. Validate each task against human labels. No error rates for extraction from filings were found.

Result format

The result is one record per document with the same fixed fields in every record, such as amount, date, party, stated risk or decision. It reports the missing and not stated rate for each field and a sample of records compared with the source text.

Requirements on the material

PropertyRequirement typeRequirement
StructureApplies in some casesrequires=unstructured; requires=semi-structured
Content typeAlways appliesrequires=text
Unit of analysisAlways appliesrequires=document
Appraises source credibilityApplies in some casesno

Not restricted on: Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialA fixed set of fields, defined before extraction, that are stated explicitly in the text.
Additional sources neededreference structure
Not for (rules)none
Not for (text)Fields are not explicit in the text and the model must infer them without those values being flagged as inferred.

Choosing and ordering

Kind of techniquePreparation technique, runs in step 5.2
Phase tagverify and prepare
Other phase tagsanalyse
Kind of findinggap; pattern
Question answeredhow much; what
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetproceed with limit stated

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: repeatable with fixed settings.

StakesCheckPerson involvedCounts as a different method
exploratoryspot checkoptionalno
decision-informingspot checkoptionalno
decision-informinghuman review sampleneeded; left out below high stakesno
decision-informingsource-span citationnoneno
decision-informingprompt variationnoneno
high-stakesspot checkoptionalno
high-stakeshuman review sampleneededno
high-stakessource-span citationnoneno
high-stakesprompt variationnoneno
high-stakesgold standard agreementconditionalno
high-stakesprovenance lognoneno
high-stakesindependent reviewneededno

Failure modes and limits

Model suitabilitymodel plus code
Typical model failureInvents a value for a field that the document does not state instead of returning not stated.
Error sourcesinput data; data transformation; language model
Certainty limitsCertainty drops when accuracy was not measured for the task (indirectness), when fields are inferred (risk of bias), or when not stated rates are high (imprecision).

Evidence and tag basis

Tested. GPT-4 performance across 27 annotation tasks and 11 datasets depended heavily on dataset and task (2023, abstract). (Pangakis et al., arXiv 2306.00176; Halterman and Keith, arXiv 2407.10747)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.