startthat.work home

PE-08 Named entity recognition (NER; entity extraction)

Tag spans of text as people, organisations, places, dates and other types, giving a table of mentions with type and position and a tagged corpus.

Prepare and extractEvidence: DocumentedPreparation technique, runs in step 5.2Strongest statement: describe

What it does

Tags spans of text as people, organisations, places, dates and other types. Artefact: a table of mentions with type and position. Read which actors recur and in what contexts.

When to use it

Use when you need to know which people, organisations or places are named in a body of text and in what contexts.

When it does not fit

Entity types must match the question. Domain terms, such as company names in filings or agency names in policy documents, are often mistagged. No domain accuracy figures were verified.

Result format

The result is a table of mentions, one row per mention, giving the text span, the entity type and the position. It ends with the actors that recur and the contexts in which they appear.

Requirements on the material

PropertyRequirement typeRequirement
StructureApplies in some casesrequires=unstructured; requires=semi-structured
Content typeAlways appliesrequires=text
Unit of analysisAlways appliesrequires=document
Appraises source credibilityApplies in some casesno
Personal informationAlways appliesrequires=no personal information; requires=personal information

Not restricted on: Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Security classification, Licence.

Minimum materialA tagger or model for the language of the text, and entity types that match the question.
Additional sources needednone
Not for (rules)none
Not for (text)Domain terms such as company names in filings or agency names in policy documents are likely to be mistagged and the entity types do not fit the question.

Choosing and ordering

Kind of techniquePreparation technique, runs in step 5.2
Phase tagverify and prepare
Other phase tagsanalyse
Kind of findingpattern
Question answeredwho; what
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to verify and prepare

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: repeatable with fixed settings.

StakesCheckPerson involvedCounts as a different method
exploratoryspot checkoptionalno
decision-informingspot checkoptionalno
decision-informinghuman review sampleneeded; left out below high stakesno
decision-informinggold standard agreementconditional; left out below high stakesno
high-stakesspot checkoptionalno
high-stakeshuman review sampleneededno
high-stakesgold standard agreementconditionalno
high-stakessecond technique with different assumptionsnoneconditional
high-stakesindependent reviewneededno
high-stakesprovenance lognoneno

Failure modes and limits

Model suitabilitymodel plus code
Typical model failureTags domain terms with the wrong type and treats each distinct spelling of a name as a separate entity.
Error sourcesinput data; statistical assumptions; language model
Certainty limitsCertainty drops when accuracy has not been measured for the domain (indirectness), when entity types do not fit the question (indirectness), or when names vary (inconsistency).

Evidence and tag basis

Documented. A textbook chapter on sequence labelling for named entities is listed on the contents page; the chapter was not opened. (Jurafsky and Martin, SLP3 contents)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.