IS-01 Entity resolution and linking (record linkage; entity matching; entity linking; coreference resolution; blocking)
Decides which records or text mentions refer to the same entity by comparing attributes, with no shared identifier required, and produces match decisions or linked clusters.
What it does
Decides which records or text mentions refer to the same entity by comparing attributes, with no shared identifier required. Steps: pre-processing, indexing (blocking), comparison, classification and evaluation. Artefact: match decisions or linked clusters. Read the match count, the uncertain pairs and the unresolved rate before counting actors.
When to use it
Use when the same people, organisations or places appear in different sources under different names or spellings and you need one list.
When it does not fit
Fits when records share enough comparable attributes. Misfit: sparse, changing or unreliable attributes. Blocking needs schema knowledge unless a schema-agnostic method is used. No accuracy figures for matching without identifiers were found.
Result format
The result is a set of match decisions or linked clusters, with the match count, the list of uncertain pairs and the unresolved rate. It gives these figures before any count of actors.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Content type | Always applies | requires=text; requires=tabular |
| Unit of analysis | Always applies | requires=record |
| Level of detail available | Always applies | requires=record-level |
| Number of sources | Always applies | requires=two or more |
| Whether sources can be joined | Always applies | requires=no shared identifier; requires=shared identifier available |
| Appraises source credibility | Applies in some cases | no |
| Personal information | Always applies | requires=no personal information; requires=personal information |
Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Security classification, Licence.
| Minimum material | records that share enough comparable attributes |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Attributes are sparse, changing or unreliable. Blocking needs schema knowledge unless a schema-agnostic method is used, and no accuracy figures for matching without identifiers were found. |
Choosing and ordering
| Kind of technique | Preparation technique, runs in step 5.2 |
|---|---|
| Phase tag | verify and prepare |
| Other phase tags | analyse |
| Kind of finding | pattern; gap |
| Question answered | who; what |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | proceed with limit stated |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: repeatable with fixed settings.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | spot check | optional | no |
| decision-informing | spot check | optional | no |
| decision-informing | repeat runs | none | no |
| decision-informing | human review sample | needed; left out below high stakes | no |
| high-stakes | spot check | optional | no |
| high-stakes | repeat runs | none | no |
| high-stakes | human review sample | needed | no |
| high-stakes | second technique with different assumptions | none | conditional |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | model plus code |
|---|---|
| Typical model failure | Merges different entities with similar names, and does not report the uncertain pairs or the unresolved rate. |
| Error sources | input data; data transformation; statistical assumptions |
| Certainty limits | Certainty drops with a high share of uncertain pairs, no labelled sample to measure match accuracy, and attributes that change over time (imprecision and risk of bias). |
Related techniques
- should follow PE-08 Named entity recognition (NER; entity extraction): text mentions need extracting before they can be linked
- pairs with IS-04 Data fusion with conflict resolution (majority voting; deciding and mediating resolution functions): fusion needs reliably resolved entities
Evidence and tag basis
Documented. A publisher note says an off-the-shelf matching system usually needs substantial adaptation. Only a book description, a survey abstract and a textbook contents page were read. (Christen, Data Matching, Springer book page; Papadakis et al., arXiv 1905.06167; Jurafsky and Martin, SLP3 contents)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.