IS-05 Truth discovery with source reliability weighting (truth finding; source reliability estimation)
Jointly estimates the true value of each object and the reliability of each source from conflicting claims and produces a truth per object and a weight per source.
What it does
Jointly estimates the true value of each object and the reliability of each source from conflicting claims. Artefact: a truth per object and a weight per source. Read low-weight sources and low-confidence truths.
When to use it
Use when many sources make conflicting claims about many objects and you need to know which values to believe and which sources to trust.
When it does not fit
Fits many objects claimed by many sources. Misfit: few claims per source, correlated sources, or objects with no conflict. Most methods start from uniform weights, which behaves like voting at first, and assume a source is equally reliable on every object. Evaluation needs ground truth, which is often absent.
Result format
The result has a table of the estimated true value for each object with a confidence level, and a table of the reliability weight for each source. It names the low-weight sources and the low-confidence truths.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Structure | Applies in some cases | requires=structured; requires=semi-structured |
| Content type | Always applies | requires=tabular; requires=text |
| Unit of analysis | Always applies | requires=claim |
| Level of detail available | Always applies | requires=aggregate tables |
| Number of sources | Always applies | requires=two or more |
| Whether sources can be joined | Always applies | requires=shared identifier available |
| Known data problems | Applies in some cases | requires=inconsistent |
| Producer interest | Changes the order | prefers=unknown; prefers=sponsored |
| Source dependence | Changes the order | prefers=independent |
| Appraises source credibility | Applies in some cases | yes |
Not restricted on: Variable types, Time structure, How the sample was chosen, Amount of material, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | many objects, each claimed by several sources |
|---|---|
| Additional sources needed | none |
| Not for (rules) | source_dependence=copied or derived :: correlated sources distort the reliability estimates |
| Not for (text) | Few claims per source, or objects with no conflict. Most methods start from uniform weights, which behaves like voting at first and assumes equal reliability on every object. Evaluation needs ground truth, which is often absent. |
Choosing and ordering
| Kind of technique | Primary technique, runs in step 5.3 |
|---|---|
| Phase tag | analyse |
| Other phase tags | none |
| Kind of finding | contradiction; option ranking |
| Question answered | what; who |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | proceed with limit stated |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: deterministic.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | input validation | none | no |
| decision-informing | input validation | none | no |
| decision-informing | spot check | optional | no |
| high-stakes | input validation | none | no |
| high-stakes | spot check | optional | no |
| high-stakes | independent review | needed | no |
| high-stakes | provenance log | none | no |
Failure modes and limits
| Model suitability | model plus code |
|---|---|
| Typical model failure | Reports source weights without a ground truth sample and treats agreement between copied sources as support. |
| Error sources | input data; statistical assumptions; source selection |
| Certainty limits | Certainty drops without ground truth to test the estimates, with dependent sources, and with few claims per source (imprecision and risk of bias). |
Related techniques
- should follow IS-06 Source dependence and copy detection (source independence analysis): dependence between sources must be checked first
- substitutes for IS-04 Data fusion with conflict resolution (majority voting; deciding and mediating resolution functions): when many claims per source allow reliability to be estimated
Evidence and tag basis
Documented. The survey groups methods as iterative, optimisation-based and probabilistic graphical model, and does not rank them. A Deep Web study found surprising inconsistency across sources it considered relatively clean (abstract only). (Li et al., A survey on truth discovery, arXiv 1505.02463; Li et al., Truth finding on the Deep Web, arXiv 1503.00303)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.