startthat.work home

IS-05 Truth discovery with source reliability weighting (truth finding; source reliability estimation)

Jointly estimates the true value of each object and the reliability of each source from conflicting claims and produces a truth per object and a weight per source.

Integrate across sourcesEvidence: DocumentedPrimary technique, runs in step 5.3Strongest statement: describe

What it does

Jointly estimates the true value of each object and the reliability of each source from conflicting claims. Artefact: a truth per object and a weight per source. Read low-weight sources and low-confidence truths.

When to use it

Use when many sources make conflicting claims about many objects and you need to know which values to believe and which sources to trust.

When it does not fit

Fits many objects claimed by many sources. Misfit: few claims per source, correlated sources, or objects with no conflict. Most methods start from uniform weights, which behaves like voting at first, and assume a source is equally reliable on every object. Evaluation needs ground truth, which is often absent.

Result format

The result has a table of the estimated true value for each object with a confidence level, and a table of the reliability weight for each source. It names the low-weight sources and the low-confidence truths.

Requirements on the material

PropertyRequirement typeRequirement
StructureApplies in some casesrequires=structured; requires=semi-structured
Content typeAlways appliesrequires=tabular; requires=text
Unit of analysisAlways appliesrequires=claim
Level of detail availableAlways appliesrequires=aggregate tables
Number of sourcesAlways appliesrequires=two or more
Whether sources can be joinedAlways appliesrequires=shared identifier available
Known data problemsApplies in some casesrequires=inconsistent
Producer interestChanges the orderprefers=unknown; prefers=sponsored
Source dependenceChanges the orderprefers=independent
Appraises source credibilityApplies in some casesyes

Not restricted on: Variable types, Time structure, How the sample was chosen, Amount of material, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialmany objects, each claimed by several sources
Additional sources needednone
Not for (rules)source_dependence=copied or derived :: correlated sources distort the reliability estimates
Not for (text)Few claims per source, or objects with no conflict. Most methods start from uniform weights, which behaves like voting at first and assumes equal reliability on every object. Evaluation needs ground truth, which is often absent.

Choosing and ordering

Kind of techniquePrimary technique, runs in step 5.3
Phase taganalyse
Other phase tagsnone
Kind of findingcontradiction; option ranking
Question answeredwhat; who
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetproceed with limit stated

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: deterministic.

StakesCheckPerson involvedCounts as a different method
exploratoryinput validationnoneno
decision-informinginput validationnoneno
decision-informingspot checkoptionalno
high-stakesinput validationnoneno
high-stakesspot checkoptionalno
high-stakesindependent reviewneededno
high-stakesprovenance lognoneno

Failure modes and limits

Model suitabilitymodel plus code
Typical model failureReports source weights without a ground truth sample and treats agreement between copied sources as support.
Error sourcesinput data; statistical assumptions; source selection
Certainty limitsCertainty drops without ground truth to test the estimates, with dependent sources, and with few claims per source (imprecision and risk of bias).

Evidence and tag basis

Documented. The survey groups methods as iterative, optimisation-based and probabilistic graphical model, and does not rank them. A Deep Web study found surprising inconsistency across sources it considered relatively clean (abstract only). (Li et al., A survey on truth discovery, arXiv 1505.02463; Li et al., Truth finding on the Deep Web, arXiv 1503.00303)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.