startthat.work home

PE-03 Near-duplicate detection (shingling; MinHash; locality-sensitive hashing; text reuse detection)

Find identical or nearly identical documents so repeats are not counted as independent evidence, giving duplicate clusters with similarity scores and a deduplicated corpus.

Prepare and extractEvidence: DocumentedPreparation technique, runs in step 5.2Strongest statement: describe

What it does

Finds identical or nearly identical documents so repeats are not counted as independent evidence. Artefact: duplicate clusters with similarity scores. Read cluster sizes, because a large cluster means syndicated or copied content.

When to use it

Use when sources syndicate, copy or republish and you need to stop repeated content counting as separate evidence.

When it does not fit

Fits sources that syndicate or republish. A low threshold merges distinct documents and a high one leaves rewrites, and banding gives false positives that need direct checking.

Result format

The result is a table of duplicate clusters, one row per cluster, giving the member documents, the cluster size and the similarity score. It ends with the largest clusters named as likely syndicated or copied content.

Requirements on the material

PropertyRequirement typeRequirement
StructureApplies in some casesrequires=unstructured; requires=semi-structured
Content typeAlways appliesrequires=text
Unit of analysisAlways appliesrequires=document
Known data problemsApplies in some casesrequires=duplicated; requires=unknown
Appraises source credibilityApplies in some casesno

Not restricted on: Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialnone
Additional sources needednone
Not for (rules)none
Not for (text)Documents are all original with no republishing, or the similarity threshold cannot be checked by reading sample pairs.

Choosing and ordering

Kind of techniquePreparation technique, runs in step 5.2
Phase tagverify and prepare
Other phase tagsnone
Kind of findingrisk; pattern
Question answeredwhat; how much
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to verify and prepare

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: deterministic.

StakesCheckPerson involvedCounts as a different method
exploratoryinput validationnoneno
decision-informinginput validationnoneno
decision-informingspot checkoptionalno
decision-informingsensitivity analysisnoneno
high-stakesinput validationnoneno
high-stakesspot checkoptionalno
high-stakessensitivity analysisnoneno
high-stakeshuman review sampleneededno
high-stakesprovenance lognoneno

Failure modes and limits

Model suitabilitymodel plus code
Typical model failureMerges distinct documents at a low threshold, or leaves rewrites unmatched at a high one, without checking sample pairs.
Error sourcesinput data; data transformation
Certainty limitsCertainty drops when the threshold has not been tested on sample pairs (imprecision) or when banding produces false positives that are not checked directly (risk of bias).

Evidence and tag basis

Documented. Course slides describe shingles of length 8 to 10 and MinHash estimation of Jaccard similarity. One 61-word sentence appeared over 60,000 times in a language model training set, and deduplication reduced memorised output about tenfold. (Stanford CS246 LSH slides; Lee et al. 2022, arXiv 2107.06499)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.