PE-03 Near-duplicate detection (shingling; MinHash; locality-sensitive hashing; text reuse detection)
Find identical or nearly identical documents so repeats are not counted as independent evidence, giving duplicate clusters with similarity scores and a deduplicated corpus.
What it does
Finds identical or nearly identical documents so repeats are not counted as independent evidence. Artefact: duplicate clusters with similarity scores. Read cluster sizes, because a large cluster means syndicated or copied content.
When to use it
Use when sources syndicate, copy or republish and you need to stop repeated content counting as separate evidence.
When it does not fit
Fits sources that syndicate or republish. A low threshold merges distinct documents and a high one leaves rewrites, and banding gives false positives that need direct checking.
Result format
The result is a table of duplicate clusters, one row per cluster, giving the member documents, the cluster size and the similarity score. It ends with the largest clusters named as likely syndicated or copied content.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Structure | Applies in some cases | requires=unstructured; requires=semi-structured |
| Content type | Always applies | requires=text |
| Unit of analysis | Always applies | requires=document |
| Known data problems | Applies in some cases | requires=duplicated; requires=unknown |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | none |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Documents are all original with no republishing, or the similarity threshold cannot be checked by reading sample pairs. |
Choosing and ordering
| Kind of technique | Preparation technique, runs in step 5.2 |
|---|---|
| Phase tag | verify and prepare |
| Other phase tags | none |
| Kind of finding | risk; pattern |
| Question answered | what; how much |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to verify and prepare |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: deterministic.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | input validation | none | no |
| decision-informing | input validation | none | no |
| decision-informing | spot check | optional | no |
| decision-informing | sensitivity analysis | none | no |
| high-stakes | input validation | none | no |
| high-stakes | spot check | optional | no |
| high-stakes | sensitivity analysis | none | no |
| high-stakes | human review sample | needed | no |
| high-stakes | provenance log | none | no |
Failure modes and limits
| Model suitability | model plus code |
|---|---|
| Typical model failure | Merges distinct documents at a low threshold, or leaves rewrites unmatched at a high one, without checking sample pairs. |
| Error sources | input data; data transformation |
| Certainty limits | Certainty drops when the threshold has not been tested on sample pairs (imprecision) or when banding produces false positives that are not checked directly (risk of bias). |
Related techniques
- pairs with IS-06 Source dependence and copy detection (source independence analysis): copy detection across sources estimates which sources depend on each other
- pairs with PJ-23 Horizon scanning (environmental scanning): removing repeated reports keeps signals independent
Evidence and tag basis
Documented. Course slides describe shingles of length 8 to 10 and MinHash estimation of Jaccard similarity. One 61-word sentence appeared over 60,000 times in a language model training set, and deduplication reduced memorised output about tenfold. (Stanford CS246 LSH slides; Lee et al. 2022, arXiv 2107.06499)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.