TC1-16 Disconfirmation search (falsification, seeking disconfirming evidence)
Looks for evidence that would contradict a favoured claim and produces a list of refuting evidence found or not found with the search used.
What it does
Looks for evidence that would contradict a favoured claim. Artefact: a list of refuting evidence found or not found, with the search used.
When to use it
Use when you hold a claim you believe and want to find out whether anything credible contradicts it before relying on it.
When it does not fit
The search is shallow, or it is run after the conclusion is fixed.
Result format
The result names the claim under test, lists the evidence that would contradict it, and records for each item whether it was found or not found. It states the search used for each item and ends with what the findings mean for the claim.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Content type | Always applies | requires=text |
| Unit of analysis | Always applies | requires=claim |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | A precisely stated claim and a statement, written before searching, of what evidence would refute it. |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Wrong choice when the search will be shallow or run after the conclusion is fixed. Not finding refuting evidence supports the claim only weakly. |
Choosing and ordering
| Kind of technique | Verification technique, runs in step 6 |
|---|---|
| Phase tag | validate |
| Other phase tags | analyse |
| Kind of finding | contradiction; gap |
| Question answered | what; why |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to gather |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | source-span citation | none | no |
| decision-informing | source-span citation | none | no |
| decision-informing | spot check | optional | no |
| decision-informing | alternative explanations check | none | no |
| high-stakes | source-span citation | none | no |
| high-stakes | spot check | optional | no |
| high-stakes | alternative explanations check | none | no |
| high-stakes | independent review | needed | no |
| high-stakes | second technique with different assumptions | none | conditional |
Failure modes and limits
| Model suitability | human judgement required |
|---|---|
| Typical model failure | Searches with terms that favour the claim, stops after a few pages, and reports that no refuting evidence was found as if it confirmed the claim. |
| Error sources | input data; source selection; coder or rater judgement |
| Certainty limits | Certainty drops when the search is shallow, restricted to sources sharing the claim's origin, or run after the conclusion was fixed (risk of bias and reporting bias). |
Related techniques
- should follow TC1-13 Analysis of competing hypotheses (ACH): the surviving hypothesis is the claim to search against
- pairs with CH-03 Devil's advocacy: devil's advocacy builds the strongest opposing case from the same weak points
Evidence and tag basis
Documented. Heuer's text states the aim of disproof within ACH; a primary source for the general technique was not opened, and no effect evidence was found. (Heuer, Psychology of Intelligence Analysis)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.