startthat.work home

TC1-16 Disconfirmation search (falsification, seeking disconfirming evidence)

Looks for evidence that would contradict a favoured claim and produces a list of refuting evidence found or not found with the search used.

Trace cause and test claimsEvidence: DocumentedVerification technique, runs in step 6Strongest statement: describe

What it does

Looks for evidence that would contradict a favoured claim. Artefact: a list of refuting evidence found or not found, with the search used.

When to use it

Use when you hold a claim you believe and want to find out whether anything credible contradicts it before relying on it.

When it does not fit

The search is shallow, or it is run after the conclusion is fixed.

Result format

The result names the claim under test, lists the evidence that would contradict it, and records for each item whether it was found or not found. It states the search used for each item and ends with what the findings mean for the claim.

Requirements on the material

PropertyRequirement typeRequirement
Content typeAlways appliesrequires=text
Unit of analysisAlways appliesrequires=claim
Appraises source credibilityApplies in some casesno

Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialA precisely stated claim and a statement, written before searching, of what evidence would refute it.
Additional sources needednone
Not for (rules)none
Not for (text)Wrong choice when the search will be shallow or run after the conclusion is fixed. Not finding refuting evidence supports the claim only weakly.

Choosing and ordering

Kind of techniqueVerification technique, runs in step 6
Phase tagvalidate
Other phase tagsanalyse
Kind of findingcontradiction; gap
Question answeredwhat; why
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to gather

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.

StakesCheckPerson involvedCounts as a different method
exploratorysource-span citationnoneno
decision-informingsource-span citationnoneno
decision-informingspot checkoptionalno
decision-informingalternative explanations checknoneno
high-stakessource-span citationnoneno
high-stakesspot checkoptionalno
high-stakesalternative explanations checknoneno
high-stakesindependent reviewneededno
high-stakessecond technique with different assumptionsnoneconditional

Failure modes and limits

Model suitabilityhuman judgement required
Typical model failureSearches with terms that favour the claim, stops after a few pages, and reports that no refuting evidence was found as if it confirmed the claim.
Error sourcesinput data; source selection; coder or rater judgement
Certainty limitsCertainty drops when the search is shallow, restricted to sources sharing the claim's origin, or run after the conclusion was fixed (risk of bias and reporting bias).

Evidence and tag basis

Documented. Heuer's text states the aim of disproof within ACH; a primary source for the general technique was not opened, and no effect evidence was found. (Heuer, Psychology of Intelligence Analysis)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.