startthat.work home

CH-03 Devil's advocacy

Builds the strongest case against the dominant view or plan, starting from its most vulnerable assumptions, and produces a labelled devil advocate finding.

ChallengeEvidence: DocumentedChallenge technique, runs in step 6Strongest statement: describe

What it does

Builds the strongest case against the dominant view or plan, starting from its most vulnerable assumptions. Artefact: findings presented to the group, labelled as devil's advocate product; read the vulnerable assumptions the advocate could not defend.

When to use it

Use when a view or plan is accepted too easily and you want the best case against it before a decision.

When it does not fit

The advocate is the plan's author (inference), or challenges are not taken seriously, which gives false confidence.

Result format

The result is a set of findings that make the strongest case against the dominant view or plan, starting from its most vulnerable assumptions and labelled as a devil's advocate product. It names the vulnerable assumptions that the advocate could not defend.

Requirements on the material

PropertyRequirement typeRequirement
Unit of analysisAlways appliesrequires=claim
Appraises source credibilityApplies in some casesno

Not restricted on: Structure, Content type, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialThe mainline judgement and its key assumptions.
Additional sources needednone
Not for (rules)none
Not for (text)The advocate is the plan author, or challenges are not taken seriously, which gives false confidence.

Choosing and ordering

Kind of techniqueChallenge technique, runs in step 6
Phase tagvalidate
Other phase tagsframe; conclude
Kind of findingrisk; contradiction
Question answeredwhat; why
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to analyse

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.

StakesCheckPerson involvedCounts as a different method
exploratoryassumption and limitation statementnoneno
decision-informingassumption and limitation statementnoneno
decision-informingindependent reviewneeded; left out below high stakesno
high-stakesassumption and limitation statementnoneno
high-stakesindependent reviewneededno
high-stakesexpert reviewneededno
high-stakessecond technique with different assumptionsnoneconditional

Failure modes and limits

Model suitabilitymodel alone
Typical model failureProduces a mild objection that the plan already answers, or argues a position the model itself drafted.
Error sourcesinput data; coder or rater judgement; language model
Certainty limitsThe CIA primer notes it takes analytic time; false confidence results when the challenge is not taken seriously, and there is no controlled evidence.

Evidence and tag basis

Documented. The CIA primer notes it takes analytic time; the UK handbook pairs it with overconfidence; no controlled evidence found. (CIA Tradecraft Primer (2009); UK Red Teaming Handbook)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.