startthat.work home

PJ-04 Delphi technique (real-time Delphi, mini-Delphi)

Collect anonymous expert estimates over two or more rounds with controlled feedback, giving the median and interquartile range for each round as a draft result.

Project forwardEvidence: DocumentedPrimary technique, runs in step 5.3Strongest statement: predict, without claiming a cause

What it does

Collects anonymous expert estimates over two or more rounds with controlled feedback. Artefact: median and interquartile range per round; treat persistent disagreement as information.

When to use it

Use when no usable data exist and you need a structured, anonymous view from a panel of experts on a quantity or event.

When it does not fit

The panel is a convenience sample with similar views, no feedback is given between rounds, or consensus is reported as accuracy.

Result format

The result is a table with one row per round giving the median and interquartile range of the expert estimates. It names any disagreement that persists across rounds and treats it as information.

Requirements on the material

PropertyRequirement typeRequirement
Unit of analysisAlways appliesrequires=topic
Appraises source credibilityApplies in some casesno

Not restricted on: Structure, Content type, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialA panel of experts with varied views, a structured questionnaire, a facilitator and at least two rounds.
Additional sources needednone
Not for (rules)none
Not for (text)The panel is a convenience sample with similar views, no feedback is given between rounds, or consensus is reported as accuracy.

Choosing and ordering

Kind of techniquePrimary technique, runs in step 5.3
Phase taganalyse
Other phase tagsnone
Kind of findingtrend; risk
Question answeredwhat next; how much
Strongest statement it supportspredict, without claiming a cause
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to gather

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.

StakesCheckPerson involvedCounts as a different method
exploratoryassumption and limitation statementnoneno
exploratoryprovenance lognoneno
decision-informingassumption and limitation statementnoneno
decision-informingprovenance lognoneno
decision-informingindependent reviewneeded; left out below high stakesno
decision-informingsensitivity analysisnoneno
high-stakesassumption and limitation statementnoneno
high-stakesprovenance lognoneno
high-stakesindependent reviewneededno
high-stakessensitivity analysisnoneno
high-stakessecond technique with different assumptionsnoneconditional
high-stakesexpert reviewneededno

Failure modes and limits

Model suitabilityhuman judgement required
Typical model failureSimulates the experts itself and reports a consensus without any real panel or feedback rounds.
Error sourcesexpert opinion; source selection
Certainty limitsCertainty drops when the panel is small or homogeneous (risk of bias), when disagreement persists (inconsistency), or when validation studies are few (indirectness).

Evidence and tag basis

Documented. Rowe and Wright (1999) report few validation studies, often without a description of conditions, so it is hard to identify which aspects relate to accuracy. The UK Futures Toolkit lists it, which shows use, not accuracy. (Rowe and Wright (1999), Delphi, abstract page; GOV.UK Futures Toolkit case study)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.