PJ-04 Delphi technique (real-time Delphi, mini-Delphi)
Collect anonymous expert estimates over two or more rounds with controlled feedback, giving the median and interquartile range for each round as a draft result.
What it does
Collects anonymous expert estimates over two or more rounds with controlled feedback. Artefact: median and interquartile range per round; treat persistent disagreement as information.
When to use it
Use when no usable data exist and you need a structured, anonymous view from a panel of experts on a quantity or event.
When it does not fit
The panel is a convenience sample with similar views, no feedback is given between rounds, or consensus is reported as accuracy.
Result format
The result is a table with one row per round giving the median and interquartile range of the expert estimates. It names any disagreement that persists across rounds and treats it as information.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Unit of analysis | Always applies | requires=topic |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Content type, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | A panel of experts with varied views, a structured questionnaire, a facilitator and at least two rounds. |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | The panel is a convenience sample with similar views, no feedback is given between rounds, or consensus is reported as accuracy. |
Choosing and ordering
| Kind of technique | Primary technique, runs in step 5.3 |
|---|---|
| Phase tag | analyse |
| Other phase tags | none |
| Kind of finding | trend; risk |
| Question answered | what next; how much |
| Strongest statement it supports | predict, without claiming a cause |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to gather |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | assumption and limitation statement | none | no |
| exploratory | provenance log | none | no |
| decision-informing | assumption and limitation statement | none | no |
| decision-informing | provenance log | none | no |
| decision-informing | independent review | needed; left out below high stakes | no |
| decision-informing | sensitivity analysis | none | no |
| high-stakes | assumption and limitation statement | none | no |
| high-stakes | provenance log | none | no |
| high-stakes | independent review | needed | no |
| high-stakes | sensitivity analysis | none | no |
| high-stakes | second technique with different assumptions | none | conditional |
| high-stakes | expert review | needed | no |
Failure modes and limits
| Model suitability | human judgement required |
|---|---|
| Typical model failure | Simulates the experts itself and reports a consensus without any real panel or feedback rounds. |
| Error sources | expert opinion; source selection |
| Certainty limits | Certainty drops when the panel is small or homogeneous (risk of bias), when disagreement persists (inconsistency), or when validation studies are few (indirectness). |
Related techniques
- pairs with TC1-14 Coherence correction and averaging of probability judgements: coherence correction and averaging of the panel probabilities
- pairs with PJ-06 Structured expert elicitation (IDEA protocol: investigate, discuss, estimate, aggregate): calibration questions and ranges from structured elicitation reduce overconfident point estimates
Evidence and tag basis
Documented. Rowe and Wright (1999) report few validation studies, often without a description of conditions, so it is hard to identify which aspects relate to accuracy. The UK Futures Toolkit lists it, which shows use, not accuracy. (Rowe and Wright (1999), Delphi, abstract page; GOV.UK Futures Toolkit case study)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.