PJ-06 Structured expert elicitation (IDEA protocol: investigate, discuss, estimate, aggregate)
Elicit numeric estimates and ranges from experts before and after discussion and aggregate them mathematically, giving aggregated estimates with intervals as a draft result.
What it does
Elicits numeric estimates and ranges from experts before and after discussion, then aggregates mathematically. Artefact: aggregated estimates with intervals.
When to use it
Use when you need numeric estimates with uncertainty ranges from a small panel of experts and can ask precise, answerable questions.
When it does not fit
Experts are not asked for ranges, or the questions are vague.
Result format
The result is a set of aggregated estimates with intervals, with one row per quantity. It shows the estimates from before and after discussion and the mathematical aggregate of them.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Unit of analysis | Always applies | requires=topic |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Content type, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | Five to ten experts, precise questions and calibration questions. |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Experts are not asked for ranges, or the questions are vague. |
Choosing and ordering
| Kind of technique | Primary technique, runs in step 5.3 |
|---|---|
| Phase tag | analyse |
| Other phase tags | none |
| Kind of finding | risk; trend |
| Question answered | how much; what next |
| Strongest statement it supports | predict, without claiming a cause |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to frame |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | assumption and limitation statement | none | no |
| exploratory | provenance log | none | no |
| decision-informing | assumption and limitation statement | none | no |
| decision-informing | provenance log | none | no |
| decision-informing | code execution for arithmetic | none | no |
| decision-informing | independent review | needed; left out below high stakes | no |
| high-stakes | assumption and limitation statement | none | no |
| high-stakes | provenance log | none | no |
| high-stakes | code execution for arithmetic | none | no |
| high-stakes | independent review | needed | no |
| high-stakes | expert review | needed | no |
| high-stakes | sensitivity analysis | none | no |
Failure modes and limits
| Model suitability | human judgement required |
|---|---|
| Typical model failure | Writes expert-like ranges itself and aggregates them as if they came from a panel. |
| Error sources | expert opinion; source selection |
| Certainty limits | Certainty drops when experts are few (imprecision), when calibration questions show overconfidence (risk of bias), or when questions are vague (indirectness). |
Related techniques
- pairs with PJ-04 Delphi technique (real-time Delphi, mini-Delphi): a multi-round anonymous design reduces the influence of dominant panel members
- should follow FQ-06 SMART check (specific, measurable, achievable, relevant, time-bound): success criteria written beforehand make each question precise enough to answer
Evidence and tag basis
Unconfirmed. Recalled from a 2018 paper by Hemming and colleagues; not opened.
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.