startthat.work home

PJ-06 Structured expert elicitation (IDEA protocol: investigate, discuss, estimate, aggregate)

Elicit numeric estimates and ranges from experts before and after discussion and aggregate them mathematically, giving aggregated estimates with intervals as a draft result.

Project forwardEvidence: UnconfirmedPrimary technique, runs in step 5.3Strongest statement: predict, without claiming a cause

What it does

Elicits numeric estimates and ranges from experts before and after discussion, then aggregates mathematically. Artefact: aggregated estimates with intervals.

When to use it

Use when you need numeric estimates with uncertainty ranges from a small panel of experts and can ask precise, answerable questions.

When it does not fit

Experts are not asked for ranges, or the questions are vague.

Result format

The result is a set of aggregated estimates with intervals, with one row per quantity. It shows the estimates from before and after discussion and the mathematical aggregate of them.

Requirements on the material

PropertyRequirement typeRequirement
Unit of analysisAlways appliesrequires=topic
Appraises source credibilityApplies in some casesno

Not restricted on: Structure, Content type, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialFive to ten experts, precise questions and calibration questions.
Additional sources needednone
Not for (rules)none
Not for (text)Experts are not asked for ranges, or the questions are vague.

Choosing and ordering

Kind of techniquePrimary technique, runs in step 5.3
Phase taganalyse
Other phase tagsnone
Kind of findingrisk; trend
Question answeredhow much; what next
Strongest statement it supportspredict, without claiming a cause
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to frame

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.

StakesCheckPerson involvedCounts as a different method
exploratoryassumption and limitation statementnoneno
exploratoryprovenance lognoneno
decision-informingassumption and limitation statementnoneno
decision-informingprovenance lognoneno
decision-informingcode execution for arithmeticnoneno
decision-informingindependent reviewneeded; left out below high stakesno
high-stakesassumption and limitation statementnoneno
high-stakesprovenance lognoneno
high-stakescode execution for arithmeticnoneno
high-stakesindependent reviewneededno
high-stakesexpert reviewneededno
high-stakessensitivity analysisnoneno

Failure modes and limits

Model suitabilityhuman judgement required
Typical model failureWrites expert-like ranges itself and aggregates them as if they came from a panel.
Error sourcesexpert opinion; source selection
Certainty limitsCertainty drops when experts are few (imprecision), when calibration questions show overconfidence (risk of bias), or when questions are vague (indirectness).

Evidence and tag basis

Unconfirmed. Recalled from a 2018 paper by Hemming and colleagues; not opened.

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.