startthat.work home

PJ-02 Reference class forecasting (outside view, base-rate forecasting) Also serves: Compare.

Replace a bottom-up estimate with the distribution of outcomes in similar completed cases, giving a base-rate distribution and the case position in it as a draft result.

Project forwardEvidence: TestedPrimary technique, runs in step 5.3Strongest statement: predict, without claiming a cause

What it does

Replaces a bottom-up estimate with the distribution of outcomes in similar past cases. Artefact: a distribution of outcomes and the case's position within it; read the base rate first, then justify departures.

When to use it

Use when you are asked how long, how much or how likely for a project or event that has many comparable precedents with recorded outcomes.

When it does not fit

The class was chosen after seeing the answer, has only a handful of cases, or several plausible classes give very different base rates that are not reported.

Result format

The result is a distribution of outcomes in similar past cases and the position of the case within that distribution. It states the base rate first and then gives the justification for any departure from it.

Requirements on the material

PropertyRequirement typeRequirement
StructureApplies in some casesrequires=structured; requires=semi-structured
Content typeAlways appliesrequires=tabular
Unit of analysisAlways appliesrequires=record
Variable typesRaises a warningprefers=ratio; prefers=counts or fractions
Level of detail availableAlways appliesrequires=aggregate tables
Appraises source credibilityApplies in some casesno

Not restricted on: Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialEnough completed comparable cases to form a distribution, not a handful; the case count is reported with the result.
Additional sources neededcomparator
Not for (rules)none
Not for (text)The reference class was chosen after seeing the answer, has only a handful of cases, or several plausible classes give very different base rates that are not reported.

Choosing and ordering

Kind of techniquePrimary technique, runs in step 5.3
Phase taganalyse
Other phase tagsnone
Kind of findingrisk; difference
Question answeredwhat next; how much
Strongest statement it supportspredict, without claiming a cause
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to gather

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.

StakesCheckPerson involvedCounts as a different method
exploratoryassumption and limitation statementnoneno
decision-informingassumption and limitation statementnoneno
decision-informingsensitivity analysisnoneno
decision-informingcode execution for arithmeticnoneno
high-stakesassumption and limitation statementnoneno
high-stakessensitivity analysisnoneno
high-stakescode execution for arithmeticnoneno
high-stakessecond technique with different assumptionsnoneconditional
high-stakesindependent reviewneededno

Failure modes and limits

Model suitabilityhuman judgement required
Typical model failurePicks a reference class that already matches the preferred answer and reports one base rate without showing the rates from other plausible classes.
Error sourcesinput data; source selection; coder or rater judgement
Certainty limitsCertainty drops when the class has few cases (imprecision), when plausible classes disagree (inconsistency), or when past cases differ from the present case in ways that matter (indirectness).

Evidence and tag basis

Tested. In the Good Judgment Project, use of comparison classes was the only principle significantly correlated with forecasting success, as summarised by a blog (secondary). PMI describes the three steps and the method's limits where few precedents exist. (AI Impacts, summary of Good Judgment Project research; PMI, From Nobel Prize to project management: reference class forecasting)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.