PJ-02 Reference class forecasting (outside view, base-rate forecasting) Also serves: Compare.
Replace a bottom-up estimate with the distribution of outcomes in similar completed cases, giving a base-rate distribution and the case position in it as a draft result.
What it does
Replaces a bottom-up estimate with the distribution of outcomes in similar past cases. Artefact: a distribution of outcomes and the case's position within it; read the base rate first, then justify departures.
When to use it
Use when you are asked how long, how much or how likely for a project or event that has many comparable precedents with recorded outcomes.
When it does not fit
The class was chosen after seeing the answer, has only a handful of cases, or several plausible classes give very different base rates that are not reported.
Result format
The result is a distribution of outcomes in similar past cases and the position of the case within that distribution. It states the base rate first and then gives the justification for any departure from it.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Structure | Applies in some cases | requires=structured; requires=semi-structured |
| Content type | Always applies | requires=tabular |
| Unit of analysis | Always applies | requires=record |
| Variable types | Raises a warning | prefers=ratio; prefers=counts or fractions |
| Level of detail available | Always applies | requires=aggregate tables |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | Enough completed comparable cases to form a distribution, not a handful; the case count is reported with the result. |
|---|---|
| Additional sources needed | comparator |
| Not for (rules) | none |
| Not for (text) | The reference class was chosen after seeing the answer, has only a handful of cases, or several plausible classes give very different base rates that are not reported. |
Choosing and ordering
| Kind of technique | Primary technique, runs in step 5.3 |
|---|---|
| Phase tag | analyse |
| Other phase tags | none |
| Kind of finding | risk; difference |
| Question answered | what next; how much |
| Strongest statement it supports | predict, without claiming a cause |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to gather |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | assumption and limitation statement | none | no |
| decision-informing | assumption and limitation statement | none | no |
| decision-informing | sensitivity analysis | none | no |
| decision-informing | code execution for arithmetic | none | no |
| high-stakes | assumption and limitation statement | none | no |
| high-stakes | sensitivity analysis | none | no |
| high-stakes | code execution for arithmetic | none | no |
| high-stakes | second technique with different assumptions | none | conditional |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | human judgement required |
|---|---|
| Typical model failure | Picks a reference class that already matches the preferred answer and reports one base rate without showing the rates from other plausible classes. |
| Error sources | input data; source selection; coder or rater judgement |
| Certainty limits | Certainty drops when the class has few cases (imprecision), when plausible classes disagree (inconsistency), or when past cases differ from the present case in ways that matter (indirectness). |
Related techniques
- pairs with PJ-01 Trend extrapolation and time-series forecasting (exponential smoothing, autoregressive integrated moving average models, trend impact analysis): the outside view tests a trend extrapolation against outcomes of similar cases
- pairs with CK-15 Sample representativeness audit (coverage check; selection bias check): a representativeness audit tests whether the chosen class covers comparable cases
Evidence and tag basis
Tested. In the Good Judgment Project, use of comparison classes was the only principle significantly correlated with forecasting success, as summarised by a blog (secondary). PMI describes the three steps and the method's limits where few precedents exist. (AI Impacts, summary of Good Judgment Project research; PMI, From Nobel Prize to project management: reference class forecasting)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.