CM1-13 Forecaster and source comparison (forecast scoring, aggregation, track record comparison) Also serves: Project forward.
Scores a forecast against those of other forecasters on resolved outcomes and aggregates them, producing an accuracy ranking and an aggregate forecast.
What it does
Compares a forecast with those of other forecasters, scored against resolved outcomes (for example with Brier scores, a squared-error score for probability forecasts). Artefact: an accuracy ranking and an aggregate; read agreement as corroboration and divergence as a prompt to check reasons.
When to use it
Use when a user has several forecasts of the same questions and asks which forecaster to trust or what the combined forecast is.
When it does not fit
The forecasts address different questions, horizons or resolution criteria (inference).
Result format
The result is an accuracy ranking of the forecasters or sources, scored against resolved outcomes (for example by Brier score), and an aggregate forecast. It names where forecasters agree, which is read as corroboration, and where they diverge, with the reasons to check.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Content type | Always applies | requires=tabular |
| Unit of analysis | Always applies | requires=source |
| Variable types | Raises a warning | prefers=counts or fractions |
| Level of detail available | Always applies | requires=record-level |
| Number of sources | Always applies | requires=two or more |
| Whether sources can be joined | Always applies | requires=shared identifier available |
| Source dependence | Changes the order | prefers=independent |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Time structure, How the sample was chosen, Amount of material, Known data problems, Producer interest, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | Forecasts from at least two sources on the same questions, with the resolved outcome of each question. |
|---|---|
| Additional sources needed | comparator |
| Not for (rules) | none |
| Not for (text) | Wrong choice when the forecasts address different questions, horizons or resolution criteria. This limit is inferred. |
Choosing and ordering
| Kind of technique | Primary technique, runs in step 5.3 |
|---|---|
| Phase tag | analyse |
| Other phase tags | validate |
| Kind of finding | difference; pattern |
| Question answered | what next; which option |
| Strongest statement it supports | predict, without claiming a cause |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to gather |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: deterministic.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | input validation | none | no |
| decision-informing | input validation | none | no |
| decision-informing | code execution for arithmetic | none | no |
| decision-informing | spot check | optional | no |
| high-stakes | input validation | none | no |
| high-stakes | code execution for arithmetic | none | no |
| high-stakes | spot check | optional | no |
| high-stakes | sensitivity analysis | none | no |
| high-stakes | second technique with different assumptions | none | conditional |
Failure modes and limits
| Model suitability | model plus code |
|---|---|
| Typical model failure | Compares forecasts that answer different questions or horizons, and computes scores in text with arithmetic errors. |
| Error sources | input data; source selection; statistical assumptions; expert opinion |
| Certainty limits | Certainty drops when few questions have resolved (imprecision), when forecasters copied each other so agreement is not independent (risk of bias), and when questions differ in difficulty (indirectness). |
Related techniques
- pairs with PJ-03 Forecasting tournaments and superforecasting practice: tournament practice supplies resolvable questions and dated probabilities to score
- pairs with PJ-02 Reference class forecasting (outside view, base-rate forecasting) Also serves: Compare.: a base rate from similar past cases gives a reference for the aggregate
Evidence and tag basis
Tested. The Good Judgment Project scored forecasts and aggregated them; results are summarised under forecasting tournaments in the Project forward family. (Wikipedia, Superforecasting)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.