startthat.work home

CM1-13 Forecaster and source comparison (forecast scoring, aggregation, track record comparison) Also serves: Project forward.

Scores a forecast against those of other forecasters on resolved outcomes and aggregates them, producing an accuracy ranking and an aggregate forecast.

CompareEvidence: TestedPrimary technique, runs in step 5.3Strongest statement: predict, without claiming a cause

What it does

Compares a forecast with those of other forecasters, scored against resolved outcomes (for example with Brier scores, a squared-error score for probability forecasts). Artefact: an accuracy ranking and an aggregate; read agreement as corroboration and divergence as a prompt to check reasons.

When to use it

Use when a user has several forecasts of the same questions and asks which forecaster to trust or what the combined forecast is.

When it does not fit

The forecasts address different questions, horizons or resolution criteria (inference).

Result format

The result is an accuracy ranking of the forecasters or sources, scored against resolved outcomes (for example by Brier score), and an aggregate forecast. It names where forecasters agree, which is read as corroboration, and where they diverge, with the reasons to check.

Requirements on the material

PropertyRequirement typeRequirement
Content typeAlways appliesrequires=tabular
Unit of analysisAlways appliesrequires=source
Variable typesRaises a warningprefers=counts or fractions
Level of detail availableAlways appliesrequires=record-level
Number of sourcesAlways appliesrequires=two or more
Whether sources can be joinedAlways appliesrequires=shared identifier available
Source dependenceChanges the orderprefers=independent
Appraises source credibilityApplies in some casesno

Not restricted on: Structure, Time structure, How the sample was chosen, Amount of material, Known data problems, Producer interest, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialForecasts from at least two sources on the same questions, with the resolved outcome of each question.
Additional sources neededcomparator
Not for (rules)none
Not for (text)Wrong choice when the forecasts address different questions, horizons or resolution criteria. This limit is inferred.

Choosing and ordering

Kind of techniquePrimary technique, runs in step 5.3
Phase taganalyse
Other phase tagsvalidate
Kind of findingdifference; pattern
Question answeredwhat next; which option
Strongest statement it supportspredict, without claiming a cause
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to gather

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: deterministic.

StakesCheckPerson involvedCounts as a different method
exploratoryinput validationnoneno
decision-informinginput validationnoneno
decision-informingcode execution for arithmeticnoneno
decision-informingspot checkoptionalno
high-stakesinput validationnoneno
high-stakescode execution for arithmeticnoneno
high-stakesspot checkoptionalno
high-stakessensitivity analysisnoneno
high-stakessecond technique with different assumptionsnoneconditional

Failure modes and limits

Model suitabilitymodel plus code
Typical model failureCompares forecasts that answer different questions or horizons, and computes scores in text with arithmetic errors.
Error sourcesinput data; source selection; statistical assumptions; expert opinion
Certainty limitsCertainty drops when few questions have resolved (imprecision), when forecasters copied each other so agreement is not independent (risk of bias), and when questions differ in difficulty (indirectness).

Evidence and tag basis

Tested. The Good Judgment Project scored forecasts and aggregated them; results are summarised under forecasting tournaments in the Project forward family. (Wikipedia, Superforecasting)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.