startthat.work home

PJ-03 Forecasting tournaments and superforecasting practice

Produce dated probability estimates for resolvable questions, score them with Brier scores and improve them by updating and aggregation, leaving a scored probability record as a draft result.

Project forwardEvidence: TestedPrimary technique, runs in step 5.3Strongest statement: predict, without claiming a cause

What it does

Produces dated probability estimates for resolvable questions, scores them with Brier scores and improves them by training, teaming, updating and aggregation. Artefact: a probability record and score; read calibration and resolution.

When to use it

Use when you need calibrated probabilities for questions that will resolve on a dated, observable outcome and forecasters can update over time.

When it does not fit

The question cannot be resolved by a dated observable outcome, the horizon is long, forecasts are stated in words only, or nothing is scored.

Result format

The result is a probability record with one row per question, giving the date, the probability estimate and, once resolved, the outcome and the Brier score. It reports calibration and resolution.

Requirements on the material

PropertyRequirement typeRequirement
Unit of analysisAlways appliesrequires=topic
Appraises source credibilityApplies in some casesno

Not restricted on: Structure, Content type, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialA question with a resolution date and criterion, one forecaster or a team, access to news and data, a scoring rule and repeated updating over time.
Additional sources needednone
Not for (rules)none
Not for (text)The question cannot be resolved by a dated observable outcome, the horizon is long, forecasts are stated in words only, or nothing is scored.

Choosing and ordering

Kind of techniquePrimary technique, runs in step 5.3
Phase taganalyse
Other phase tagsvalidate
Kind of findingrisk; trend
Question answeredwhat next
Strongest statement it supportspredict, without claiming a cause
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to frame

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.

StakesCheckPerson involvedCounts as a different method
exploratoryassumption and limitation statementnoneno
decision-informingassumption and limitation statementnoneno
decision-informingcode execution for arithmeticnoneno
decision-informingprovenance lognoneno
high-stakesassumption and limitation statementnoneno
high-stakescode execution for arithmeticnoneno
high-stakesprovenance lognoneno
high-stakessecond technique with different assumptionsnoneconditional
high-stakesindependent reviewneededno

Failure modes and limits

Model suitabilityhuman judgement required
Typical model failureStates a single probability once without dates, updates or a scoring rule, which cannot be calibrated.
Error sourcesexpert opinion; input data
Certainty limitsCertainty drops when few questions have resolved (imprecision), when the horizon is long (indirectness), or when forecasters share information or views (risk of bias).

Evidence and tag basis

Tested. See the evidence section for effect sizes and caveats. (AI Impacts, summary of Good Judgment Project research; Wikipedia, Superforecaster)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.