PJ-03 Forecasting tournaments and superforecasting practice
Produce dated probability estimates for resolvable questions, score them with Brier scores and improve them by updating and aggregation, leaving a scored probability record as a draft result.
What it does
Produces dated probability estimates for resolvable questions, scores them with Brier scores and improves them by training, teaming, updating and aggregation. Artefact: a probability record and score; read calibration and resolution.
When to use it
Use when you need calibrated probabilities for questions that will resolve on a dated, observable outcome and forecasters can update over time.
When it does not fit
The question cannot be resolved by a dated observable outcome, the horizon is long, forecasts are stated in words only, or nothing is scored.
Result format
The result is a probability record with one row per question, giving the date, the probability estimate and, once resolved, the outcome and the Brier score. It reports calibration and resolution.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Unit of analysis | Always applies | requires=topic |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Content type, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | A question with a resolution date and criterion, one forecaster or a team, access to news and data, a scoring rule and repeated updating over time. |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | The question cannot be resolved by a dated observable outcome, the horizon is long, forecasts are stated in words only, or nothing is scored. |
Choosing and ordering
| Kind of technique | Primary technique, runs in step 5.3 |
|---|---|
| Phase tag | analyse |
| Other phase tags | validate |
| Kind of finding | risk; trend |
| Question answered | what next |
| Strongest statement it supports | predict, without claiming a cause |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to frame |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | assumption and limitation statement | none | no |
| decision-informing | assumption and limitation statement | none | no |
| decision-informing | code execution for arithmetic | none | no |
| decision-informing | provenance log | none | no |
| high-stakes | assumption and limitation statement | none | no |
| high-stakes | code execution for arithmetic | none | no |
| high-stakes | provenance log | none | no |
| high-stakes | second technique with different assumptions | none | conditional |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | human judgement required |
|---|---|
| Typical model failure | States a single probability once without dates, updates or a scoring rule, which cannot be calibrated. |
| Error sources | expert opinion; input data |
| Certainty limits | Certainty drops when few questions have resolved (imprecision), when the horizon is long (indirectness), or when forecasters share information or views (risk of bias). |
Related techniques
- pairs with TC1-14 Coherence correction and averaging of probability judgements: coherence correction and averaging improve the probabilities of several forecasters
- pairs with CM1-13 Forecaster and source comparison (forecast scoring, aggregation, track record comparison) Also serves: Project forward.: comparing forecasters against resolved outcomes shows which are calibrated
Evidence and tag basis
Tested. See the evidence section for effect sizes and caveats. (AI Impacts, summary of Good Judgment Project research; Wikipedia, Superforecaster)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.