TC1-14 Coherence correction and averaging of probability judgements
Adjusts each analyst's probabilities on mutually exclusive hypotheses so they sum to one, averages across analysts, and produces a corrected aggregated judgement.
What it does
Adjusts each analyst's probability judgements so that those on mutually exclusive hypotheses sum to one, and averages across analysts. Artefact: a corrected, aggregated judgement.
When to use it
Use when several analysts have given probability estimates on the same set of hypotheses and you want one combined, internally consistent judgement.
When it does not fit
Judgements are not on mutually exclusive, exhaustive hypotheses, or the judges share the same bias.
Result format
The result is a table with one row per hypothesis and one column per analyst, showing each analyst's probability as given and as corrected so that the probabilities on mutually exclusive hypotheses sum to one. It ends with a corrected, aggregated probability for each hypothesis, averaged across analysts.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Content type | Always applies | requires=tabular |
| Unit of analysis | Always applies | requires=claim |
| Variable types | Raises a warning | prefers=counts or fractions |
| Level of detail available | Always applies | requires=aggregate tables |
| Number of sources | Always applies | requires=two or more |
| Source dependence | Changes the order | prefers=independent |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Time structure, How the sample was chosen, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | Probability judgements from at least two analysts on the same mutually exclusive and exhaustive hypotheses. More analysts improve the average. |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Wrong choice when judgements are not on mutually exclusive and exhaustive hypotheses, or when the judges share the same bias. |
Choosing and ordering
| Kind of technique | Primary technique, runs in step 5.3 |
|---|---|
| Phase tag | analyse |
| Other phase tags | conclude |
| Kind of finding | option ranking; difference |
| Question answered | how much; what |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to gather |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: deterministic.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | input validation | none | no |
| decision-informing | input validation | none | no |
| decision-informing | code execution for arithmetic | none | no |
| decision-informing | sensitivity analysis | none | no |
| high-stakes | input validation | none | no |
| high-stakes | code execution for arithmetic | none | no |
| high-stakes | sensitivity analysis | none | no |
| high-stakes | spot check | optional | no |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | model plus code |
|---|---|
| Typical model failure | Averages judgements that share one source or bias, or normalises probabilities over a hypothesis set that is not exhaustive. |
| Error sources | input data; expert opinion; statistical assumptions |
| Certainty limits | Certainty drops when analysts are not independent, when hypotheses overlap or omit possibilities, or when the evidence comes from one small homogeneous laboratory sample (indirectness). |
Related techniques
- should follow TC1-13 Analysis of competing hypotheses (ACH): the matrix supplies the hypotheses on which judgements are collected
- pairs with TC1-15 Bayesian updating (likelihood reasoning, likelihood ratio, Bayes factor): Bayesian updating gives coherent probabilities from a single analyst
Evidence and tag basis
Tested. Reported to cut error by about 35 per cent (coherence correction) and about 61 per cent (averaging all 50 corrected judgements) in one laboratory task. Small homogeneous sample; attribution of the paper is unresolved. (Cambridge Core paper on what works and what fails in improving analysts' accuracy)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.