startthat.work home

PJ-25 Leading indicators and signposts (indicators of change, trigger points)

Select measurable quantities that change before an outcome of interest and set thresholds beforehand, giving a table of indicators with lead time, threshold, source, frequency and owner as a draft result.

Project forwardEvidence: DocumentedPrimary technique, runs in step 5.3Strongest statement: predict, without claiming a cause

What it does

Selects measurable quantities that change before the outcome of interest, with thresholds set beforehand. Artefact: a table of indicators with lead time, threshold, source, frequency and owner; the CIA primer favours the hypothesis with the most indicators appearing.

When to use it

Use when you need to monitor for a future outcome or scenario and want measurable early indicators with preset trigger points.

When it does not fit

The same indicator appears under every hypothesis, the lead is shorter than the response time, thresholds are set after the fact, or the false alarm rate is not reported.

Result format

The result is a table with one row per indicator, giving its lead time, threshold, source, frequency and owner. It ends by naming the hypothesis with the most indicators appearing.

Requirements on the material

PropertyRequirement typeRequirement
StructureApplies in some casesrequires=structured; requires=semi-structured
Content typeAlways appliesrequires=time series; requires=tabular
Unit of analysisAlways appliesrequires=variable
Variable typesRaises a warningprefers=ratio; prefers=interval; prefers=counts or fractions
Time structureApplies in some casesrequires=aggregate time series; requires=continuous stream
Level of detail availableAlways appliesrequires=aggregate tables
Appraises source credibilityApplies in some casesno

Not restricted on: How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialA defined outcome, candidate indicators with regularly published data, and enough history to check lead time and false alarm rate.
Additional sources needednone
Not for (rules)none
Not for (text)The same indicator appears under every hypothesis, the lead is shorter than the response time, thresholds are set after the fact, or the false alarm rate is not reported.

Choosing and ordering

Kind of techniquePrimary technique, runs in step 5.3
Phase taganalyse
Other phase tagsconclude
Kind of findingtrend; risk
Question answeredwhat next; what
Strongest statement it supportspredict, without claiming a cause
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to analyse

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.

StakesCheckPerson involvedCounts as a different method
exploratoryinput validationnoneno
decision-informinginput validationnoneno
decision-informingheld-out validationnoneno
decision-informingsensitivity analysisnoneno
high-stakesinput validationnoneno
high-stakesheld-out validationnoneno
high-stakessensitivity analysisnoneno
high-stakesleakage checknoneno
high-stakessecond technique with different assumptionsnoneconditional
high-stakesassumption and limitation statementnoneno

Failure modes and limits

Model suitabilitymodel plus code
Typical model failureSelects indicators that appear under every hypothesis, or sets thresholds after seeing the outcome.
Error sourcesinput data; statistical assumptions; expert opinion
Certainty limitsCertainty drops when history is too short to estimate lead time (imprecision), thresholds were set after the event (risk of bias), or indicator performance has no tested evidence (indirectness).

Evidence and tag basis

Documented. Described in the CIA primer and referred to in the PHIA framework as monitoring scenarios with indicators and triggers; no source on indicator performance was opened. (CIA Tradecraft Primer (2009); UK PHIA Professional Development Framework (2019))

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.