PJ-25 Leading indicators and signposts (indicators of change, trigger points)
Select measurable quantities that change before an outcome of interest and set thresholds beforehand, giving a table of indicators with lead time, threshold, source, frequency and owner as a draft result.
What it does
Selects measurable quantities that change before the outcome of interest, with thresholds set beforehand. Artefact: a table of indicators with lead time, threshold, source, frequency and owner; the CIA primer favours the hypothesis with the most indicators appearing.
When to use it
Use when you need to monitor for a future outcome or scenario and want measurable early indicators with preset trigger points.
When it does not fit
The same indicator appears under every hypothesis, the lead is shorter than the response time, thresholds are set after the fact, or the false alarm rate is not reported.
Result format
The result is a table with one row per indicator, giving its lead time, threshold, source, frequency and owner. It ends by naming the hypothesis with the most indicators appearing.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Structure | Applies in some cases | requires=structured; requires=semi-structured |
| Content type | Always applies | requires=time series; requires=tabular |
| Unit of analysis | Always applies | requires=variable |
| Variable types | Raises a warning | prefers=ratio; prefers=interval; prefers=counts or fractions |
| Time structure | Applies in some cases | requires=aggregate time series; requires=continuous stream |
| Level of detail available | Always applies | requires=aggregate tables |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | A defined outcome, candidate indicators with regularly published data, and enough history to check lead time and false alarm rate. |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | The same indicator appears under every hypothesis, the lead is shorter than the response time, thresholds are set after the fact, or the false alarm rate is not reported. |
Choosing and ordering
| Kind of technique | Primary technique, runs in step 5.3 |
|---|---|
| Phase tag | analyse |
| Other phase tags | conclude |
| Kind of finding | trend; risk |
| Question answered | what next; what |
| Strongest statement it supports | predict, without claiming a cause |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to analyse |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | input validation | none | no |
| decision-informing | input validation | none | no |
| decision-informing | held-out validation | none | no |
| decision-informing | sensitivity analysis | none | no |
| high-stakes | input validation | none | no |
| high-stakes | held-out validation | none | no |
| high-stakes | sensitivity analysis | none | no |
| high-stakes | leakage check | none | no |
| high-stakes | second technique with different assumptions | none | conditional |
| high-stakes | assumption and limitation statement | none | no |
Failure modes and limits
| Model suitability | model plus code |
|---|---|
| Typical model failure | Selects indicators that appear under every hypothesis, or sets thresholds after seeing the outcome. |
| Error sources | input data; statistical assumptions; expert opinion |
| Certainty limits | Certainty drops when history is too short to estimate lead time (imprecision), thresholds were set after the event (risk of bias), or indicator performance has no tested evidence (indirectness). |
Related techniques
- should follow PJ-13 Scenario matrix (two by two scenario axes, alternative futures analysis): scenario signposts supply the outcomes the indicators must separate
- pairs with TC1-13 Analysis of competing hypotheses (ACH): analysis of competing hypotheses shows which indicator discriminates between hypotheses
Evidence and tag basis
Documented. Described in the CIA primer and referred to in the PHIA framework as monitoring scenarios with indicators and triggers; no source on indicator performance was opened. (CIA Tradecraft Primer (2009); UK PHIA Professional Development Framework (2019))
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.