startthat.work home

PJ-24 Weak signal detection (early warning signs, wild cards)

Identify small ambiguous early indications that may precede a discontinuity, giving a watchlist with a strength rating over time as an appraised evidence base.

Project forwardEvidence: UnconfirmedPrimary technique, runs in step 5.3Strongest statement: describe

What it does

Identifies small ambiguous early indications that may precede a discontinuity. Artefact: a watchlist with strength rating over time; check whether a signal is gaining independent sources.

When to use it

Use when you are watching a defined domain for faint early indications and want to see whether any is gaining independent sources.

When it does not fit

Signals are chosen after the event, or signals that did not materialise are not recorded, so the hit rate cannot be known.

Result format

The result is a watchlist with one row per weak signal, giving the signal and its strength rating over time. It states for each signal whether it is gaining independent sources.

Requirements on the material

PropertyRequirement typeRequirement
StructureApplies in some casesrequires=unstructured; requires=semi-structured
Content typeAlways appliesrequires=text
Unit of analysisAlways appliesrequires=document; requires=source
Number of sourcesAlways appliesrequires=two or more
Source dependenceChanges the orderprefers=independent
Appraises source credibilityApplies in some casesno

Not restricted on: Variable types, Time structure, How the sample was chosen, Level of detail available, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialA defined watch domain, a baseline and a signal log kept over time.
Additional sources needednone
Not for (rules)none
Not for (text)Signals are chosen after the event, or signals that did not materialise are not recorded, so the hit rate cannot be known.

Choosing and ordering

Kind of techniquePrimary technique, runs in step 5.3
Phase taganalyse
Other phase tagsgather
Kind of findingtrend; gap; risk
Question answeredwhat next; what
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to gather

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.

StakesCheckPerson involvedCounts as a different method
exploratoryassumption and limitation statementnoneno
decision-informingassumption and limitation statementnoneno
decision-informingprovenance lognoneno
decision-informingindependent reviewneeded; left out below high stakesno
high-stakesassumption and limitation statementnoneno
high-stakesprovenance lognoneno
high-stakesindependent reviewneededno
high-stakessecond technique with different assumptionsnoneconditional
high-stakesexpert reviewneededno

Failure modes and limits

Model suitabilityhuman judgement required
Typical model failureSelects signals after an event occurs and omits signals that did not materialise.
Error sourcessource selection; expert opinion; language model
Certainty limitsConfidence is low because no study shows how often weak signals precede events (indirectness), and unrecorded misses inflate the apparent hit rate (reporting bias).

Evidence and tag basis

Unconfirmed. Credited to Ansoff (1975) from memory. No study of how often weak signals precede events was found.

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.