PJ-24 Weak signal detection (early warning signs, wild cards)
Identify small ambiguous early indications that may precede a discontinuity, giving a watchlist with a strength rating over time as an appraised evidence base.
What it does
Identifies small ambiguous early indications that may precede a discontinuity. Artefact: a watchlist with strength rating over time; check whether a signal is gaining independent sources.
When to use it
Use when you are watching a defined domain for faint early indications and want to see whether any is gaining independent sources.
When it does not fit
Signals are chosen after the event, or signals that did not materialise are not recorded, so the hit rate cannot be known.
Result format
The result is a watchlist with one row per weak signal, giving the signal and its strength rating over time. It states for each signal whether it is gaining independent sources.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Structure | Applies in some cases | requires=unstructured; requires=semi-structured |
| Content type | Always applies | requires=text |
| Unit of analysis | Always applies | requires=document; requires=source |
| Number of sources | Always applies | requires=two or more |
| Source dependence | Changes the order | prefers=independent |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Variable types, Time structure, How the sample was chosen, Level of detail available, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | A defined watch domain, a baseline and a signal log kept over time. |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Signals are chosen after the event, or signals that did not materialise are not recorded, so the hit rate cannot be known. |
Choosing and ordering
| Kind of technique | Primary technique, runs in step 5.3 |
|---|---|
| Phase tag | analyse |
| Other phase tags | gather |
| Kind of finding | trend; gap; risk |
| Question answered | what next; what |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to gather |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | assumption and limitation statement | none | no |
| decision-informing | assumption and limitation statement | none | no |
| decision-informing | provenance log | none | no |
| decision-informing | independent review | needed; left out below high stakes | no |
| high-stakes | assumption and limitation statement | none | no |
| high-stakes | provenance log | none | no |
| high-stakes | independent review | needed | no |
| high-stakes | second technique with different assumptions | none | conditional |
| high-stakes | expert review | needed | no |
Failure modes and limits
| Model suitability | human judgement required |
|---|---|
| Typical model failure | Selects signals after an event occurs and omits signals that did not materialise. |
| Error sources | source selection; expert opinion; language model |
| Certainty limits | Confidence is low because no study shows how often weak signals precede events (indirectness), and unrecorded misses inflate the apparent hit rate (reporting bias). |
Related techniques
- should follow PJ-23 Horizon scanning (environmental scanning): a scan produces the signal log that is rated here
- pairs with IS-06 Source dependence and copy detection (source independence analysis): copy detection tests whether a strengthening signal has independent sources
Evidence and tag basis
Unconfirmed. Credited to Ansoff (1975) from memory. No study of how often weak signals precede events was found.
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.