PJ-19 Policy stress-testing
Run a policy or plan through defined adverse or alternative conditions, giving a table of performance by condition and the cheapest changes that remove failures as a draft comparison matrix.
What it does
Runs a policy or plan through defined adverse or alternative conditions. Artefact: a table of performance by condition; read the failure conditions and the cheapest changes that remove them.
When to use it
Use when you have a specified plan and need to know under which conditions it fails and what small changes would make it hold up.
When it does not fit
Conditions are mild versions of the status quo, success criteria are set after seeing results, or only the plan's designers run the test.
Result format
The result is a table with one row per adverse or alternative condition, giving how the policy or plan performs under that condition. It names the conditions under which the policy fails and the cheapest changes that remove each failure.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Unit of analysis | Always applies | requires=topic |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Content type, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | A specified plan, a set of conditions and success criteria stated before the test. |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Conditions are mild versions of the status quo, success criteria are set after seeing results, or only the plan designers run the test. |
Choosing and ordering
| Kind of technique | Primary technique, runs in step 5.3 |
|---|---|
| Phase tag | analyse |
| Other phase tags | validate |
| Kind of finding | risk; difference |
| Question answered | which option; what next |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to frame |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | assumption and limitation statement | none | no |
| decision-informing | assumption and limitation statement | none | no |
| decision-informing | independent review | needed; left out below high stakes | no |
| decision-informing | source-span citation | none | no |
| high-stakes | assumption and limitation statement | none | no |
| high-stakes | independent review | needed | no |
| high-stakes | source-span citation | none | no |
| high-stakes | expert review | needed | no |
| high-stakes | provenance log | none | no |
Failure modes and limits
| Model suitability | human judgement required |
|---|---|
| Typical model failure | Chooses mild conditions and sets success criteria after seeing how the plan performs. |
| Error sources | expert opinion; source selection; language model |
| Certainty limits | Confidence drops when conditions are chosen by the plan designers (risk of bias) or when no effectiveness evidence exists for the method (indirectness). |
Related techniques
- should follow PJ-12 Scenario planning (intuitive logics, Shell approach): scenarios supply the adverse and alternative conditions
- pairs with CH-08 Premortem (prospective hindsight, inversion) Also serves: Project forward.: a premortem finds failure causes that the defined conditions miss
- pairs with PJ-21 Robust decision making, adaptive pathways and info-gap analysis: a model-based exploration finds failure conditions beyond the chosen ones
Evidence and tag basis
Documented. Listed in the UK Futures Toolkit; no effectiveness evidence found. (GOV.UK Futures Toolkit case study)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.