startthat.work home

PJ-19 Policy stress-testing

Run a policy or plan through defined adverse or alternative conditions, giving a table of performance by condition and the cheapest changes that remove failures as a draft comparison matrix.

Project forwardEvidence: DocumentedPrimary technique, runs in step 5.3Strongest statement: describe

What it does

Runs a policy or plan through defined adverse or alternative conditions. Artefact: a table of performance by condition; read the failure conditions and the cheapest changes that remove them.

When to use it

Use when you have a specified plan and need to know under which conditions it fails and what small changes would make it hold up.

When it does not fit

Conditions are mild versions of the status quo, success criteria are set after seeing results, or only the plan's designers run the test.

Result format

The result is a table with one row per adverse or alternative condition, giving how the policy or plan performs under that condition. It names the conditions under which the policy fails and the cheapest changes that remove each failure.

Requirements on the material

PropertyRequirement typeRequirement
Unit of analysisAlways appliesrequires=topic
Appraises source credibilityApplies in some casesno

Not restricted on: Structure, Content type, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialA specified plan, a set of conditions and success criteria stated before the test.
Additional sources needednone
Not for (rules)none
Not for (text)Conditions are mild versions of the status quo, success criteria are set after seeing results, or only the plan designers run the test.

Choosing and ordering

Kind of techniquePrimary technique, runs in step 5.3
Phase taganalyse
Other phase tagsvalidate
Kind of findingrisk; difference
Question answeredwhich option; what next
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to frame

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.

StakesCheckPerson involvedCounts as a different method
exploratoryassumption and limitation statementnoneno
decision-informingassumption and limitation statementnoneno
decision-informingindependent reviewneeded; left out below high stakesno
decision-informingsource-span citationnoneno
high-stakesassumption and limitation statementnoneno
high-stakesindependent reviewneededno
high-stakessource-span citationnoneno
high-stakesexpert reviewneededno
high-stakesprovenance lognoneno

Failure modes and limits

Model suitabilityhuman judgement required
Typical model failureChooses mild conditions and sets success criteria after seeing how the plan performs.
Error sourcesexpert opinion; source selection; language model
Certainty limitsConfidence drops when conditions are chosen by the plan designers (risk of bias) or when no effectiveness evidence exists for the method (indirectness).

Evidence and tag basis

Documented. Listed in the UK Futures Toolkit; no effectiveness evidence found. (GOV.UK Futures Toolkit case study)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.