AO-09 Longlist-to-shortlist options analysis (options appraisal)
Generates a wide set of options, filters them against objectives and critical success factors, and appraises a shortlist of about five, producing a longlist and shortlist with red, amber and green ratings.
What it does
Generates a wide set of options, filters them against objectives and critical success factors, and appraises a shortlist of about five. Artefact: a longlist and shortlist with red, amber, green ratings.
When to use it
Use when there are more candidates than can be appraised in detail, such as projects or initiatives competing for funding, and a defensible shortlist is needed before detailed scoring.
When it does not fit
Objectives are output-based or not measurable, so options cannot be screened.
Result format
The result has a longlist of options and a shortlist of about five, with a red, amber or green rating for each option against each objective and critical success factor. It states why each option was dropped or kept, in a comparison matrix.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Unit of analysis | Always applies | requires=topic |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Content type, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | Measurable objectives and critical success factors to screen against, with expert and affected-party input, often gathered in workshops. |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Wrong choice when objectives are output-based or not measurable, so options cannot be screened. |
Choosing and ordering
| Kind of technique | Primary technique, runs in step 5.3 |
|---|---|
| Phase tag | analyse |
| Other phase tags | frame |
| Kind of finding | option ranking |
| Question answered | which option |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to frame |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | assumption and limitation statement | none | no |
| decision-informing | assumption and limitation statement | none | no |
| decision-informing | source-span citation | none | no |
| decision-informing | human review sample | needed; left out below high stakes | no |
| high-stakes | assumption and limitation statement | none | no |
| high-stakes | source-span citation | none | no |
| high-stakes | human review sample | needed | no |
| high-stakes | independent review | needed | no |
| high-stakes | expert review | needed | no |
Failure modes and limits
| Model suitability | human judgement required |
|---|---|
| Typical model failure | Screens options against vague objectives so that ratings are asserted and not tested, and drops an option without a recorded reason. |
| Error sources | input data; source selection; coder or rater judgement; expert opinion |
| Certainty limits | Certainty drops when screening criteria are not measurable (indirectness), when options were generated by a narrow group (risk of bias), and when only part of the guidance was read. |
Related techniques
- should follow FQ-02 Decision-first framing (decision-focused or value-focused question design): the decision and its objectives set the screening criteria
- pairs with AO-12 Multi-criteria decision analysis (MCDA): weighted scoring appraises the shortlist more finely than red, amber and green ratings
Evidence and tag basis
Documented. Described in the UK Green Book, of which only part was read. (HM Treasury, The Green Book 2020)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.