CH-09 Structured self-critique
Asks about sources of uncertainty, evidence, gaps, critical assumptions and alternative hypotheses, and produces a written critique of one own analysis.
What it does
Asks about sources of uncertainty, evidence, gaps, critical assumptions and alternative hypotheses. Artefact: a written critique of one's own analysis.
When to use it
Use when you have drafted an analysis and want to find its gaps and weak assumptions before it is reviewed by others.
When it does not fit
It is applied dishonestly; the handbook says it is only as good as the honesty with which it is applied.
Result format
The result is a written critique of the analyst's own analysis that covers the sources of uncertainty, the evidence, the gaps, the critical assumptions and the alternative hypotheses. Each of these headings has its own section.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Unit of analysis | Always applies | requires=claim |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Content type, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | A draft analysis. |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | It is applied dishonestly, since the handbook says it is only as good as the honesty with which it is applied. |
Choosing and ordering
| Kind of technique | Challenge technique, runs in step 6 |
|---|---|
| Phase tag | validate |
| Other phase tags | conclude |
| Kind of finding | gap; risk |
| Question answered | what; why |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to analyse |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | assumption and limitation statement | none | no |
| decision-informing | assumption and limitation statement | none | no |
| decision-informing | independent review | needed; left out below high stakes | no |
| high-stakes | assumption and limitation statement | none | no |
| high-stakes | independent review | needed | no |
| high-stakes | expert review | needed | no |
| high-stakes | second technique with different assumptions | none | conditional |
Failure modes and limits
| Model suitability | model alone |
|---|---|
| Typical model failure | Reports that the analysis is sound and lists minor gaps, because the model reviews its own output with the same biases. |
| Error sources | input data; coder or rater judgement; language model |
| Certainty limits | A self-critique is not independent of the analysis it reviews; the result carries risk of bias and the technique overlaps with other checks. |
Related techniques
- pairs with CH-01 Key assumptions check (assumptions check): A key assumptions check covers the assumptions part of the critique
- pairs with CH-13 Cognitive bias checks and debiasing (including consider-the-opposite): A bias checklist adds known judgement biases to the critique
Evidence and tag basis
Documented. Overlaps with a key assumptions check plus alternative hypotheses (inference). (UK Red Teaming Handbook)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.