TC1-23 Ladder of inference
Traces how a conclusion was reached from observed data through selected data and interpretation to belief and produces a stepwise list.
What it does
Traces how a conclusion was reached, from observed data through selected data and interpretation to belief. Artefact: a stepwise list; the step where data selection or interpretation was added is the likely error.
When to use it
Use when a conclusion seems firmly held and you want to find the step where data were selected or interpreted that may have introduced an error.
When it does not fit
A conclusion cannot be linked to observed data.
Result format
The result is a stepwise list from the observed data, through the data selected and the interpretation added, to the belief held, with the content of each step written out. It names the step where data selection or interpretation was added as the likely error.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Content type | Always applies | requires=text |
| Unit of analysis | Always applies | requires=claim |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | A stated conclusion and an account of the data behind it. |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Wrong choice when a conclusion cannot be linked to any observed data. The reconstruction is a prompt for inquiry and not evidence of bias. |
Choosing and ordering
| Kind of technique | Verification technique, runs in step 6 |
|---|---|
| Phase tag | validate |
| Other phase tags | analyse |
| Kind of finding | gap; contradiction |
| Question answered | why |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to gather |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | spot check | optional | no |
| decision-informing | spot check | optional | no |
| decision-informing | source-span citation | none | no |
| high-stakes | spot check | optional | no |
| high-stakes | source-span citation | none | no |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | human judgement required |
|---|---|
| Typical model failure | Reconstructs a plausible ladder after the fact that the author did not follow, and attributes data selection to bias without evidence. |
| Error sources | input data; coder or rater judgement; language model |
| Certainty limits | The ladder is a reconstruction of reasoning and describes it only. Certainty drops when the account of the data is supplied by the person whose reasoning is traced. |
Related techniques
- pairs with CH-13 Cognitive bias checks and debiasing (including consider-the-opposite): bias checks name the judgement biases at the rungs where selection happened
- pairs with CH-12 Socratic questioning (Paul and Elder's six question types): Socratic questions elicit the account of the data and reasoning
Evidence and tag basis
Practice. Attributed to Argyris; the page gives no rungs, evidence or criticism. (Wikipedia, Ladder of inference)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.