startthat.work home

TC1-23 Ladder of inference

Traces how a conclusion was reached from observed data through selected data and interpretation to belief and produces a stepwise list.

Trace cause and test claimsEvidence: PracticeVerification technique, runs in step 6Strongest statement: describe

What it does

Traces how a conclusion was reached, from observed data through selected data and interpretation to belief. Artefact: a stepwise list; the step where data selection or interpretation was added is the likely error.

When to use it

Use when a conclusion seems firmly held and you want to find the step where data were selected or interpreted that may have introduced an error.

When it does not fit

A conclusion cannot be linked to observed data.

Result format

The result is a stepwise list from the observed data, through the data selected and the interpretation added, to the belief held, with the content of each step written out. It names the step where data selection or interpretation was added as the likely error.

Requirements on the material

PropertyRequirement typeRequirement
Content typeAlways appliesrequires=text
Unit of analysisAlways appliesrequires=claim
Appraises source credibilityApplies in some casesno

Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialA stated conclusion and an account of the data behind it.
Additional sources needednone
Not for (rules)none
Not for (text)Wrong choice when a conclusion cannot be linked to any observed data. The reconstruction is a prompt for inquiry and not evidence of bias.

Choosing and ordering

Kind of techniqueVerification technique, runs in step 6
Phase tagvalidate
Other phase tagsanalyse
Kind of findinggap; contradiction
Question answeredwhy
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to gather

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.

StakesCheckPerson involvedCounts as a different method
exploratoryspot checkoptionalno
decision-informingspot checkoptionalno
decision-informingsource-span citationnoneno
high-stakesspot checkoptionalno
high-stakessource-span citationnoneno
high-stakesindependent reviewneededno

Failure modes and limits

Model suitabilityhuman judgement required
Typical model failureReconstructs a plausible ladder after the fact that the author did not follow, and attributes data selection to bias without evidence.
Error sourcesinput data; coder or rater judgement; language model
Certainty limitsThe ladder is a reconstruction of reasoning and describes it only. Certainty drops when the account of the data is supplied by the person whose reasoning is traced.

Evidence and tag basis

Practice. Attributed to Argyris; the page gives no rungs, evidence or criticism. (Wikipedia, Ladder of inference)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.