startthat.work home

TC1-26 Thought experiments

Constructs an imagined case to test a principle and produces a judgement about the case that counts as evidence on the principle only if reasoners agree.

Trace cause and test claimsEvidence: UnconfirmedPrimary technique, runs in step 5.3Strongest statement: describe

What it does

Constructs an imagined case to test a principle. Artefact: a judgement about the case, which is evidence about the principle only if reasoners agree.

When to use it

Use when a principle cannot be tested on real cases and you want to see whether it gives acceptable answers in a constructed one.

When it does not fit

Details of the case, not the principle, determine the judgement.

Result format

The result describes the imagined case, states the principle it tests and gives the judgement about the case. It notes whether the reasoners agree, since the judgement is evidence about the principle only if they do.

Requirements on the material

PropertyRequirement typeRequirement
Content typeAlways appliesrequires=text
Unit of analysisAlways appliesrequires=claim
Appraises source credibilityApplies in some casesno

Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialA principle to test and a constructed case that isolates the feature of interest.
Additional sources needednone
Not for (rules)none
Not for (text)Wrong choice when details of the case, and not the principle, determine the judgement.

Choosing and ordering

Kind of techniquePrimary technique, runs in step 5.3
Phase taganalyse
Other phase tagsnone
Kind of findingcontradiction; gap
Question answeredwhat; why
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to analyse

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.

StakesCheckPerson involvedCounts as a different method
exploratoryassumption and limitation statementnoneno
decision-informingassumption and limitation statementnoneno
decision-informingspot checkoptionalno
high-stakesassumption and limitation statementnoneno
high-stakesspot checkoptionalno
high-stakesindependent reviewneededno

Failure modes and limits

Model suitabilityhuman judgement required
Typical model failureBuilds a case whose incidental details produce the judgement and counts the model's own reaction as evidence about the principle.
Error sourcescoder or rater judgement; expert opinion; language model
Certainty limitsA judgement from one reasoner is weak evidence. Certainty drops when reasoners disagree or when the result changes with details of the case.

Evidence and tag basis

Unconfirmed. The source entry failed to load, so no claim was verified.

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.