startthat.work home

Trace cause and test claims

This is the largest family. It holds techniques for finding the cause of a problem (five whys, fishbone diagrams, fault trees), for testing rival explanations against evidence (process tracing, counterfactual analysis) and for estimating the effect of a change (difference-in-differences). The techniques support different strengths of statement, and the Dictionary explains how each technique's strongest statement is recorded.

34 techniques. Open one to see what it does, when it fits and what it needs from your material.

IDTechniquePurposeEvidence status
TC1-01Five whysAsks why repeatedly about one stated problem and records a single chain of causes, leaving a candidate root cause as a result.Practice
TC1-02Fishbone diagram (Ishikawa diagram, cause and effect diagram)Groups brainstormed causes of one problem under category branches and produces a fishbone diagram with the problem as its head.Documented
TC1-03Fault tree analysis Also serves: Assess actors, options and risks.Works downward from one undesired top event through AND and OR gates to produce a gate tree with minimal cut sets.Documented
TC1-04Process tracing (diagnostic tests: straw in the wind, hoop, smoking gun, doubly decisive)Tests what each rival hypothesis predicts against diagnostic evidence within one case and produces a test result per hypothesis in a hypotheses matrix.Documented
TC1-05Counterfactual analysis (potential outcomes, minimal rewrite of history) Also serves: Compare.Sets the actual outcome against an argued account of what would have happened had one factor differed and produces a counterfactual scenario.Documented
TC1-06Difference-in-differences Also serves: Compare.Compares the change over time in a treated group with the change in a control group and produces an interaction coefficient read as the treatment effect.Documented
TC1-07Quasi-experimental designs (natural experiment, instrumental variables, regression discontinuity)Estimates an effect from observational data in which exposure was assigned as if at random (natural experiment, instrument or cutoff) and produces an effect estimate under stated assumptions.Unconfirmed
TC1-08Confounder identification with directed acyclic graphs (adjustment sets, backdoor criterion)Draws assumed causal links as a directed acyclic graph and produces a diagram and an adjustment set of variables to include and to leave out.Unconfirmed
TC1-09Logic model and theory of changeMaps the expected causal chain from inputs to outcomes with the assumption behind each step and produces a logic model or theory of change.Documented
TC1-10Contribution analysisBuilds a reasoned line of evidence that a program contributed to an outcome and produces an evidenced contribution story.Documented
TC1-11Realist evaluation (context, mechanism, outcome configurations)Tests what works for whom and in what circumstances through context, mechanism and outcome configurations and produces a refined program theory.Documented
TC1-12Process evaluation and impact evaluationExamines how an intervention was delivered and estimates what difference it made, and produces findings on delivery quality or an attributed effect estimate.Documented
TC1-13Analysis of competing hypotheses (ACH)Lists plausible hypotheses, rates each evidence item against each, and produces a hypothesis by evidence matrix aimed at disproving hypotheses.Tested
TC1-14Coherence correction and averaging of probability judgementsAdjusts each analyst's probabilities on mutually exclusive hypotheses so they sum to one, averages across analysts, and produces a corrected aggregated judgement.Tested
TC1-15Bayesian updating (likelihood reasoning, likelihood ratio, Bayes factor)Revises the probability of a hypothesis in proportion to how much more likely the evidence is under it than under alternatives and produces a posterior probability or odds.Unconfirmed
TC1-16Disconfirmation search (falsification, seeking disconfirming evidence)Looks for evidence that would contradict a favoured claim and produces a list of refuting evidence found or not found with the search used.Documented
TC1-17Sensitivity analysis (robustness check, leave-one-out)Re-runs an analysis under different assumptions or inclusion rules and produces a table of results by assumption showing whether the conclusion reverses.Documented
TC1-18Internal consistency check (term from earlier work)Tests whether the claims, figures and assumptions within one analysis are compatible with each other and produces a list of contradictions or inconsistent figures.Unconfirmed
TC1-19Toulmin argument analysisLabels the parts of an argument as claim, ground, warrant, backing, rebuttal and qualifier and produces a diagram per argument.Documented
TC1-20Argument mapping (argument diagramming)Draws claims, reasons and objections as a diagram and produces an argument map that shows isolated claims and reasons not connected to any conclusion.Unconfirmed
TC1-21Fallacy analysis (informal fallacies, pragma-dialectical rules)Compares an argument with a list of known defective patterns and produces a list of suspected defects, each read as a prompt for missing support.Documented
TC1-22Abduction and inference to the best explanationRanks candidate explanations of an observation on simplicity, generality and coherence and produces a ranked list of explanations.Documented
TC1-23Ladder of inferenceTraces how a conclusion was reached from observed data through selected data and interpretation to belief and produces a stepwise list.Practice
TC1-24Reflective equilibriumAdjusts principles and case judgements against each other until they cohere and produces a revised set of principles and judgements.Documented
TC1-25First principles reasoningReplaces inherited convention with a derivation from basic premises and produces a derivation chain with the base premises listed as assumptions.Practice
TC1-26Thought experimentsConstructs an imagined case to test a principle and produces a judgement about the case that counts as evidence on the principle only if reasoners agree.Unconfirmed
TC1-27Fermi estimation and order-of-magnitude checks (back-of-the-envelope estimate)Breaks an unknown quantity into factors that can be estimated, multiplies them, and produces a number with a plausible range used as a bound on a detailed result.Tested
TC1-28Dimensional and unit checksChecks that both sides of a relation have the same units and produces a unit balance in which a mismatch flags an error in the derivation.Documented
TC-01GRIM test (granularity-related inconsistency of means)Checks whether a reported mean of whole-number data can arise from the reported sample size and item count and produces a flag for possible or impossible means.Documented
TC-02SPRITE (Sample Parameter Reconstruction via Iterative TEchniques)Reconstructs possible datasets that match reported descriptive statistics and produces plausible samples showing the range of possible item distributions.Documented
TC-03statcheck (recomputation of reported p-values)Recomputes p-values from reported test statistics and degrees of freedom and produces a list of inconsistencies and gross inconsistencies.Tested
TC-04Claim and citation verification against source text (fact verification; claim checking; citation support check; quotation accuracy check)Labels each claim as supported, refuted or not enough information against retrieved evidence, or rates whether a cited source supports its claim, and produces a label with evidence sentences.Tested
TC-05Subgroup and Simpson's paradox check (stratified reanalysis; confounding check; collapsibility check)Recomputes an association within subgroups to see whether the pooled direction reverses or vanishes and produces pooled and stratified estimates side by side.Documented
TC-06Ecological fallacy check (aggregation bias check)Decides whether a claim about people is supported by group-level data and produces a decision on whether the claim is group-level or individual-level.Documented