TC1-17 Sensitivity analysis (robustness check, leave-one-out)
Re-runs an analysis under different assumptions or inclusion rules and produces a table of results by assumption showing whether the conclusion reverses.
What it does
Re-runs an analysis under different assumptions or inclusion rules. Artefact: a table of results by assumption; read whether the conclusion reverses.
When to use it
Use when you want to know how much a result depends on the choices made in producing it.
When it does not fit
Only convenient assumptions are varied.
Result format
The result is a table with one row per changed assumption or inclusion rule and the result obtained under each, alongside the original result. It ends by stating whether the conclusion reverses under any row.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Unit of analysis | Always applies | requires=dataset; requires=claim |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Content type, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | A completed analysis that can be re-run and a written list of the uncertain choices to vary. |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Wrong choice when only convenient assumptions are varied. Agreement among variants of one analysis is consistency and not corroboration. |
Choosing and ordering
| Kind of technique | Verification technique, runs in step 6 |
|---|---|
| Phase tag | validate |
| Other phase tags | none |
| Kind of finding | difference; risk |
| Question answered | how much; what |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | proceed with limit stated |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: deterministic.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | input validation | none | no |
| decision-informing | input validation | none | no |
| decision-informing | code execution for arithmetic | none | no |
| decision-informing | spot check | optional | no |
| high-stakes | input validation | none | no |
| high-stakes | code execution for arithmetic | none | no |
| high-stakes | spot check | optional | no |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | model plus code |
|---|---|
| Typical model failure | Varies only convenient assumptions and reports that the result holds, or changes several assumptions at once so the cause of a change is unclear. |
| Error sources | input data; statistical assumptions; data transformation |
| Certainty limits | Certainty drops when only favourable alternatives are tried, when choices are not listed in advance, or when the original analysis is itself unsound. |
Related techniques
- pairs with TC1-06 Difference-in-differences Also serves: Compare.: varying pre-periods and controls tests the parallel trends assumption
- pairs with TC1-18 Internal consistency check (term from earlier work): sensitivity varies the choices and the consistency check tests the figures themselves
Evidence and tag basis
Documented. The Cochrane Handbook advises it because meta-analysis results depend on many judgements. (Cochrane Handbook, chapter 10)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.