CK-12 Specification sweep across models and prompts (LLM hacking test; multiverse analysis; specification curve)
Repeat the downstream statistical test across several models, prompts and settings, and produce error rates for false positive, false negative, wrong-sign and wrong-magnitude conclusions.
What it does
Repeats the downstream statistical test across several models, prompts and settings and reports the share of conclusions that change. Artefact: error rates for false positive, false negative, wrong-sign and wrong-magnitude conclusions. Read results near a significance threshold with extra care.
When to use it
Use when a significance claim rests on language model labels and you need to know how often the conclusion would change under other defensible choices.
When it does not fit
Use for any significance claim built on LLM labels. The multiverse and specification curve names are general knowledge and were not confirmed.
Result format
The result is a table with one row per model, prompt and setting combination, showing the downstream test result, followed by error rates for false positive, false negative, wrong-sign and wrong-magnitude conclusions. It states the share of conclusions that change and names the results near the significance threshold.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Content type | Always applies | requires=text; requires=tabular |
| Unit of analysis | Always applies | requires=document; requires=record |
| Level of detail available | Always applies | requires=record-level |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | human-coded annotations for a sample and at least several model and prompt specifications |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Wrong choice when the analyst will select the favourable prompt after seeing results. Results near a significance threshold need extra care. |
Choosing and ordering
| Kind of technique | Verification technique, runs in step 6 |
|---|---|
| Phase tag | validate |
| Other phase tags | conclude |
| Kind of finding | risk; difference |
| Question answered | how much |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | proceed with limit stated |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | repeat runs | none | no |
| decision-informing | repeat runs | none | no |
| decision-informing | sensitivity analysis | none | no |
| decision-informing | gold standard agreement | conditional; left out below high stakes | no |
| high-stakes | repeat runs | none | no |
| high-stakes | sensitivity analysis | none | no |
| high-stakes | gold standard agreement | conditional | no |
| high-stakes | multiple comparison correction | none | no |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | human judgement required |
|---|---|
| Typical model failure | The analyst selects the favourable prompt. |
| Error sources | input data; language model; statistical assumptions |
| Certainty limits | Certainty drops when conclusions change across specifications (inconsistency) and when human annotations are missing (risk of bias). |
Related techniques
- should follow MM-08 LLM coding and zero-shot classification of text (LLM annotation; LLM-assisted deductive coding; few-shot classification; natural language inference classification): the labels and downstream test come from the coding step
- pairs with CK-06 Bias correction of model labels with a human-coded subsample (design-based supervised learning (DSL); prediction-powered inference (PPI); regression estimator correction): bias correction is the remedy when the sweep shows label error matters
- pairs with CK-13 Multiple-testing correction (Benjamini-Hochberg false discovery rate control; Bonferroni; Holm): many specifications multiply the number of tests
Evidence and tag basis
Tested. Across 13 million labels from 18 LLMs and 2,361 hypotheses, about 31 per cent of hypotheses reached incorrect conclusions with state-of-the-art models and about half with smaller models. A few prompt paraphrases made nearly any result significant, and human annotations were the most important of 21 safeguards (2025 preprint). (Baumann et al. 2025, arXiv 2509.08825)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.