startthat.work home

CK-12 Specification sweep across models and prompts (LLM hacking test; multiverse analysis; specification curve)

Repeat the downstream statistical test across several models, prompts and settings, and produce error rates for false positive, false negative, wrong-sign and wrong-magnitude conclusions.

Check extracted resultsEvidence: TestedVerification technique, runs in step 6Strongest statement: describe

What it does

Repeats the downstream statistical test across several models, prompts and settings and reports the share of conclusions that change. Artefact: error rates for false positive, false negative, wrong-sign and wrong-magnitude conclusions. Read results near a significance threshold with extra care.

When to use it

Use when a significance claim rests on language model labels and you need to know how often the conclusion would change under other defensible choices.

When it does not fit

Use for any significance claim built on LLM labels. The multiverse and specification curve names are general knowledge and were not confirmed.

Result format

The result is a table with one row per model, prompt and setting combination, showing the downstream test result, followed by error rates for false positive, false negative, wrong-sign and wrong-magnitude conclusions. It states the share of conclusions that change and names the results near the significance threshold.

Requirements on the material

PropertyRequirement typeRequirement
Content typeAlways appliesrequires=text; requires=tabular
Unit of analysisAlways appliesrequires=document; requires=record
Level of detail availableAlways appliesrequires=record-level
Appraises source credibilityApplies in some casesno

Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialhuman-coded annotations for a sample and at least several model and prompt specifications
Additional sources needednone
Not for (rules)none
Not for (text)Wrong choice when the analyst will select the favourable prompt after seeing results. Results near a significance threshold need extra care.

Choosing and ordering

Kind of techniqueVerification technique, runs in step 6
Phase tagvalidate
Other phase tagsconclude
Kind of findingrisk; difference
Question answeredhow much
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetproceed with limit stated

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.

StakesCheckPerson involvedCounts as a different method
exploratoryrepeat runsnoneno
decision-informingrepeat runsnoneno
decision-informingsensitivity analysisnoneno
decision-informinggold standard agreementconditional; left out below high stakesno
high-stakesrepeat runsnoneno
high-stakessensitivity analysisnoneno
high-stakesgold standard agreementconditionalno
high-stakesmultiple comparison correctionnoneno
high-stakesindependent reviewneededno

Failure modes and limits

Model suitabilityhuman judgement required
Typical model failureThe analyst selects the favourable prompt.
Error sourcesinput data; language model; statistical assumptions
Certainty limitsCertainty drops when conclusions change across specifications (inconsistency) and when human annotations are missing (risk of bias).

Evidence and tag basis

Tested. Across 13 million labels from 18 LLMs and 2,361 hypotheses, about 31 per cent of hypotheses reached incorrect conclusions with state-of-the-art models and about half with smaller models. A few prompt paraphrases made nearly any result significant, and human annotations were the most important of 21 safeguards (2025 preprint). (Baumann et al. 2025, arXiv 2509.08825)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.