CK-10 Repeated-run consistency test (test-retest; intra-coder reliability; TARr@N and TARa@N (raw and parsed answer agreement); settings pinning)
Run the same prompt and input several times with model version, temperature and seed recorded, and produce agreement rates and accuracy spread in a validation report.
What it does
Runs the same prompt and input N times, with model version, temperature and seed fixed and recorded, and measures agreement. Artefact: agreement rates and accuracy spread. Read both the spread and whether outputs are identical strings.
When to use it
Use when a conclusion depends on a language model step and you need to know whether it gives the same answer each time.
When it does not fit
Use for any LLM step on which a conclusion depends. Misfit: only one run exists. Pinning settings reduces variance but does not remove it.
Result format
The result gives the number of runs, the fixed and recorded model version, temperature and seed, the agreement rate across runs and the spread in accuracy. It also states whether the outputs were identical strings.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Content type | Always applies | requires=text; requires=tabular |
| Unit of analysis | Always applies | requires=record; requires=document; requires=claim |
| Level of detail available | Always applies | requires=record-level |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | at least 5 repeated runs on a sample of items |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Wrong choice when only one run exists and none can be made. Pinning settings reduces variance but does not remove it, and stable output is not correct output. |
Choosing and ordering
| Kind of technique | Verification technique, runs in step 6 |
|---|---|
| Phase tag | validate |
| Other phase tags | analyse |
| Kind of finding | risk |
| Question answered | how much |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to analyse |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | repeat runs | none | no |
| decision-informing | repeat runs | none | no |
| decision-informing | prompt variation | none | no |
| decision-informing | spot check | optional | no |
| high-stakes | repeat runs | none | no |
| high-stakes | prompt variation | none | no |
| high-stakes | spot check | optional | no |
| high-stakes | sensitivity analysis | none | no |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | model plus code |
|---|---|
| Typical model failure | Reports a single run as the result, or assumes temperature 0 gives identical output. |
| Error sources | input data; language model |
| Certainty limits | Certainty drops when runs disagree (inconsistency) and when agreement is read as accuracy (indirectness). |
Related techniques
- should follow MM-08 LLM coding and zero-shot classification of text (LLM annotation; LLM-assisted deductive coding; few-shot classification; natural language inference classification): the model coding step is what gets repeated
- pairs with CK-11 Prompt stability scoring (prompt perturbation; paraphrase test): prompt paraphrases test a different source of instability
- pairs with CK-12 Specification sweep across models and prompts (LLM hacking test; multiverse analysis; specification curve): a specification sweep shows downstream effects of instability
Evidence and tag basis
Tested. Across five LLMs, eight tasks and ten runs each, accuracy varied by up to 15 per cent between runs, the best-to-worst gap reached 70 per cent, and no model was consistent on all tasks (2024). (Atil et al. 2024, arXiv 2408.04667; Reiss, arXiv 2304.11085)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.