startthat.work home

CK-10 Repeated-run consistency test (test-retest; intra-coder reliability; TARr@N and TARa@N (raw and parsed answer agreement); settings pinning)

Run the same prompt and input several times with model version, temperature and seed recorded, and produce agreement rates and accuracy spread in a validation report.

Check extracted resultsEvidence: TestedVerification technique, runs in step 6Strongest statement: describe

What it does

Runs the same prompt and input N times, with model version, temperature and seed fixed and recorded, and measures agreement. Artefact: agreement rates and accuracy spread. Read both the spread and whether outputs are identical strings.

When to use it

Use when a conclusion depends on a language model step and you need to know whether it gives the same answer each time.

When it does not fit

Use for any LLM step on which a conclusion depends. Misfit: only one run exists. Pinning settings reduces variance but does not remove it.

Result format

The result gives the number of runs, the fixed and recorded model version, temperature and seed, the agreement rate across runs and the spread in accuracy. It also states whether the outputs were identical strings.

Requirements on the material

PropertyRequirement typeRequirement
Content typeAlways appliesrequires=text; requires=tabular
Unit of analysisAlways appliesrequires=record; requires=document; requires=claim
Level of detail availableAlways appliesrequires=record-level
Appraises source credibilityApplies in some casesno

Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialat least 5 repeated runs on a sample of items
Additional sources needednone
Not for (rules)none
Not for (text)Wrong choice when only one run exists and none can be made. Pinning settings reduces variance but does not remove it, and stable output is not correct output.

Choosing and ordering

Kind of techniqueVerification technique, runs in step 6
Phase tagvalidate
Other phase tagsanalyse
Kind of findingrisk
Question answeredhow much
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to analyse

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.

StakesCheckPerson involvedCounts as a different method
exploratoryrepeat runsnoneno
decision-informingrepeat runsnoneno
decision-informingprompt variationnoneno
decision-informingspot checkoptionalno
high-stakesrepeat runsnoneno
high-stakesprompt variationnoneno
high-stakesspot checkoptionalno
high-stakessensitivity analysisnoneno
high-stakesindependent reviewneededno

Failure modes and limits

Model suitabilitymodel plus code
Typical model failureReports a single run as the result, or assumes temperature 0 gives identical output.
Error sourcesinput data; language model
Certainty limitsCertainty drops when runs disagree (inconsistency) and when agreement is read as accuracy (indirectness).

Evidence and tag basis

Tested. Across five LLMs, eight tasks and ten runs each, accuracy varied by up to 15 per cent between runs, the best-to-worst gap reached 70 per cent, and no model was consistent on all tasks (2024). (Atil et al. 2024, arXiv 2408.04667; Reiss, arXiv 2304.11085)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.