startthat.work home

CK-11 Prompt stability scoring (prompt perturbation; paraphrase test)

Re-run a task with paraphrased prompts and apply reliability metrics to the labels, and produce a prompt stability score in a validation report.

Check extracted resultsEvidence: TestedVerification technique, runs in step 6Strongest statement: describe

What it does

Re-runs the task with paraphrased prompts and applies reliability metrics to the labels. Artefact: a stability score. Read it before scaling a classification prompt to a large corpus.

When to use it

Use when a classification prompt will be applied to a large corpus and you need to know whether wording changes the labels.

When it does not fit

Fits a classification prompt that will be used on a large corpus. Misfit: paraphrases that differ in substance. The paraphrase set defines what counts as a small change.

Result format

The result lists the paraphrased prompts used and gives a stability score for the labels across them. It states whether the score is high enough to scale the classification prompt to the large corpus.

Requirements on the material

PropertyRequirement typeRequirement
Content typeAlways appliesrequires=text
Unit of analysisAlways appliesrequires=document; requires=record
Level of detail availableAlways appliesrequires=record-level
Appraises source credibilityApplies in some casesno

Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialat least 5 paraphrased prompts and a sample of items to label under each
Additional sources needednone
Not for (rules)none
Not for (text)Wrong choice when the paraphrases differ in substance, because the paraphrase set defines what counts as a small change.

Choosing and ordering

Kind of techniqueVerification technique, runs in step 6
Phase tagvalidate
Other phase tagsanalyse
Kind of findingrisk
Question answeredhow much
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to analyse

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.

StakesCheckPerson involvedCounts as a different method
exploratoryprompt variationnoneno
decision-informingprompt variationnoneno
decision-informingrepeat runsnoneno
decision-informingspot checkoptionalno
high-stakesprompt variationnoneno
high-stakesrepeat runsnoneno
high-stakesspot checkoptionalno
high-stakessensitivity analysisnoneno
high-stakesindependent reviewneededno

Failure modes and limits

Model suitabilitymodel plus code
Typical model failureResults change with the wording and nobody notices.
Error sourcesinput data; language model
Certainty limitsCertainty drops when labels change across paraphrases (inconsistency) and when the paraphrase set is narrow (indirectness).

Evidence and tag basis

Tested. A prompt stability score was tested on about 3.1 million rows across six datasets and twelve outcomes (2024); the scores were not on the abstract page. (Barrie, Palaiologou and Tornberg 2024, arXiv 2407.02039)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.