CK-11 Prompt stability scoring (prompt perturbation; paraphrase test)
Re-run a task with paraphrased prompts and apply reliability metrics to the labels, and produce a prompt stability score in a validation report.
What it does
Re-runs the task with paraphrased prompts and applies reliability metrics to the labels. Artefact: a stability score. Read it before scaling a classification prompt to a large corpus.
When to use it
Use when a classification prompt will be applied to a large corpus and you need to know whether wording changes the labels.
When it does not fit
Fits a classification prompt that will be used on a large corpus. Misfit: paraphrases that differ in substance. The paraphrase set defines what counts as a small change.
Result format
The result lists the paraphrased prompts used and gives a stability score for the labels across them. It states whether the score is high enough to scale the classification prompt to the large corpus.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Content type | Always applies | requires=text |
| Unit of analysis | Always applies | requires=document; requires=record |
| Level of detail available | Always applies | requires=record-level |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | at least 5 paraphrased prompts and a sample of items to label under each |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Wrong choice when the paraphrases differ in substance, because the paraphrase set defines what counts as a small change. |
Choosing and ordering
| Kind of technique | Verification technique, runs in step 6 |
|---|---|
| Phase tag | validate |
| Other phase tags | analyse |
| Kind of finding | risk |
| Question answered | how much |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to analyse |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | prompt variation | none | no |
| decision-informing | prompt variation | none | no |
| decision-informing | repeat runs | none | no |
| decision-informing | spot check | optional | no |
| high-stakes | prompt variation | none | no |
| high-stakes | repeat runs | none | no |
| high-stakes | spot check | optional | no |
| high-stakes | sensitivity analysis | none | no |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | model plus code |
|---|---|
| Typical model failure | Results change with the wording and nobody notices. |
| Error sources | input data; language model |
| Certainty limits | Certainty drops when labels change across paraphrases (inconsistency) and when the paraphrase set is narrow (indirectness). |
Related techniques
- should follow MM-08 LLM coding and zero-shot classification of text (LLM annotation; LLM-assisted deductive coding; few-shot classification; natural language inference classification): the prompt under test comes from the coding step
- pairs with CK-10 Repeated-run consistency test (test-retest; intra-coder reliability; TARr@N and TARa@N (raw and parsed answer agreement); settings pinning): repeated runs test stability across runs and not across wording
- pairs with CK-12 Specification sweep across models and prompts (LLM hacking test; multiverse analysis; specification curve): a specification sweep shows downstream effects of prompt choice
Evidence and tag basis
Tested. A prompt stability score was tested on about 3.1 million rows across six datasets and twelve outcomes (2024); the scores were not on the abstract page. (Barrie, Palaiologou and Tornberg 2024, arXiv 2407.02039)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.