CK-05 Out-of-sample validation (train and test split; k-fold cross-validation; bootstrap; permutation test)
Tune prompts, rules or models on one set of items and score them on a held-out set, and produce an out-of-sample error, interval or null distribution in a validation report.
What it does
Tunes prompts, rules or models on one set of items and scores them on another. Artefact: out-of-sample error, interval or null distribution. Read the gap between training and test results.
When to use it
Use when a model or a pattern found by search needs an honest estimate of how it performs on items it has not seen.
When it does not fit
Use for any model or any pattern found by search. Random splits leak when items are near-duplicates or share a source (inference).
Result format
The result reports the out-of-sample error, interval or null distribution for the tuned prompts, rules or model, beside the result on the training sample. It states the gap between the training and test results and says whether the held-out test sample has now been opened.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Unit of analysis | Always applies | requires=record; requires=document |
| Level of detail available | Always applies | requires=record-level |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Content type, Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | enough items that both the training and the held-out part support an error estimate, usually several hundred |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Wrong choice when items are near-duplicates or share a source, because a random split then leaks. Reporting training performance, or reporting the best of many tested patterns, is a misuse. |
Choosing and ordering
| Kind of technique | Verification technique, runs in step 6 |
|---|---|
| Phase tag | validate |
| Other phase tags | analyse |
| Kind of finding | risk |
| Question answered | how much; which option |
| Strongest statement it supports | predict, without claiming a cause |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to analyse |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: repeatable with fixed settings.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | input validation | none | no |
| exploratory | leakage check | none | no |
| decision-informing | input validation | none | no |
| decision-informing | leakage check | none | no |
| decision-informing | code execution for arithmetic | none | no |
| high-stakes | input validation | none | no |
| high-stakes | leakage check | none | no |
| high-stakes | code execution for arithmetic | none | no |
| high-stakes | sensitivity analysis | none | no |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | model plus code |
|---|---|
| Typical model failure | Reuses the examples that shaped the prompt as test items, reports training performance as evidence, or tests many patterns and reports the best. |
| Error sources | input data; data transformation; statistical assumptions |
| Certainty limits | Certainty drops when the split leaks (risk of bias), when the held-out part is small (imprecision) and when the future data differ from the test data (indirectness). |
Related techniques
- conflicts with CK-07 Split-sample design for measures discovered from text (split-sample framework): both spend the single-use held-out test sample; should follow :: family:Measure and model :: the modelled result is what gets tested
- pairs with CK-14 Data leakage audit (train-test contamination check; model information sheet): a leakage audit checks that the split is clean
- should follow PE-03 Near-duplicate detection (shingling; MinHash; locality-sensitive hashing; text reuse detection): near-duplicates must be removed before splitting
Evidence and tag basis
Documented. In V-fold cross-validation all predictions are made on data out of sample. Cross-validation documentation was not opened. (Grimmer and Stewart 2013, Political Analysis)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.