startthat.work home

CK-05 Out-of-sample validation (train and test split; k-fold cross-validation; bootstrap; permutation test)

Tune prompts, rules or models on one set of items and score them on a held-out set, and produce an out-of-sample error, interval or null distribution in a validation report.

Check extracted resultsEvidence: DocumentedVerification technique, runs in step 6Strongest statement: predict, without claiming a cause

What it does

Tunes prompts, rules or models on one set of items and scores them on another. Artefact: out-of-sample error, interval or null distribution. Read the gap between training and test results.

When to use it

Use when a model or a pattern found by search needs an honest estimate of how it performs on items it has not seen.

When it does not fit

Use for any model or any pattern found by search. Random splits leak when items are near-duplicates or share a source (inference).

Result format

The result reports the out-of-sample error, interval or null distribution for the tuned prompts, rules or model, beside the result on the training sample. It states the gap between the training and test results and says whether the held-out test sample has now been opened.

Requirements on the material

PropertyRequirement typeRequirement
Unit of analysisAlways appliesrequires=record; requires=document
Level of detail availableAlways appliesrequires=record-level
Appraises source credibilityApplies in some casesno

Not restricted on: Structure, Content type, Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialenough items that both the training and the held-out part support an error estimate, usually several hundred
Additional sources needednone
Not for (rules)none
Not for (text)Wrong choice when items are near-duplicates or share a source, because a random split then leaks. Reporting training performance, or reporting the best of many tested patterns, is a misuse.

Choosing and ordering

Kind of techniqueVerification technique, runs in step 6
Phase tagvalidate
Other phase tagsanalyse
Kind of findingrisk
Question answeredhow much; which option
Strongest statement it supportspredict, without claiming a cause
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to analyse

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: repeatable with fixed settings.

StakesCheckPerson involvedCounts as a different method
exploratoryinput validationnoneno
exploratoryleakage checknoneno
decision-informinginput validationnoneno
decision-informingleakage checknoneno
decision-informingcode execution for arithmeticnoneno
high-stakesinput validationnoneno
high-stakesleakage checknoneno
high-stakescode execution for arithmeticnoneno
high-stakessensitivity analysisnoneno
high-stakesindependent reviewneededno

Failure modes and limits

Model suitabilitymodel plus code
Typical model failureReuses the examples that shaped the prompt as test items, reports training performance as evidence, or tests many patterns and reports the best.
Error sourcesinput data; data transformation; statistical assumptions
Certainty limitsCertainty drops when the split leaks (risk of bias), when the held-out part is small (imprecision) and when the future data differ from the test data (indirectness).

Evidence and tag basis

Documented. In V-fold cross-validation all predictions are made on data out of sample. Cross-validation documentation was not opened. (Grimmer and Stewart 2013, Political Analysis)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.