startthat.work home

Check library

A check tests one result for a specific weakness, for example by running the technique a second time or by comparing a sample with the source. Each technique lists the checks it needs at each stakes level, and a higher level includes the checks of the level below. Step 6 applies them. The last two columns say whether a person must be involved and whether the check counts as corroboration of a finding.

human_involvement says whether a check needs a person. needed: the process cannot do it. It applies at the high-stakes level only, where the process asks the user to review a sample and lists the check as not done if the user does not. Below the high-stakes level the plan leaves it out. optional: code or the model can do it, and a person can too. conditional: needed only when the stated condition holds, and then it follows the rule for needed. none: the process does it. Every check the plan lists that needs no person (including spot checks and repeat runs) is done by the process.

CheckWhat it testsPerson involvedCorroboration
spot check
A person or code compares a few outputs with the source by hand.
accuracyoptional
A person or code can do it. Without a person the process does it by code or by re-reading the source.
no
input validation
Code checks that the input meets the technique's declared format, ranges and units.
validity of inputnone
no
repeat runs
Run the same technique several times with the same settings or different seeds.
stabilitynone
no
prompt variation
Rephrase the instruction or reorder the input and compare outputs.
stabilitynone
no
human review sample
A person reviews a random sample of outputs and records the error rate.
accuracyneeded
A person reviews the sample.
no
gold standard agreement
Compare outputs with labels made by trained people, and report agreement.
accuracyconditional
Needed unless the material already contains labels made by trained people, when it can be computed.
no
second technique with different assumptions
Apply a second technique that differs in assumptions, data or direction of bias, and compare results.
independencenone
conditional
sensitivity analysis
Change key assumptions, thresholds or inclusion rules and report how the result moves.
robustnessnone
no
source-span citation
Tie each statement to the exact passage or cell that supports it.
traceabilitynone
no
code execution for arithmetic
Do calculations in code, not in the model's text.
accuracynone
no
held-out validation
Test on data kept unseen during development, once.
validitynone
no
leakage check
Check for information from the test data reaching development: preprocessing across sets, duplicates, features unavailable at prediction time, time order.
validitynone
no
multiple comparison correction
Adjust results when many tests or subgroups are examined.
validitynone
no
provenance log
Record the input, technique, settings and time for each result.
traceabilitynone
no
assumption and limitation statement
State linchpin assumptions, what happens if they are wrong, and what would change the finding.
transparencynone
no
alternative explanations check
List and assess plausible alternative hypotheses or explanations.
validitynone
no
independent review
A reviewer who did not produce the result checks it.
independenceneeded
A person who did not produce the result reviews it.
no
expert review
A subject expert reviews the result and its interpretation.
validityneeded
A subject expert reviews it.
no