Check library
A check tests one result for a specific weakness, for example by running the technique a second time or by comparing a sample with the source. Each technique lists the checks it needs at each stakes level, and a higher level includes the checks of the level below. Step 6 applies them. The last two columns say whether a person must be involved and whether the check counts as corroboration of a finding.
human_involvement says whether a check needs a person. needed: the process cannot do it. It applies at the high-stakes level only, where the process asks the user to review a sample and lists the check as not done if the user does not. Below the high-stakes level the plan leaves it out. optional: code or the model can do it, and a person can too. conditional: needed only when the stated condition holds, and then it follows the rule for needed. none: the process does it. Every check the plan lists that needs no person (including spot checks and repeat runs) is done by the process.
| Check | What it tests | Person involved | Corroboration |
|---|---|---|---|
| spot check A person or code compares a few outputs with the source by hand. | accuracy | optional A person or code can do it. Without a person the process does it by code or by re-reading the source. | no |
| input validation Code checks that the input meets the technique's declared format, ranges and units. | validity of input | none | no |
| repeat runs Run the same technique several times with the same settings or different seeds. | stability | none | no |
| prompt variation Rephrase the instruction or reorder the input and compare outputs. | stability | none | no |
| human review sample A person reviews a random sample of outputs and records the error rate. | accuracy | needed A person reviews the sample. | no |
| gold standard agreement Compare outputs with labels made by trained people, and report agreement. | accuracy | conditional Needed unless the material already contains labels made by trained people, when it can be computed. | no |
| second technique with different assumptions Apply a second technique that differs in assumptions, data or direction of bias, and compare results. | independence | none | conditional |
| sensitivity analysis Change key assumptions, thresholds or inclusion rules and report how the result moves. | robustness | none | no |
| source-span citation Tie each statement to the exact passage or cell that supports it. | traceability | none | no |
| code execution for arithmetic Do calculations in code, not in the model's text. | accuracy | none | no |
| held-out validation Test on data kept unseen during development, once. | validity | none | no |
| leakage check Check for information from the test data reaching development: preprocessing across sets, duplicates, features unavailable at prediction time, time order. | validity | none | no |
| multiple comparison correction Adjust results when many tests or subgroups are examined. | validity | none | no |
| provenance log Record the input, technique, settings and time for each result. | traceability | none | no |
| assumption and limitation statement State linchpin assumptions, what happens if they are wrong, and what would change the finding. | transparency | none | no |
| alternative explanations check List and assess plausible alternative hypotheses or explanations. | validity | none | no |
| independent review A reviewer who did not produce the result checks it. | independence | needed A person who did not produce the result reviews it. | no |
| expert review A subject expert reviews the result and its interpretation. | validity | needed A subject expert reviews it. | no |