Check extracted results
These techniques measure how far coders, human or model, agree, score a model against a human coder, put a confidence interval around an agreement figure, build a human-coded sample as a standard, test on held-out items and correct bias in model labels using a human-coded subsample. They are how the process tests its own coding and labelling.
29 techniques. Open one to see what it does, when it fits and what it needs from your material.
| ID | Technique | Purpose | Evidence status |
|---|---|---|---|
| CK-01 | Inter-coder agreement statistics (percentage agreement; Cohen's kappa; weighted kappa; Scott's pi; Fleiss' kappa; Krippendorff's alpha; intercoder reliability testing) | Measure how far coders, human or model, give the same code to the same items, corrected for chance, and produce an agreement coefficient per code with a bootstrap interval in a validation report. | Documented |
| CK-02 | Model versus human coding agreement protocol (LLM-as-coder reliability test; intra-coder agreement across runs) | Score a model as one coder against a human coder or panel and against itself across repeated runs, and produce human-human, model-human and model-model agreement in a validation report. | Tested |
| CK-03 | Bootstrap confidence intervals for agreement and accuracy (bootstrap resampling of items) | Quantify sampling uncertainty in an agreement or accuracy figure computed from a review sample by resampling items, and produce an interval around the coefficient. | Documented |
| CK-04 | Gold-standard validation sample (ground truth set; human-coded validation set) | Build a random or stratified human-coded subset against which model output is scored and produce a labelled set. | Documented |
| CK-05 | Out-of-sample validation (train and test split; k-fold cross-validation; bootstrap; permutation test) | Tune prompts, rules or models on one set of items and score them on a held-out set, and produce an out-of-sample error, interval or null distribution in a validation report. | Documented |
| CK-06 | Bias correction of model labels with a human-coded subsample (design-based supervised learning (DSL); prediction-powered inference (PPI); regression estimator correction) | Combine model labels for all items with human labels for a random subsample so that downstream estimates and intervals stay valid, and produce a corrected estimate with an interval. | Tested |
| CK-07 | Split-sample design for measures discovered from text (split-sample framework) | Discover a measure such as categories or topics in one random part of the corpus and estimate its effect in the other, and produce an effect estimate that does not overfit the discovery. | Documented |
| CK-08 | Semantic, predictive and convergent validity checks (construct validity checks) | Test whether an extracted category or measure behaves as the concept it claims to measure, and produce rater verdicts and comparisons with a known event or established measure in a validation report. | Documented |
| CK-09 | Cluster validation indexes (silhouette score; adjusted Rand index; adjusted mutual information; V-measure; stability analysis) | Score how well a clustering fits the data or agrees with known labels, and produce a score per clustering for comparison across cluster numbers and algorithms. | Documented |
| CK-10 | Repeated-run consistency test (test-retest; intra-coder reliability; TARr@N and TARa@N (raw and parsed answer agreement); settings pinning) | Run the same prompt and input several times with model version, temperature and seed recorded, and produce agreement rates and accuracy spread in a validation report. | Tested |
| CK-11 | Prompt stability scoring (prompt perturbation; paraphrase test) | Re-run a task with paraphrased prompts and apply reliability metrics to the labels, and produce a prompt stability score in a validation report. | Tested |
| CK-12 | Specification sweep across models and prompts (LLM hacking test; multiverse analysis; specification curve) | Repeat the downstream statistical test across several models, prompts and settings, and produce error rates for false positive, false negative, wrong-sign and wrong-magnitude conclusions. | Tested |
| CK-13 | Multiple-testing correction (Benjamini-Hochberg false discovery rate control; Bonferroni; Holm) | Adjust the significance threshold when many comparisons are made, and produce adjusted p-values or a list of retained results. | Documented |
| CK-14 | Data leakage audit (train-test contamination check; model information sheet) | Confirm that no information from the test items or the outcome entered the model, prompt or features, and produce a leakage checklist and a model information sheet. | Documented |
| CK-15 | Sample representativeness audit (coverage check; selection bias check) | Compare the analysed sample with the population it claims to describe and with the steps that selected it, and produce a comparison table on known variables and a list of exclusion steps. | Documented |
| CK-16 | Entailment-based consistency check (textual entailment; NLI (natural language inference) consistency; SummaC; FactCC; faithfulness checking) | Score whether each sentence of a summary or answer is supported by the source sentences, and produce a consistency score per sentence and per summary. | Tested |
| CK-17 | Source-span and quotation check by code (quote-and-verify; extractive grounding; citation verification) | Require each extracted statement to point to the exact passage it came from, confirm by string match that the passage exists, and produce a pass or fail list per statement. | Documented |
| CK-18 | Numeric and attribution recomputation by code (numeric fidelity check; deterministic recomputation) | Extract every number, date, name and quotation from an output by code and confirm that each appears in, or can be derived from, the source, and produce a list of unmatched figures and names. | Unconfirmed |
| CK-19 | Atomic claim decomposition and scoring (FActScore; atomic fact checking) | Split generated text into atomic facts and score the share that a reliable knowledge source supports, and produce a supported proportion and a list of unsupported facts. | Tested |
| CK-20 | Sampling-consistency hallucination detection (SelfCheckGPT; self-consistency check) | Draw several stochastic responses to the same prompt and flag statements that the other samples do not support, and produce sentence-level consistency scores. | Tested |
| CK-21 | Reference-free grounding metrics for retrieval-augmented answers (Ragas; faithfulness, answer relevance and context relevance scoring) | Score whether retrieval finds relevant focused passages, whether the answer uses them faithfully and the quality of the answer, and produce a score per dimension without human reference answers. | Documented |
| CK-22 | LLM-as-judge with human calibration (model-graded evaluation) | Use a strong model to grade outputs against a rubric and check the judge against human grades on a subsample, and produce judge scores with judge-human agreement. | Tested |
| CK-23 | Confidence calibration check (reliability diagram; expected calibration error; temperature scaling; P(True); verbalised confidence) | Check that a stated confidence of 80 per cent matches being right about 80 per cent of the time, rescale confidence if not, and produce a calibration curve and error. | Tested |
| CK-24 | Review sampling design (random audit sample; stratified sample; risk-based sampling; acceptance sampling; rule of three) | Choose which outputs to inspect and how many so that the observed error rate has a stated bound, and produce an error-rate estimate with an upper bound. | Documented |
| CK-25 | Error taxonomy and error analysis (failure mode analysis; confusion analysis) | Classify reviewed failures into defined types such as invented fact, wrong attribution, omission, wrong scope and arithmetic, and produce error counts by type. | Documented |
| CK-26 | Independent human review of model output (expert review; adjudication; human oversight) | Specify who reviews which outputs, with what rubric and with authority to overrule, and produce a review log with decisions. | Documented |
| CK-27 | Adversarial testing of an extraction pipeline (red-teaming; chaos testing) | Probe an extraction pipeline with hostile or odd inputs such as instructions inserted in source text, contradictory documents and missing data, and produce a list of failures found. | Documented |
| CK-28 | Provenance tracking and audit logging (W3C (World Wide Web Consortium) PROV; lineage; record keeping) | Record which source, prompt, model version, settings, reviewer and decision produced each output, and produce a provenance graph or log that traces any claim back to its source. | Documented |
| CK-29 | Scripted reproducible workflow (reproducible analysis; version control; literate programming) | Record data, code and environment so that another person or a later run can regenerate every figure, and produce a project record that can be rerun from the raw data. | Practice |