CK-15 Sample representativeness audit (coverage check; selection bias check)
Compare the analysed sample with the population it claims to describe and with the steps that selected it, and produce a comparison table on known variables and a list of exclusion steps.
What it does
Compares the analysed sample with the population it claims to describe and with the steps that selected it. Artefact: a comparison table on known variables and a list of exclusion steps. Read the gaps between sample and population.
When to use it
Use when results from a sample will be read as describing a wider population or a full set of documents.
When it does not fit
Misfit: no known population benchmark. In LLM pipelines, retrieval or truncation steps can create selection silently (inference).
Result format
The result has a comparison table of the analysed sample against the population on known variables, with the gap in each row, and a list of the exclusion steps that selected the sample. It states where the gaps between sample and population are largest.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Content type | Always applies | requires=text; requires=tabular |
| Unit of analysis | Always applies | requires=dataset |
| Level of detail available | Always applies | requires=record-level |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | at least one population benchmark on a known variable and a record of how items were selected |
|---|---|
| Additional sources needed | comparator |
| Not for (rules) | none |
| Not for (text) | Wrong choice when no population benchmark exists. In language model pipelines, retrieval or truncation steps can create selection without being noticed. |
Choosing and ordering
| Kind of technique | Verification technique, runs in step 6 |
|---|---|
| Phase tag | validate |
| Other phase tags | verify and prepare |
| Kind of finding | gap; risk |
| Question answered | what; who |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | proceed with limit stated |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | spot check | optional | no |
| decision-informing | spot check | optional | no |
| decision-informing | code execution for arithmetic | none | no |
| decision-informing | assumption and limitation statement | none | no |
| high-stakes | spot check | optional | no |
| high-stakes | code execution for arithmetic | none | no |
| high-stakes | assumption and limitation statement | none | no |
| high-stakes | sensitivity analysis | none | no |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | human judgement required |
|---|---|
| Typical model failure | Summarises the documents it was given as if they were all the relevant documents. |
| Error sources | input data; source selection |
| Certainty limits | Certainty drops when the benchmark is out of date or on different definitions (indirectness) and when selection steps are undocumented (risk of bias). |
Related techniques
- should follow PE-01 Corpus construction and sampling (document selection; search string design; sampling frame design): corpus construction records how documents entered the analysis
- pairs with MM-06 Survey weighting and subgroup estimation (design weights; calibration; post-stratification; relative standard error): weighting is one remedy for the gaps found
- pairs with CK-24 Review sampling design (random audit sample; stratified sample; risk-based sampling; acceptance sampling; rule of three): sampling design affects what is covered
Evidence and tag basis
Documented. Only the point that validation sets should be random or stratified so that they represent the corpus was verified. (Grimmer and Stewart 2013, Political Analysis)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.