CK-03 Bootstrap confidence intervals for agreement and accuracy (bootstrap resampling of items)
Quantify sampling uncertainty in an agreement or accuracy figure computed from a review sample by resampling items, and produce an interval around the coefficient.
What it does
Quantifies sampling uncertainty in a reliability or accuracy figure computed from a finite review sample, by resampling items. Artefact: an interval around the coefficient. Read the lower bound against the threshold needed.
When to use it
Use when you have an agreement or accuracy figure from a small review sample and need to know how precise it is.
When it does not fit
Needs independent items. Clustered items, such as many snippets from one document, need cluster resampling (inference). Small samples give wide intervals.
Result format
The result gives, for each reliability or accuracy figure, the point value and its bootstrap interval, with the sample size beside it. It states whether the lower bound of each interval meets the threshold needed.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Unit of analysis | Always applies | requires=record; requires=document; requires=claim |
| Level of detail available | Always applies | requires=record-level |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Content type, Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | at least 30 independent reviewed items |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Wrong choice when items are clustered, such as many snippets from one document, unless clusters are resampled. Small samples give wide intervals that must not be hidden. |
Choosing and ordering
| Kind of technique | Verification technique, runs in step 6 |
|---|---|
| Phase tag | validate |
| Other phase tags | none |
| Kind of finding | risk |
| Question answered | how much |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | proceed with limit stated |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: repeatable with fixed settings.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | input validation | none | no |
| exploratory | repeat runs | none | no |
| decision-informing | input validation | none | no |
| decision-informing | repeat runs | none | no |
| decision-informing | code execution for arithmetic | none | no |
| high-stakes | input validation | none | no |
| high-stakes | repeat runs | none | no |
| high-stakes | code execution for arithmetic | none | no |
| high-stakes | sensitivity analysis | none | no |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | model plus code |
|---|---|
| Typical model failure | Reports an interval without computing it, which gives invented precision. |
| Error sources | input data; statistical assumptions |
| Certainty limits | Certainty drops when items are not independent (risk of bias) and when the reviewed sample is small (imprecision). |
Related techniques
- should follow CK-01 Inter-coder agreement statistics (percentage agreement; Cohen's kappa; weighted kappa; Scott's pi; Fleiss' kappa; Krippendorff's alpha; intercoder reliability testing): the coefficient needs an interval
- should follow CK-24 Review sampling design (random audit sample; stratified sample; risk-based sampling; acceptance sampling; rule of three): the review sample design sets how items were drawn
- pairs with CK-04 Gold-standard validation sample (ground truth set; human-coded validation set): gold-standard accuracy figures need intervals
Evidence and tag basis
Documented. The source recommends bootstrapping for alpha and reporting confidence intervals. General bootstrap theory was not opened. (Artstein and Poesio 2008, Inter-coder agreement for computational linguistics)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.