CK-17 Source-span and quotation check by code (quote-and-verify; extractive grounding; citation verification)
Require each extracted statement to point to the exact passage it came from, confirm by string match that the passage exists, and produce a pass or fail list per statement.
What it does
Requires each extracted statement to point to the exact passage it came from, then confirms by string match that the passage exists. A reviewer or entailment check then confirms the passage supports the claim. Artefact: a pass or fail list per statement.
When to use it
Use when a model has produced statements with quotations or citations and you need to know whether the quoted passages really exist in the source.
When it does not fit
Needs the source as text. Misfit: the quoted span is a paraphrase, or exists but does not support the claim.
Result format
The result is a pass or fail list with one row per extracted statement, giving the statement, the exact passage it points to, whether that passage exists by string match and whether the passage supports the claim. It ends with the list of statements that failed.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Content type | Always applies | requires=text |
| Unit of analysis | Always applies | requires=claim |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | the source in machine-readable text |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Wrong choice when the source is available only as images or scans. A quoted span that exists may still not support the claim, so a reviewer or entailment check must follow. |
Choosing and ordering
| Kind of technique | Verification technique, runs in step 6 |
|---|---|
| Phase tag | validate |
| Other phase tags | report |
| Kind of finding | contradiction; gap |
| Question answered | what |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to verify and prepare |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: repeatable with fixed settings.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | input validation | none | no |
| decision-informing | input validation | none | no |
| decision-informing | source-span citation | none | no |
| decision-informing | spot check | optional | no |
| high-stakes | input validation | none | no |
| high-stakes | source-span citation | none | no |
| high-stakes | spot check | optional | no |
| high-stakes | human review sample | needed | no |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | model plus code |
|---|---|
| Typical model failure | Fabricates quotations that look verbatim, or attaches real quotes to claims they do not support. |
| Error sources | input data; data transformation; language model |
| Certainty limits | Certainty drops when passages are paraphrases and not exact matches (indirectness) and when matched passages are not read for support (risk of bias). |
Related techniques
- should follow PE-10 Schema-based structured extraction with an LLM (LLM (large language model) information extraction; form filling from documents): extracted fields should carry source spans
- should follow PE-12 Model summarisation of documents (abstractive summarisation; multi-document summarisation): summary statements should be tied to passages
- should follow PE-13 Retrieval-augmented analysis (RAG (retrieval-augmented generation); grounded question answering over a corpus): cited passages are checked for existence
- pairs with CK-16 Entailment-based consistency check (textual entailment; NLI (natural language inference) consistency; SummaC; FactCC; faithfulness checking): entailment checks whether the passage supports the claim
- pairs with TC-04 Claim and citation verification against source text (fact verification; claim checking; citation support check; quotation accuracy check): claim verification judges support against evidence
Evidence and tag basis
Documented. One approach extracts both the supporting source span and the inconsistent span, and human evaluation found the span extraction helpful (abstract). NIST requires review and verification of sources and citations in outputs. (Kryscinski et al., arXiv 1910.12840; NIST AI 600-1)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.