PE-12 Model summarisation of documents (abstractive summarisation; multi-document summarisation)
Write a condensed text that paraphrases the content of one or many documents, giving a summary as a draft result.
What it does
Writes a condensed text that paraphrases the content of one or many documents. Artefact: a summary. Read it only beside a check of each statement against the source.
When to use it
Use when you need a quick orientation to a set of documents before deciding how to analyse them.
When it does not fit
Use for orientation, not as a finding. No verified measure for current models on multi-document corpora was found.
Result format
The result is a condensed text that paraphrases the content of the documents. It must be accompanied by a check of each statement in the summary against the source, and it is presented as orientation and not as a finding.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Structure | Applies in some cases | requires=unstructured; requires=semi-structured |
| Content type | Always applies | requires=text |
| Unit of analysis | Always applies | requires=document |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | none |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | The summary is used as a finding, or statements in it are not checked against the source. |
Choosing and ordering
| Kind of technique | Preparation technique, runs in step 5.2 |
|---|---|
| Phase tag | verify and prepare |
| Other phase tags | analyse; report |
| Kind of finding | pattern |
| Question answered | what |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | proceed with limit stated |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | spot check | optional | no |
| decision-informing | spot check | optional | no |
| decision-informing | source-span citation | none | no |
| decision-informing | human review sample | needed; left out below high stakes | no |
| decision-informing | prompt variation | none | no |
| high-stakes | spot check | optional | no |
| high-stakes | source-span citation | none | no |
| high-stakes | human review sample | needed | no |
| high-stakes | prompt variation | none | no |
| high-stakes | independent review | needed | no |
| high-stakes | provenance log | none | no |
Failure modes and limits
| Model suitability | human judgement required |
|---|---|
| Typical model failure | Adds statements that the documents do not contain and under-uses information in the middle of long inputs. |
| Error sources | input data; language model |
| Certainty limits | Certainty is low because all neural summarisers tested produced hallucinated content (risk of bias) and long inputs are used unevenly (inconsistency). |
Related techniques
- pairs with CK-17 Source-span and quotation check by code (quote-and-verify; extractive grounding; citation verification): a source-span check by code catches summary statements that no passage supports
- pairs with CK-19 Atomic claim decomposition and scoring (FActScore; atomic fact checking): atomic claim decomposition scores each summary statement against the source
Evidence and tag basis
Tested. Human evaluation found substantial hallucinated content in all neural summarisation systems tested (2020). Models use information in the middle of long inputs less reliably (2023). (Maynez et al. 2020, ACL; Liu et al. 2023, Lost in the middle)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.