HE1-02 Quality of information check
Rates the completeness, accuracy and corroboration of each source and produces a source database with a statement of what is known and unknown.
What it does
Rates the completeness, accuracy and corroboration of each source. Artefact: a source database and a statement of what is known and unknown; read items with single-source support.
When to use it
Use when you need to know which parts of the evidence rest on a single source and how much of the picture is missing.
When it does not fit
The output is a count of sources, not an assessment of each. Many sources do not replace thoroughly examined good information.
Result format
The result is a source database with one row per source, rating its completeness, accuracy and corroboration. It ends with a statement of what is known and what is unknown, and names the items that rest on single-source support.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Content type | Always applies | requires=text |
| Unit of analysis | Always applies | requires=source |
| Producer interest | Changes the order | prefers=sponsored; prefers=self-interested seller; prefers=unknown |
| Source dependence | Changes the order | prefers=copied or derived; prefers=unknown |
| Origin | Changes the order | prefers=secondary; prefers=tertiary; prefers=unknown |
| Appraises source credibility | Applies in some cases | yes |
Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | none |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | The output is only a count of sources. Many sources do not replace a few thoroughly examined good ones. |
Choosing and ordering
| Kind of technique | Preparation technique, runs in step 5.2 |
|---|---|
| Phase tag | verify and prepare |
| Other phase tags | none |
| Kind of finding | gap; contradiction |
| Question answered | what |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to gather |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | spot check | optional | no |
| decision-informing | spot check | optional | no |
| decision-informing | human review sample | needed; left out below high stakes | no |
| high-stakes | spot check | optional | no |
| high-stakes | human review sample | needed | no |
| high-stakes | independent review | needed | no |
| high-stakes | second technique with different assumptions | none | conditional |
Failure modes and limits
| Model suitability | human judgement required |
|---|---|
| Typical model failure | Rates completeness against what the model already knows instead of against the question, and treats the number of sources as quality. |
| Error sources | input data; coder or rater judgement; source selection |
| Certainty limits | Certainty drops when ratings rest on one assessor, when sources are not independent, or when excluded reporting was never reviewed (reporting bias). |
Related techniques
- pairs with HE1-01 Admiralty system (source reliability and information credibility grading): the grade adds a source reliability letter to this completeness and accuracy rating
Evidence and tag basis
Documented. Described in the CIA primer without effect evidence. Equivalence to the PHIA 'Analytical Confidence Method' is unconfirmed because that document does not describe the method. (CIA Tradecraft Primer (2009); UK PHIA Professional Development Framework (2019))
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.