TC1-18 Internal consistency check (term from earlier work)
Tests whether the claims, figures and assumptions within one analysis are compatible with each other and produces a list of contradictions or inconsistent figures.
What it does
Tests whether the claims, figures and assumptions within one analysis are compatible with each other. Artefact: a list of contradictions or inconsistent figures.
When to use it
Use when you have a finished draft analysis and want to check that its numbers and statements do not contradict one another.
When it does not fit
Consistent results are read as correct (consistency does not show truth).
Result format
The result is a list of contradictions or inconsistent figures found within the analysis, each naming the two claims, figures or assumptions that conflict and where each appears. If none is found, it says so and names what was compared.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Content type | Always applies | requires=text; requires=tabular |
| Unit of analysis | Always applies | requires=document; requires=claim |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | A complete draft analysis with its figures and assumptions. |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Misused when consistent results are read as correct, because consistency does not show truth. |
Choosing and ordering
| Kind of technique | Verification technique, runs in step 6 |
|---|---|
| Phase tag | validate |
| Other phase tags | none |
| Kind of finding | contradiction |
| Question answered | what |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to analyse |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: repeatable with fixed settings.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | spot check | optional | no |
| decision-informing | spot check | optional | no |
| decision-informing | code execution for arithmetic | none | no |
| decision-informing | human review sample | needed; left out below high stakes | no |
| high-stakes | spot check | optional | no |
| high-stakes | code execution for arithmetic | none | no |
| high-stakes | human review sample | needed | no |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | model plus code |
|---|---|
| Typical model failure | Misses contradictions that need arithmetic across tables, or reports an apparent contradiction that is only a difference in definitions. |
| Error sources | input data; language model; coder or rater judgement |
| Certainty limits | A clean result shows only that the parts agree. Certainty of any conclusion is not raised when the same inputs and assumptions underlie every part. |
Related techniques
- pairs with CK-18 Numeric and attribution recomputation by code (numeric fidelity check; deterministic recomputation): recomputation checks figures against the source while this check tests them against each other
- pairs with TC1-28 Dimensional and unit checks: unit checks catch contradictions that the figures hide
Evidence and tag basis
Unconfirmed. No source defines it as a named technique. Related documented checks are cross-consistency assessment in morphological analysis (Wikipedia, Morphological analysis (problem-solving)), dimensional checks (below) and coherence correction (above).
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.