TC-01 GRIM test (granularity-related inconsistency of means)
Checks whether a reported mean of whole-number data can arise from the reported sample size and item count and produces a flag for possible or impossible means.
What it does
Checks whether a reported mean of whole-number data can arise from the reported sample size. Artefact: a flag for possible or impossible means. Read a flag as a reporting error to query, not as proof of misconduct.
When to use it
Use when a report gives means from whole-number scales such as Likert items and you want to know whether they are arithmetically possible.
When it does not fit
Fits whole-number data such as Likert items with small samples. Misfit: non-integer data, or large samples where many means are possible. Needs the reported sample size and item count.
Result format
The result is a table with one row per reported mean, giving the mean, the sample size and a flag of possible or impossible. It states that a flag is a reporting error to query and not proof of misconduct.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Content type | Always applies | requires=text; requires=tabular |
| Unit of analysis | Always applies | requires=claim |
| Variable types | Raises a warning | prefers=ordinal; prefers=counts or fractions |
| Level of detail available | Always applies | requires=aggregate tables |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | A reported mean with its decimals, the sample size and the number of items, for whole-number data. Most informative when sample size times item count is small. |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Wrong choice for non-integer data, or for large samples where nearly every mean is possible. A flag is a reporting error to query and not proof of misconduct. |
Choosing and ordering
| Kind of technique | Verification technique, runs in step 6 |
|---|---|
| Phase tag | validate |
| Other phase tags | verify and prepare |
| Kind of finding | contradiction |
| Question answered | what |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | proceed with limit stated |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: deterministic.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | input validation | none | no |
| decision-informing | input validation | none | no |
| decision-informing | code execution for arithmetic | none | no |
| decision-informing | spot check | optional | no |
| high-stakes | input validation | none | no |
| high-stakes | code execution for arithmetic | none | no |
| high-stakes | spot check | optional | no |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | model plus code |
|---|---|
| Typical model failure | Applies the test to data that are not whole numbers or ignores rounding rules, and reports false inconsistencies. |
| Error sources | input data; data transformation |
| Certainty limits | A flag can come from rounding, a typing error or missing participants and does not show misconduct. Certainty drops when the sample size or item count is not stated. |
Related techniques
- pairs with TC-03 statcheck (recomputation of reported p-values): statcheck tests the reported test statistics while this test checks the reported means
- should follow CK-18 Numeric and attribution recomputation by code (numeric fidelity check; deterministic recomputation): recomputation of reported figures confirms that the numbers were extracted correctly
Evidence and tag basis
Documented. In a preprint, 71 of 260 psychology articles were testable and 36 of those 71 (50.7 per cent) had at least one inconsistent mean. (Brown and Heathers, PeerJ preprint 2064)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.