HE-01 Grey literature appraisal checklist (AACODS (Authority, Accuracy, Coverage, Objectivity, Date, Significance))
Scores a non-peer-reviewed item on six criteria (authority, accuracy, coverage, objectivity, date, significance) and produces a completed checklist.
What it does
Scores a non-peer-reviewed item on six criteria, each answered yes, no or unclear. Artefact: a completed checklist. Read the Objectivity and Accuracy answers for unstated bias and unsupported claims.
When to use it
Use when you must decide whether to rely on a government, organisation or consultancy report that has no design-specific checklist.
When it does not fit
Fits an item with no design-specific checklist. Misfit: the item is a study whose design has its own checklist. The checklist has no sponsorship item, so the analyst must record funder and author affiliations at the Authority and Objectivity steps.
Result format
The result is a completed checklist with one row per criterion (authority, accuracy, coverage, objectivity, date, significance), each answered yes, no or unclear. It states what the objectivity and accuracy answers show about unstated bias and unsupported claims.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Content type | Always applies | requires=text |
| Unit of analysis | Always applies | requires=document; requires=source |
| Producer interest | Changes the order | prefers=sponsored; prefers=self-interested seller; prefers=unknown |
| Origin | Changes the order | prefers=secondary; prefers=tertiary |
| Document genre | Changes the order | requires=government or agency report; requires=organisation report; requires=news or commentary; requires=other |
| Appraises source credibility | Applies in some cases | yes |
Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Source dependence, Language, Personal information, Security classification, Licence.
| Minimum material | none |
|---|---|
| Additional sources needed | none |
| Not for (rules) | document_genre=research article|evidence synthesis :: a design-specific checklist applies to these genres |
| Not for (text) | The item is a study whose design has its own checklist. The checklist has no sponsorship item, so record funder and author affiliations at the authority and objectivity steps. |
Choosing and ordering
| Kind of technique | Preparation technique, runs in step 5.2 |
|---|---|
| Phase tag | verify and prepare |
| Other phase tags | none |
| Kind of finding | risk; gap |
| Question answered | what; who |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | proceed with limit stated |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | spot check | optional | no |
| decision-informing | spot check | optional | no |
| decision-informing | repeat runs | none | no |
| decision-informing | prompt variation | none | no |
| high-stakes | spot check | optional | no |
| high-stakes | repeat runs | none | no |
| high-stakes | prompt variation | none | no |
| high-stakes | human review sample | needed | no |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | model alone |
|---|---|
| Typical model failure | Answers yes on authority and objectivity from the report's own claims, and does not record funder or affiliation. |
| Error sources | input data; coder or rater judgement; language model |
| Certainty limits | Certainty drops when the checklist is read as a score (it has no scoring rule), when the funder is unknown, and when one assessor answers every item (risk of bias). |
Related techniques
- substitutes for HE1-03 CRAAP test (currency, relevance, authority, accuracy, purpose): when document_genre is government or agency report or organisation report and a structured checklist is wanted
- pairs with CM-01 Sponsor versus non-sponsor comparison (meta-epidemiological sponsorship analysis): funder and affiliation records support a comparison of sponsored and non-sponsored findings
Evidence and tag basis
Documented. Written by Tyndall at Flinders University in 2010, with no scoring rule or inclusion threshold. No study of its validity or reliability was found. (AACODS checklist)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.