startthat.work home

HE-01 Grey literature appraisal checklist (AACODS (Authority, Accuracy, Coverage, Objectivity, Date, Significance))

Scores a non-peer-reviewed item on six criteria (authority, accuracy, coverage, objectivity, date, significance) and produces a completed checklist.

Handle evidenceEvidence: DocumentedPreparation technique, runs in step 5.2Strongest statement: describe

What it does

Scores a non-peer-reviewed item on six criteria, each answered yes, no or unclear. Artefact: a completed checklist. Read the Objectivity and Accuracy answers for unstated bias and unsupported claims.

When to use it

Use when you must decide whether to rely on a government, organisation or consultancy report that has no design-specific checklist.

When it does not fit

Fits an item with no design-specific checklist. Misfit: the item is a study whose design has its own checklist. The checklist has no sponsorship item, so the analyst must record funder and author affiliations at the Authority and Objectivity steps.

Result format

The result is a completed checklist with one row per criterion (authority, accuracy, coverage, objectivity, date, significance), each answered yes, no or unclear. It states what the objectivity and accuracy answers show about unstated bias and unsupported claims.

Requirements on the material

PropertyRequirement typeRequirement
Content typeAlways appliesrequires=text
Unit of analysisAlways appliesrequires=document; requires=source
Producer interestChanges the orderprefers=sponsored; prefers=self-interested seller; prefers=unknown
OriginChanges the orderprefers=secondary; prefers=tertiary
Document genreChanges the orderrequires=government or agency report; requires=organisation report; requires=news or commentary; requires=other
Appraises source credibilityApplies in some casesyes

Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Source dependence, Language, Personal information, Security classification, Licence.

Minimum materialnone
Additional sources needednone
Not for (rules)document_genre=research article|evidence synthesis :: a design-specific checklist applies to these genres
Not for (text)The item is a study whose design has its own checklist. The checklist has no sponsorship item, so record funder and author affiliations at the authority and objectivity steps.

Choosing and ordering

Kind of techniquePreparation technique, runs in step 5.2
Phase tagverify and prepare
Other phase tagsnone
Kind of findingrisk; gap
Question answeredwhat; who
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetproceed with limit stated

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.

StakesCheckPerson involvedCounts as a different method
exploratoryspot checkoptionalno
decision-informingspot checkoptionalno
decision-informingrepeat runsnoneno
decision-informingprompt variationnoneno
high-stakesspot checkoptionalno
high-stakesrepeat runsnoneno
high-stakesprompt variationnoneno
high-stakeshuman review sampleneededno
high-stakesindependent reviewneededno

Failure modes and limits

Model suitabilitymodel alone
Typical model failureAnswers yes on authority and objectivity from the report's own claims, and does not record funder or affiliation.
Error sourcesinput data; coder or rater judgement; language model
Certainty limitsCertainty drops when the checklist is read as a score (it has no scoring rule), when the funder is unknown, and when one assessor answers every item (risk of bias).

Evidence and tag basis

Documented. Written by Tyndall at Flinders University in 2010, with no scoring rule or inclusion threshold. No study of its validity or reliability was found. (AACODS checklist)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.