startthat.work home

HE-02 Critical appraisal checklist by study design (CASP (Critical Appraisal Skills Programme) checklists; JBI (Joanna Briggs Institute) checklists)

Answers design-specific questions on validity, results and relevance for one study and produces a completed checklist for the evidence base.

Handle evidenceEvidence: DocumentedPreparation technique, runs in step 5.2Strongest statement: describe

What it does

Answers design-specific questions on validity, results and relevance. JBI lists 15 checklists, including prevalence, analytical cross-sectional and qualitative studies and three for textual evidence (expert opinion, narrative, policy). Artefact: item answers. Read the validity items first.

When to use it

Use when you need to judge how far to trust a single study, such as a randomised trial, a cohort study, a qualitative study or an expert opinion paper, and its design has a checklist.

When it does not fit

Choose the checklist by the design the study used. Neither source gives a scoring rule, and the items in the JBI textual evidence checklists were not read.

Result format

The result is a list of answers to the checklist questions that fit the study design, with the validity items listed first and then the results and relevance items. Each answer is given beside its item.

Requirements on the material

PropertyRequirement typeRequirement
Content typeAlways appliesrequires=text
Unit of analysisAlways appliesrequires=document
OriginChanges the orderprefers=primary
Document genreChanges the orderrequires=research article; requires=evidence synthesis; requires=other
Appraises source credibilityApplies in some casesyes

Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Language, Personal information, Security classification, Licence.

Minimum materialnone
Additional sources needednone
Not for (rules)none
Not for (text)The checklist does not match the design the study used. Neither source gives a scoring rule, so a summed score should not be used as a threshold.

Choosing and ordering

Kind of techniquePreparation technique, runs in step 5.2
Phase tagverify and prepare
Other phase tagsnone
Kind of findingrisk
Question answeredwhat; how much
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetproceed with limit stated

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.

StakesCheckPerson involvedCounts as a different method
exploratoryspot checkoptionalno
decision-informingspot checkoptionalno
decision-informinghuman review sampleneeded; left out below high stakesno
high-stakesspot checkoptionalno
high-stakeshuman review sampleneededno
high-stakesindependent reviewneededno
high-stakessecond technique with different assumptionsnoneconditional

Failure modes and limits

Model suitabilityhuman judgement required
Typical model failurePicks the wrong checklist for the design, or answers items from the abstract without the methods section.
Error sourcesinput data; coder or rater judgement
Certainty limitsCertainty drops when items are answered from reports that omit methods, when one appraiser works alone, and when the checklist is turned into a score (risk of bias).

Evidence and tag basis

Documented. CASP describes its checklists as a guide and not a replacement for judgement. JBI gives no scoring rule on the page read. (JBI critical appraisal tools; CASP knowledge hub)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.