HE-02 Critical appraisal checklist by study design (CASP (Critical Appraisal Skills Programme) checklists; JBI (Joanna Briggs Institute) checklists)
Answers design-specific questions on validity, results and relevance for one study and produces a completed checklist for the evidence base.
What it does
Answers design-specific questions on validity, results and relevance. JBI lists 15 checklists, including prevalence, analytical cross-sectional and qualitative studies and three for textual evidence (expert opinion, narrative, policy). Artefact: item answers. Read the validity items first.
When to use it
Use when you need to judge how far to trust a single study, such as a randomised trial, a cohort study, a qualitative study or an expert opinion paper, and its design has a checklist.
When it does not fit
Choose the checklist by the design the study used. Neither source gives a scoring rule, and the items in the JBI textual evidence checklists were not read.
Result format
The result is a list of answers to the checklist questions that fit the study design, with the validity items listed first and then the results and relevance items. Each answer is given beside its item.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Content type | Always applies | requires=text |
| Unit of analysis | Always applies | requires=document |
| Origin | Changes the order | prefers=primary |
| Document genre | Changes the order | requires=research article; requires=evidence synthesis; requires=other |
| Appraises source credibility | Applies in some cases | yes |
Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Language, Personal information, Security classification, Licence.
| Minimum material | none |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | The checklist does not match the design the study used. Neither source gives a scoring rule, so a summed score should not be used as a threshold. |
Choosing and ordering
| Kind of technique | Preparation technique, runs in step 5.2 |
|---|---|
| Phase tag | verify and prepare |
| Other phase tags | none |
| Kind of finding | risk |
| Question answered | what; how much |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | proceed with limit stated |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | spot check | optional | no |
| decision-informing | spot check | optional | no |
| decision-informing | human review sample | needed; left out below high stakes | no |
| high-stakes | spot check | optional | no |
| high-stakes | human review sample | needed | no |
| high-stakes | independent review | needed | no |
| high-stakes | second technique with different assumptions | none | conditional |
Failure modes and limits
| Model suitability | human judgement required |
|---|---|
| Typical model failure | Picks the wrong checklist for the design, or answers items from the abstract without the methods section. |
| Error sources | input data; coder or rater judgement |
| Certainty limits | Certainty drops when items are answered from reports that omit methods, when one appraiser works alone, and when the checklist is turned into a score (risk of bias). |
Related techniques
- pairs with HE1-07 Evidence hierarchies and certainty grading (study design levels, risk-of-bias assessment, Grading of Recommendations Assessment, Development and Evaluation or GRADE): item answers feed the certainty rating
- should follow HE1-08 Systematic review: appraisal applies to the included studies of a review
Evidence and tag basis
Documented. CASP describes its checklists as a guide and not a replacement for judgement. JBI gives no scoring rule on the page read. (JBI critical appraisal tools; CASP knowledge hub)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.