HE1-07 Evidence hierarchies and certainty grading (study design levels, risk-of-bias assessment, Grading of Recommendations Assessment, Development and Evaluation or GRADE)
Ranks study designs and rates confidence in a body of evidence from high to very low, and produces a certainty rating with reasons for downgrading.
What it does
Ranks study designs and rates confidence in a body of evidence. Artefact: a level or a certainty rating from high to very low; read the reasons for downgrading, not only the label.
When to use it
Use when you need to say how much confidence a body of studies deserves and why.
When it does not fit
The question has no randomised designs, as with policy descriptions.
Result format
The result gives, for the body of evidence, the study design level and a certainty rating from high to very low. It lists the reasons for any downgrading beside the rating, so the label is not given alone.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Content type | Always applies | requires=text |
| Unit of analysis | Always applies | requires=document; requires=topic |
| Number of sources | Always applies | requires=two or more |
| Origin | Changes the order | prefers=primary |
| Document genre | Changes the order | prefers=research article; prefers=evidence synthesis |
| Appraises source credibility | Applies in some cases | yes |
Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Level of detail available, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Language, Personal information, Security classification, Licence.
| Minimum material | none |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | The question has no randomised designs, as with policy description, so a design hierarchy ranks everything low. |
Choosing and ordering
| Kind of technique | Verification technique, runs in step 6 |
|---|---|
| Phase tag | validate |
| Other phase tags | conclude |
| Kind of finding | risk; gap |
| Question answered | how much |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to gather |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | spot check | optional | no |
| decision-informing | spot check | optional | no |
| decision-informing | human review sample | needed; left out below high stakes | no |
| high-stakes | spot check | optional | no |
| high-stakes | human review sample | needed | no |
| high-stakes | independent review | needed | no |
| high-stakes | second technique with different assumptions | none | conditional |
Failure modes and limits
| Model suitability | human judgement required |
|---|---|
| Typical model failure | Applies a certainty label without the domain reasoning, or downgrades once for a study design and again for the same weakness. |
| Error sources | input data; coder or rater judgement; source selection |
| Certainty limits | Certainty ratings are themselves judgements. Check that each downgrade names a domain (risk of bias, inconsistency, indirectness, imprecision, reporting bias). |
Related techniques
- should follow HE-02 Critical appraisal checklist by study design (CASP (Critical Appraisal Skills Programme) checklists; JBI (Joanna Briggs Institute) checklists): risk of bias must be assessed for each study before certainty is rated
- pairs with HE1-08 Systematic review: certainty grading is the final step of a systematic review
Evidence and tag basis
Documented. Cochrane rapid review guidance has reviewers grade certainty of evidence; the five GRADE domains were not verified. (Cochrane Rapid Reviews Methods Group guidance (2020))
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.