HE-03 Appraisal of a systematic review (AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews 2); ROBIS (Risk Of Bias In Systematic reviews))
Rates confidence in a systematic review with AMSTAR 2 (high to critically low) or concern in four domains with ROBIS, and produces a rating or domain judgements.
What it does
AMSTAR 2 rates confidence in a review from 16 items and seven critical domains as high, moderate, low or critically low. ROBIS rates concern in four domains and gives no summary score. Artefact: a rating or domain judgements. Read which critical items failed or which domains raise concern.
When to use it
Use when you need to decide whether a published systematic review can be relied on.
When it does not fit
Fits healthcare reviews of interventions. Misfit: a consultancy review that is not systematic. In one ROBIS pilot two reviewers agreed on 26 of 40 domain ratings (65 per cent). The Cochrane Handbook recommends neither instrument over the other.
Result format
The result is a confidence rating of high, moderate, low or critically low for the review, or a judgement of concern for each of the four domains, depending on the tool used. It names the critical items that failed or the domains that raise concern.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Content type | Always applies | requires=text |
| Unit of analysis | Always applies | requires=document |
| Origin | Changes the order | prefers=secondary |
| Document genre | Changes the order | requires=evidence synthesis |
| Appraises source credibility | Applies in some cases | yes |
Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Language, Personal information, Security classification, Licence.
| Minimum material | none |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | The review is not systematic, such as a consultancy review, or it does not concern healthcare interventions (ROBIS is not validated outside healthcare). Assessors can disagree: in one pilot two reviewers agreed on 26 of 40 domain ratings. |
Choosing and ordering
| Kind of technique | Preparation technique, runs in step 5.2 |
|---|---|
| Phase tag | verify and prepare |
| Other phase tags | none |
| Kind of finding | risk |
| Question answered | what; how much |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | proceed with limit stated |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | spot check | optional | no |
| decision-informing | spot check | optional | no |
| decision-informing | human review sample | needed; left out below high stakes | no |
| high-stakes | spot check | optional | no |
| high-stakes | human review sample | needed | no |
| high-stakes | independent review | needed | no |
| high-stakes | second technique with different assumptions | none | conditional |
Failure modes and limits
| Model suitability | human judgement required |
|---|---|
| Typical model failure | Rates critical items as met without reading the methods, and gives a single score for ROBIS, which has none. |
| Error sources | input data; coder or rater judgement |
| Certainty limits | Certainty drops with low inter-assessor agreement, a single assessor, and application outside healthcare (indirectness). |
Related techniques
- pairs with HE-07 Overview of reviews (umbrella review; tertiary review): each review in an overview needs this appraisal
- pairs with HE1-07 Evidence hierarchies and certainty grading (study design levels, risk-of-bias assessment, Grading of Recommendations Assessment, Development and Evaluation or GRADE): the review rating feeds certainty of a body of evidence
Evidence and tag basis
Documented. ROBIS is not validated outside healthcare. (AMSTAR 2, BMJ 2017; ROBIS, J Clin Epidemiol; Cochrane Handbook chapter V)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.