startthat.work home

HE-03 Appraisal of a systematic review (AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews 2); ROBIS (Risk Of Bias In Systematic reviews))

Rates confidence in a systematic review with AMSTAR 2 (high to critically low) or concern in four domains with ROBIS, and produces a rating or domain judgements.

Handle evidenceEvidence: DocumentedPreparation technique, runs in step 5.2Strongest statement: describe

What it does

AMSTAR 2 rates confidence in a review from 16 items and seven critical domains as high, moderate, low or critically low. ROBIS rates concern in four domains and gives no summary score. Artefact: a rating or domain judgements. Read which critical items failed or which domains raise concern.

When to use it

Use when you need to decide whether a published systematic review can be relied on.

When it does not fit

Fits healthcare reviews of interventions. Misfit: a consultancy review that is not systematic. In one ROBIS pilot two reviewers agreed on 26 of 40 domain ratings (65 per cent). The Cochrane Handbook recommends neither instrument over the other.

Result format

The result is a confidence rating of high, moderate, low or critically low for the review, or a judgement of concern for each of the four domains, depending on the tool used. It names the critical items that failed or the domains that raise concern.

Requirements on the material

PropertyRequirement typeRequirement
Content typeAlways appliesrequires=text
Unit of analysisAlways appliesrequires=document
OriginChanges the orderprefers=secondary
Document genreChanges the orderrequires=evidence synthesis
Appraises source credibilityApplies in some casesyes

Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Language, Personal information, Security classification, Licence.

Minimum materialnone
Additional sources needednone
Not for (rules)none
Not for (text)The review is not systematic, such as a consultancy review, or it does not concern healthcare interventions (ROBIS is not validated outside healthcare). Assessors can disagree: in one pilot two reviewers agreed on 26 of 40 domain ratings.

Choosing and ordering

Kind of techniquePreparation technique, runs in step 5.2
Phase tagverify and prepare
Other phase tagsnone
Kind of findingrisk
Question answeredwhat; how much
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetproceed with limit stated

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.

StakesCheckPerson involvedCounts as a different method
exploratoryspot checkoptionalno
decision-informingspot checkoptionalno
decision-informinghuman review sampleneeded; left out below high stakesno
high-stakesspot checkoptionalno
high-stakeshuman review sampleneededno
high-stakesindependent reviewneededno
high-stakessecond technique with different assumptionsnoneconditional

Failure modes and limits

Model suitabilityhuman judgement required
Typical model failureRates critical items as met without reading the methods, and gives a single score for ROBIS, which has none.
Error sourcesinput data; coder or rater judgement
Certainty limitsCertainty drops with low inter-assessor agreement, a single assessor, and application outside healthcare (indirectness).

Evidence and tag basis

Documented. ROBIS is not validated outside healthcare. (AMSTAR 2, BMJ 2017; ROBIS, J Clin Epidemiol; Cochrane Handbook chapter V)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.