startthat.work home

Handle evidence

These techniques grade the reliability of a source and the credibility of its information, check a source against criteria such as currency, authority and purpose, read what other sources say about a publisher, and compare independent lines of evidence. Use them when sources vary in quality and the answer has to say how far each can be trusted.

29 techniques. Open one to see what it does, when it fits and what it needs from your material.

IDTechniquePurposeEvidence status
HE1-01Admiralty system (source reliability and information credibility grading)Assigns each report a two-character grade for source reliability (A to F) and information credibility (1 to 6) and produces an appraised evidence base.Documented
HE1-02Quality of information checkRates the completeness, accuracy and corroboration of each source and produces a source database with a statement of what is known and unknown.Documented
HE1-03CRAAP test (currency, relevance, authority, accuracy, purpose)Screens one source against five criteria (currency, relevance, authority, accuracy, purpose) and produces a checklist judgement for the evidence base.Documented
HE1-04SIFT (stop, investigate the source, find better coverage, trace claims to the original; claim verification and fact-checking)Moves from a claim to its publisher, to other coverage and to the original source, and produces a short credibility judgement with the original context located.Practice
HE1-05Lateral readingLeaves the page to read about the publisher in other sources and produces a judgement of publisher credibility.Unconfirmed
HE1-06Triangulation (source, method, investigator and theory triangulation; corroboration)Compares independent lines of evidence on one question and produces a convergence table of agreement, disagreement and reasons.Unconfirmed
HE1-07Evidence hierarchies and certainty grading (study design levels, risk-of-bias assessment, Grading of Recommendations Assessment, Development and Evaluation or GRADE)Ranks study designs and rates confidence in a body of evidence from high to very low, and produces a certainty rating with reasons for downgrading.Documented
HE1-08Systematic reviewCollects all eligible studies by a pre-specified, reproducible search, appraises and synthesises them, and produces a screening log, an evidence table and a synthesis.Documented
HE1-09Rapid reviewRuns a systematic review with streamlined steps to meet a deadline and produces a narrative synthesis that states the shortcuts taken.Documented
HE1-10Scoping review (evidence mapping)Maps the extent, types and gaps of evidence on a broad topic without judging the strength of findings, and produces a charted table.Unconfirmed
HE1-11Meta-analysisCombines effect estimates from separate studies into a weighted average with a confidence interval and produces a forest plot, a pooled effect and heterogeneity statistics.Documented
HE1-12Narrative, thematic and qualitative evidence synthesis (including meta-ethnography)Combines findings across studies in words or themes when pooling numbers is not appropriate, and produces a structured narrative or theme set linked back to studies.Documented
HE1-13Reflexive thematic analysis (Braun and Clarke)Identifies and interprets patterns of meaning across qualitative data in six phases and produces themes with supporting extracts.Documented
HE1-14Framework analysis (Framework Method)Sorts coded qualitative data into a case by theme matrix and produces the matrix for reading across cases and themes.Documented
HE1-15Content analysis (conventional, directed and summative)Codes text into categories, then counts or interprets category use, and produces category frequencies or a category structure.Unconfirmed
HE1-16Discourse analysis (critical discourse analysis)Examines how language constructs topics and positions actors and produces an interpretive account of framing checked against quoted passages.Unconfirmed
HE1-17Grounded theory coding (open, axial and selective coding; constant comparison)Builds categories and a theory from data through open, axial and selective coding with constant comparison and produces codes, memos and a theory.Unconfirmed
HE1-18Document and policy analysisReads documents systematically to extract facts, positions, changes or stated reasons using their provenance and purpose, and produces an extraction table or chronology.Unconfirmed
HE1-19Bibliometric analysis (citation analysis, science mapping, co-citation, co-word analysis)Counts and maps publications, authors and citations to show the structure and growth of a field and produces performance tables and network maps.Unconfirmed
HE1-20Trend and time-series decomposition Also serves: Project forward.Separates a series into trend, seasonal and irregular parts and produces plots and a fitted trend.Unconfirmed
HE1-21Distribution, outlier and base-rate checksExamines the shape, extremes and baseline frequency of data before a figure is interpreted and produces histograms, summary statistics, flagged outliers and a base rate.Unconfirmed
HE-01Grey literature appraisal checklist (AACODS (Authority, Accuracy, Coverage, Objectivity, Date, Significance))Scores a non-peer-reviewed item on six criteria (authority, accuracy, coverage, objectivity, date, significance) and produces a completed checklist.Documented
HE-02Critical appraisal checklist by study design (CASP (Critical Appraisal Skills Programme) checklists; JBI (Joanna Briggs Institute) checklists)Answers design-specific questions on validity, results and relevance for one study and produces a completed checklist for the evidence base.Documented
HE-03Appraisal of a systematic review (AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews 2); ROBIS (Risk Of Bias In Systematic reviews))Rates confidence in a systematic review with AMSTAR 2 (high to critically low) or concern in four domains with ROBIS, and produces a rating or domain judgements.Documented
HE-04Survey disclosure completeness check (AAPOR (American Association for Public Opinion Research) disclosure elements)Tests whether a survey report states the sponsor, question wording, population, sampling frame, mode, dates, sample size, margin of error and weighting, and produces a list of disclosed and missing items.Documented
HE-05Margin of error and credibility interval checkChecks that a reported margin of error matches the sampling method and produces a judgement on whether the figure is valid.Documented
HE-06Opt-in panel respondent quality check (bogus respondent screen)Looks for implausible answers that show careless or fraudulent respondents inflating rare responses in an opt-in survey and produces a judgement of inflation risk.Documented
HE-07Overview of reviews (umbrella review; tertiary review)Uses systematic reviews as the unit of search, inclusion and analysis and produces a summary of reviews with a quality rating per review and an overlap map.Documented
HE-08Living systematic reviewKeeps a systematic review continuously updated as new studies appear and produces a versioned evidence summary and an update log.Documented