Techniques
A technique is a named way of analysing something, such as five whys, difference-in-differences or lateral reading. The catalogue holds 236 of them. Each has a plain description of what it does and what it produces, a statement of when it fits and when it does not, the conditions your material must meet, and a result format that tells the model what a run of the technique must contain.
The catalogue records enough about each technique for the process to do three things without a person: decide whether the technique can run on your material (step 2), choose the ones that fit your request (step 3), and put them in a workable order with the right checks (steps 4 to 6). The process pages explain those steps.
How the techniques are grouped
The catalogue has 12 families. A family groups techniques by what they do for a request, such as framing a question, comparing cases, or checking results. The family does not decide when a technique runs. That depends on the kind of technique (question, preparation, primary, verification or challenge), which the dictionary defines.
| Family | Techniques | What the family is for |
|---|---|---|
| Frame the question | 6 | Turn a vague topic into a clear question, before any analysis starts. |
| Decompose and map | 21 | Break a topic into parts and show how the parts relate. |
| Compare | 17 | Set cases, periods or groups side by side to see what differs. |
| Trace cause and test claims | 34 | Look for causes, and test whether a claim holds. |
| Handle evidence | 29 | Judge how far to trust sources, and prepare them for use. |
| Project forward | 25 | Estimate what may happen next. |
| Assess actors, options and risks | 19 | Work out who is affected, what the choices are and what could go wrong. |
| Challenge | 15 | Test a conclusion by arguing against it. |
| Prepare and extract | 14 | Get the material ready, so that counts and themes describe what you think they do. |
| Measure and model | 16 | Analyse records with statistics and models. |
| Integrate across sources | 11 | Combine sources that overlap or disagree. |
| Check extracted results | 29 | Check results that came from coding or extraction, including model output. |
| Total | 236 |
How to read a technique page
Each technique page has these parts, in this order.
- The header gives the identifier, the family, the evidence status, the kind of technique and the step it runs in, and the strongest statement the technique can support. Evidence status says how well the technique itself is supported: tested (a published study measured its effect or accuracy), documented (described in a text or guidance, without a test), practice (used with no published description found) or unconfirmed (no source could be opened). Across the catalogue, 33 are tested, 150 documented, 13 practice and 40 unconfirmed.
- What it does, when to use it and when it does not fit describe the technique in the words a user would use.
- Result format says what a run must produce.
- Requirements on the material list what the material must be like, such as record-level data or more than one source. Step 2 uses these to remove techniques that cannot run.
- Choosing and ordering and library checks by stakes show how the process places the technique and which checks it needs at each stakes level.
- Failure modes and limits, related techniques, and evidence and tag basis say where the technique goes wrong, which techniques it pairs with or substitutes for, and how the tags were decided. Most tags are inference unless the page names a source.
The property names in the requirements table link to their definitions in the dictionary.
Find a technique
Filter the table by name, purpose or family, or open a family from the list above. The table lists all 236 techniques.
Using the catalogue without the skill
You can choose techniques yourself, following the same steps as the process. Use by hand explains how, from the brief to the answer.
| ID | Technique | Family | Purpose |
|---|---|---|---|
| FQ-01 | PICO family (population, intervention, comparison, outcome; variants PICOT, PEO, PICo, SPIDER) | Frame the question | Splits a vague topic into named slots (population, intervention or exposure, comparison, outcome) and produces a one-sentence structured question and a slot table. |
| FQ-02 | Decision-first framing (decision-focused or value-focused question design) | Frame the question | Starts from the decision to be made and works back to a list of questions, each tagged with the option it could change. |
| FQ-03 | Definitional analysis (explication, concept clarification, conceptual analysis) | Frame the question | Replaces ambiguous key terms with precise working definitions and produces a glossary with borderline cases. |
| FQ-04 | Reframing and issue redefinition (changing the level of analysis or baseline; five-step issue redefinition) | Frame the question | Restates the question at another level, with another baseline, or by rephrasing, broadening, narrowing and reversing it, and produces answers side by side. |
| FQ-05 | Multiple mental models (applying explicit models from several fields to one topic) | Frame the question | Reads one topic through several explicit models from different fields and produces one reading per model. |
| FQ-06 | SMART check (specific, measurable, achievable, relevant, time-bound) | Frame the question | Checks that an objective or question names a scope, a measure, a feasible target and a time limit, and produces the draft rewritten with each criterion answered. |
| DM-01 | MECE decomposition and issue tree (mutually exclusive, collectively exhaustive; logic tree, problem tree; related: work breakdown structure) | Decompose and map | Divides a topic into sub-parts that do not overlap and together cover the whole, repeated at each level, and produces a tree. |
| DM-02 | Hypothesis tree (hypothesis-driven decomposition, claim decomposition) | Decompose and map | Splits a claim into the sub-claims that must all be true for it to be true and produces a tree of claims that goes into a claims register. |
| DM-03 | Morphological analysis (Zwicky box, general morphological analysis) Also serves: Project forward. | Decompose and map | Lists the parameters of a topic and the possible values of each, removes value pairs judged inconsistent, and produces a grid and a reduced set of consistent configurations. |
| DM-04 | Value chain analysis (Porter) | Decompose and map | Divides the activities of one business unit into five primary and four support activities and produces a value chain showing where cost and differentiation concentrate. |
| DM-05 | Taxonomy building (classification scheme, typology, controlled vocabulary) | Decompose and map | Builds a scheme of classes from observable characteristics, assigns items to the classes and produces a hierarchy and a draft codebook. |
| DM-06 | Affinity diagram (KJ method) | Decompose and map | Groups many recorded items, such as survey comments or interview quotes, into clusters that form from the relationships among the items, and produces labelled clusters. |
| DM-07 | Lens application (sorting claims by a classification lens, for example input versus outcome) | Decompose and map | Sorts a set of claims or measures by chosen categories, such as input or outcome, and produces a table of claims by category. |
| DM-08 | Two by two matrix (quadrant analysis) Also serves: Assess actors, options and risks; Project forward. | Decompose and map | Places items on two chosen dimensions so that each falls into one of four cells and produces a quadrant table. |
| DM-09 | PESTLE (political, economic, social, technological, legal, environmental; the shorter PEST omits legal and environmental; outside-in thinking) | Decompose and map | Sorts external factors into political, economic, social, technological, legal and environmental categories and produces a categorised list. |
| DM-10 | SWOT (strengths, weaknesses, opportunities, threats) | Decompose and map | Sorts factors into internal strengths and weaknesses and external opportunities and threats and produces a two by two matrix. |
| DM-11 | Porter's Five Forces (industry structure analysis) | Decompose and map | Rates rivalry, new entrants, substitutes, buyer power and supplier power and produces a five-force diagram with each force rated. |
| DM-12 | Cynefin framework | Decompose and map | Classifies a situation, or each part of it, as clear, complicated, complex, chaotic or confusion by the relation of cause and effect and produces a domain assignment. |
| DM-13 | System archetypes (limits to growth, shifting the burden, fixes that fail, and others) | Decompose and map | Matches an observed behaviour pattern to a recurring feedback structure and produces a named pattern with the loop structure to change. |
| DM-14 | Leverage points (Meadows' twelve places to intervene in a system) | Decompose and map | Ranks types of intervention in a system from least to most effective and produces an ordered list showing the level at which a proposed intervention acts. |
| DM-15 | Wardley mapping | Decompose and map | Positions the components of a service by visibility to the user and by evolution stage, with dependency lines, and produces a two-axis map. |
| DM-16 | Levels-of-analysis models (Marr's three levels, micro, meso and macro levels) | Decompose and map | Examines a topic at several distinct levels, each answering a different kind of question, and produces findings organised by level. |
| DM-17 | Process mapping (flowcharts, swimlane diagrams, value stream maps) | Decompose and map | Records the sequence of steps, inputs, outputs and responsible roles in a process and produces a flow diagram showing handoffs, delays and rework loops. |
| DM-18 | Design structure matrix (dependency structure matrix) | Decompose and map | Records which element depends on or supplies which in a square matrix, reorders it to reveal clusters or sequence, and produces a reordered matrix. |
| DM-19 | Causal loop diagram Also serves: Trace cause and test claims. | Decompose and map | Draws variables and signed causal links so that reinforcing and balancing loops and delays can be identified, and produces a diagram with loop labels. |
| DM-20 | Stock and flow modelling (system dynamics, including simulation) Also serves: Project forward. | Decompose and map | Represents accumulations that change only through inflows and outflows and, when simulated, produces a diagram and simulated time paths. |
| DM-21 | Social network analysis (sociogram, network analysis) Also serves: Assess actors, options and risks. | Decompose and map | Represents actors as nodes and relationships as ties, measures structure with graph metrics, and produces a sociogram, an entity table and metrics. |
| CM1-01 | Historical analogy and parallel analysis (likenesses and differences, analogical explanation) | Compare | Places the present topic beside a past case and rates each likeness and difference, producing a likenesses and differences table that shows where a lesson from the past case may and may not be applied. |
| CM1-02 | Mill's method of agreement and most different systems design (MDSD) | Compare | Looks across dissimilar cases that share an outcome for the factor present in all of them, producing a table of candidate factors by case in which the common factor is the candidate cause. |
| CM1-03 | Mill's method of difference and most similar systems design (MSSD; joint method of agreement and difference) | Compare | Looks for the factor present where an outcome occurs and absent in an otherwise similar case where it does not, producing a short list of the remaining differences between matched cases. |
| CM1-04 | Concomitant variation and method of residues (Mill) | Compare | Infers a link when a factor varies with the effect (concomitant variation) or attributes the unexplained part of an effect to the remaining circumstances (residues), producing a co-movement pattern or a residual. |
| CM1-05 | Comparative case study (structured focused comparison, small-N comparison) | Compare | Examines a small number of cases with the same questions for each, producing a case by case matrix that shows consistent patterns and deviant cases. |
| CM1-06 | Case selection techniques (extreme, deviant, diverse, index, most similar) | Compare | Chooses which cases to compare by their position on a variable distribution or on a model prediction, producing a chosen case set with the rule used. |
| CM1-07 | Qualitative comparative analysis (QCA; crisp-set, fuzzy-set and multi-value variants) | Compare | Uses set theory and Boolean algebra to find combinations of conditions that are necessary or sufficient for an outcome across a medium number of cases, producing a truth table and minimal solution formulas with consistency and coverage scores. |
| CM1-08 | Cross-case synthesis (replication logic) | Compare | Combines completed within-case write-ups into one cross-case matrix, producing a matrix that shows repeating patterns and diverging cases. |
| CM1-09 | Contrast with a structural opposite (term from earlier work) | Compare | Sets the topic beside a case or position that differs from it on a stated defining structure, producing a table of contrasting features that shows what each assumes or lacks. |
| CM1-10 | Cross-sectional and longitudinal comparison (cohort and panel comparison; cross-cycle comparison) | Compare | Compares different units at one time (cross-sectional) or the same units repeatedly (longitudinal), producing group differences at one time or change paths over time. |
| CM1-11 | Before and after comparison (pre-post design) | Compare | Compares an outcome for the same unit before and after an event, producing a change estimate. |
| CM1-12 | Benchmarking and peer comparison (best-practice comparison, competitor analysis) | Compare | Compares an organisation processes and metrics with those of peers or leaders, producing a gap table of own against benchmark performance. |
| CM1-13 | Forecaster and source comparison (forecast scoring, aggregation, track record comparison) Also serves: Project forward. | Compare | Scores a forecast against those of other forecasters on resolved outcomes and aggregates them, producing an accuracy ranking and an aggregate forecast. |
| CM1-14 | Absence analysis (argument from silence, the dog that did not bark) Also serves: Handle evidence. | Compare | Draws a conclusion from an item missing from a document or comparison set where it would be expected if a claim were true, producing a list of expected but absent items. |
| CM1-15 | Gap analysis | Compare | Compares current with desired performance, producing a prioritised list of gaps and the means to close each. |
| CM1-16 | Maturity model and capability assessment (Capability Maturity Model Integration, CMMI) | Compare | Rates processes against staged levels, producing a maturity level or capability profile that shows where an organisation stands and what to improve. |
| CM-01 | Sponsor versus non-sponsor comparison (meta-epidemiological sponsorship analysis) | Compare | Compares the share of favourable results and conclusions between sponsored and independently funded reports on one topic, producing a risk ratio or odds ratio. |
| TC1-01 | Five whys | Trace cause and test claims | Asks why repeatedly about one stated problem and records a single chain of causes, leaving a candidate root cause as a result. |
| TC1-02 | Fishbone diagram (Ishikawa diagram, cause and effect diagram) | Trace cause and test claims | Groups brainstormed causes of one problem under category branches and produces a fishbone diagram with the problem as its head. |
| TC1-03 | Fault tree analysis Also serves: Assess actors, options and risks. | Trace cause and test claims | Works downward from one undesired top event through AND and OR gates to produce a gate tree with minimal cut sets. |
| TC1-04 | Process tracing (diagnostic tests: straw in the wind, hoop, smoking gun, doubly decisive) | Trace cause and test claims | Tests what each rival hypothesis predicts against diagnostic evidence within one case and produces a test result per hypothesis in a hypotheses matrix. |
| TC1-05 | Counterfactual analysis (potential outcomes, minimal rewrite of history) Also serves: Compare. | Trace cause and test claims | Sets the actual outcome against an argued account of what would have happened had one factor differed and produces a counterfactual scenario. |
| TC1-06 | Difference-in-differences Also serves: Compare. | Trace cause and test claims | Compares the change over time in a treated group with the change in a control group and produces an interaction coefficient read as the treatment effect. |
| TC1-07 | Quasi-experimental designs (natural experiment, instrumental variables, regression discontinuity) | Trace cause and test claims | Estimates an effect from observational data in which exposure was assigned as if at random (natural experiment, instrument or cutoff) and produces an effect estimate under stated assumptions. |
| TC1-08 | Confounder identification with directed acyclic graphs (adjustment sets, backdoor criterion) | Trace cause and test claims | Draws assumed causal links as a directed acyclic graph and produces a diagram and an adjustment set of variables to include and to leave out. |
| TC1-09 | Logic model and theory of change | Trace cause and test claims | Maps the expected causal chain from inputs to outcomes with the assumption behind each step and produces a logic model or theory of change. |
| TC1-10 | Contribution analysis | Trace cause and test claims | Builds a reasoned line of evidence that a program contributed to an outcome and produces an evidenced contribution story. |
| TC1-11 | Realist evaluation (context, mechanism, outcome configurations) | Trace cause and test claims | Tests what works for whom and in what circumstances through context, mechanism and outcome configurations and produces a refined program theory. |
| TC1-12 | Process evaluation and impact evaluation | Trace cause and test claims | Examines how an intervention was delivered and estimates what difference it made, and produces findings on delivery quality or an attributed effect estimate. |
| TC1-13 | Analysis of competing hypotheses (ACH) | Trace cause and test claims | Lists plausible hypotheses, rates each evidence item against each, and produces a hypothesis by evidence matrix aimed at disproving hypotheses. |
| TC1-14 | Coherence correction and averaging of probability judgements | Trace cause and test claims | Adjusts each analyst's probabilities on mutually exclusive hypotheses so they sum to one, averages across analysts, and produces a corrected aggregated judgement. |
| TC1-15 | Bayesian updating (likelihood reasoning, likelihood ratio, Bayes factor) | Trace cause and test claims | Revises the probability of a hypothesis in proportion to how much more likely the evidence is under it than under alternatives and produces a posterior probability or odds. |
| TC1-16 | Disconfirmation search (falsification, seeking disconfirming evidence) | Trace cause and test claims | Looks for evidence that would contradict a favoured claim and produces a list of refuting evidence found or not found with the search used. |
| TC1-17 | Sensitivity analysis (robustness check, leave-one-out) | Trace cause and test claims | Re-runs an analysis under different assumptions or inclusion rules and produces a table of results by assumption showing whether the conclusion reverses. |
| TC1-18 | Internal consistency check (term from earlier work) | Trace cause and test claims | Tests whether the claims, figures and assumptions within one analysis are compatible with each other and produces a list of contradictions or inconsistent figures. |
| TC1-19 | Toulmin argument analysis | Trace cause and test claims | Labels the parts of an argument as claim, ground, warrant, backing, rebuttal and qualifier and produces a diagram per argument. |
| TC1-20 | Argument mapping (argument diagramming) | Trace cause and test claims | Draws claims, reasons and objections as a diagram and produces an argument map that shows isolated claims and reasons not connected to any conclusion. |
| TC1-21 | Fallacy analysis (informal fallacies, pragma-dialectical rules) | Trace cause and test claims | Compares an argument with a list of known defective patterns and produces a list of suspected defects, each read as a prompt for missing support. |
| TC1-22 | Abduction and inference to the best explanation | Trace cause and test claims | Ranks candidate explanations of an observation on simplicity, generality and coherence and produces a ranked list of explanations. |
| TC1-23 | Ladder of inference | Trace cause and test claims | Traces how a conclusion was reached from observed data through selected data and interpretation to belief and produces a stepwise list. |
| TC1-24 | Reflective equilibrium | Trace cause and test claims | Adjusts principles and case judgements against each other until they cohere and produces a revised set of principles and judgements. |
| TC1-25 | First principles reasoning | Trace cause and test claims | Replaces inherited convention with a derivation from basic premises and produces a derivation chain with the base premises listed as assumptions. |
| TC1-26 | Thought experiments | Trace cause and test claims | Constructs an imagined case to test a principle and produces a judgement about the case that counts as evidence on the principle only if reasoners agree. |
| TC1-27 | Fermi estimation and order-of-magnitude checks (back-of-the-envelope estimate) | Trace cause and test claims | Breaks an unknown quantity into factors that can be estimated, multiplies them, and produces a number with a plausible range used as a bound on a detailed result. |
| TC1-28 | Dimensional and unit checks | Trace cause and test claims | Checks that both sides of a relation have the same units and produces a unit balance in which a mismatch flags an error in the derivation. |
| TC-01 | GRIM test (granularity-related inconsistency of means) | Trace cause and test claims | Checks whether a reported mean of whole-number data can arise from the reported sample size and item count and produces a flag for possible or impossible means. |
| TC-02 | SPRITE (Sample Parameter Reconstruction via Iterative TEchniques) | Trace cause and test claims | Reconstructs possible datasets that match reported descriptive statistics and produces plausible samples showing the range of possible item distributions. |
| TC-03 | statcheck (recomputation of reported p-values) | Trace cause and test claims | Recomputes p-values from reported test statistics and degrees of freedom and produces a list of inconsistencies and gross inconsistencies. |
| TC-04 | Claim and citation verification against source text (fact verification; claim checking; citation support check; quotation accuracy check) | Trace cause and test claims | Labels each claim as supported, refuted or not enough information against retrieved evidence, or rates whether a cited source supports its claim, and produces a label with evidence sentences. |
| TC-05 | Subgroup and Simpson's paradox check (stratified reanalysis; confounding check; collapsibility check) | Trace cause and test claims | Recomputes an association within subgroups to see whether the pooled direction reverses or vanishes and produces pooled and stratified estimates side by side. |
| TC-06 | Ecological fallacy check (aggregation bias check) | Trace cause and test claims | Decides whether a claim about people is supported by group-level data and produces a decision on whether the claim is group-level or individual-level. |
| HE1-01 | Admiralty system (source reliability and information credibility grading) | Handle evidence | Assigns each report a two-character grade for source reliability (A to F) and information credibility (1 to 6) and produces an appraised evidence base. |
| HE1-02 | Quality of information check | Handle evidence | Rates the completeness, accuracy and corroboration of each source and produces a source database with a statement of what is known and unknown. |
| HE1-03 | CRAAP test (currency, relevance, authority, accuracy, purpose) | Handle evidence | Screens one source against five criteria (currency, relevance, authority, accuracy, purpose) and produces a checklist judgement for the evidence base. |
| HE1-04 | SIFT (stop, investigate the source, find better coverage, trace claims to the original; claim verification and fact-checking) | Handle evidence | Moves from a claim to its publisher, to other coverage and to the original source, and produces a short credibility judgement with the original context located. |
| HE1-05 | Lateral reading | Handle evidence | Leaves the page to read about the publisher in other sources and produces a judgement of publisher credibility. |
| HE1-06 | Triangulation (source, method, investigator and theory triangulation; corroboration) | Handle evidence | Compares independent lines of evidence on one question and produces a convergence table of agreement, disagreement and reasons. |
| HE1-07 | Evidence hierarchies and certainty grading (study design levels, risk-of-bias assessment, Grading of Recommendations Assessment, Development and Evaluation or GRADE) | Handle evidence | Ranks study designs and rates confidence in a body of evidence from high to very low, and produces a certainty rating with reasons for downgrading. |
| HE1-08 | Systematic review | Handle evidence | Collects all eligible studies by a pre-specified, reproducible search, appraises and synthesises them, and produces a screening log, an evidence table and a synthesis. |
| HE1-09 | Rapid review | Handle evidence | Runs a systematic review with streamlined steps to meet a deadline and produces a narrative synthesis that states the shortcuts taken. |
| HE1-10 | Scoping review (evidence mapping) | Handle evidence | Maps the extent, types and gaps of evidence on a broad topic without judging the strength of findings, and produces a charted table. |
| HE1-11 | Meta-analysis | Handle evidence | Combines effect estimates from separate studies into a weighted average with a confidence interval and produces a forest plot, a pooled effect and heterogeneity statistics. |
| HE1-12 | Narrative, thematic and qualitative evidence synthesis (including meta-ethnography) | Handle evidence | Combines findings across studies in words or themes when pooling numbers is not appropriate, and produces a structured narrative or theme set linked back to studies. |
| HE1-13 | Reflexive thematic analysis (Braun and Clarke) | Handle evidence | Identifies and interprets patterns of meaning across qualitative data in six phases and produces themes with supporting extracts. |
| HE1-14 | Framework analysis (Framework Method) | Handle evidence | Sorts coded qualitative data into a case by theme matrix and produces the matrix for reading across cases and themes. |
| HE1-15 | Content analysis (conventional, directed and summative) | Handle evidence | Codes text into categories, then counts or interprets category use, and produces category frequencies or a category structure. |
| HE1-16 | Discourse analysis (critical discourse analysis) | Handle evidence | Examines how language constructs topics and positions actors and produces an interpretive account of framing checked against quoted passages. |
| HE1-17 | Grounded theory coding (open, axial and selective coding; constant comparison) | Handle evidence | Builds categories and a theory from data through open, axial and selective coding with constant comparison and produces codes, memos and a theory. |
| HE1-18 | Document and policy analysis | Handle evidence | Reads documents systematically to extract facts, positions, changes or stated reasons using their provenance and purpose, and produces an extraction table or chronology. |
| HE1-19 | Bibliometric analysis (citation analysis, science mapping, co-citation, co-word analysis) | Handle evidence | Counts and maps publications, authors and citations to show the structure and growth of a field and produces performance tables and network maps. |
| HE1-20 | Trend and time-series decomposition Also serves: Project forward. | Handle evidence | Separates a series into trend, seasonal and irregular parts and produces plots and a fitted trend. |
| HE1-21 | Distribution, outlier and base-rate checks | Handle evidence | Examines the shape, extremes and baseline frequency of data before a figure is interpreted and produces histograms, summary statistics, flagged outliers and a base rate. |
| HE-01 | Grey literature appraisal checklist (AACODS (Authority, Accuracy, Coverage, Objectivity, Date, Significance)) | Handle evidence | Scores a non-peer-reviewed item on six criteria (authority, accuracy, coverage, objectivity, date, significance) and produces a completed checklist. |
| HE-02 | Critical appraisal checklist by study design (CASP (Critical Appraisal Skills Programme) checklists; JBI (Joanna Briggs Institute) checklists) | Handle evidence | Answers design-specific questions on validity, results and relevance for one study and produces a completed checklist for the evidence base. |
| HE-03 | Appraisal of a systematic review (AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews 2); ROBIS (Risk Of Bias In Systematic reviews)) | Handle evidence | Rates confidence in a systematic review with AMSTAR 2 (high to critically low) or concern in four domains with ROBIS, and produces a rating or domain judgements. |
| HE-04 | Survey disclosure completeness check (AAPOR (American Association for Public Opinion Research) disclosure elements) | Handle evidence | Tests whether a survey report states the sponsor, question wording, population, sampling frame, mode, dates, sample size, margin of error and weighting, and produces a list of disclosed and missing items. |
| HE-05 | Margin of error and credibility interval check | Handle evidence | Checks that a reported margin of error matches the sampling method and produces a judgement on whether the figure is valid. |
| HE-06 | Opt-in panel respondent quality check (bogus respondent screen) | Handle evidence | Looks for implausible answers that show careless or fraudulent respondents inflating rare responses in an opt-in survey and produces a judgement of inflation risk. |
| HE-07 | Overview of reviews (umbrella review; tertiary review) | Handle evidence | Uses systematic reviews as the unit of search, inclusion and analysis and produces a summary of reviews with a quality rating per review and an overlap map. |
| HE-08 | Living systematic review | Handle evidence | Keeps a systematic review continuously updated as new studies appear and produces a versioned evidence summary and an update log. |
| PJ-01 | Trend extrapolation and time-series forecasting (exponential smoothing, autoregressive integrated moving average models, trend impact analysis) | Project forward | Extend the measured history of one indicator with a stated functional form to give a central projection with prediction intervals, as a draft result. |
| PJ-02 | Reference class forecasting (outside view, base-rate forecasting) Also serves: Compare. | Project forward | Replace a bottom-up estimate with the distribution of outcomes in similar completed cases, giving a base-rate distribution and the case position in it as a draft result. |
| PJ-03 | Forecasting tournaments and superforecasting practice | Project forward | Produce dated probability estimates for resolvable questions, score them with Brier scores and improve them by updating and aggregation, leaving a scored probability record as a draft result. |
| PJ-04 | Delphi technique (real-time Delphi, mini-Delphi) | Project forward | Collect anonymous expert estimates over two or more rounds with controlled feedback, giving the median and interquartile range for each round as a draft result. |
| PJ-05 | Prediction markets (information markets) | Project forward | Read the price of a defined contract as a probability for an event, giving a price series and a liquidity-qualified probability as a draft result. |
| PJ-06 | Structured expert elicitation (IDEA protocol: investigate, discuss, estimate, aggregate) | Project forward | Elicit numeric estimates and ranges from experts before and after discussion and aggregate them mathematically, giving aggregated estimates with intervals as a draft result. |
| PJ-07 | Technology S-curve analysis (logistic growth, technology substitution curve) | Project forward | Fit a logistic curve to an adoption or performance series to estimate a ceiling and an inflection date, as a draft result. |
| PJ-08 | Bass diffusion model | Project forward | Forecast the adoption path of a product or practice from external-influence and imitation parameters and a market potential, giving a cumulative adoption curve and peak timing as a draft result. |
| PJ-09 | Hype cycle (technology expectations curve) | Project forward | Place a technology on a five-phase expectations curve with a years-to-mainstream band, giving a placement to compare with measured attention data as a draft result. |
| PJ-10 | Trajectory analysis (technological trajectories, envelope curves; term not standard) | Project forward | Track a performance metric across product generations and extend the envelope of best performance, giving a trajectory chart and rate of improvement as a draft result. |
| PJ-11 | Agent-based modelling and Monte Carlo simulation (microsimulation) | Project forward | Simulate agents or random draws many times to give an outcome distribution with tail probabilities and the inputs that account for the spread, as a draft result. |
| PJ-12 | Scenario planning (intuitive logics, Shell approach) | Project forward | Build a small set of internally consistent plausible futures from critical uncertainties so that a strategy can be examined in each, giving named narrative scenarios as a draft result. |
| PJ-13 | Scenario matrix (two by two scenario axes, alternative futures analysis) | Project forward | Cross the two most critical and uncertain forces to give four scenario quadrants with narratives and signposts, as a draft result. |
| PJ-14 | Alternative futures archetypes (Manoa school, four generic futures) | Project forward | Apply a fixed set of generic futures to a topic to give a short image of the topic under each archetype, as a draft result. |
| PJ-15 | Futures wheel (implications wheel) | Project forward | Map first, second and third-order consequences of one change in rings, giving a radial diagram of consequences as a draft result. |
| PJ-16 | Causal layered analysis | Project forward | Examine a topic at four depths (litany, social and structural causes, worldview and myth) for the present and each alternative future, giving a four-row table as a draft result. |
| PJ-17 | Cross-impact analysis (cross-impact matrix) | Project forward | Adjust the probability of each event in a set according to how every other event raises or lowers it, giving adjusted event probabilities and an influence matrix as a draft result. |
| PJ-18 | Backcasting | Project forward | Fix a desired end state and work backwards to the steps and conditions that connect it to the present, giving a dated pathway as a draft result. |
| PJ-19 | Policy stress-testing | Project forward | Run a policy or plan through defined adverse or alternative conditions, giving a table of performance by condition and the cheapest changes that remove failures as a draft comparison matrix. |
| PJ-20 | Wargaming and simulation exercises (policy gaming, tabletop exercise) | Project forward | Have players take the roles of competing actors in a structured scenario, giving a record of decisions and surprises as a draft result. |
| PJ-21 | Robust decision making, adaptive pathways and info-gap analysis | Project forward | Test many model runs or futures to find the conditions under which a strategy fails and design strategies that adapt, giving vulnerability scenarios and pathway maps with trigger points as a draft result. |
| PJ-22 | Technology roadmapping | Project forward | Lay out how markets, products, technologies and resources are expected to evolve on a common time axis, giving a layered time chart with dependencies as a draft result. |
| PJ-23 | Horizon scanning (environmental scanning) | Project forward | Search systematically for early signs of change in sources outside the topic usual field, giving a signal database with source, date and rating as a collected evidence base. |
| PJ-24 | Weak signal detection (early warning signs, wild cards) | Project forward | Identify small ambiguous early indications that may precede a discontinuity, giving a watchlist with a strength rating over time as an appraised evidence base. |
| PJ-25 | Leading indicators and signposts (indicators of change, trigger points) | Project forward | Select measurable quantities that change before an outcome of interest and set thresholds beforehand, giving a table of indicators with lead time, threshold, source, frequency and owner as a draft result. |
| AO-01 | Stakeholder analysis (stakeholder mapping) | Assess actors, options and risks | Identifies who is affected by or can affect a topic and classifies each by interest, influence and attitude, producing a stakeholder register and grid. |
| AO-02 | Power-interest grid (influence-interest grid, power-impact grid) | Assess actors, options and risks | Plots actors on power and interest to choose an engagement strategy for each, producing a two by two grid. |
| AO-03 | Stakeholder salience model (Mitchell, Agle and Wood, 1997) | Assess actors, options and risks | Ranks actors by legitimacy, power and urgency, producing a diagram with eight classes from definitive to non-stakeholder. |
| AO-04 | Political economy analysis | Assess actors, options and risks | Explains why a problem persists by analysing structures, institutions, actors, interests, incentives and power relations, producing a synthesis of key propositions, pathways of change, and maps of blockers and champions. |
| AO-05 | Customer journey mapping | Assess actors, options and risks | Turns a persona goals, actions, thoughts and emotions across stages into a visual map, producing a map with phases, touchpoints, pain points and opportunities. |
| AO-06 | Personas | Assess actors, options and risks | Condenses research on user groups into a fictional but realistic description, producing a persona profile with goals, context and concerns. |
| AO-07 | Jobs to be done (JTBD) | Assess actors, options and risks | Frames demand around the progress a customer is trying to make in a circumstance, producing a job statement that shows which jobs are poorly served. |
| AO-08 | Eightfold path (Bardach) | Assess actors, options and risks | Structures a policy question into eight steps from defining the problem to telling the story, producing a written options analysis with criteria, projected outcomes and trade-offs. |
| AO-09 | Longlist-to-shortlist options analysis (options appraisal) | Assess actors, options and risks | Generates a wide set of options, filters them against objectives and critical success factors, and appraises a shortlist of about five, producing a longlist and shortlist with red, amber and green ratings. |
| AO-10 | Cost-benefit analysis | Assess actors, options and risks | Values all costs and benefits to society in money, discounts them and compares options by net present value, producing net present value per option with sensitivity results. |
| AO-11 | Cost-effectiveness and cost-utility analysis | Assess actors, options and risks | Compares options on cost per unit of one quantified effect when benefits cannot be monetised, producing cost per unit of effect for each option. |
| AO-12 | Multi-criteria decision analysis (MCDA) | Assess actors, options and risks | Scores options against several weighted criteria, producing weights, scores and a ranking, with a test of how the ranking changes when weights change. |
| AO-13 | Decision tree and expected value | Assess actors, options and risks | Lays out decisions, chance events and outcomes and computes expected value or utility, producing a decision tree that shows the alternative with the best expected value. |
| AO-14 | Real options analysis | Assess actors, options and risks | Values the right but not the obligation to expand, delay, switch or abandon an investment, producing an option value added to a standard valuation. |
| AO-15 | Risk matrix (likelihood and consequence matrix, risk heat map) | Assess actors, options and risks | Rates each risk by plotting likelihood against consequence, producing a grid of ratings that shows which risks need treatment first. |
| AO-16 | Failure mode and effects analysis (FMEA) | Assess actors, options and risks | Lists the failure modes of each function or component with causes and effects and ranks them, producing a worksheet with ranked failure modes that shows single points of failure. |
| AO-17 | Bow-tie analysis | Assess actors, options and risks | Maps causes, a central event, consequences and the barriers on each path, producing a cause, event and consequence diagram that shows pathways lacking a barrier. |
| AO-18 | Threat modelling (STRIDE: spoofing, tampering, repudiation, information disclosure, denial of service, elevation of privilege) | Assess actors, options and risks | Identifies security threats using six categories mapped to security properties, producing a list of threats by category. |
| CH-01 | Key assumptions check (assumptions check) | Challenge | Lists the stated and unstated premises of the current line, challenges each, and produces a refined assumptions register with the conditions that would invalidate each assumption. |
| CH-02 | Strategic assumption surfacing and testing (SAST) | Challenge | Lists each interested group assumptions, rates them, has opposing groups debate them in four stages, and produces a rated assumptions register. |
| CH-03 | Devil's advocacy | Challenge | Builds the strongest case against the dominant view or plan, starting from its most vulnerable assumptions, and produces a labelled devil advocate finding. |
| CH-04 | Team A and Team B | Challenge | Has two teams each build the best case for a competing view and debate before a jury of peers, and produces competing papers and debate findings. |
| CH-05 | Red team analysis (red team, red cell) | Challenge | Models how an adversary or other actor would decide by reasoning in that actor first-person voice, or examines a plan from outside the planning team, and produces first-person memos predicting decisions. |
| CH-06 | High-impact and low-probability analysis Also serves: Project forward. | Challenge | Works out plausible pathways to an unlikely but consequential event, with triggers and indicators, and produces pathways and an indicator list. |
| CH-07 | What if analysis Also serves: Project forward. | Challenge | Assumes an unexpected event has occurred, explains how it could have come about, and produces scenarios with indicators and consequence assessments. |
| CH-08 | Premortem (prospective hindsight, inversion) Also serves: Project forward. | Challenge | Asks a team to assume the plan has already failed and to write the reasons, and produces a grouped list of failure reasons and draft assumptions. |
| CH-09 | Structured self-critique | Challenge | Asks about sources of uncertainty, evidence, gaps, critical assumptions and alternative hypotheses, and produces a written critique of one own analysis. |
| CH-10 | Rapoport's rules (fair restatement of an opposing position) Also serves: Compare. | Challenge | Requires a fair restatement of the opposing position, a list of points of agreement and of what was learned before any rebuttal, and produces a restatement and an agreement list. |
| CH-11 | Steelmanning Also serves: Compare. | Challenge | Builds the strongest version of a position the analyst disagrees with, responds to it, and produces a strengthened opposing argument and a response. |
| CH-12 | Socratic questioning (Paul and Elder's six question types) | Challenge | Asks a sequence of questions that expose unclear terms, hidden assumptions and unsupported claims, and produces a record of questions and answers and a draft assumptions register. |
| CH-13 | Cognitive bias checks and debiasing (including consider-the-opposite) | Challenge | Runs a conclusion through a list of known judgement biases, applies a counter-procedure, and produces a checklist of marked items where disconfirming evidence should be sought. |
| CH-14 | Structured brainstorming | Challenge | Generates and clusters many ideas about a focal question before any critique and produces clustered, prioritised ideas. |
| CH-15 | Group process safeguards (everyone speaks once before anyone speaks twice, mindguarding) | Challenge | Controls how a group forms views, with each person contributing in turn, leaders withholding early preferences and evaluators rotating, and produces independent individual views. |
| PE-01 | Corpus construction and sampling (document selection; search string design; sampling frame design) | Prepare and extract | Define which documents enter the analysis so that counts and themes describe a stated population, giving a corpus table with counts and an inclusion log. |
| PE-02 | Preprocessing sensitivity testing (text cleaning; tokenisation; stopword removal; stemming; boilerplate removal; preText) | Prepare and extract | Convert raw text into countable units and re-run the main result under several defensible cleaning regimes, giving cleaned text, a document-term matrix, a rule log and a sensitivity result. |
| PE-03 | Near-duplicate detection (shingling; MinHash; locality-sensitive hashing; text reuse detection) | Prepare and extract | Find identical or nearly identical documents so repeats are not counted as independent evidence, giving duplicate clusters with similarity scores and a deduplicated corpus. |
| PE-04 | Data profiling and rule-based validation (data audit; data quality assessment; edit checks) | Prepare and extract | Describe the structure, completeness, validity and consistency of each field and test explicit rules, giving a profile table per column and a list of records that break each rule. |
| PE-05 | Missing data diagnosis and treatment (missingness analysis; imputation; complete case analysis; non-response weighting) | Prepare and extract | Measure how much is missing, where and for whom, then choose deletion, imputation or weighting, giving the missing share by field and subgroup and a cleaned dataset. |
| PE-06 | Keyword in context (KWIC; concordance) | Prepare and extract | List every occurrence of a keyword or phrase with a fixed number of surrounding words, giving a table of document, position and context. |
| PE-07 | Collocation and n-gram extraction (phrase detection; association measures) | Prepare and extract | Find word sequences that occur together more often than chance, giving ranked phrases with association scores as a feature set. |
| PE-08 | Named entity recognition (NER; entity extraction) | Prepare and extract | Tag spans of text as people, organisations, places, dates and other types, giving a table of mentions with type and position and a tagged corpus. |
| PE-09 | Relation and event extraction (information extraction; open information extraction; temporal expression extraction) | Prepare and extract | Extract who did what to whom, with arguments and time, as structured rows, giving an event or relation table and a coded corpus. |
| PE-10 | Schema-based structured extraction with an LLM (LLM (large language model) information extraction; form filling from documents) | Prepare and extract | Fill a fixed set of fields, such as amount, date, party, stated risk or decision, for each document, giving one record per document and the share of missing or not stated values. |
| PE-11 | Knowledge graph construction from text (entity-relation graph; ontology population) | Prepare and extract | Convert resolved entities and extracted relations into a graph of typed nodes and edges, each linked to its source sentence, giving a graph with provenance as a draft result. |
| PE-12 | Model summarisation of documents (abstractive summarisation; multi-document summarisation) | Prepare and extract | Write a condensed text that paraphrases the content of one or many documents, giving a summary as a draft result. |
| PE-13 | Retrieval-augmented analysis (RAG (retrieval-augmented generation); grounded question answering over a corpus) | Prepare and extract | Retrieve passages relevant to a question and have a model answer from them with citations, giving an answer with quoted passages and a claims register. |
| MM-01 | Exploratory data analysis with plots and distribution checks (EDA (exploratory data analysis); summary statistics; Q-Q plot; scatterplot matrix) | Measure and model | Examine a cleaned table with plots and summary statistics to find shape, outliers, trends, subgroup differences and broken assumptions, and produce a draft result of plots and summaries. |
| MM-02 | Algorithmic outlier and anomaly detection (Isolation Forest; Local Outlier Factor; Elliptic Envelope; One-Class SVM (support vector machine)) | Measure and model | Score each record by how far it differs from the rest, using distance, density, isolation or a covariance estimate that resists outliers, and produce an outlier score and label per record. |
| MM-03 | Clustering of records (k-means; hierarchical (agglomerative) clustering; DBSCAN (density-based spatial clustering of applications with noise); HDBSCAN (hierarchical DBSCAN)) | Measure and model | Group records by distance or density without predefined categories and produce a cluster label per record, with centroids, a dendrogram or a noise label. |
| MM-04 | Dimensionality reduction (PCA (principal component analysis); factor analysis; non-negative matrix factorisation) | Measure and model | Replace many correlated numeric fields with a few components and produce loadings, explained variance and a score per record. |
| MM-05 | Survival and duration analysis (Kaplan-Meier estimator; Nelson-Aalen estimator; Cox proportional hazards; time-to-event analysis) | Measure and model | Estimate the time until an event while using records whose event has not yet been seen, and produce survival curves with intervals and a median duration. |
| MM-06 | Survey weighting and subgroup estimation (design weights; calibration; post-stratification; relative standard error) | Measure and model | Apply survey weights so that sample results represent the population and produce weighted estimates with standard errors by subgroup. |
| MM-07 | Supervised classification and dictionary methods (text classification; fine-tuned classifier; lexicon method) | Measure and model | Assign documents to predefined categories using human-labelled examples or word lists and produce coded documents with confidence and a confusion matrix. |
| MM-08 | LLM coding and zero-shot classification of text (LLM annotation; LLM-assisted deductive coding; few-shot classification; natural language inference classification) | Measure and model | Apply a human-written codebook or category descriptions to text with a large language model and produce codes, rationales and per-code agreement with human coders. |
| MM-09 | Sentiment analysis (polarity detection; opinion mining; affect lexicon) | Measure and model | Score text as positive, negative or neutral, or by emotion, and produce scores per document with aggregates by group or period. |
| MM-10 | Stance detection (position detection) | Measure and model | Label whether each text favours, opposes or is neutral toward a named target and produce a stance label per text and target. |
| MM-11 | Framing analysis at scale (frame detection; issue framing; media frames coding) | Measure and model | Identify which aspects of an issue each text emphasises using a defined frame scheme and produce frame labels with frame shares by source and period. |
| MM-12 | Probabilistic topic modelling (LDA (latent Dirichlet allocation); STM (structural topic model); mixed-membership model) | Measure and model | Model each document as a mixture of topics and each topic as a distribution over words, and produce topic word lists and document-topic proportions. |
| MM-13 | Embedding-based topic modelling (BERTopic; Top2Vec; clustering-based topic models) | Measure and model | Embed documents with a neural model, cluster the embeddings and describe each cluster by distinctive words, and produce topics with representative words, representative documents and an outlier group. |
| MM-14 | Document clustering (computer-assisted clustering; k-means on document vectors) | Measure and model | Group similar documents without predefined categories using counts or embeddings and produce cluster assignments, centroids and exemplar documents. |
| MM-15 | Computer-assisted theme building (reading-assisted thematic analysis; human-guided topic labelling) | Measure and model | Have a person read exemplars from computer-generated groups, then merge, split and name them, and produce a named theme set with exemplar quotations and a count per theme. |
| MM-16 | Word embeddings for concept measurement and semantic change (diachronic embeddings; concept direction) | Measure and model | Measure how terms relate in meaning within a corpus or across periods and produce nearest neighbours, similarity to seed words and change scores. |
| IS-01 | Entity resolution and linking (record linkage; entity matching; entity linking; coreference resolution; blocking) | Integrate across sources | Decides which records or text mentions refer to the same entity by comparing attributes, with no shared identifier required, and produces match decisions or linked clusters. |
| IS-02 | Schema matching and ontology matching (schema mapping; field mapping) | Integrate across sources | Establishes which fields or concepts in different sources mean the same thing and produces a mapping table of equivalent, partly equivalent and unmapped fields. |
| IS-03 | Common data model harmonisation (OMOP (Observational Medical Outcomes Partnership) common data model; data harmonisation) | Integrate across sources | Transforms source datasets into one shared structure and vocabulary so that the same analysis runs on each and produces a standardised dataset with a coverage report. |
| IS-04 | Data fusion with conflict resolution (majority voting; deciding and mediating resolution functions) | Integrate across sources | Combines records on the same entity into one record, resolving conflicting values by a rule, and produces a fused record per entity with uncertain and contradictory values marked. |
| IS-05 | Truth discovery with source reliability weighting (truth finding; source reliability estimation) | Integrate across sources | Jointly estimates the true value of each object and the reliability of each source from conflicting claims and produces a truth per object and a weight per source. |
| IS-06 | Source dependence and copy detection (source independence analysis) | Integrate across sources | Estimates whether sources copy from each other so that their agreement is not counted as independent support and produces a dependence estimate for source pairs or groups. |
| IS-07 | Convergent comparison of qualitative and quantitative results (convergent parallel design; concurrent triangulation) | Integrate across sources | Analyses qualitative and quantitative results on the same topic separately, then merges them to compare and produces a comparison of agreement, partial agreement and discordance by topic. |
| IS-08 | Joint display | Integrate across sources | Places quantitative and qualitative results in one table, figure or matrix with shared row labels and produces a matrix that shows where the strands agree, expand each other or conflict. |
| IS-09 | Quantitising and qualitising (data transformation) | Integrate across sources | Converts one data type into the other, for example by counting coded themes, and produces a single-type dataset of frequencies or profiles. |
| IS-10 | Explanatory sequential integration (explanatory sequential design) | Integrate across sources | Uses a completed quantitative result, especially an unexpected or extreme one, to choose the qualitative material read to explain it and produces an explanation linked to the numeric pattern. |
| IS-11 | Exploratory sequential integration (exploratory sequential design) | Integrate across sources | Uses qualitative results to decide what is measured in a later quantitative stage and produces measures or hypotheses derived from themes. |
| CK-01 | Inter-coder agreement statistics (percentage agreement; Cohen's kappa; weighted kappa; Scott's pi; Fleiss' kappa; Krippendorff's alpha; intercoder reliability testing) | Check extracted results | Measure how far coders, human or model, give the same code to the same items, corrected for chance, and produce an agreement coefficient per code with a bootstrap interval in a validation report. |
| CK-02 | Model versus human coding agreement protocol (LLM-as-coder reliability test; intra-coder agreement across runs) | Check extracted results | Score a model as one coder against a human coder or panel and against itself across repeated runs, and produce human-human, model-human and model-model agreement in a validation report. |
| CK-03 | Bootstrap confidence intervals for agreement and accuracy (bootstrap resampling of items) | Check extracted results | Quantify sampling uncertainty in an agreement or accuracy figure computed from a review sample by resampling items, and produce an interval around the coefficient. |
| CK-04 | Gold-standard validation sample (ground truth set; human-coded validation set) | Check extracted results | Build a random or stratified human-coded subset against which model output is scored and produce a labelled set. |
| CK-05 | Out-of-sample validation (train and test split; k-fold cross-validation; bootstrap; permutation test) | Check extracted results | Tune prompts, rules or models on one set of items and score them on a held-out set, and produce an out-of-sample error, interval or null distribution in a validation report. |
| CK-06 | Bias correction of model labels with a human-coded subsample (design-based supervised learning (DSL); prediction-powered inference (PPI); regression estimator correction) | Check extracted results | Combine model labels for all items with human labels for a random subsample so that downstream estimates and intervals stay valid, and produce a corrected estimate with an interval. |
| CK-07 | Split-sample design for measures discovered from text (split-sample framework) | Check extracted results | Discover a measure such as categories or topics in one random part of the corpus and estimate its effect in the other, and produce an effect estimate that does not overfit the discovery. |
| CK-08 | Semantic, predictive and convergent validity checks (construct validity checks) | Check extracted results | Test whether an extracted category or measure behaves as the concept it claims to measure, and produce rater verdicts and comparisons with a known event or established measure in a validation report. |
| CK-09 | Cluster validation indexes (silhouette score; adjusted Rand index; adjusted mutual information; V-measure; stability analysis) | Check extracted results | Score how well a clustering fits the data or agrees with known labels, and produce a score per clustering for comparison across cluster numbers and algorithms. |
| CK-10 | Repeated-run consistency test (test-retest; intra-coder reliability; TARr@N and TARa@N (raw and parsed answer agreement); settings pinning) | Check extracted results | Run the same prompt and input several times with model version, temperature and seed recorded, and produce agreement rates and accuracy spread in a validation report. |
| CK-11 | Prompt stability scoring (prompt perturbation; paraphrase test) | Check extracted results | Re-run a task with paraphrased prompts and apply reliability metrics to the labels, and produce a prompt stability score in a validation report. |
| CK-12 | Specification sweep across models and prompts (LLM hacking test; multiverse analysis; specification curve) | Check extracted results | Repeat the downstream statistical test across several models, prompts and settings, and produce error rates for false positive, false negative, wrong-sign and wrong-magnitude conclusions. |
| CK-13 | Multiple-testing correction (Benjamini-Hochberg false discovery rate control; Bonferroni; Holm) | Check extracted results | Adjust the significance threshold when many comparisons are made, and produce adjusted p-values or a list of retained results. |
| CK-14 | Data leakage audit (train-test contamination check; model information sheet) | Check extracted results | Confirm that no information from the test items or the outcome entered the model, prompt or features, and produce a leakage checklist and a model information sheet. |
| CK-15 | Sample representativeness audit (coverage check; selection bias check) | Check extracted results | Compare the analysed sample with the population it claims to describe and with the steps that selected it, and produce a comparison table on known variables and a list of exclusion steps. |
| CK-16 | Entailment-based consistency check (textual entailment; NLI (natural language inference) consistency; SummaC; FactCC; faithfulness checking) | Check extracted results | Score whether each sentence of a summary or answer is supported by the source sentences, and produce a consistency score per sentence and per summary. |
| CK-17 | Source-span and quotation check by code (quote-and-verify; extractive grounding; citation verification) | Check extracted results | Require each extracted statement to point to the exact passage it came from, confirm by string match that the passage exists, and produce a pass or fail list per statement. |
| CK-18 | Numeric and attribution recomputation by code (numeric fidelity check; deterministic recomputation) | Check extracted results | Extract every number, date, name and quotation from an output by code and confirm that each appears in, or can be derived from, the source, and produce a list of unmatched figures and names. |
| CK-19 | Atomic claim decomposition and scoring (FActScore; atomic fact checking) | Check extracted results | Split generated text into atomic facts and score the share that a reliable knowledge source supports, and produce a supported proportion and a list of unsupported facts. |
| CK-20 | Sampling-consistency hallucination detection (SelfCheckGPT; self-consistency check) | Check extracted results | Draw several stochastic responses to the same prompt and flag statements that the other samples do not support, and produce sentence-level consistency scores. |
| CK-21 | Reference-free grounding metrics for retrieval-augmented answers (Ragas; faithfulness, answer relevance and context relevance scoring) | Check extracted results | Score whether retrieval finds relevant focused passages, whether the answer uses them faithfully and the quality of the answer, and produce a score per dimension without human reference answers. |
| CK-22 | LLM-as-judge with human calibration (model-graded evaluation) | Check extracted results | Use a strong model to grade outputs against a rubric and check the judge against human grades on a subsample, and produce judge scores with judge-human agreement. |
| CK-23 | Confidence calibration check (reliability diagram; expected calibration error; temperature scaling; P(True); verbalised confidence) | Check extracted results | Check that a stated confidence of 80 per cent matches being right about 80 per cent of the time, rescale confidence if not, and produce a calibration curve and error. |
| CK-24 | Review sampling design (random audit sample; stratified sample; risk-based sampling; acceptance sampling; rule of three) | Check extracted results | Choose which outputs to inspect and how many so that the observed error rate has a stated bound, and produce an error-rate estimate with an upper bound. |
| CK-25 | Error taxonomy and error analysis (failure mode analysis; confusion analysis) | Check extracted results | Classify reviewed failures into defined types such as invented fact, wrong attribution, omission, wrong scope and arithmetic, and produce error counts by type. |
| CK-26 | Independent human review of model output (expert review; adjudication; human oversight) | Check extracted results | Specify who reviews which outputs, with what rubric and with authority to overrule, and produce a review log with decisions. |
| CK-27 | Adversarial testing of an extraction pipeline (red-teaming; chaos testing) | Check extracted results | Probe an extraction pipeline with hostile or odd inputs such as instructions inserted in source text, contradictory documents and missing data, and produce a list of failures found. |
| CK-28 | Provenance tracking and audit logging (W3C (World Wide Web Consortium) PROV; lineage; record keeping) | Check extracted results | Record which source, prompt, model version, settings, reviewer and decision produced each output, and produce a provenance graph or log that traces any claim back to its source. |
| CK-29 | Scripted reproducible workflow (reproducible analysis; version control; literate programming) | Check extracted results | Record data, code and environment so that another person or a later run can regenerate every figure, and produce a project record that can be rerun from the raw data. |
| PE-14 | De-identification and re-identification risk assessment (data situation audit; direct and indirect identifier removal; disclosure risk assessment) | Prepare and extract | Removes or alters identifying details and assesses the risk that a person can still be identified in the context of release, giving a de-identified dataset and a re-identification risk statement. |
| AO-19 | Ranking uncertainty analysis (stochastic multicriteria acceptability analysis; SMAA; rank acceptability index) | Assess actors, options and risks | Samples weights and score uncertainty many times and reports how often each option takes each rank, giving rank acceptability indices that show how stable a multi-criteria ranking is. |
How the data is organised
Each technique has 52 columns in master/master_tagged.csv: identity and text (ID, family, subgroup, name, what it does, fit test, status and evidence), 44 tag columns, and the result format. Tag columns fall into five groups: input profile, selection, sequencing, robustness and the identity fields purpose and use when.
The schema and the scripts keep their original field names. This table gives the name used in this document for each.
| Name in the schema | Name in this document |
|---|---|
| Gate, gate class | Requirement, requirement type |
| Hard gate | Always applies |
| Conditional | Applies in some cases |
| Weight | Changes the order |
| Advisory | Raises a warning |
| Material profile, facet | Description of the material, property |
| Claim level needed, claim ceiling | Strength of statement needed, strongest statement a technique supports |
| Associate; predict without implying cause; estimate an intervention effect; explain mechanism or counterfactual | Show a link between things; predict, without claiming a cause; estimate the effect of an action; explain how or why |
| Stakes tier | Stakes |
| Insight type | Kind of finding |
| Representation | Structure |
| Access depth | Level of detail available |
| Linkage | Whether sources can be joined |
| Collection basis | How the sample was chosen |
| Volume class, data condition flags | Amount of material, known data problems |
| Artefact; consumes; produces | Working document; needs as input; makes as output |
| Output specification | Result format |
| Answer basis | What the answer is based on |
| Convergence matrix; dissonance; silence | Agreement table; conflict; not covered |
| Phase (frame, gather, verify and prepare, analyse, validate) | Kind of technique (section 2) |
| Pick, held | Chosen technique, reserve list |
| Review | Follow-up round |
The CSV is rebuilt from master_base.csv and the tags_*.json files. The result formats are stored in master/specs/out_*.json and are added after the rebuild.
Working documents. The schema defines 21 types, each tagged with a kind: brief (charter or question), prepared material (data source inventory, raw dataset or corpus, data dictionary, codebook, data quality report, screening log, cleaned dataset, feature set, training sample, held-out test sample, coded corpus, entity table, evidence base, claims register), finding (assumptions register, hypotheses matrix, comparison matrix, result, validation report) and report (final report). The held-out test sample is single use.