startthat.work home

Techniques

A technique is a named way of analysing something, such as five whys, difference-in-differences or lateral reading. The catalogue holds 236 of them. Each has a plain description of what it does and what it produces, a statement of when it fits and when it does not, the conditions your material must meet, and a result format that tells the model what a run of the technique must contain.

The catalogue records enough about each technique for the process to do three things without a person: decide whether the technique can run on your material (step 2), choose the ones that fit your request (step 3), and put them in a workable order with the right checks (steps 4 to 6). The process pages explain those steps.

How the techniques are grouped

The catalogue has 12 families. A family groups techniques by what they do for a request, such as framing a question, comparing cases, or checking results. The family does not decide when a technique runs. That depends on the kind of technique (question, preparation, primary, verification or challenge), which the dictionary defines.

FamilyTechniquesWhat the family is for
Frame the question6Turn a vague topic into a clear question, before any analysis starts.
Decompose and map21Break a topic into parts and show how the parts relate.
Compare17Set cases, periods or groups side by side to see what differs.
Trace cause and test claims34Look for causes, and test whether a claim holds.
Handle evidence29Judge how far to trust sources, and prepare them for use.
Project forward25Estimate what may happen next.
Assess actors, options and risks19Work out who is affected, what the choices are and what could go wrong.
Challenge15Test a conclusion by arguing against it.
Prepare and extract14Get the material ready, so that counts and themes describe what you think they do.
Measure and model16Analyse records with statistics and models.
Integrate across sources11Combine sources that overlap or disagree.
Check extracted results29Check results that came from coding or extraction, including model output.
Total236

How to read a technique page

Each technique page has these parts, in this order.

  • The header gives the identifier, the family, the evidence status, the kind of technique and the step it runs in, and the strongest statement the technique can support. Evidence status says how well the technique itself is supported: tested (a published study measured its effect or accuracy), documented (described in a text or guidance, without a test), practice (used with no published description found) or unconfirmed (no source could be opened). Across the catalogue, 33 are tested, 150 documented, 13 practice and 40 unconfirmed.
  • What it does, when to use it and when it does not fit describe the technique in the words a user would use.
  • Result format says what a run must produce.
  • Requirements on the material list what the material must be like, such as record-level data or more than one source. Step 2 uses these to remove techniques that cannot run.
  • Choosing and ordering and library checks by stakes show how the process places the technique and which checks it needs at each stakes level.
  • Failure modes and limits, related techniques, and evidence and tag basis say where the technique goes wrong, which techniques it pairs with or substitutes for, and how the tags were decided. Most tags are inference unless the page names a source.

The property names in the requirements table link to their definitions in the dictionary.

Find a technique

Filter the table by name, purpose or family, or open a family from the list above. The table lists all 236 techniques.

Using the catalogue without the skill

You can choose techniques yourself, following the same steps as the process. Use by hand explains how, from the brief to the answer.

IDTechniqueFamilyPurpose
FQ-01PICO family (population, intervention, comparison, outcome; variants PICOT, PEO, PICo, SPIDER)Frame the questionSplits a vague topic into named slots (population, intervention or exposure, comparison, outcome) and produces a one-sentence structured question and a slot table.
FQ-02Decision-first framing (decision-focused or value-focused question design)Frame the questionStarts from the decision to be made and works back to a list of questions, each tagged with the option it could change.
FQ-03Definitional analysis (explication, concept clarification, conceptual analysis)Frame the questionReplaces ambiguous key terms with precise working definitions and produces a glossary with borderline cases.
FQ-04Reframing and issue redefinition (changing the level of analysis or baseline; five-step issue redefinition)Frame the questionRestates the question at another level, with another baseline, or by rephrasing, broadening, narrowing and reversing it, and produces answers side by side.
FQ-05Multiple mental models (applying explicit models from several fields to one topic)Frame the questionReads one topic through several explicit models from different fields and produces one reading per model.
FQ-06SMART check (specific, measurable, achievable, relevant, time-bound)Frame the questionChecks that an objective or question names a scope, a measure, a feasible target and a time limit, and produces the draft rewritten with each criterion answered.
DM-01MECE decomposition and issue tree (mutually exclusive, collectively exhaustive; logic tree, problem tree; related: work breakdown structure)Decompose and mapDivides a topic into sub-parts that do not overlap and together cover the whole, repeated at each level, and produces a tree.
DM-02Hypothesis tree (hypothesis-driven decomposition, claim decomposition)Decompose and mapSplits a claim into the sub-claims that must all be true for it to be true and produces a tree of claims that goes into a claims register.
DM-03Morphological analysis (Zwicky box, general morphological analysis) Also serves: Project forward.Decompose and mapLists the parameters of a topic and the possible values of each, removes value pairs judged inconsistent, and produces a grid and a reduced set of consistent configurations.
DM-04Value chain analysis (Porter)Decompose and mapDivides the activities of one business unit into five primary and four support activities and produces a value chain showing where cost and differentiation concentrate.
DM-05Taxonomy building (classification scheme, typology, controlled vocabulary)Decompose and mapBuilds a scheme of classes from observable characteristics, assigns items to the classes and produces a hierarchy and a draft codebook.
DM-06Affinity diagram (KJ method)Decompose and mapGroups many recorded items, such as survey comments or interview quotes, into clusters that form from the relationships among the items, and produces labelled clusters.
DM-07Lens application (sorting claims by a classification lens, for example input versus outcome)Decompose and mapSorts a set of claims or measures by chosen categories, such as input or outcome, and produces a table of claims by category.
DM-08Two by two matrix (quadrant analysis) Also serves: Assess actors, options and risks; Project forward.Decompose and mapPlaces items on two chosen dimensions so that each falls into one of four cells and produces a quadrant table.
DM-09PESTLE (political, economic, social, technological, legal, environmental; the shorter PEST omits legal and environmental; outside-in thinking)Decompose and mapSorts external factors into political, economic, social, technological, legal and environmental categories and produces a categorised list.
DM-10SWOT (strengths, weaknesses, opportunities, threats)Decompose and mapSorts factors into internal strengths and weaknesses and external opportunities and threats and produces a two by two matrix.
DM-11Porter's Five Forces (industry structure analysis)Decompose and mapRates rivalry, new entrants, substitutes, buyer power and supplier power and produces a five-force diagram with each force rated.
DM-12Cynefin frameworkDecompose and mapClassifies a situation, or each part of it, as clear, complicated, complex, chaotic or confusion by the relation of cause and effect and produces a domain assignment.
DM-13System archetypes (limits to growth, shifting the burden, fixes that fail, and others)Decompose and mapMatches an observed behaviour pattern to a recurring feedback structure and produces a named pattern with the loop structure to change.
DM-14Leverage points (Meadows' twelve places to intervene in a system)Decompose and mapRanks types of intervention in a system from least to most effective and produces an ordered list showing the level at which a proposed intervention acts.
DM-15Wardley mappingDecompose and mapPositions the components of a service by visibility to the user and by evolution stage, with dependency lines, and produces a two-axis map.
DM-16Levels-of-analysis models (Marr's three levels, micro, meso and macro levels)Decompose and mapExamines a topic at several distinct levels, each answering a different kind of question, and produces findings organised by level.
DM-17Process mapping (flowcharts, swimlane diagrams, value stream maps)Decompose and mapRecords the sequence of steps, inputs, outputs and responsible roles in a process and produces a flow diagram showing handoffs, delays and rework loops.
DM-18Design structure matrix (dependency structure matrix)Decompose and mapRecords which element depends on or supplies which in a square matrix, reorders it to reveal clusters or sequence, and produces a reordered matrix.
DM-19Causal loop diagram Also serves: Trace cause and test claims.Decompose and mapDraws variables and signed causal links so that reinforcing and balancing loops and delays can be identified, and produces a diagram with loop labels.
DM-20Stock and flow modelling (system dynamics, including simulation) Also serves: Project forward.Decompose and mapRepresents accumulations that change only through inflows and outflows and, when simulated, produces a diagram and simulated time paths.
DM-21Social network analysis (sociogram, network analysis) Also serves: Assess actors, options and risks.Decompose and mapRepresents actors as nodes and relationships as ties, measures structure with graph metrics, and produces a sociogram, an entity table and metrics.
CM1-01Historical analogy and parallel analysis (likenesses and differences, analogical explanation)ComparePlaces the present topic beside a past case and rates each likeness and difference, producing a likenesses and differences table that shows where a lesson from the past case may and may not be applied.
CM1-02Mill's method of agreement and most different systems design (MDSD)CompareLooks across dissimilar cases that share an outcome for the factor present in all of them, producing a table of candidate factors by case in which the common factor is the candidate cause.
CM1-03Mill's method of difference and most similar systems design (MSSD; joint method of agreement and difference)CompareLooks for the factor present where an outcome occurs and absent in an otherwise similar case where it does not, producing a short list of the remaining differences between matched cases.
CM1-04Concomitant variation and method of residues (Mill)CompareInfers a link when a factor varies with the effect (concomitant variation) or attributes the unexplained part of an effect to the remaining circumstances (residues), producing a co-movement pattern or a residual.
CM1-05Comparative case study (structured focused comparison, small-N comparison)CompareExamines a small number of cases with the same questions for each, producing a case by case matrix that shows consistent patterns and deviant cases.
CM1-06Case selection techniques (extreme, deviant, diverse, index, most similar)CompareChooses which cases to compare by their position on a variable distribution or on a model prediction, producing a chosen case set with the rule used.
CM1-07Qualitative comparative analysis (QCA; crisp-set, fuzzy-set and multi-value variants)CompareUses set theory and Boolean algebra to find combinations of conditions that are necessary or sufficient for an outcome across a medium number of cases, producing a truth table and minimal solution formulas with consistency and coverage scores.
CM1-08Cross-case synthesis (replication logic)CompareCombines completed within-case write-ups into one cross-case matrix, producing a matrix that shows repeating patterns and diverging cases.
CM1-09Contrast with a structural opposite (term from earlier work)CompareSets the topic beside a case or position that differs from it on a stated defining structure, producing a table of contrasting features that shows what each assumes or lacks.
CM1-10Cross-sectional and longitudinal comparison (cohort and panel comparison; cross-cycle comparison)CompareCompares different units at one time (cross-sectional) or the same units repeatedly (longitudinal), producing group differences at one time or change paths over time.
CM1-11Before and after comparison (pre-post design)CompareCompares an outcome for the same unit before and after an event, producing a change estimate.
CM1-12Benchmarking and peer comparison (best-practice comparison, competitor analysis)CompareCompares an organisation processes and metrics with those of peers or leaders, producing a gap table of own against benchmark performance.
CM1-13Forecaster and source comparison (forecast scoring, aggregation, track record comparison) Also serves: Project forward.CompareScores a forecast against those of other forecasters on resolved outcomes and aggregates them, producing an accuracy ranking and an aggregate forecast.
CM1-14Absence analysis (argument from silence, the dog that did not bark) Also serves: Handle evidence.CompareDraws a conclusion from an item missing from a document or comparison set where it would be expected if a claim were true, producing a list of expected but absent items.
CM1-15Gap analysisCompareCompares current with desired performance, producing a prioritised list of gaps and the means to close each.
CM1-16Maturity model and capability assessment (Capability Maturity Model Integration, CMMI)CompareRates processes against staged levels, producing a maturity level or capability profile that shows where an organisation stands and what to improve.
CM-01Sponsor versus non-sponsor comparison (meta-epidemiological sponsorship analysis)CompareCompares the share of favourable results and conclusions between sponsored and independently funded reports on one topic, producing a risk ratio or odds ratio.
TC1-01Five whysTrace cause and test claimsAsks why repeatedly about one stated problem and records a single chain of causes, leaving a candidate root cause as a result.
TC1-02Fishbone diagram (Ishikawa diagram, cause and effect diagram)Trace cause and test claimsGroups brainstormed causes of one problem under category branches and produces a fishbone diagram with the problem as its head.
TC1-03Fault tree analysis Also serves: Assess actors, options and risks.Trace cause and test claimsWorks downward from one undesired top event through AND and OR gates to produce a gate tree with minimal cut sets.
TC1-04Process tracing (diagnostic tests: straw in the wind, hoop, smoking gun, doubly decisive)Trace cause and test claimsTests what each rival hypothesis predicts against diagnostic evidence within one case and produces a test result per hypothesis in a hypotheses matrix.
TC1-05Counterfactual analysis (potential outcomes, minimal rewrite of history) Also serves: Compare.Trace cause and test claimsSets the actual outcome against an argued account of what would have happened had one factor differed and produces a counterfactual scenario.
TC1-06Difference-in-differences Also serves: Compare.Trace cause and test claimsCompares the change over time in a treated group with the change in a control group and produces an interaction coefficient read as the treatment effect.
TC1-07Quasi-experimental designs (natural experiment, instrumental variables, regression discontinuity)Trace cause and test claimsEstimates an effect from observational data in which exposure was assigned as if at random (natural experiment, instrument or cutoff) and produces an effect estimate under stated assumptions.
TC1-08Confounder identification with directed acyclic graphs (adjustment sets, backdoor criterion)Trace cause and test claimsDraws assumed causal links as a directed acyclic graph and produces a diagram and an adjustment set of variables to include and to leave out.
TC1-09Logic model and theory of changeTrace cause and test claimsMaps the expected causal chain from inputs to outcomes with the assumption behind each step and produces a logic model or theory of change.
TC1-10Contribution analysisTrace cause and test claimsBuilds a reasoned line of evidence that a program contributed to an outcome and produces an evidenced contribution story.
TC1-11Realist evaluation (context, mechanism, outcome configurations)Trace cause and test claimsTests what works for whom and in what circumstances through context, mechanism and outcome configurations and produces a refined program theory.
TC1-12Process evaluation and impact evaluationTrace cause and test claimsExamines how an intervention was delivered and estimates what difference it made, and produces findings on delivery quality or an attributed effect estimate.
TC1-13Analysis of competing hypotheses (ACH)Trace cause and test claimsLists plausible hypotheses, rates each evidence item against each, and produces a hypothesis by evidence matrix aimed at disproving hypotheses.
TC1-14Coherence correction and averaging of probability judgementsTrace cause and test claimsAdjusts each analyst's probabilities on mutually exclusive hypotheses so they sum to one, averages across analysts, and produces a corrected aggregated judgement.
TC1-15Bayesian updating (likelihood reasoning, likelihood ratio, Bayes factor)Trace cause and test claimsRevises the probability of a hypothesis in proportion to how much more likely the evidence is under it than under alternatives and produces a posterior probability or odds.
TC1-16Disconfirmation search (falsification, seeking disconfirming evidence)Trace cause and test claimsLooks for evidence that would contradict a favoured claim and produces a list of refuting evidence found or not found with the search used.
TC1-17Sensitivity analysis (robustness check, leave-one-out)Trace cause and test claimsRe-runs an analysis under different assumptions or inclusion rules and produces a table of results by assumption showing whether the conclusion reverses.
TC1-18Internal consistency check (term from earlier work)Trace cause and test claimsTests whether the claims, figures and assumptions within one analysis are compatible with each other and produces a list of contradictions or inconsistent figures.
TC1-19Toulmin argument analysisTrace cause and test claimsLabels the parts of an argument as claim, ground, warrant, backing, rebuttal and qualifier and produces a diagram per argument.
TC1-20Argument mapping (argument diagramming)Trace cause and test claimsDraws claims, reasons and objections as a diagram and produces an argument map that shows isolated claims and reasons not connected to any conclusion.
TC1-21Fallacy analysis (informal fallacies, pragma-dialectical rules)Trace cause and test claimsCompares an argument with a list of known defective patterns and produces a list of suspected defects, each read as a prompt for missing support.
TC1-22Abduction and inference to the best explanationTrace cause and test claimsRanks candidate explanations of an observation on simplicity, generality and coherence and produces a ranked list of explanations.
TC1-23Ladder of inferenceTrace cause and test claimsTraces how a conclusion was reached from observed data through selected data and interpretation to belief and produces a stepwise list.
TC1-24Reflective equilibriumTrace cause and test claimsAdjusts principles and case judgements against each other until they cohere and produces a revised set of principles and judgements.
TC1-25First principles reasoningTrace cause and test claimsReplaces inherited convention with a derivation from basic premises and produces a derivation chain with the base premises listed as assumptions.
TC1-26Thought experimentsTrace cause and test claimsConstructs an imagined case to test a principle and produces a judgement about the case that counts as evidence on the principle only if reasoners agree.
TC1-27Fermi estimation and order-of-magnitude checks (back-of-the-envelope estimate)Trace cause and test claimsBreaks an unknown quantity into factors that can be estimated, multiplies them, and produces a number with a plausible range used as a bound on a detailed result.
TC1-28Dimensional and unit checksTrace cause and test claimsChecks that both sides of a relation have the same units and produces a unit balance in which a mismatch flags an error in the derivation.
TC-01GRIM test (granularity-related inconsistency of means)Trace cause and test claimsChecks whether a reported mean of whole-number data can arise from the reported sample size and item count and produces a flag for possible or impossible means.
TC-02SPRITE (Sample Parameter Reconstruction via Iterative TEchniques)Trace cause and test claimsReconstructs possible datasets that match reported descriptive statistics and produces plausible samples showing the range of possible item distributions.
TC-03statcheck (recomputation of reported p-values)Trace cause and test claimsRecomputes p-values from reported test statistics and degrees of freedom and produces a list of inconsistencies and gross inconsistencies.
TC-04Claim and citation verification against source text (fact verification; claim checking; citation support check; quotation accuracy check)Trace cause and test claimsLabels each claim as supported, refuted or not enough information against retrieved evidence, or rates whether a cited source supports its claim, and produces a label with evidence sentences.
TC-05Subgroup and Simpson's paradox check (stratified reanalysis; confounding check; collapsibility check)Trace cause and test claimsRecomputes an association within subgroups to see whether the pooled direction reverses or vanishes and produces pooled and stratified estimates side by side.
TC-06Ecological fallacy check (aggregation bias check)Trace cause and test claimsDecides whether a claim about people is supported by group-level data and produces a decision on whether the claim is group-level or individual-level.
HE1-01Admiralty system (source reliability and information credibility grading)Handle evidenceAssigns each report a two-character grade for source reliability (A to F) and information credibility (1 to 6) and produces an appraised evidence base.
HE1-02Quality of information checkHandle evidenceRates the completeness, accuracy and corroboration of each source and produces a source database with a statement of what is known and unknown.
HE1-03CRAAP test (currency, relevance, authority, accuracy, purpose)Handle evidenceScreens one source against five criteria (currency, relevance, authority, accuracy, purpose) and produces a checklist judgement for the evidence base.
HE1-04SIFT (stop, investigate the source, find better coverage, trace claims to the original; claim verification and fact-checking)Handle evidenceMoves from a claim to its publisher, to other coverage and to the original source, and produces a short credibility judgement with the original context located.
HE1-05Lateral readingHandle evidenceLeaves the page to read about the publisher in other sources and produces a judgement of publisher credibility.
HE1-06Triangulation (source, method, investigator and theory triangulation; corroboration)Handle evidenceCompares independent lines of evidence on one question and produces a convergence table of agreement, disagreement and reasons.
HE1-07Evidence hierarchies and certainty grading (study design levels, risk-of-bias assessment, Grading of Recommendations Assessment, Development and Evaluation or GRADE)Handle evidenceRanks study designs and rates confidence in a body of evidence from high to very low, and produces a certainty rating with reasons for downgrading.
HE1-08Systematic reviewHandle evidenceCollects all eligible studies by a pre-specified, reproducible search, appraises and synthesises them, and produces a screening log, an evidence table and a synthesis.
HE1-09Rapid reviewHandle evidenceRuns a systematic review with streamlined steps to meet a deadline and produces a narrative synthesis that states the shortcuts taken.
HE1-10Scoping review (evidence mapping)Handle evidenceMaps the extent, types and gaps of evidence on a broad topic without judging the strength of findings, and produces a charted table.
HE1-11Meta-analysisHandle evidenceCombines effect estimates from separate studies into a weighted average with a confidence interval and produces a forest plot, a pooled effect and heterogeneity statistics.
HE1-12Narrative, thematic and qualitative evidence synthesis (including meta-ethnography)Handle evidenceCombines findings across studies in words or themes when pooling numbers is not appropriate, and produces a structured narrative or theme set linked back to studies.
HE1-13Reflexive thematic analysis (Braun and Clarke)Handle evidenceIdentifies and interprets patterns of meaning across qualitative data in six phases and produces themes with supporting extracts.
HE1-14Framework analysis (Framework Method)Handle evidenceSorts coded qualitative data into a case by theme matrix and produces the matrix for reading across cases and themes.
HE1-15Content analysis (conventional, directed and summative)Handle evidenceCodes text into categories, then counts or interprets category use, and produces category frequencies or a category structure.
HE1-16Discourse analysis (critical discourse analysis)Handle evidenceExamines how language constructs topics and positions actors and produces an interpretive account of framing checked against quoted passages.
HE1-17Grounded theory coding (open, axial and selective coding; constant comparison)Handle evidenceBuilds categories and a theory from data through open, axial and selective coding with constant comparison and produces codes, memos and a theory.
HE1-18Document and policy analysisHandle evidenceReads documents systematically to extract facts, positions, changes or stated reasons using their provenance and purpose, and produces an extraction table or chronology.
HE1-19Bibliometric analysis (citation analysis, science mapping, co-citation, co-word analysis)Handle evidenceCounts and maps publications, authors and citations to show the structure and growth of a field and produces performance tables and network maps.
HE1-20Trend and time-series decomposition Also serves: Project forward.Handle evidenceSeparates a series into trend, seasonal and irregular parts and produces plots and a fitted trend.
HE1-21Distribution, outlier and base-rate checksHandle evidenceExamines the shape, extremes and baseline frequency of data before a figure is interpreted and produces histograms, summary statistics, flagged outliers and a base rate.
HE-01Grey literature appraisal checklist (AACODS (Authority, Accuracy, Coverage, Objectivity, Date, Significance))Handle evidenceScores a non-peer-reviewed item on six criteria (authority, accuracy, coverage, objectivity, date, significance) and produces a completed checklist.
HE-02Critical appraisal checklist by study design (CASP (Critical Appraisal Skills Programme) checklists; JBI (Joanna Briggs Institute) checklists)Handle evidenceAnswers design-specific questions on validity, results and relevance for one study and produces a completed checklist for the evidence base.
HE-03Appraisal of a systematic review (AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews 2); ROBIS (Risk Of Bias In Systematic reviews))Handle evidenceRates confidence in a systematic review with AMSTAR 2 (high to critically low) or concern in four domains with ROBIS, and produces a rating or domain judgements.
HE-04Survey disclosure completeness check (AAPOR (American Association for Public Opinion Research) disclosure elements)Handle evidenceTests whether a survey report states the sponsor, question wording, population, sampling frame, mode, dates, sample size, margin of error and weighting, and produces a list of disclosed and missing items.
HE-05Margin of error and credibility interval checkHandle evidenceChecks that a reported margin of error matches the sampling method and produces a judgement on whether the figure is valid.
HE-06Opt-in panel respondent quality check (bogus respondent screen)Handle evidenceLooks for implausible answers that show careless or fraudulent respondents inflating rare responses in an opt-in survey and produces a judgement of inflation risk.
HE-07Overview of reviews (umbrella review; tertiary review)Handle evidenceUses systematic reviews as the unit of search, inclusion and analysis and produces a summary of reviews with a quality rating per review and an overlap map.
HE-08Living systematic reviewHandle evidenceKeeps a systematic review continuously updated as new studies appear and produces a versioned evidence summary and an update log.
PJ-01Trend extrapolation and time-series forecasting (exponential smoothing, autoregressive integrated moving average models, trend impact analysis)Project forwardExtend the measured history of one indicator with a stated functional form to give a central projection with prediction intervals, as a draft result.
PJ-02Reference class forecasting (outside view, base-rate forecasting) Also serves: Compare.Project forwardReplace a bottom-up estimate with the distribution of outcomes in similar completed cases, giving a base-rate distribution and the case position in it as a draft result.
PJ-03Forecasting tournaments and superforecasting practiceProject forwardProduce dated probability estimates for resolvable questions, score them with Brier scores and improve them by updating and aggregation, leaving a scored probability record as a draft result.
PJ-04Delphi technique (real-time Delphi, mini-Delphi)Project forwardCollect anonymous expert estimates over two or more rounds with controlled feedback, giving the median and interquartile range for each round as a draft result.
PJ-05Prediction markets (information markets)Project forwardRead the price of a defined contract as a probability for an event, giving a price series and a liquidity-qualified probability as a draft result.
PJ-06Structured expert elicitation (IDEA protocol: investigate, discuss, estimate, aggregate)Project forwardElicit numeric estimates and ranges from experts before and after discussion and aggregate them mathematically, giving aggregated estimates with intervals as a draft result.
PJ-07Technology S-curve analysis (logistic growth, technology substitution curve)Project forwardFit a logistic curve to an adoption or performance series to estimate a ceiling and an inflection date, as a draft result.
PJ-08Bass diffusion modelProject forwardForecast the adoption path of a product or practice from external-influence and imitation parameters and a market potential, giving a cumulative adoption curve and peak timing as a draft result.
PJ-09Hype cycle (technology expectations curve)Project forwardPlace a technology on a five-phase expectations curve with a years-to-mainstream band, giving a placement to compare with measured attention data as a draft result.
PJ-10Trajectory analysis (technological trajectories, envelope curves; term not standard)Project forwardTrack a performance metric across product generations and extend the envelope of best performance, giving a trajectory chart and rate of improvement as a draft result.
PJ-11Agent-based modelling and Monte Carlo simulation (microsimulation)Project forwardSimulate agents or random draws many times to give an outcome distribution with tail probabilities and the inputs that account for the spread, as a draft result.
PJ-12Scenario planning (intuitive logics, Shell approach)Project forwardBuild a small set of internally consistent plausible futures from critical uncertainties so that a strategy can be examined in each, giving named narrative scenarios as a draft result.
PJ-13Scenario matrix (two by two scenario axes, alternative futures analysis)Project forwardCross the two most critical and uncertain forces to give four scenario quadrants with narratives and signposts, as a draft result.
PJ-14Alternative futures archetypes (Manoa school, four generic futures)Project forwardApply a fixed set of generic futures to a topic to give a short image of the topic under each archetype, as a draft result.
PJ-15Futures wheel (implications wheel)Project forwardMap first, second and third-order consequences of one change in rings, giving a radial diagram of consequences as a draft result.
PJ-16Causal layered analysisProject forwardExamine a topic at four depths (litany, social and structural causes, worldview and myth) for the present and each alternative future, giving a four-row table as a draft result.
PJ-17Cross-impact analysis (cross-impact matrix)Project forwardAdjust the probability of each event in a set according to how every other event raises or lowers it, giving adjusted event probabilities and an influence matrix as a draft result.
PJ-18BackcastingProject forwardFix a desired end state and work backwards to the steps and conditions that connect it to the present, giving a dated pathway as a draft result.
PJ-19Policy stress-testingProject forwardRun a policy or plan through defined adverse or alternative conditions, giving a table of performance by condition and the cheapest changes that remove failures as a draft comparison matrix.
PJ-20Wargaming and simulation exercises (policy gaming, tabletop exercise)Project forwardHave players take the roles of competing actors in a structured scenario, giving a record of decisions and surprises as a draft result.
PJ-21Robust decision making, adaptive pathways and info-gap analysisProject forwardTest many model runs or futures to find the conditions under which a strategy fails and design strategies that adapt, giving vulnerability scenarios and pathway maps with trigger points as a draft result.
PJ-22Technology roadmappingProject forwardLay out how markets, products, technologies and resources are expected to evolve on a common time axis, giving a layered time chart with dependencies as a draft result.
PJ-23Horizon scanning (environmental scanning)Project forwardSearch systematically for early signs of change in sources outside the topic usual field, giving a signal database with source, date and rating as a collected evidence base.
PJ-24Weak signal detection (early warning signs, wild cards)Project forwardIdentify small ambiguous early indications that may precede a discontinuity, giving a watchlist with a strength rating over time as an appraised evidence base.
PJ-25Leading indicators and signposts (indicators of change, trigger points)Project forwardSelect measurable quantities that change before an outcome of interest and set thresholds beforehand, giving a table of indicators with lead time, threshold, source, frequency and owner as a draft result.
AO-01Stakeholder analysis (stakeholder mapping)Assess actors, options and risksIdentifies who is affected by or can affect a topic and classifies each by interest, influence and attitude, producing a stakeholder register and grid.
AO-02Power-interest grid (influence-interest grid, power-impact grid)Assess actors, options and risksPlots actors on power and interest to choose an engagement strategy for each, producing a two by two grid.
AO-03Stakeholder salience model (Mitchell, Agle and Wood, 1997)Assess actors, options and risksRanks actors by legitimacy, power and urgency, producing a diagram with eight classes from definitive to non-stakeholder.
AO-04Political economy analysisAssess actors, options and risksExplains why a problem persists by analysing structures, institutions, actors, interests, incentives and power relations, producing a synthesis of key propositions, pathways of change, and maps of blockers and champions.
AO-05Customer journey mappingAssess actors, options and risksTurns a persona goals, actions, thoughts and emotions across stages into a visual map, producing a map with phases, touchpoints, pain points and opportunities.
AO-06PersonasAssess actors, options and risksCondenses research on user groups into a fictional but realistic description, producing a persona profile with goals, context and concerns.
AO-07Jobs to be done (JTBD)Assess actors, options and risksFrames demand around the progress a customer is trying to make in a circumstance, producing a job statement that shows which jobs are poorly served.
AO-08Eightfold path (Bardach)Assess actors, options and risksStructures a policy question into eight steps from defining the problem to telling the story, producing a written options analysis with criteria, projected outcomes and trade-offs.
AO-09Longlist-to-shortlist options analysis (options appraisal)Assess actors, options and risksGenerates a wide set of options, filters them against objectives and critical success factors, and appraises a shortlist of about five, producing a longlist and shortlist with red, amber and green ratings.
AO-10Cost-benefit analysisAssess actors, options and risksValues all costs and benefits to society in money, discounts them and compares options by net present value, producing net present value per option with sensitivity results.
AO-11Cost-effectiveness and cost-utility analysisAssess actors, options and risksCompares options on cost per unit of one quantified effect when benefits cannot be monetised, producing cost per unit of effect for each option.
AO-12Multi-criteria decision analysis (MCDA)Assess actors, options and risksScores options against several weighted criteria, producing weights, scores and a ranking, with a test of how the ranking changes when weights change.
AO-13Decision tree and expected valueAssess actors, options and risksLays out decisions, chance events and outcomes and computes expected value or utility, producing a decision tree that shows the alternative with the best expected value.
AO-14Real options analysisAssess actors, options and risksValues the right but not the obligation to expand, delay, switch or abandon an investment, producing an option value added to a standard valuation.
AO-15Risk matrix (likelihood and consequence matrix, risk heat map)Assess actors, options and risksRates each risk by plotting likelihood against consequence, producing a grid of ratings that shows which risks need treatment first.
AO-16Failure mode and effects analysis (FMEA)Assess actors, options and risksLists the failure modes of each function or component with causes and effects and ranks them, producing a worksheet with ranked failure modes that shows single points of failure.
AO-17Bow-tie analysisAssess actors, options and risksMaps causes, a central event, consequences and the barriers on each path, producing a cause, event and consequence diagram that shows pathways lacking a barrier.
AO-18Threat modelling (STRIDE: spoofing, tampering, repudiation, information disclosure, denial of service, elevation of privilege)Assess actors, options and risksIdentifies security threats using six categories mapped to security properties, producing a list of threats by category.
CH-01Key assumptions check (assumptions check)ChallengeLists the stated and unstated premises of the current line, challenges each, and produces a refined assumptions register with the conditions that would invalidate each assumption.
CH-02Strategic assumption surfacing and testing (SAST)ChallengeLists each interested group assumptions, rates them, has opposing groups debate them in four stages, and produces a rated assumptions register.
CH-03Devil's advocacyChallengeBuilds the strongest case against the dominant view or plan, starting from its most vulnerable assumptions, and produces a labelled devil advocate finding.
CH-04Team A and Team BChallengeHas two teams each build the best case for a competing view and debate before a jury of peers, and produces competing papers and debate findings.
CH-05Red team analysis (red team, red cell)ChallengeModels how an adversary or other actor would decide by reasoning in that actor first-person voice, or examines a plan from outside the planning team, and produces first-person memos predicting decisions.
CH-06High-impact and low-probability analysis Also serves: Project forward.ChallengeWorks out plausible pathways to an unlikely but consequential event, with triggers and indicators, and produces pathways and an indicator list.
CH-07What if analysis Also serves: Project forward.ChallengeAssumes an unexpected event has occurred, explains how it could have come about, and produces scenarios with indicators and consequence assessments.
CH-08Premortem (prospective hindsight, inversion) Also serves: Project forward.ChallengeAsks a team to assume the plan has already failed and to write the reasons, and produces a grouped list of failure reasons and draft assumptions.
CH-09Structured self-critiqueChallengeAsks about sources of uncertainty, evidence, gaps, critical assumptions and alternative hypotheses, and produces a written critique of one own analysis.
CH-10Rapoport's rules (fair restatement of an opposing position) Also serves: Compare.ChallengeRequires a fair restatement of the opposing position, a list of points of agreement and of what was learned before any rebuttal, and produces a restatement and an agreement list.
CH-11Steelmanning Also serves: Compare.ChallengeBuilds the strongest version of a position the analyst disagrees with, responds to it, and produces a strengthened opposing argument and a response.
CH-12Socratic questioning (Paul and Elder's six question types)ChallengeAsks a sequence of questions that expose unclear terms, hidden assumptions and unsupported claims, and produces a record of questions and answers and a draft assumptions register.
CH-13Cognitive bias checks and debiasing (including consider-the-opposite)ChallengeRuns a conclusion through a list of known judgement biases, applies a counter-procedure, and produces a checklist of marked items where disconfirming evidence should be sought.
CH-14Structured brainstormingChallengeGenerates and clusters many ideas about a focal question before any critique and produces clustered, prioritised ideas.
CH-15Group process safeguards (everyone speaks once before anyone speaks twice, mindguarding)ChallengeControls how a group forms views, with each person contributing in turn, leaders withholding early preferences and evaluators rotating, and produces independent individual views.
PE-01Corpus construction and sampling (document selection; search string design; sampling frame design)Prepare and extractDefine which documents enter the analysis so that counts and themes describe a stated population, giving a corpus table with counts and an inclusion log.
PE-02Preprocessing sensitivity testing (text cleaning; tokenisation; stopword removal; stemming; boilerplate removal; preText)Prepare and extractConvert raw text into countable units and re-run the main result under several defensible cleaning regimes, giving cleaned text, a document-term matrix, a rule log and a sensitivity result.
PE-03Near-duplicate detection (shingling; MinHash; locality-sensitive hashing; text reuse detection)Prepare and extractFind identical or nearly identical documents so repeats are not counted as independent evidence, giving duplicate clusters with similarity scores and a deduplicated corpus.
PE-04Data profiling and rule-based validation (data audit; data quality assessment; edit checks)Prepare and extractDescribe the structure, completeness, validity and consistency of each field and test explicit rules, giving a profile table per column and a list of records that break each rule.
PE-05Missing data diagnosis and treatment (missingness analysis; imputation; complete case analysis; non-response weighting)Prepare and extractMeasure how much is missing, where and for whom, then choose deletion, imputation or weighting, giving the missing share by field and subgroup and a cleaned dataset.
PE-06Keyword in context (KWIC; concordance)Prepare and extractList every occurrence of a keyword or phrase with a fixed number of surrounding words, giving a table of document, position and context.
PE-07Collocation and n-gram extraction (phrase detection; association measures)Prepare and extractFind word sequences that occur together more often than chance, giving ranked phrases with association scores as a feature set.
PE-08Named entity recognition (NER; entity extraction)Prepare and extractTag spans of text as people, organisations, places, dates and other types, giving a table of mentions with type and position and a tagged corpus.
PE-09Relation and event extraction (information extraction; open information extraction; temporal expression extraction)Prepare and extractExtract who did what to whom, with arguments and time, as structured rows, giving an event or relation table and a coded corpus.
PE-10Schema-based structured extraction with an LLM (LLM (large language model) information extraction; form filling from documents)Prepare and extractFill a fixed set of fields, such as amount, date, party, stated risk or decision, for each document, giving one record per document and the share of missing or not stated values.
PE-11Knowledge graph construction from text (entity-relation graph; ontology population)Prepare and extractConvert resolved entities and extracted relations into a graph of typed nodes and edges, each linked to its source sentence, giving a graph with provenance as a draft result.
PE-12Model summarisation of documents (abstractive summarisation; multi-document summarisation)Prepare and extractWrite a condensed text that paraphrases the content of one or many documents, giving a summary as a draft result.
PE-13Retrieval-augmented analysis (RAG (retrieval-augmented generation); grounded question answering over a corpus)Prepare and extractRetrieve passages relevant to a question and have a model answer from them with citations, giving an answer with quoted passages and a claims register.
MM-01Exploratory data analysis with plots and distribution checks (EDA (exploratory data analysis); summary statistics; Q-Q plot; scatterplot matrix)Measure and modelExamine a cleaned table with plots and summary statistics to find shape, outliers, trends, subgroup differences and broken assumptions, and produce a draft result of plots and summaries.
MM-02Algorithmic outlier and anomaly detection (Isolation Forest; Local Outlier Factor; Elliptic Envelope; One-Class SVM (support vector machine))Measure and modelScore each record by how far it differs from the rest, using distance, density, isolation or a covariance estimate that resists outliers, and produce an outlier score and label per record.
MM-03Clustering of records (k-means; hierarchical (agglomerative) clustering; DBSCAN (density-based spatial clustering of applications with noise); HDBSCAN (hierarchical DBSCAN))Measure and modelGroup records by distance or density without predefined categories and produce a cluster label per record, with centroids, a dendrogram or a noise label.
MM-04Dimensionality reduction (PCA (principal component analysis); factor analysis; non-negative matrix factorisation)Measure and modelReplace many correlated numeric fields with a few components and produce loadings, explained variance and a score per record.
MM-05Survival and duration analysis (Kaplan-Meier estimator; Nelson-Aalen estimator; Cox proportional hazards; time-to-event analysis)Measure and modelEstimate the time until an event while using records whose event has not yet been seen, and produce survival curves with intervals and a median duration.
MM-06Survey weighting and subgroup estimation (design weights; calibration; post-stratification; relative standard error)Measure and modelApply survey weights so that sample results represent the population and produce weighted estimates with standard errors by subgroup.
MM-07Supervised classification and dictionary methods (text classification; fine-tuned classifier; lexicon method)Measure and modelAssign documents to predefined categories using human-labelled examples or word lists and produce coded documents with confidence and a confusion matrix.
MM-08LLM coding and zero-shot classification of text (LLM annotation; LLM-assisted deductive coding; few-shot classification; natural language inference classification)Measure and modelApply a human-written codebook or category descriptions to text with a large language model and produce codes, rationales and per-code agreement with human coders.
MM-09Sentiment analysis (polarity detection; opinion mining; affect lexicon)Measure and modelScore text as positive, negative or neutral, or by emotion, and produce scores per document with aggregates by group or period.
MM-10Stance detection (position detection)Measure and modelLabel whether each text favours, opposes or is neutral toward a named target and produce a stance label per text and target.
MM-11Framing analysis at scale (frame detection; issue framing; media frames coding)Measure and modelIdentify which aspects of an issue each text emphasises using a defined frame scheme and produce frame labels with frame shares by source and period.
MM-12Probabilistic topic modelling (LDA (latent Dirichlet allocation); STM (structural topic model); mixed-membership model)Measure and modelModel each document as a mixture of topics and each topic as a distribution over words, and produce topic word lists and document-topic proportions.
MM-13Embedding-based topic modelling (BERTopic; Top2Vec; clustering-based topic models)Measure and modelEmbed documents with a neural model, cluster the embeddings and describe each cluster by distinctive words, and produce topics with representative words, representative documents and an outlier group.
MM-14Document clustering (computer-assisted clustering; k-means on document vectors)Measure and modelGroup similar documents without predefined categories using counts or embeddings and produce cluster assignments, centroids and exemplar documents.
MM-15Computer-assisted theme building (reading-assisted thematic analysis; human-guided topic labelling)Measure and modelHave a person read exemplars from computer-generated groups, then merge, split and name them, and produce a named theme set with exemplar quotations and a count per theme.
MM-16Word embeddings for concept measurement and semantic change (diachronic embeddings; concept direction)Measure and modelMeasure how terms relate in meaning within a corpus or across periods and produce nearest neighbours, similarity to seed words and change scores.
IS-01Entity resolution and linking (record linkage; entity matching; entity linking; coreference resolution; blocking)Integrate across sourcesDecides which records or text mentions refer to the same entity by comparing attributes, with no shared identifier required, and produces match decisions or linked clusters.
IS-02Schema matching and ontology matching (schema mapping; field mapping)Integrate across sourcesEstablishes which fields or concepts in different sources mean the same thing and produces a mapping table of equivalent, partly equivalent and unmapped fields.
IS-03Common data model harmonisation (OMOP (Observational Medical Outcomes Partnership) common data model; data harmonisation)Integrate across sourcesTransforms source datasets into one shared structure and vocabulary so that the same analysis runs on each and produces a standardised dataset with a coverage report.
IS-04Data fusion with conflict resolution (majority voting; deciding and mediating resolution functions)Integrate across sourcesCombines records on the same entity into one record, resolving conflicting values by a rule, and produces a fused record per entity with uncertain and contradictory values marked.
IS-05Truth discovery with source reliability weighting (truth finding; source reliability estimation)Integrate across sourcesJointly estimates the true value of each object and the reliability of each source from conflicting claims and produces a truth per object and a weight per source.
IS-06Source dependence and copy detection (source independence analysis)Integrate across sourcesEstimates whether sources copy from each other so that their agreement is not counted as independent support and produces a dependence estimate for source pairs or groups.
IS-07Convergent comparison of qualitative and quantitative results (convergent parallel design; concurrent triangulation)Integrate across sourcesAnalyses qualitative and quantitative results on the same topic separately, then merges them to compare and produces a comparison of agreement, partial agreement and discordance by topic.
IS-08Joint displayIntegrate across sourcesPlaces quantitative and qualitative results in one table, figure or matrix with shared row labels and produces a matrix that shows where the strands agree, expand each other or conflict.
IS-09Quantitising and qualitising (data transformation)Integrate across sourcesConverts one data type into the other, for example by counting coded themes, and produces a single-type dataset of frequencies or profiles.
IS-10Explanatory sequential integration (explanatory sequential design)Integrate across sourcesUses a completed quantitative result, especially an unexpected or extreme one, to choose the qualitative material read to explain it and produces an explanation linked to the numeric pattern.
IS-11Exploratory sequential integration (exploratory sequential design)Integrate across sourcesUses qualitative results to decide what is measured in a later quantitative stage and produces measures or hypotheses derived from themes.
CK-01Inter-coder agreement statistics (percentage agreement; Cohen's kappa; weighted kappa; Scott's pi; Fleiss' kappa; Krippendorff's alpha; intercoder reliability testing)Check extracted resultsMeasure how far coders, human or model, give the same code to the same items, corrected for chance, and produce an agreement coefficient per code with a bootstrap interval in a validation report.
CK-02Model versus human coding agreement protocol (LLM-as-coder reliability test; intra-coder agreement across runs)Check extracted resultsScore a model as one coder against a human coder or panel and against itself across repeated runs, and produce human-human, model-human and model-model agreement in a validation report.
CK-03Bootstrap confidence intervals for agreement and accuracy (bootstrap resampling of items)Check extracted resultsQuantify sampling uncertainty in an agreement or accuracy figure computed from a review sample by resampling items, and produce an interval around the coefficient.
CK-04Gold-standard validation sample (ground truth set; human-coded validation set)Check extracted resultsBuild a random or stratified human-coded subset against which model output is scored and produce a labelled set.
CK-05Out-of-sample validation (train and test split; k-fold cross-validation; bootstrap; permutation test)Check extracted resultsTune prompts, rules or models on one set of items and score them on a held-out set, and produce an out-of-sample error, interval or null distribution in a validation report.
CK-06Bias correction of model labels with a human-coded subsample (design-based supervised learning (DSL); prediction-powered inference (PPI); regression estimator correction)Check extracted resultsCombine model labels for all items with human labels for a random subsample so that downstream estimates and intervals stay valid, and produce a corrected estimate with an interval.
CK-07Split-sample design for measures discovered from text (split-sample framework)Check extracted resultsDiscover a measure such as categories or topics in one random part of the corpus and estimate its effect in the other, and produce an effect estimate that does not overfit the discovery.
CK-08Semantic, predictive and convergent validity checks (construct validity checks)Check extracted resultsTest whether an extracted category or measure behaves as the concept it claims to measure, and produce rater verdicts and comparisons with a known event or established measure in a validation report.
CK-09Cluster validation indexes (silhouette score; adjusted Rand index; adjusted mutual information; V-measure; stability analysis)Check extracted resultsScore how well a clustering fits the data or agrees with known labels, and produce a score per clustering for comparison across cluster numbers and algorithms.
CK-10Repeated-run consistency test (test-retest; intra-coder reliability; TARr@N and TARa@N (raw and parsed answer agreement); settings pinning)Check extracted resultsRun the same prompt and input several times with model version, temperature and seed recorded, and produce agreement rates and accuracy spread in a validation report.
CK-11Prompt stability scoring (prompt perturbation; paraphrase test)Check extracted resultsRe-run a task with paraphrased prompts and apply reliability metrics to the labels, and produce a prompt stability score in a validation report.
CK-12Specification sweep across models and prompts (LLM hacking test; multiverse analysis; specification curve)Check extracted resultsRepeat the downstream statistical test across several models, prompts and settings, and produce error rates for false positive, false negative, wrong-sign and wrong-magnitude conclusions.
CK-13Multiple-testing correction (Benjamini-Hochberg false discovery rate control; Bonferroni; Holm)Check extracted resultsAdjust the significance threshold when many comparisons are made, and produce adjusted p-values or a list of retained results.
CK-14Data leakage audit (train-test contamination check; model information sheet)Check extracted resultsConfirm that no information from the test items or the outcome entered the model, prompt or features, and produce a leakage checklist and a model information sheet.
CK-15Sample representativeness audit (coverage check; selection bias check)Check extracted resultsCompare the analysed sample with the population it claims to describe and with the steps that selected it, and produce a comparison table on known variables and a list of exclusion steps.
CK-16Entailment-based consistency check (textual entailment; NLI (natural language inference) consistency; SummaC; FactCC; faithfulness checking)Check extracted resultsScore whether each sentence of a summary or answer is supported by the source sentences, and produce a consistency score per sentence and per summary.
CK-17Source-span and quotation check by code (quote-and-verify; extractive grounding; citation verification)Check extracted resultsRequire each extracted statement to point to the exact passage it came from, confirm by string match that the passage exists, and produce a pass or fail list per statement.
CK-18Numeric and attribution recomputation by code (numeric fidelity check; deterministic recomputation)Check extracted resultsExtract every number, date, name and quotation from an output by code and confirm that each appears in, or can be derived from, the source, and produce a list of unmatched figures and names.
CK-19Atomic claim decomposition and scoring (FActScore; atomic fact checking)Check extracted resultsSplit generated text into atomic facts and score the share that a reliable knowledge source supports, and produce a supported proportion and a list of unsupported facts.
CK-20Sampling-consistency hallucination detection (SelfCheckGPT; self-consistency check)Check extracted resultsDraw several stochastic responses to the same prompt and flag statements that the other samples do not support, and produce sentence-level consistency scores.
CK-21Reference-free grounding metrics for retrieval-augmented answers (Ragas; faithfulness, answer relevance and context relevance scoring)Check extracted resultsScore whether retrieval finds relevant focused passages, whether the answer uses them faithfully and the quality of the answer, and produce a score per dimension without human reference answers.
CK-22LLM-as-judge with human calibration (model-graded evaluation)Check extracted resultsUse a strong model to grade outputs against a rubric and check the judge against human grades on a subsample, and produce judge scores with judge-human agreement.
CK-23Confidence calibration check (reliability diagram; expected calibration error; temperature scaling; P(True); verbalised confidence)Check extracted resultsCheck that a stated confidence of 80 per cent matches being right about 80 per cent of the time, rescale confidence if not, and produce a calibration curve and error.
CK-24Review sampling design (random audit sample; stratified sample; risk-based sampling; acceptance sampling; rule of three)Check extracted resultsChoose which outputs to inspect and how many so that the observed error rate has a stated bound, and produce an error-rate estimate with an upper bound.
CK-25Error taxonomy and error analysis (failure mode analysis; confusion analysis)Check extracted resultsClassify reviewed failures into defined types such as invented fact, wrong attribution, omission, wrong scope and arithmetic, and produce error counts by type.
CK-26Independent human review of model output (expert review; adjudication; human oversight)Check extracted resultsSpecify who reviews which outputs, with what rubric and with authority to overrule, and produce a review log with decisions.
CK-27Adversarial testing of an extraction pipeline (red-teaming; chaos testing)Check extracted resultsProbe an extraction pipeline with hostile or odd inputs such as instructions inserted in source text, contradictory documents and missing data, and produce a list of failures found.
CK-28Provenance tracking and audit logging (W3C (World Wide Web Consortium) PROV; lineage; record keeping)Check extracted resultsRecord which source, prompt, model version, settings, reviewer and decision produced each output, and produce a provenance graph or log that traces any claim back to its source.
CK-29Scripted reproducible workflow (reproducible analysis; version control; literate programming)Check extracted resultsRecord data, code and environment so that another person or a later run can regenerate every figure, and produce a project record that can be rerun from the raw data.
PE-14De-identification and re-identification risk assessment (data situation audit; direct and indirect identifier removal; disclosure risk assessment)Prepare and extractRemoves or alters identifying details and assesses the risk that a person can still be identified in the context of release, giving a de-identified dataset and a re-identification risk statement.
AO-19Ranking uncertainty analysis (stochastic multicriteria acceptability analysis; SMAA; rank acceptability index)Assess actors, options and risksSamples weights and score uncertainty many times and reports how often each option takes each rank, giving rank acceptability indices that show how stable a multi-criteria ranking is.

How the data is organised

Each technique has 52 columns in master/master_tagged.csv: identity and text (ID, family, subgroup, name, what it does, fit test, status and evidence), 44 tag columns, and the result format. Tag columns fall into five groups: input profile, selection, sequencing, robustness and the identity fields purpose and use when.

The schema and the scripts keep their original field names. This table gives the name used in this document for each.

Name in the schema Name in this document
Gate, gate class Requirement, requirement type
Hard gate Always applies
Conditional Applies in some cases
Weight Changes the order
Advisory Raises a warning
Material profile, facet Description of the material, property
Claim level needed, claim ceiling Strength of statement needed, strongest statement a technique supports
Associate; predict without implying cause; estimate an intervention effect; explain mechanism or counterfactual Show a link between things; predict, without claiming a cause; estimate the effect of an action; explain how or why
Stakes tier Stakes
Insight type Kind of finding
Representation Structure
Access depth Level of detail available
Linkage Whether sources can be joined
Collection basis How the sample was chosen
Volume class, data condition flags Amount of material, known data problems
Artefact; consumes; produces Working document; needs as input; makes as output
Output specification Result format
Answer basis What the answer is based on
Convergence matrix; dissonance; silence Agreement table; conflict; not covered
Phase (frame, gather, verify and prepare, analyse, validate) Kind of technique (section 2)
Pick, held Chosen technique, reserve list
Review Follow-up round

The CSV is rebuilt from master_base.csv and the tags_*.json files. The result formats are stored in master/specs/out_*.json and are added after the rebuild.

Working documents. The schema defines 21 types, each tagged with a kind: brief (charter or question), prepared material (data source inventory, raw dataset or corpus, data dictionary, codebook, data quality report, screening log, cleaned dataset, feature set, training sample, held-out test sample, coded corpus, entity table, evidence base, claims register), finding (assumptions register, hypotheses matrix, comparison matrix, result, validation report) and report (final report). The held-out test sample is single use.