startthat.work home

Fields

52 fields. Every technique page shows the values of these fields for that technique. A field has a requirement type, which says what the process does with it: some fields remove a technique that does not fit your material (step 2), some change the order in which techniques are offered (step 3), some raise a warning, and some are information only. The table below defines the types, and each field then shows its type, its definition, the rule used to tag it and its allowed values. Field keys are the names used in the schema and the catalogue file. The dictionary overview has a worked example.

Requirement types

TypeName in the schemaMeaning
Always applieshard gateThe technique is removed when the material or job does not meet its requires value, or when the material is unknown on that facet and the user cannot supply it.
Applies in some casesconditionalThe requirement applies only under the condition stated in the field definition; otherwise the field is ignored.
Changes the orderweightNever removes a technique. It changes ranking only.
Raises a warningadvisoryNever removes a technique. It raises a warning shown with the shortlist.
Not usednot applicableThe field is not used for matching.

Identity fields

Technique ID Not used

Schema name: technique_id

Stable identifier of one technique: the family prefix and a running number, as used in the two earlier reports.

Tagging rule. Copy the identifier from the earlier report. Never renumber or reuse an identifier.

Technique name Not used

Schema name: technique_name

Preferred name of the technique, followed by alternative names in brackets, separated by semicolons.

Tagging rule. Use the name in the earlier report. Put alternative names and common abbreviations in brackets so that a search on the user's own words can match.

Family Not used

Schema name: family

The family the technique belongs to. A technique has exactly one family, which groups techniques by the operation they perform and does not order them.

Tagging rule. Use the family given in the earlier report. Do not infer a new family.

ValueMeaning
frame the questionActs on the question and its terms before evidence is examined.
decompose and mapSplits a topic into parts, sorts items, places them in a structure or models relationships.
compareSets one case, period, standard or forecaster against another.
trace cause and test claimsTraces causal chains, estimates effects or tests whether a claim is true.
handle evidenceGrades sources, combines evidence and analyses documents, text or data.
project forwardProjects a quantity, generates alternative futures or monitors for change.
assess actors, options and risksAssesses who matters, which option is best and what could go wrong.
challengeTests the current view or plan by arguing against it.
prepare and extractTurns raw text, tables or reports into a form that can be analysed.
measure and modelMeasures, summarises or models prepared data or text.
integrate across sourcesCombines or reconciles results from two or more sources.
check extracted resultsChecks the accuracy, reliability or validity of extracted or modelled results.

Purpose Not used

Schema name: purpose

One plain sentence stating what the technique does and what artefact it produces.

Tagging rule. Write one sentence in plain words that starts with a verb. Name the artefact produced.

Use when Not used

Schema name: use_when

A short phrase, in the words a user would use, describing the situations in which the technique is the right choice.

Tagging rule. Write one phrase starting with a user situation, for example a question a user might ask. Do not repeat the purpose.

Subgroup Not used

Schema name: subgroup

Subgroup within the family, copied from the earlier reports. Used for display only.

Tagging rule. Not tagged by the schema taggers; carried from the master text file.

What it does Not used

Schema name: what_it_does

Longer description of the technique and its artefact, copied from the earlier reports. Read by the model after selection.

Tagging rule. Not tagged by the schema taggers; carried from the master text file.

Fit test Not used

Schema name: fit_test

Condition that tells whether the technique fits, and the misfit cases. Copied from the earlier reports.

Tagging rule. Not tagged by the schema taggers; carried from the master text file.

Status and evidence Not used

Schema name: status_and_evidence

Source and standing of the technique as given in the earlier reports. Evidence status is copied, not upgraded.

Tagging rule. Not tagged by the schema taggers; carried from the master text file.

Input profile fields

Structure Applies in some cases

Schema name: representation (Representation)

How the material is stored: whether it has a fixed schema, a loose structure or none. This is a different property from content type and from document genre.

Tagging rule. List the representations the technique needs as requires, and any it handles better as prefers. Write any when the technique runs on whatever form the material is in.

ValueMeaning
structuredFixed fields or columns with a declared schema, such as a database table or CSV file.
semi-structuredSelf-describing records with variable fields, such as JSON, XML, or documents with headings and tables.
unstructuredNo schema; free text or media in which structure must be extracted.

Content type Always applies

Schema name: content_type

The kind of content in the material. Values are not mutually exclusive, because one item can have several (a geospatial time series is geospatial and a time series); tag every value the technique needs.

Tagging rule. List each content type the technique operates on as requires. Write any only if the technique works on every content type.

ValueMeaning
textNatural language words.
tabularRows and columns of values.
time seriesValues indexed by time order.
geospatialValues tied to coordinates or areas.
networkNodes and the links between them.
imageStill images.
audioSound recordings.
videoMoving images with or without sound.

Unit of analysis Always applies

Schema name: unit_of_analysis

The unit that one output of the technique describes and that the material must supply or allow to be extracted. Each value names a different unit; one material can offer several.

Tagging rule. List the unit the technique works on as requires. A claim-level technique requires claim unless the material is already split into claims.

ValueMeaning
recordOne row, case or respondent.
variableOne measured attribute across records.
documentOne report, article, page or file.
claimOne assertion that can be true or false.
sourceOne origin of information, such as a publisher, dataset or person.
datasetA whole table or collection treated as one object.
topicA subject or question, not tied to one item.

Variable types Raises a warning

Schema name: variable_types

The measurement types present among the variables. Advisory and applied per variable, not as a dataset-level gate, because scale type depends on the question and the extra information available.

Tagging rule. List the types the technique's estimator assumes as prefers. Write any if the technique does not depend on measurement type. Binary variables are tagged nominal.

ValueMeaning
nominalUnordered categories, including binary.
ordinalOrdered categories with unequal or unknown spacing.
intervalEqual spacing with an arbitrary zero.
ratioEqual spacing with a true zero.
counts or fractionsNon-negative integer counts, or fractions bounded by 0 and 1 such as percentages.

Time structure Applies in some cases

Schema name: time_structure

How observations relate to time. The DDI Time Method terms are the base, flattened so that no value contains another; continuous stream is not a DDI term.

Tagging rule. Require panel or cohort if the technique follows the same units over time. Write any if the technique does not use time.

ValueMeaning
cross-sectionOne observation of each unit at one point in time.
repeated cross-sectionDifferent samples of the same population at several points in time.
panelThe same sample of units observed at several points in time.
cohortA group defined by a shared event or birth period, followed over time.
aggregate time seriesExternally observable aggregate indicators recorded in time order, such as economic indicators or vital statistics.
continuous streamObservations arriving without a fixed period (not a DDI term).
not applicableNo time dimension is relevant to the material.
unknownThe time structure cannot be determined.

How the sample was chosen Applies in some cases

Schema name: collection_basis (Collection basis)

How the units in the material were selected. It decides whether estimates for a population are valid; descriptive use is unaffected.

Tagging rule. Require probability or census if the technique produces population estimates with confidence intervals. Write any for descriptive techniques.

ValueMeaning
probabilityUnits selected with known, non-zero chances by a random mechanism.
quotaUnits selected non-randomly to fill preset group counts.
purposiveUnits chosen by the researcher for stated characteristics.
availability or convenienceUnits chosen because they were easy for the researcher to reach.
respondent-assistedUnits recruited through referrals from earlier units, such as snowball sampling.
self-selected or opt-inUnits that chose to take part, including opt-in panels and volunteer responses.
mixed probability and non-probabilityA design that combines both kinds.
censusEvery unit in the universe is included.
non-sampledNot collected as a sample: administrative, transactional, sensor or scraped data.
unknownThe selection process is not documented.

Level of detail available Always applies

Schema name: access_depth (Access depth)

The deepest level of detail the material gives access to. Values are ordered from narrative only to record-level, and a requirement means that level or deeper.

Tagging rule. Require the shallowest level the technique can work from. Record-level techniques such as regression or deduplication require record-level.

ValueMeaning
narrative onlyProse description with no tables of figures.
aggregate tablesPublished totals, rates or summary tables.
record-levelUnderlying rows or records.

Number of sources Always applies

Schema name: source_count

Whether the technique works within one source or needs several. Separate from linkage.

Tagging rule. Write requires=two or more for cross-source techniques. Write any if the technique works with one source or many.

ValueMeaning
oneA single source.
two or moreTwo or more separate sources.

Whether sources can be joined Always applies

Schema name: linkage (Linkage)

Whether units in different sources can be joined through a shared identifier. Separate from the number of sources.

Tagging rule. Require shared identifier available for joins and record linkage. Write any if the technique does not join sources.

ValueMeaning
shared identifier availableA key such as an ID, code or name exists in both sources.
no shared identifierNo usable key; matching would need inexact matching.
not applicableOnly one source.
unknownNot yet checked.

Amount of material Applies in some cases

Schema name: volume_class (Volume class)

Whether a person can read the whole material or it needs automation. It is separate from unit of analysis.

Tagging rule. Require needs automation for techniques that only work at scale, or readable by a person for techniques done by hand. Write any otherwise.

ValueMeaning
readable by a personSmall enough for a person to read in full in the time available.
needs automationToo large to read in full; processing needs code or a model.

Known data problems Applies in some cases

Schema name: data_condition_flags (Data condition flags)

Known quality problems in the material, from the ABS Data Quality Framework dimensions. None known and unknown cannot be combined with other values. A technique that repairs a problem requires its flag.

Tagging rule. Require a flag if the technique exists to handle it, for example imputation requires missing data. Use not_for_rules to exclude a technique that fails when a flag is present. Write any if condition does not matter.

ValueMeaning
none knownNo problem has been identified.
unknownCondition has not been assessed.
missing dataValues or records are absent.
inconsistentThe material contradicts itself or other sources.
undocumentedVariable meaning, units or collection process are not documented.
out of dateThe reference period is too old for the question.
duplicatedUnits appear more than once.
measurement errorValues are known or suspected to be inaccurate or imprecise.

Producer interest Changes the order

Schema name: producer_interest

The relationship of the producer of a source to its funder or subject. It changes how much weight a result deserves, not whether a technique runs.

Tagging rule. Write any for nearly all techniques. Require a value only if the technique appraises it, in which case set appraises_source_credibility to yes.

ValueMeaning
independentThe producer has no stake in the finding.
sponsoredFunded by a party with an interest in the finding.
self-interested sellerThe producer sells a product or service the finding supports.
unknownNot determined.

Source dependence Changes the order

Schema name: source_dependence

Whether sources rest on the same upstream origin. It changes the weight of agreement between sources.

Tagging rule. Write any for nearly all techniques. Require independent only for corroboration techniques that count agreement between sources.

ValueMeaning
independentSources were produced separately from different evidence.
copied or derivedOne source copies, cites or is built from another.
unknownNot determined.

Origin Changes the order

Schema name: origin

How close the source is to the original event or data. It changes the weight of a result.

Tagging rule. Write any unless the technique appraises provenance. Record derived-from links in the data source inventory, not here.

ValueMeaning
primaryOriginal account, data or research.
secondaryInterprets or reports primary material.
tertiaryCollects and distils primary and secondary sources, such as an encyclopedia or handbook.
unknownNot determined.

Document genre Changes the order

Schema name: document_genre

The form of a document, chosen by the document's form and not by its publisher. Where two values fit, use the earlier one in the list.

Tagging rule. Write any for nearly all techniques. List genres only for appraisal techniques that apply to one genre, for example a systematic review appraisal tool.

ValueMeaning
evidence synthesisSystematic review, meta-analysis or overview of reviews.
research articleReport of original research, peer reviewed or preprint.
legal or regulatory textLegislation, regulation or court decision.
government or agency reportReport or statistical release by a public body.
organisation reportReport by a company, think tank, industry body or non-government organisation.
reference workEncyclopedia, textbook or handbook.
news or commentaryJournalism, opinion or blog post.
transcript or testimonyInterview, hearing or meeting record.
social media or forum postShort public posts and replies.
data releaseA dataset or table with at most brief documentation.
otherFits none of the above.

Appraises source credibility Applies in some cases

Schema name: appraises_source_credibility

Whether the technique appraises source credibility, independence or sponsorship. When yes, producer interest, source dependence and origin become required inputs for this technique.

Tagging rule. Write yes if the technique's purpose is to grade, test or compare the credibility, independence or sponsorship of sources. Otherwise write no.

ValueMeaning
yesThe technique appraises credibility, independence or sponsorship.
noThe technique does not.

Language Applies in some cases

Schema name: language

Whether the material is in one language or several. Language-specific tools are recorded in requires_minimum.

Tagging rule. Require multiple only for cross-language techniques such as translation checks. Write any for all others.

ValueMeaning
singleAll material is in one language.
multipleThe material has more than one language.

Personal information Always applies

Schema name: privacy (Privacy)

Whether the material contains personal information, applied as a pre-filter before matching. On a technique, the cell lists the classes the technique may run on; any means all.

Tagging rule. Write any unless the technique links, discloses or re-identifies individuals, in which case list only the classes it may run on, using requires.

ValueMeaning
no personal informationNothing that could identify an individual.
personal informationInformation or an opinion that could identify an individual.
sensitive informationPersonal information about racial or ethnic origin, political opinions, religion, union membership, sexual orientation, criminal record, health or genetic data, or some biometric data.
unknownNot assessed.

Security classification Not used

Schema name: security_classification

The protective marking of the material, applied as a pre-filter before matching. On a technique, the cell lists the markings it may be used with. Not used in matching or asked at intake: all 236 techniques are tagged any. Kept so that a deployment can add a requirement later.

Tagging rule. Write any unless the technique requires copying material to an uncontrolled service, in which case list only the lowest markings using requires.

ValueMeaning
unclassifiedNo protective marking.
officialMarked OFFICIAL under the PSPF.
official sensitiveMarked OFFICIAL: Sensitive under the PSPF.
higher than official sensitiveAny marking above OFFICIAL: Sensitive; the full list is unconfirmed.
unknownNot assessed.

Licence Not used

Schema name: licence

The terms under which the material may be used, applied as a pre-filter before matching. It is a different instrument from privacy and security. Not used in matching or asked at intake: all 236 techniques are tagged any. Kept so that a deployment can add a requirement later.

Tagging rule. Write any unless the technique copies, adapts or republishes the material, in which case require open.

ValueMeaning
openOpen licence that allows the intended use.
restricted by copyrightCopyright limits copying or adaptation.
restricted by contractA contract or terms of use limit use.
unknownTerms not checked.

Selection fields

Kind of technique (main phase) Applies in some cases

Schema name: primary_phase (Primary phase)

The one phase in which the technique is normally used. Phases are ordered; the seven-phase list replaces the five steps in the earlier draft.

Tagging rule. Pick the phase in which the technique produces its main artefact. If a technique is used equally in two phases, pick the earlier one and list the other in other_phases.

ValueMeaning
frameSettle the question, scope, terms and assumptions before evidence is examined.
gatherFind, collect and inventory the material.
verify and prepareCheck quality and fitness of the material, then clean, extract and structure it. Ends when the material is ready for analysis. Material that cannot be read is reported in the answer basis and the techniques that need it are dropped.
analyseExplore, measure, model, compare or trace relationships in the prepared material.
validateCheck the result before concluding: held-out validation, agreement with a standard, sensitivity and alternative techniques.
concludeState what the findings support, with a confidence label and a one-line reason for each conclusion.
reportCommunicate the conclusion, its limits and the assumptions behind it.

Other permitted phases Applies in some cases

Schema name: other_phases

Further phases in which the technique may be used. Excludes the primary phase.

Tagging rule. List every other phase in which the technique is a normal choice. Write none if it is used in the primary phase only.

ValueMeaning
frameSettle the question, scope, terms and assumptions before evidence is examined.
gatherFind, collect and inventory the material.
verify and prepareCheck quality and fitness of the material, then clean, extract and structure it. Ends when the material is ready for analysis. Material that cannot be read is reported in the answer basis and the techniques that need it are dropped.
analyseExplore, measure, model, compare or trace relationships in the prepared material.
validateCheck the result before concluding: held-out validation, agreement with a standard, sensitivity and alternative techniques.
concludeState what the findings support, with a confidence label and a one-line reason for each conclusion.
reportCommunicate the conclusion, its limits and the assumptions behind it.
noneNo other phase.

Kind of finding Changes the order

Schema name: insight_type (Insight type)

What the output is read for. Several values are allowed; matching uses it as a weight because tags are inferred.

Tagging rule. List every insight type the output is read for, with the main one first. Write any only if the technique is general purpose.

ValueMeaning
patternA regularity in the material.
differenceHow cases, periods or sources differ.
contradictionStatements or sources that cannot both be true.
causeA cause or effect relationship.
trendDirection of change over time.
gapSomething expected that is missing.
riskWhat could go wrong or where a source is weak.
option rankingOptions placed in order.

Question answered Changes the order

Schema name: question_answered

The question the output addresses. Several values are allowed.

Tagging rule. List every question the output answers, with the main one first.

ValueMeaning
whatWhat is the case, or what is in the material.
whyWhy something happened.
how muchThe size of a quantity or effect.
what nextWhat is likely to happen.
whoWhich actors matter.
which optionWhich choice is best.

Strongest statement the technique supports Raises a warning

Schema name: claim_ceiling (Claim ceiling)

The strongest claim the technique can support under ideal conditions. It is a maximum for the technique. The answer does not lower it or judge the material. In the matching step this field never removes a technique. The skill reports how many techniques that passed the other gates reach the claim level needed, and warns when none does, because preparation and appraisal techniques support a stronger final claim without reaching it themselves. The claim level needed also adds checks by stakes tier.

Tagging rule. Pick the highest claim level the technique can support if every assumption is met. A technique that only summarises is describe, even if users misuse it for cause.

ValueMeaning
describeSummarise what is in the material or the world without relating variables.
associateShow that variables, events or claims go together, without saying why.
predict without implying causeForecast or classify new cases from patterns. The prediction never supports a causal reading.
estimate an intervention effectEstimate how an outcome changes when an intervention is applied (Pearl rung 2, Maryland levels 3 to 5, Nesta level 3).
explain mechanism or counterfactualSay how or why, or what would have happened otherwise in a case (Pearl rung 3, Nesta level 4).

Requires minimum amounts Applies in some cases

Schema name: requires_minimum

Minimum amounts the technique needs, such as sample size, number of variables, number of sources or number of periods. Free text so that each can carry its own unit.

Tagging rule. Write each minimum with its unit, for example at least 30 records per group. Write none if no minimum is known.

Additional sources needed Applies in some cases

Schema name: external_material_needed (Additional sources needed)

What the technique needs beyond the user's material. The process finds it in 5.2, or the user supplies it when none is found. None cannot be combined with other values.

Tagging rule. List what the technique needs beyond the user's material. Write none if the material is enough.

ValueMeaning
noneNothing beyond the input.
comparatorA second case, group, period or standard to compare with.
reference structureA codebook, schema, rule set, protocol or population benchmark.
earlier periodData or documents from before the period of interest.

Not for rules Always applies

Schema name: not_for_rules

Machine-checkable exclusions. The technique is removed when all clauses of any one exclusion are true for the job.

Tagging rule. Write each exclusion as clauses of the form facet=value or facet=value1|value2 joined by &, exclusions separated by ;, and a reason after ::. Keys may be any facet key or claim_level_needed, stakes_tier, insight_wanted. Write none if there are no exclusions.

Not for, in words Raises a warning

Schema name: not_for_text

Fit test and misuse signs that cannot be written as rules, in words. Carried over from the fit test column of the earlier reports.

Tagging rule. Write signs that the technique is the wrong choice or is being misused, in one or two sentences. Do not repeat anything already in not_for_rules.

Sequencing fields

Needs as input Always applies

Schema name: consumes (Consumes)

Artefacts the technique reads, each with the state it needs, written as type:state. An artefact type marked single use is spent by the first technique that consumes it.

Tagging rule. List each artefact the technique needs before it can start, as type:state. If the technique starts from the user's question alone, write charter or question:agreed.

Makes as output Not used

Schema name: produces (Produces)

Artefacts the technique creates, each with the state it reaches, written as type:state.

Tagging rule. List each artefact the technique outputs, as type:state. Every technique produces at least one artefact.

Requires Always applies

Schema name: requires

Hard prerequisites beyond the artefacts consumed: technique IDs that must have run, or artefact types that must exist, written as artefact:type:state. The technique must not run until all are met.

Tagging rule. Write technique IDs or artefact:type:state items separated by ;. Do not repeat artefacts already in consumes. Write none if there is no extra prerequisite.

Loop target Not used

Schema name: loop_target

What happens when the technique's fit test fails. Exactly one outcome.

Tagging rule. Pick the outcome that follows a failed fit test. Use stop only when no earlier phase can repair the failure.

ValueMeaning
proceed with limit statedContinue and record the failed fit test as a limit on the claim.
return to frameRework the question or scope.
return to gatherCollect more or different material.
return to verify and prepareClean, extract or restructure the material again.
return to analyseTry another technique or settings on the same prepared material.
stop because the data cannot answer the questionReport the limit; do not continue.

Robustness fields

Model suitability Applies in some cases

Schema name: model_suitability

The most demanding level of support the whole technique needs, from least to most oversight. The skill compares it with what the deployment offers.

Tagging rule. Pick the most demanding level any step needs. If one step needs a human judgement, write human judgement required.

ValueMeaning
model aloneA language model can do every step with no code and no person.
model plus codeNeeds code execution for at least one step, such as arithmetic or sampling.
human judgement requiredNeeds a person to supply or confirm a judgement.

Typical model failure Not used

Schema name: typical_model_failure

The most likely way an AI model gets this technique wrong, in words.

Tagging rule. Write one or two sentences naming a specific failure, for example inventing a source or miscounting. Write unknown if no failure is known.

Repeatability Not used

Schema name: repeatability

How far the same input gives the same output.

Tagging rule. Pick deterministic if code gives the same output every time. Pick repeatable with fixed settings if a model or seed gives the same output when its settings are fixed. Otherwise pick judgement dependent.

ValueMeaning
deterministicThe same input always gives the same output.
repeatable with fixed settingsThe same input gives the same output when settings such as seed or temperature are fixed.
judgement dependentOutput depends on a person's or a model's judgement and varies between runs or between people.

Minimum checks by tier Not used

Schema name: minimum_checks

For each stakes tier, the full list of check codes required at that tier (not an increment), written as tier:code. Codes come from the check library. Tier contents are the schema designer's design.

Tagging rule. Give at least one check per tier, and each higher tier must contain every check of the tier below. For a deterministic technique use input validation, not repeat runs.

Error sources Not used

Schema name: error_sources

What the technique's errors depend on. Used to test whether a second technique is independent corroboration. Values are not mutually exclusive.

Tagging rule. List every source of error the technique depends on, at least input data.

ValueMeaning
input dataThe same underlying records or documents.
data transformationThe same cleaning, coding or extraction step.
statistical assumptionsThe same modelling assumptions, such as linearity.
coder or rater judgementOne person's or one rater group's judgement.
language modelThe same language model or model family.
source selectionThe same choice of which sources to include.
expert opinionThe same experts or panel.

Certainty limits Not used

Schema name: certainty_limits

What can make a conclusion from this technique less reliable, in words, mapped where possible to GRADE domains: risk of bias, inconsistency, indirectness, imprecision and reporting bias. Used to write the one-line reason for the confidence label.

Tagging rule. Write one or two sentences naming the conditions under which confidence in the result drops. Write none known only if the tagger has no basis.

Evidence status Raises a warning

Schema name: evidence_status

How well the technique itself is supported. The four values are unchanged from the earlier reports.

Tagging rule. Copy the status from the earlier report. Do not upgrade it.

ValueMeaning
TestedAt least one published study in the notes measured the technique's effect or accuracy.
DocumentedDescribed in a text on the technique, official guidance or secondary summary, without a test.
PracticeUsed in practice with no published description found.
UnconfirmedNo source was opened or a source failed to load.

Tag basis Not used

Schema name: tag_basis

For each field group, whether the tags for this technique were taken from a source or inferred. Exactly one entry per group.

Tagging rule. Write one entry per group: input profile, selection, sequencing, robustness. Write sourced only if a source states the values for the technique itself; otherwise write inference.

ValueMeaning
input profile sourcedInput profile values for this technique come from a source.
input profile inferenceInput profile values for this technique are inferred.
selection sourcedSelection values come from a source.
selection inferenceSelection values are inferred.
sequencing sourcedSequencing values come from a source.
sequencing inferenceSequencing values are inferred.
robustness sourcedRobustness values come from a source.
robustness inferenceRobustness values are inferred.

Application fields

Result format Not used

Schema name: output_spec (Output specification)

Two to three sentences describing the shape of the result: its parts, tables or lists, what each row or cell holds, and what it must name or end with. Describes the result only, not how to run the technique.

Tagging rule. Written from what_it_does and produces. Starts with "The result".