startthat.work home

CK-07 Split-sample design for measures discovered from text (split-sample framework)

Discover a measure such as categories or topics in one random part of the corpus and estimate its effect in the other, and produce an effect estimate that does not overfit the discovery.

Check extracted resultsEvidence: DocumentedPrimary technique, runs in step 5.3Strongest statement: show a link between things

What it does

Discovers the measure (categories or topics) in one random part of the corpus and estimates the effect in the other. Artefact: an effect estimate that does not overfit the discovery. Read whether the effect persists in the estimation sample.

When to use it

Use when a measure found in text will be used as a treatment or outcome in an effect or comparison claim.

When it does not fit

Fits when a text-derived measure is the treatment or outcome in a causal claim. Only the abstract was read.

Result format

The result names the measure, such as categories or topics, discovered in the discovery sample and gives the effect estimate from the separate estimation sample. It states whether the effect persists in the estimation sample.

Requirements on the material

PropertyRequirement typeRequirement
StructureApplies in some casesrequires=unstructured; requires=semi-structured
Content typeAlways appliesrequires=text
Unit of analysisAlways appliesrequires=document
Level of detail availableAlways appliesrequires=record-level
Appraises source credibilityApplies in some casesno

Not restricted on: Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialenough documents to split into a discovery part and an estimation part, each large enough for its task
Additional sources needednone
Not for (rules)none
Not for (text)Wrong choice when the categories are discovered and tested on the same documents. Only the abstract of the source was read, so the design details are unconfirmed.

Choosing and ordering

Kind of techniquePrimary technique, runs in step 5.3
Phase taganalyse
Other phase tagsvalidate
Kind of findingcause; difference
Question answeredwhy; how much
Strongest statement it supportsshow a link between things
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to analyse

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: repeatable with fixed settings.

StakesCheckPerson involvedCounts as a different method
exploratoryinput validationnoneno
exploratoryleakage checknoneno
decision-informinginput validationnoneno
decision-informingleakage checknoneno
decision-informingassumption and limitation statementnoneno
high-stakesinput validationnoneno
high-stakesleakage checknoneno
high-stakesassumption and limitation statementnoneno
high-stakessensitivity analysisnoneno
high-stakesindependent reviewneededno

Failure modes and limits

Model suitabilitymodel plus code
Typical model failureDiscovers the categories and tests them on the same documents.
Error sourcesinput data; data transformation; statistical assumptions
Certainty limitsCertainty drops when the split is not random (risk of bias), when the estimation part is small (imprecision) and when identification of the effect rests on assumptions outside the technique (indirectness).

Evidence and tag basis

Documented. Estimating the latent representation on the same data "may introduce an identification problem or overfit" (2018). (Egami et al. 2018, arXiv 1802.02163)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.