CK-07 Split-sample design for measures discovered from text (split-sample framework)
Discover a measure such as categories or topics in one random part of the corpus and estimate its effect in the other, and produce an effect estimate that does not overfit the discovery.
What it does
Discovers the measure (categories or topics) in one random part of the corpus and estimates the effect in the other. Artefact: an effect estimate that does not overfit the discovery. Read whether the effect persists in the estimation sample.
When to use it
Use when a measure found in text will be used as a treatment or outcome in an effect or comparison claim.
When it does not fit
Fits when a text-derived measure is the treatment or outcome in a causal claim. Only the abstract was read.
Result format
The result names the measure, such as categories or topics, discovered in the discovery sample and gives the effect estimate from the separate estimation sample. It states whether the effect persists in the estimation sample.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Structure | Applies in some cases | requires=unstructured; requires=semi-structured |
| Content type | Always applies | requires=text |
| Unit of analysis | Always applies | requires=document |
| Level of detail available | Always applies | requires=record-level |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | enough documents to split into a discovery part and an estimation part, each large enough for its task |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Wrong choice when the categories are discovered and tested on the same documents. Only the abstract of the source was read, so the design details are unconfirmed. |
Choosing and ordering
| Kind of technique | Primary technique, runs in step 5.3 |
|---|---|
| Phase tag | analyse |
| Other phase tags | validate |
| Kind of finding | cause; difference |
| Question answered | why; how much |
| Strongest statement it supports | show a link between things |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to analyse |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: repeatable with fixed settings.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | input validation | none | no |
| exploratory | leakage check | none | no |
| decision-informing | input validation | none | no |
| decision-informing | leakage check | none | no |
| decision-informing | assumption and limitation statement | none | no |
| high-stakes | input validation | none | no |
| high-stakes | leakage check | none | no |
| high-stakes | assumption and limitation statement | none | no |
| high-stakes | sensitivity analysis | none | no |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | model plus code |
|---|---|
| Typical model failure | Discovers the categories and tests them on the same documents. |
| Error sources | input data; data transformation; statistical assumptions |
| Certainty limits | Certainty drops when the split is not random (risk of bias), when the estimation part is small (imprecision) and when identification of the effect rests on assumptions outside the technique (indirectness). |
Related techniques
- conflicts with CK-05 Out-of-sample validation (train and test split; k-fold cross-validation; bootstrap; permutation test): both spend the single-use held-out test sample
- should follow MM-12 Probabilistic topic modelling (LDA (latent Dirichlet allocation); STM (structural topic model); mixed-membership model): topic models are a typical way to discover the measure
- should follow MM-14 Document clustering (computer-assisted clustering; k-means on document vectors): document clusters are a typical way to discover the measure
- pairs with CK-14 Data leakage audit (train-test contamination check; model information sheet): a leakage audit confirms the estimation part was not seen
Evidence and tag basis
Documented. Estimating the latent representation on the same data "may introduce an identification problem or overfit" (2018). (Egami et al. 2018, arXiv 1802.02163)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.