startthat.work home

PE-01 Corpus construction and sampling (document selection; search string design; sampling frame design)

Define which documents enter the analysis so that counts and themes describe a stated population, giving a corpus table with counts and an inclusion log.

Prepare and extractEvidence: DocumentedPreparation technique, runs in step 5.2Strongest statement: describe

What it does

Defines which documents enter the analysis, so that counts and themes describe a stated population. Artefact: a corpus table with counts per source, period and document type, and an inclusion log. Read coverage, imbalance and gaps before reading any result.

When to use it

Use when you are about to count or code documents and must decide which documents represent the population you are asking about.

When it does not fit

Test: name the population the corpus represents and count documents per source and period. Misfit: the corpus is whatever was easiest to retrieve, one source supplies most documents, or the search string contains the conclusion terms. For a fixed coding budget, code more unique documents instead of using more coders per document.

Result format

The result is a corpus table with one row per source, period and document type, giving the document count in each, followed by an inclusion log listing what was included or excluded and why. It names the population the corpus represents and ends with the coverage gaps and imbalances.

Requirements on the material

PropertyRequirement typeRequirement
StructureApplies in some casesrequires=unstructured; requires=semi-structured
Content typeAlways appliesrequires=text
Unit of analysisAlways appliesrequires=document
Appraises source credibilityApplies in some casesno

Not restricted on: Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialnone
Additional sources needednone
Not for (rules)none
Not for (text)The corpus is whatever was easiest to retrieve, one source supplies most documents, or the search string contains the conclusion terms.

Choosing and ordering

Kind of techniquePreparation technique, runs in step 5.2
Phase taggather
Other phase tagsverify and prepare
Kind of findinggap; risk
Question answeredwhat
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to gather

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.

StakesCheckPerson involvedCounts as a different method
exploratoryspot checkoptionalno
decision-informingspot checkoptionalno
decision-informingcode execution for arithmeticnoneno
decision-informingprovenance lognoneno
decision-informingsensitivity analysisnoneno
high-stakesspot checkoptionalno
high-stakescode execution for arithmeticnoneno
high-stakesprovenance lognoneno
high-stakessensitivity analysisnoneno
high-stakesindependent reviewneededno
high-stakeshuman review sampleneededno

Failure modes and limits

Model suitabilityhuman judgement required
Typical model failureTreats the supplied documents as the whole population and overstates themes drawn from a few documents.
Error sourcesinput data; source selection; coder or rater judgement
Certainty limitsCertainty drops when one source supplies most documents (risk of bias), when coverage by period or type is uneven (indirectness), or when the search string reflects the expected answer (reporting bias).

Evidence and tag basis

Documented. Representation is the first of five tasks in the text-as-data framework; the coding-budget result is from an abstract. (Barberá et al. 2021 via EconPapers; Stanford Political Science: Text as Data)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.