PE-01 Corpus construction and sampling (document selection; search string design; sampling frame design)
Define which documents enter the analysis so that counts and themes describe a stated population, giving a corpus table with counts and an inclusion log.
What it does
Defines which documents enter the analysis, so that counts and themes describe a stated population. Artefact: a corpus table with counts per source, period and document type, and an inclusion log. Read coverage, imbalance and gaps before reading any result.
When to use it
Use when you are about to count or code documents and must decide which documents represent the population you are asking about.
When it does not fit
Test: name the population the corpus represents and count documents per source and period. Misfit: the corpus is whatever was easiest to retrieve, one source supplies most documents, or the search string contains the conclusion terms. For a fixed coding budget, code more unique documents instead of using more coders per document.
Result format
The result is a corpus table with one row per source, period and document type, giving the document count in each, followed by an inclusion log listing what was included or excluded and why. It names the population the corpus represents and ends with the coverage gaps and imbalances.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Structure | Applies in some cases | requires=unstructured; requires=semi-structured |
| Content type | Always applies | requires=text |
| Unit of analysis | Always applies | requires=document |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | none |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | The corpus is whatever was easiest to retrieve, one source supplies most documents, or the search string contains the conclusion terms. |
Choosing and ordering
| Kind of technique | Preparation technique, runs in step 5.2 |
|---|---|
| Phase tag | gather |
| Other phase tags | verify and prepare |
| Kind of finding | gap; risk |
| Question answered | what |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to gather |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | spot check | optional | no |
| decision-informing | spot check | optional | no |
| decision-informing | code execution for arithmetic | none | no |
| decision-informing | provenance log | none | no |
| decision-informing | sensitivity analysis | none | no |
| high-stakes | spot check | optional | no |
| high-stakes | code execution for arithmetic | none | no |
| high-stakes | provenance log | none | no |
| high-stakes | sensitivity analysis | none | no |
| high-stakes | independent review | needed | no |
| high-stakes | human review sample | needed | no |
Failure modes and limits
| Model suitability | human judgement required |
|---|---|
| Typical model failure | Treats the supplied documents as the whole population and overstates themes drawn from a few documents. |
| Error sources | input data; source selection; coder or rater judgement |
| Certainty limits | Certainty drops when one source supplies most documents (risk of bias), when coverage by period or type is uneven (indirectness), or when the search string reflects the expected answer (reporting bias). |
Related techniques
- pairs with CK-15 Sample representativeness audit (coverage check; selection bias check): a representativeness audit tests whether the corpus covers the stated population
- pairs with PE-03 Near-duplicate detection (shingling; MinHash; locality-sensitive hashing; text reuse detection): removing near-duplicates after selection stops repeats counting as separate documents
Evidence and tag basis
Documented. Representation is the first of five tasks in the text-as-data framework; the coding-budget result is from an abstract. (Barberá et al. 2021 via EconPapers; Stanford Political Science: Text as Data)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.