startthat.work home

HE1-18 Document and policy analysis

Reads documents systematically to extract facts, positions, changes or stated reasons using their provenance and purpose, and produces an extraction table or chronology.

Handle evidenceEvidence: UnconfirmedPrimary technique, runs in step 5.3Strongest statement: describe

What it does

Reads documents systematically to extract facts, positions, changes or stated reasons, using their provenance and purpose. Artefact: an extraction table or chronology; read the author, audience and purpose before the content.

When to use it

Use when the question is what laws, reports or regulations say, when they changed, or why an institution says it acted.

When it does not fit

It is used as proof of what happened in practice, when it shows only what an institution says or did.

Result format

The result is an extraction table or chronology with one row per document, giving the facts, positions, changes or stated reasons found in it. Each row names the author, audience and purpose of the document, which are stated before its content.

Requirements on the material

PropertyRequirement typeRequirement
Content typeAlways appliesrequires=text
Unit of analysisAlways appliesrequires=document
OriginChanges the orderprefers=primary
Document genreChanges the orderprefers=legal or regulatory text; prefers=government or agency report; prefers=organisation report
Appraises source credibilityApplies in some casesno

Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Language, Personal information, Security classification, Licence.

Minimum materialnone
Additional sources needednone
Not for (rules)none
Not for (text)The documents are used as proof of what happened in practice, when they show only what an institution says it did.

Choosing and ordering

Kind of techniquePrimary technique, runs in step 5.3
Phase taganalyse
Other phase tagsnone
Kind of findingpattern; trend; gap
Question answeredwhat; why
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetproceed with limit stated

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.

StakesCheckPerson involvedCounts as a different method
exploratoryspot checkoptionalno
decision-informingspot checkoptionalno
decision-informingrepeat runsnoneno
decision-informingprompt variationnoneno
high-stakesspot checkoptionalno
high-stakesrepeat runsnoneno
high-stakesprompt variationnoneno
high-stakeshuman review sampleneededno
high-stakesindependent reviewneededno

Failure modes and limits

Model suitabilitymodel plus code
Typical model failureQuotes text that is not in the document, or reads a stated reason as the true reason.
Error sourcesinput data; language model; coder or rater judgement
Certainty limitsCertainty drops when documents are secondary or edited, when authorship and purpose are unknown, and when practice is inferred from policy (indirectness).

Evidence and tag basis

Unconfirmed. A commonly cited guidance paper (Bowen 2009) was recalled and not opened.

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.