startthat.work home

CK-08 Semantic, predictive and convergent validity checks (construct validity checks)

Test whether an extracted category or measure behaves as the concept it claims to measure, and produce rater verdicts and comparisons with a known event or established measure in a validation report.

Check extracted resultsEvidence: DocumentedVerification technique, runs in step 6Strongest statement: describe

What it does

Tests whether an extracted category or measure behaves as the concept it claims to measure. Artefact: raters' verdicts on pairs of items, and comparisons with a known event or an established measure. Read agreement with the benchmark.

When to use it

Use when a topic, theme or score from text needs evidence that it means what you say it means.

When it does not fit

Needs an external benchmark. Misfit: the benchmark is derived from the same model, or measures something related but different. Semantic validity is human rating of coherence, predictive validity is movement with expected events, and convergent validity is agreement with an established measure.

Result format

The result has a section for each validity check that applies, giving the raters' verdicts on pairs of items, the comparison with a known event or an established measure, and the agreement with that benchmark. It names the benchmark used and states whether the category or measure behaves as the concept it claims to measure.

Requirements on the material

PropertyRequirement typeRequirement
Content typeAlways appliesrequires=text; requires=tabular
Unit of analysisAlways appliesrequires=topic; requires=document
Level of detail availableAlways appliesrequires=record-level
Appraises source credibilityApplies in some casesno

Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialan external benchmark and enough raters or item pairs to judge agreement
Additional sources neededcomparator
Not for (rules)none
Not for (text)Wrong choice when the benchmark comes from the same model or measures something related but different. Semantic validity is human rating of coherence, and the other two types need an outside measure.

Choosing and ordering

Kind of techniqueVerification technique, runs in step 6
Phase tagvalidate
Other phase tagsnone
Kind of findingpattern; difference
Question answeredwhat; which option
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to gather

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.

StakesCheckPerson involvedCounts as a different method
exploratoryspot checkoptionalno
decision-informingspot checkoptionalno
decision-informinghuman review sampleneeded; left out below high stakesno
decision-informinggold standard agreementconditional; left out below high stakesno
high-stakesspot checkoptionalno
high-stakeshuman review sampleneededno
high-stakesgold standard agreementconditionalno
high-stakessecond technique with different assumptionsnoneconditional
high-stakesexpert reviewneededno

Failure modes and limits

Model suitabilityhuman judgement required
Typical model failureConfirms its own construct by producing agreeing evidence.
Error sourcesinput data; coder or rater judgement; expert opinion
Certainty limitsCertainty drops when the benchmark is derived from the same model (risk of bias), when it measures a related concept (indirectness) and when few raters judged (imprecision).

Evidence and tag basis

Documented. Worked examples include raters judging document pairs and topic use against committee jurisdiction. (Grimmer and Stewart 2013, Political Analysis)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.