CK-08 Semantic, predictive and convergent validity checks (construct validity checks)
Test whether an extracted category or measure behaves as the concept it claims to measure, and produce rater verdicts and comparisons with a known event or established measure in a validation report.
What it does
Tests whether an extracted category or measure behaves as the concept it claims to measure. Artefact: raters' verdicts on pairs of items, and comparisons with a known event or an established measure. Read agreement with the benchmark.
When to use it
Use when a topic, theme or score from text needs evidence that it means what you say it means.
When it does not fit
Needs an external benchmark. Misfit: the benchmark is derived from the same model, or measures something related but different. Semantic validity is human rating of coherence, predictive validity is movement with expected events, and convergent validity is agreement with an established measure.
Result format
The result has a section for each validity check that applies, giving the raters' verdicts on pairs of items, the comparison with a known event or an established measure, and the agreement with that benchmark. It names the benchmark used and states whether the category or measure behaves as the concept it claims to measure.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Content type | Always applies | requires=text; requires=tabular |
| Unit of analysis | Always applies | requires=topic; requires=document |
| Level of detail available | Always applies | requires=record-level |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | an external benchmark and enough raters or item pairs to judge agreement |
|---|---|
| Additional sources needed | comparator |
| Not for (rules) | none |
| Not for (text) | Wrong choice when the benchmark comes from the same model or measures something related but different. Semantic validity is human rating of coherence, and the other two types need an outside measure. |
Choosing and ordering
| Kind of technique | Verification technique, runs in step 6 |
|---|---|
| Phase tag | validate |
| Other phase tags | none |
| Kind of finding | pattern; difference |
| Question answered | what; which option |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to gather |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | spot check | optional | no |
| decision-informing | spot check | optional | no |
| decision-informing | human review sample | needed; left out below high stakes | no |
| decision-informing | gold standard agreement | conditional; left out below high stakes | no |
| high-stakes | spot check | optional | no |
| high-stakes | human review sample | needed | no |
| high-stakes | gold standard agreement | conditional | no |
| high-stakes | second technique with different assumptions | none | conditional |
| high-stakes | expert review | needed | no |
Failure modes and limits
| Model suitability | human judgement required |
|---|---|
| Typical model failure | Confirms its own construct by producing agreeing evidence. |
| Error sources | input data; coder or rater judgement; expert opinion |
| Certainty limits | Certainty drops when the benchmark is derived from the same model (risk of bias), when it measures a related concept (indirectness) and when few raters judged (imprecision). |
Related techniques
- pairs with MM-12 Probabilistic topic modelling (LDA (latent Dirichlet allocation); STM (structural topic model); mixed-membership model): probabilistic topics need all three validity types
- pairs with MM-13 Embedding-based topic modelling (BERTopic; Top2Vec; clustering-based topic models): embedding topics lack validated automatic scores
Evidence and tag basis
Documented. Worked examples include raters judging document pairs and topic use against committee jurisdiction. (Grimmer and Stewart 2013, Political Analysis)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.