HE-04 Survey disclosure completeness check (AAPOR (American Association for Public Opinion Research) disclosure elements)
Tests whether a survey report states the sponsor, question wording, population, sampling frame, mode, dates, sample size, margin of error and weighting, and produces a list of disclosed and missing items.
What it does
Tests whether a survey report states the sponsor, question wording, population, sampling frame, mode, dates, sample size, margin of error and weighting. Artefact: a list of disclosed and missing items. Read each missing item as a finding.
When to use it
Use when you want to know whether a published survey gives enough information to be assessed.
When it does not fit
Fits any publicly released survey. It tests disclosure, not accuracy.
Result format
The result is a list of the items the survey report should disclose (sponsor, question wording, population, sampling frame, mode, dates, sample size, margin of error and weighting), each marked as disclosed or missing. Each missing item is stated as a finding.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Content type | Always applies | requires=text |
| Unit of analysis | Always applies | requires=document |
| Producer interest | Changes the order | prefers=sponsored; prefers=self-interested seller; prefers=unknown |
| Document genre | Changes the order | prefers=organisation report; prefers=government or agency report; prefers=news or commentary |
| Appraises source credibility | Applies in some cases | yes |
Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Level of detail available, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Source dependence, Origin, Language, Personal information, Security classification, Licence.
| Minimum material | none |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | The aim is to judge accuracy. The check tests disclosure, not whether the survey is right. |
Choosing and ordering
| Kind of technique | Preparation technique, runs in step 5.2 |
|---|---|
| Phase tag | verify and prepare |
| Other phase tags | none |
| Kind of finding | gap |
| Question answered | what; who |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | proceed with limit stated |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: judgement dependent.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | spot check | optional | no |
| decision-informing | spot check | optional | no |
| decision-informing | repeat runs | none | no |
| decision-informing | prompt variation | none | no |
| high-stakes | spot check | optional | no |
| high-stakes | repeat runs | none | no |
| high-stakes | prompt variation | none | no |
| high-stakes | human review sample | needed | no |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | model alone |
|---|---|
| Typical model failure | Marks an element as disclosed because the report uses a related word, and does not read each missing item as a finding. |
| Error sources | input data; language model |
| Certainty limits | Certainty drops when the source report is a press summary and the full methods statement exists elsewhere (reporting bias). |
Related techniques
- pairs with HE-05 Margin of error and credibility interval check: disclosed sampling method decides which margin of error is valid
- pairs with HE-06 Opt-in panel respondent quality check (bogus respondent screen): disclosed mode and panel type point to the respondent quality check
Evidence and tag basis
Documented. The standards list what must be disclosed at release and what must be supplied on request within 30 days. (AAPOR disclosure standards)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.