MM-06 Survey weighting and subgroup estimation (design weights; calibration; post-stratification; relative standard error)
Apply survey weights so that sample results represent the population and produce weighted estimates with standard errors by subgroup.
What it does
Applies weights so that sample results represent the population, and reports estimates with sampling error by subgroup. Artefact: weighted estimates with standard errors. Read each estimate with its precision, and whether the subgroup count supports it.
When to use it
Use when you have survey microdata and need population figures, with precision, for the whole sample or for subgroups.
When it does not fit
Use the weights supplied by the survey producer. Misfit: a relative standard error too high for the subgroup. Design questions need a person.
Result format
The result is a table of weighted estimates by subgroup, each with its standard error. It states the subgroup count behind each estimate and whether that count supports the estimate.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Structure | Applies in some cases | requires=structured |
| Content type | Always applies | requires=tabular |
| Unit of analysis | Always applies | requires=record |
| How the sample was chosen | Applies in some cases | requires=probability; requires=mixed probability and non-probability |
| Level of detail available | Always applies | requires=record-level |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Variable types, Time structure, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | the producer-supplied weights and design variables, and enough responses in each subgroup to give a usable relative standard error |
|---|---|
| Additional sources needed | reference structure |
| Not for (rules) | none |
| Not for (text) | Wrong choice when no weights or design information are supplied. A subgroup whose relative standard error is too high should not be reported. Design and calibration choices need a person. |
Choosing and ordering
| Kind of technique | Primary technique, runs in step 5.3 |
|---|---|
| Phase tag | analyse |
| Other phase tags | none |
| Kind of finding | difference; pattern |
| Question answered | how much; who |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to gather |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: deterministic.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | input validation | none | no |
| decision-informing | input validation | none | no |
| decision-informing | code execution for arithmetic | none | no |
| decision-informing | spot check | optional | no |
| high-stakes | input validation | none | no |
| high-stakes | code execution for arithmetic | none | no |
| high-stakes | spot check | optional | no |
| high-stakes | sensitivity analysis | none | no |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | human judgement required |
|---|---|
| Typical model failure | Ignores the weights and reports raw sample percentages as population figures. |
| Error sources | input data; statistical assumptions; source selection |
| Certainty limits | Certainty drops with high non-response (risk of bias), small subgroup counts (imprecision) and weights that do not match the target population (indirectness). |
Related techniques
- should follow PE-04 Data profiling and rule-based validation (data audit; data quality assessment; edit checks): weights and codes must be present and valid
- pairs with HE-05 Margin of error and credibility interval check: the margin of error check tests whether a published precision figure suits the sampling method
- pairs with CK-15 Sample representativeness audit (coverage check; selection bias check): a representativeness audit shows where weighting must correct coverage
Evidence and tag basis
Tested. In an ABS evaluation, calibration with auxiliary information lowered relative standard errors by the order of 12.5 per cent and key-item non-response bias from 1.2 to 0.3 per cent. (ABS Methodological News, Dec 2009)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.