startthat.work home

MM-06 Survey weighting and subgroup estimation (design weights; calibration; post-stratification; relative standard error)

Apply survey weights so that sample results represent the population and produce weighted estimates with standard errors by subgroup.

Measure and modelEvidence: TestedPrimary technique, runs in step 5.3Strongest statement: describe

What it does

Applies weights so that sample results represent the population, and reports estimates with sampling error by subgroup. Artefact: weighted estimates with standard errors. Read each estimate with its precision, and whether the subgroup count supports it.

When to use it

Use when you have survey microdata and need population figures, with precision, for the whole sample or for subgroups.

When it does not fit

Use the weights supplied by the survey producer. Misfit: a relative standard error too high for the subgroup. Design questions need a person.

Result format

The result is a table of weighted estimates by subgroup, each with its standard error. It states the subgroup count behind each estimate and whether that count supports the estimate.

Requirements on the material

PropertyRequirement typeRequirement
StructureApplies in some casesrequires=structured
Content typeAlways appliesrequires=tabular
Unit of analysisAlways appliesrequires=record
How the sample was chosenApplies in some casesrequires=probability; requires=mixed probability and non-probability
Level of detail availableAlways appliesrequires=record-level
Appraises source credibilityApplies in some casesno

Not restricted on: Variable types, Time structure, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialthe producer-supplied weights and design variables, and enough responses in each subgroup to give a usable relative standard error
Additional sources neededreference structure
Not for (rules)none
Not for (text)Wrong choice when no weights or design information are supplied. A subgroup whose relative standard error is too high should not be reported. Design and calibration choices need a person.

Choosing and ordering

Kind of techniquePrimary technique, runs in step 5.3
Phase taganalyse
Other phase tagsnone
Kind of findingdifference; pattern
Question answeredhow much; who
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetreturn to gather

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: deterministic.

StakesCheckPerson involvedCounts as a different method
exploratoryinput validationnoneno
decision-informinginput validationnoneno
decision-informingcode execution for arithmeticnoneno
decision-informingspot checkoptionalno
high-stakesinput validationnoneno
high-stakescode execution for arithmeticnoneno
high-stakesspot checkoptionalno
high-stakessensitivity analysisnoneno
high-stakesindependent reviewneededno

Failure modes and limits

Model suitabilityhuman judgement required
Typical model failureIgnores the weights and reports raw sample percentages as population figures.
Error sourcesinput data; statistical assumptions; source selection
Certainty limitsCertainty drops with high non-response (risk of bias), small subgroup counts (imprecision) and weights that do not match the target population (indirectness).

Evidence and tag basis

Tested. In an ABS evaluation, calibration with auxiliary information lowered relative standard errors by the order of 12.5 per cent and key-item non-response bias from 1.2 to 0.3 per cent. (ABS Methodological News, Dec 2009)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.