startthat.work home

HE1-21 Distribution, outlier and base-rate checks

Examines the shape, extremes and baseline frequency of data before a figure is interpreted and produces histograms, summary statistics, flagged outliers and a base rate.

Handle evidenceEvidence: UnconfirmedPreparation technique, runs in step 5.2Strongest statement: describe

What it does

Examines the shape, extremes and baseline frequency of data before a figure is interpreted. Artefact: histograms, summary statistics, flagged outliers and a base rate; read whether the mean is representative and whether a rare event rate is high only against a small base.

When to use it

Use when you are about to quote a mean or a rare event rate and need to know whether it is representative and what the base is.

When it does not fit

Outliers are deleted without cause.

Result format

The result has histograms, summary statistics, a list of flagged outliers and a base rate for the data. It states whether the mean is representative and whether a rare event rate is high only against a small base.

Requirements on the material

PropertyRequirement typeRequirement
StructureApplies in some casesrequires=structured
Content typeAlways appliesrequires=tabular; requires=time series
Unit of analysisAlways appliesrequires=variable; requires=dataset
Variable typesRaises a warningprefers=interval; prefers=ratio; prefers=counts or fractions
Level of detail availableAlways appliesrequires=aggregate tables
Appraises source credibilityApplies in some casesno

Not restricted on: Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialnone
Additional sources needednone
Not for (rules)none
Not for (text)Outliers are deleted without cause, or the summary replaces a look at the records behind it.

Choosing and ordering

Kind of techniquePreparation technique, runs in step 5.2
Phase tagverify and prepare
Other phase tagsanalyse
Kind of findingpattern; risk
Question answeredwhat; how much
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetproceed with limit stated

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: deterministic.

StakesCheckPerson involvedCounts as a different method
exploratoryinput validationnoneno
decision-informinginput validationnoneno
decision-informingspot checkoptionalno
high-stakesinput validationnoneno
high-stakesspot checkoptionalno
high-stakesindependent reviewneededno
high-stakesprovenance lognoneno

Failure modes and limits

Model suitabilitymodel plus code
Typical model failureDescribes distribution shape without computing it, and drops outliers that are real values.
Error sourcesinput data; data transformation; statistical assumptions
Certainty limitsCertainty drops when the data are already aggregated, when outliers cannot be traced to records, or when no base rate is available (imprecision).

Evidence and tag basis

Unconfirmed. No single primary source was opened.

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.