HE1-21 Distribution, outlier and base-rate checks
Examines the shape, extremes and baseline frequency of data before a figure is interpreted and produces histograms, summary statistics, flagged outliers and a base rate.
What it does
Examines the shape, extremes and baseline frequency of data before a figure is interpreted. Artefact: histograms, summary statistics, flagged outliers and a base rate; read whether the mean is representative and whether a rare event rate is high only against a small base.
When to use it
Use when you are about to quote a mean or a rare event rate and need to know whether it is representative and what the base is.
When it does not fit
Outliers are deleted without cause.
Result format
The result has histograms, summary statistics, a list of flagged outliers and a base rate for the data. It states whether the mean is representative and whether a rare event rate is high only against a small base.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Structure | Applies in some cases | requires=structured |
| Content type | Always applies | requires=tabular; requires=time series |
| Unit of analysis | Always applies | requires=variable; requires=dataset |
| Variable types | Raises a warning | prefers=interval; prefers=ratio; prefers=counts or fractions |
| Level of detail available | Always applies | requires=aggregate tables |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | none |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Outliers are deleted without cause, or the summary replaces a look at the records behind it. |
Choosing and ordering
| Kind of technique | Preparation technique, runs in step 5.2 |
|---|---|
| Phase tag | verify and prepare |
| Other phase tags | analyse |
| Kind of finding | pattern; risk |
| Question answered | what; how much |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | proceed with limit stated |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: deterministic.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | input validation | none | no |
| decision-informing | input validation | none | no |
| decision-informing | spot check | optional | no |
| high-stakes | input validation | none | no |
| high-stakes | spot check | optional | no |
| high-stakes | independent review | needed | no |
| high-stakes | provenance log | none | no |
Failure modes and limits
| Model suitability | model plus code |
|---|---|
| Typical model failure | Describes distribution shape without computing it, and drops outliers that are real values. |
| Error sources | input data; data transformation; statistical assumptions |
| Certainty limits | Certainty drops when the data are already aggregated, when outliers cannot be traced to records, or when no base rate is available (imprecision). |
Related techniques
- should follow PE-04 Data profiling and rule-based validation (data audit; data quality assessment; edit checks): profiling and rule-based validation catch format errors first
- pairs with MM-02 Algorithmic outlier and anomaly detection (Isolation Forest; Local Outlier Factor; Elliptic Envelope; One-Class SVM (support vector machine)): algorithmic outlier detection extends the flagged outlier list
Evidence and tag basis
Unconfirmed. No single primary source was opened.
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.