startthat.work home

MM-01 Exploratory data analysis with plots and distribution checks (EDA (exploratory data analysis); summary statistics; Q-Q plot; scatterplot matrix)

Examine a cleaned table with plots and summary statistics to find shape, outliers, trends, subgroup differences and broken assumptions, and produce a draft result of plots and summaries.

Measure and modelEvidence: DocumentedPrimary technique, runs in step 5.3Strongest statement: describe

What it does

Examines a cleaned table with plots and summaries to find shape, outliers, trends, subgroup differences and violated assumptions before modelling. Artefact: plots (histogram, box plot, scatterplot matrix, run sequence plot) and summary statistics. Read for structure that summaries hide.

When to use it

Use when you have a cleaned table and want to see what it contains before choosing or fitting any model.

When it does not fit

Use before modelling. Treat findings as hypotheses, because reporting the most striking exploratory finding as a test result is the misuse. Plots of very large tables overplot and need aggregation.

Result format

The result has plots of the cleaned table, such as histograms, box plots, scatterplot matrices and run sequence plots, and a table of summary statistics. It ends with a list of the shapes, outliers, trends, subgroup differences and violated assumptions found, each stated as a hypothesis.

Requirements on the material

PropertyRequirement typeRequirement
StructureApplies in some casesrequires=structured
Content typeAlways appliesrequires=tabular; requires=time series
Unit of analysisAlways appliesrequires=dataset; requires=variable
Level of detail availableAlways appliesrequires=record-level
Appraises source credibilityApplies in some casesno

Not restricted on: Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.

Minimum materialat least one variable and enough records to show a distribution, usually 30 or more
Additional sources needednone
Not for (rules)none
Not for (text)Wrong choice when the most striking exploratory finding is going to be reported as a tested result. Very large tables overplot and need aggregation before plotting.

Choosing and ordering

Kind of techniquePrimary technique, runs in step 5.3
Phase taganalyse
Other phase tagsverify and prepare
Kind of findingpattern; risk; difference
Question answeredwhat
Strongest statement it supportsdescribe
Needs as input
Makes as output
Must exist firstNone
Loop targetproceed with limit stated

Library checks by stakes

Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: deterministic.

StakesCheckPerson involvedCounts as a different method
exploratoryinput validationnoneno
decision-informinginput validationnoneno
decision-informingspot checkoptionalno
high-stakesinput validationnoneno
high-stakesspot checkoptionalno
high-stakesassumption and limitation statementnoneno
high-stakesindependent reviewneededno

Failure modes and limits

Model suitabilityhuman judgement required
Typical model failureDescribes plots it did not generate, computes summary statistics by mental arithmetic, or invents an explanation for noise.
Error sourcesinput data; data transformation; statistical assumptions
Certainty limitsCertainty drops when findings are selected after looking (risk of bias), when overplotting obscures structure in plots (imprecision) and when the table is not representative of the question (indirectness).

Evidence and tag basis

Documented. Anscombe's four datasets share a mean, variance, correlation of 0.816 and fitted line, yet differ when plotted. The chapter contents from the National Institute of Standards and Technology (NIST) list the plots and tests. Chart generation was the lowest-scoring category in TableBench (GPT-4-Turbo 62.00 against 86.3 for humans). (NIST EDA chapter contents; Wikipedia: Anscombe's quartet; TableBench, arXiv 2408.09174)

Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.