MM-01 Exploratory data analysis with plots and distribution checks (EDA (exploratory data analysis); summary statistics; Q-Q plot; scatterplot matrix)
Examine a cleaned table with plots and summary statistics to find shape, outliers, trends, subgroup differences and broken assumptions, and produce a draft result of plots and summaries.
What it does
Examines a cleaned table with plots and summaries to find shape, outliers, trends, subgroup differences and violated assumptions before modelling. Artefact: plots (histogram, box plot, scatterplot matrix, run sequence plot) and summary statistics. Read for structure that summaries hide.
When to use it
Use when you have a cleaned table and want to see what it contains before choosing or fitting any model.
When it does not fit
Use before modelling. Treat findings as hypotheses, because reporting the most striking exploratory finding as a test result is the misuse. Plots of very large tables overplot and need aggregation.
Result format
The result has plots of the cleaned table, such as histograms, box plots, scatterplot matrices and run sequence plots, and a table of summary statistics. It ends with a list of the shapes, outliers, trends, subgroup differences and violated assumptions found, each stated as a hypothesis.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Structure | Applies in some cases | requires=structured |
| Content type | Always applies | requires=tabular; requires=time series |
| Unit of analysis | Always applies | requires=dataset; requires=variable |
| Level of detail available | Always applies | requires=record-level |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | at least one variable and enough records to show a distribution, usually 30 or more |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Wrong choice when the most striking exploratory finding is going to be reported as a tested result. Very large tables overplot and need aggregation before plotting. |
Choosing and ordering
| Kind of technique | Primary technique, runs in step 5.3 |
|---|---|
| Phase tag | analyse |
| Other phase tags | verify and prepare |
| Kind of finding | pattern; risk; difference |
| Question answered | what |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | proceed with limit stated |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: deterministic.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | input validation | none | no |
| decision-informing | input validation | none | no |
| decision-informing | spot check | optional | no |
| high-stakes | input validation | none | no |
| high-stakes | spot check | optional | no |
| high-stakes | assumption and limitation statement | none | no |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | human judgement required |
|---|---|
| Typical model failure | Describes plots it did not generate, computes summary statistics by mental arithmetic, or invents an explanation for noise. |
| Error sources | input data; data transformation; statistical assumptions |
| Certainty limits | Certainty drops when findings are selected after looking (risk of bias), when overplotting obscures structure in plots (imprecision) and when the table is not representative of the question (indirectness). |
Related techniques
- should follow PE-04 Data profiling and rule-based validation (data audit; data quality assessment; edit checks): profiling and rule checks give the quality findings that plots then explore
- pairs with MM-02 Algorithmic outlier and anomaly detection (Isolation Forest; Local Outlier Factor; Elliptic Envelope; One-Class SVM (support vector machine)): plots show outliers and the algorithm scores each record
- pairs with CK-13 Multiple-testing correction (Benjamini-Hochberg false discovery rate control; Bonferroni; Holm): findings from many exploratory comparisons need multiple-testing correction before they are treated as tests
- substitutes for HE1-21 Distribution, outlier and base-rate checks: when the material is record-level tabular data and a full set of plots is wanted beyond distribution and base-rate checks
Evidence and tag basis
Documented. Anscombe's four datasets share a mean, variance, correlation of 0.816 and fitted line, yet differ when plotted. The chapter contents from the National Institute of Standards and Technology (NIST) list the plots and tests. Chart generation was the lowest-scoring category in TableBench (GPT-4-Turbo 62.00 against 86.3 for humans). (NIST EDA chapter contents; Wikipedia: Anscombe's quartet; TableBench, arXiv 2408.09174)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.