PE-04 Data profiling and rule-based validation (data audit; data quality assessment; edit checks)
Describe the structure, completeness, validity and consistency of each field and test explicit rules, giving a profile table per column and a list of records that break each rule.
What it does
Describes the structure, completeness, validity and consistency of each field and tests explicit rules such as end date after start date. Artefact: a profile table per column and a list of records that break each rule. Read for missing patterns, impossible values, duplicate keys and inconsistent codes.
When to use it
Use when you have received a table or dataset and need to know its quality before any analysis.
When it does not fit
Always applicable as a first step. Profiling shows that a value is odd, not that it is wrong. The analyst chooses the rule set, and it is never complete.
Result format
The result has a profile table with one row per column, giving its type, completeness, valid range and consistency, and a list of the records that break each stated rule. It ends by naming the missing patterns, impossible values, duplicate keys and inconsistent codes found.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Structure | Applies in some cases | requires=structured; requires=semi-structured |
| Content type | Always applies | requires=tabular |
| Unit of analysis | Always applies | requires=dataset; requires=variable |
| Level of detail available | Always applies | requires=record-level |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | none |
|---|---|
| Additional sources needed | reference structure |
| Not for (rules) | none |
| Not for (text) | The profile is read as proof that a value is wrong; it shows only that a value is unusual, and the rule set chosen by the analyst is never complete. |
Choosing and ordering
| Kind of technique | Preparation technique, runs in step 5.2 |
|---|---|
| Phase tag | verify and prepare |
| Other phase tags | none |
| Kind of finding | gap; contradiction; risk |
| Question answered | what |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | return to gather |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: deterministic.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | input validation | none | no |
| decision-informing | input validation | none | no |
| decision-informing | spot check | optional | no |
| decision-informing | provenance log | none | no |
| high-stakes | input validation | none | no |
| high-stakes | spot check | optional | no |
| high-stakes | provenance log | none | no |
| high-stakes | human review sample | needed | no |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | model plus code |
|---|---|
| Typical model failure | Reports that odd values are errors and writes a short rule set as if it were complete. |
| Error sources | input data; data transformation |
| Certainty limits | Certainty about quality drops when the rule set is short (reporting bias) or when field definitions are undocumented (indirectness). |
Related techniques
- pairs with HE1-21 Distribution, outlier and base-rate checks: distribution and outlier checks examine the shape of values that profiling flags
- pairs with MM-02 Algorithmic outlier and anomaly detection (Isolation Forest; Local Outlier Factor; Elliptic Envelope; One-Class SVM (support vector machine)): algorithmic outlier detection finds unusual values that explicit rules miss
Evidence and tag basis
Documented. The Australian Bureau of Statistics (ABS) framework defines seven quality dimensions. RADAR names five artefact classes: missing data, bad values, outliers, inconsistent formatting and inconsistent logic. The column profile layout is practice with no authoritative definition. (ABS data quality framework; RADAR, arXiv 2506.08249)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.