CK-09 Cluster validation indexes (silhouette score; adjusted Rand index; adjusted mutual information; V-measure; stability analysis)
Score how well a clustering fits the data or agrees with known labels, and produce a score per clustering for comparison across cluster numbers and algorithms.
What it does
Scores how well a clustering fits the data or agrees with known labels. Artefact: a score per clustering. Read relative to alternatives such as other numbers of clusters.
When to use it
Use when you have run one or more clusterings and need to choose among them or judge whether any is good.
When it does not fit
Compare several cluster numbers and algorithms. External indices need ground-truth labels, and the adjusted Rand index is near 0 for random labellings. The silhouette section of the documentation was not read.
Result format
The result is a table with one row per clustering, such as each number of clusters or algorithm tried, and a column for each validation score. It names the clustering that scores best relative to the alternatives and notes which scores need ground-truth labels.
Requirements on the material
| Property | Requirement type | Requirement |
|---|---|---|
| Content type | Always applies | requires=text; requires=tabular |
| Unit of analysis | Always applies | requires=record; requires=document |
| Level of detail available | Always applies | requires=record-level |
| Appraises source credibility | Applies in some cases | no |
Not restricted on: Structure, Variable types, Time structure, How the sample was chosen, Number of sources, Whether sources can be joined, Amount of material, Known data problems, Producer interest, Source dependence, Origin, Document genre, Language, Personal information, Security classification, Licence.
| Minimum material | at least two clusterings to compare, and ground-truth labels for the external indices |
|---|---|
| Additional sources needed | none |
| Not for (rules) | none |
| Not for (text) | Wrong choice when only one clustering exists and no labels are available, because a score then has nothing to be read against. The external indices need ground-truth labels. |
Choosing and ordering
| Kind of technique | Verification technique, runs in step 6 |
|---|---|
| Phase tag | validate |
| Other phase tags | analyse |
| Kind of finding | option ranking |
| Question answered | which option |
| Strongest statement it supports | describe |
| Needs as input |
|
| Makes as output |
|
| Must exist first | None |
| Loop target | proceed with limit stated |
Library checks by stakes
Each stakes level includes the checks of the level below it. Checks that need a person apply at high stakes only, so the process leaves them out at lower stakes (decision 38). Repeatability: repeatable with fixed settings.
| Stakes | Check | Person involved | Counts as a different method |
|---|---|---|---|
| exploratory | input validation | none | no |
| exploratory | repeat runs | none | no |
| decision-informing | input validation | none | no |
| decision-informing | repeat runs | none | no |
| decision-informing | sensitivity analysis | none | no |
| high-stakes | input validation | none | no |
| high-stakes | repeat runs | none | no |
| high-stakes | sensitivity analysis | none | no |
| high-stakes | second technique with different assumptions | none | conditional |
| high-stakes | independent review | needed | no |
Failure modes and limits
| Model suitability | model plus code |
|---|---|
| Typical model failure | Reports a score without computing it. |
| Error sources | input data; statistical assumptions |
| Certainty limits | Certainty drops when only one clustering is scored (indirectness), when scores change across restarts (inconsistency) and when labels used as truth are weak (risk of bias). |
Related techniques
- should follow MM-03 Clustering of records (k-means; hierarchical (agglomerative) clustering; DBSCAN (density-based spatial clustering of applications with noise); HDBSCAN (hierarchical DBSCAN)): record clusters are what is scored
- should follow MM-14 Document clustering (computer-assisted clustering; k-means on document vectors): document clusters are what is scored
- should follow MM-13 Embedding-based topic modelling (BERTopic; Top2Vec; clustering-based topic models): embedding topic clusters are what is scored
Evidence and tag basis
Documented. External metrics are documented in the clustering guide. (scikit-learn clustering)
Tag basis: input profile inference; selection inference; sequencing inference; robustness inference. All tag values are inference unless a source is named.