Project: Reduce Dimensions and Validate a Cluster Solution

Cluster project definition of done

Use a non-sensitive public or synthetic dataset. Choose a task where grouping could support exploration, not automated decisions about people.

• Compare raw scaled features with a PCA or SVD representation.
• Evaluate at least k-means and one hierarchical or density method.
• Measure internal quality, stability and segment interpretability.
• Publish limitations and avoid assigning unsupported persona labels.

Validation needs three kinds of evidence

Internal metrics measure geometry. Silhouette, Davies-Bouldin and within-cluster dispersion help compare candidate partitions under the same representation. They do not prove business usefulness and can favour compact shapes.

Stability measures whether a result persists across seeds, samples or small perturbations. Compare pairwise co-assignment or adjusted Rand after aligning sample coverage. Unstable clusters may still expose a continuum, but they should not be presented as fixed natural categories.

External or domain evidence asks whether groups differ on information not used to create them and whether that difference supports a legitimate action. Hold the external field out of clustering to avoid circular validation. A difference can be statistically detectable yet operationally trivial.

PCA supports inspection of variance and lower-dimensional visualization. Fit it inside the analysis boundary and report loadings. If a component is dominated by one scale or data-quality artifact, fix the representation rather than naming the component creatively.

SVD is often preferable for sparse matrices such as text or transactions. Keep the sparse input sparse and track explained variance, reconstruction limits and memory.

Create a reproducible cluster comparison

The built-in wine dataset is used for mechanics. Its known classes are withheld from fitting and used only for an optional external comparison.

Install the lesson dependencies

python -m pip install pandas scikit-learn

Compare scaled and PCA spaces

import pandas as pd
from sklearn.cluster import KMeans
from sklearn.datasets import load_wine
from sklearn.decomposition import PCA
from sklearn.metrics import silhouette_score, adjusted_rand_score
from sklearn.preprocessing import StandardScaler

X, known = load_wine(return_X_y=True)
scaled = StandardScaler().fit_transform(X)
pca = PCA(n_components=0.90, random_state=42).fit(scaled)
spaces = {"scaled": scaled, "pca90": pca.transform(scaled)}

rows = []
for space_name, matrix in spaces.items():
    for seed in [1, 7, 42]:
        labels = KMeans(n_clusters=3, n_init=20, random_state=seed).fit_predict(matrix)
        rows.append({
            "space": space_name,
            "seed": seed,
            "dimensions": matrix.shape[1],
            "silhouette": silhouette_score(matrix, labels),
            "external_ari": adjusted_rand_score(known, labels),
        })

report = pd.DataFrame(rows)
print("PCA components:", pca.n_components_)
print(report.round(3).to_string(index=False))

Expected report shape

PCA components: <number selected to retain at least 90% variance>
space  seed  dimensions  silhouette  external_ari
scaled    1          13       <value>         <value>
... six rows total ...

The known classes are not proof of the correct clustering. Adjusted Rand only shows agreement with that external labelling, which may represent a different purpose.

Required challenge tests

A polished cluster chart is not enough. Run these tests and keep the failures.

• Bootstrap the dataset and compare co-assignment for shared records.
• Remove each high-loading feature group and note structural changes.
• Compare at least two candidate cluster counts or density settings.
• Inspect nearest-to-centroid records and boundary cases in original units.

Complete the cluster portfolio project

Write a report that begins with the exploratory question, not the algorithm name.

• Document preprocessing and why each feature belongs.
• Show candidate methods and parameters, including rejected solutions.
• Describe clusters with distributions and uncertainty rather than fictional personalities.
• State whether the result supports exploration, a limited pilot or no action.

Checkpoint evidence

• Representation and algorithm comparison.
• Stability and external-evidence report.
• Reproducible notebook or script plus cautious conclusion.

Official references and further reading

• scikit-learn clustering evaluation (Official internal and external metric guidance)
• scikit-learn PCA (Official component and variance guidance)
• scikit-learn adjusted Rand (Official chance-adjusted agreement definition)

Review note for Project: Reduce Dimensions and Validate a Cluster Solution: recheck the linked documentation after a dependency changes the relevant API, metric or modelling assumption, then record the tested version beside your result.