Hierarchical Clustering, Linkage and Dendrograms

Read a hierarchy without treating the dendrogram as truth

You will generate a linkage matrix, inspect merge distances and cut the hierarchy into clusters.

• Distinguish agglomerative and divisive clustering.
• Explain single, complete, average and Ward linkage.
• Interpret dendrogram merge height and leaf order correctly.
• Evaluate a cut with stability, geometry and domain evidence.

Linkage defines distance between groups

Agglomerative clustering begins with one cluster per record and repeatedly merges the closest pair. Divisive clustering starts with all records together and splits them. Agglomerative methods are more widely available in standard Python tooling.

Single linkage uses the closest pair between clusters and can find irregular chains, but one bridge of points can connect distant groups. Complete linkage uses the farthest pair and tends to create compact clusters. Average linkage averages pairwise distances. Ward linkage merges the pair causing the smallest increase in within-cluster variance and is tied to Euclidean geometry.

A dendrogram’s vertical merge height reflects dissimilarity under the chosen method. Horizontal leaf order is often rearranged for readability and does not itself measure distance. A long vertical gap can suggest a cut, but it is not an automatic proof of the correct number of clusters.

Hierarchical clustering can be expensive because pairwise relationships grow quickly with records. Sample thoughtfully or use scalable alternatives for large data. Never fit a small convenience sample and present its hierarchy as the whole population without explaining the sampling design.

Once two groups merge in standard agglomerative clustering, the merge is not undone. Early noise can affect the whole tree. Compare linkage methods and bootstrap samples to see which relationships persist.

Inspect the linkage matrix

SciPy’s linkage output records the merged cluster IDs, merge distance and new cluster size. This numeric view is testable even without drawing a chart.

Install the lesson dependencies

python -m pip install numpy scipy scikit-learn

Build a Ward hierarchy and cut it

import numpy as np
from scipy.cluster.hierarchy import linkage, fcluster
from sklearn.preprocessing import StandardScaler

X = np.array([
    [1.0, 1.0], [1.2, 0.8], [0.8, 1.1],
    [5.0, 5.0], [5.1, 4.8], [4.9, 5.2],
])
X_ready = StandardScaler().fit_transform(X)
tree = linkage(X_ready, method="ward")
labels = fcluster(tree, t=2, criterion="maxclust")

print("linkage shape:", tree.shape)
print("last merge distance:", round(float(tree[-1, 2]), 3))
print("two-cluster labels:", labels.tolist())
print("cluster sizes:", np.bincount(labels)[1:].tolist())

Expected hierarchy facts

linkage shape: (5, 4)
last merge distance: <positive value larger than early within-group merges>
two-cluster labels: <three matching labels followed by three matching labels>
cluster sizes: [3, 3] in label-dependent order

Six starting points require five merges, so the linkage matrix has five rows. Label numbers may swap without changing the partition.

Dendrogram interpretation errors

A visually attractive tree can still encode a poor distance choice or a sample artefact.

• Ward linkage with a non-Euclidean precomputed distance violates its variance interpretation.
• Single-link chaining can unite groups through a thin bridge of noise.
• Reading horizontal spacing between leaves as distance is incorrect.
• Naming clusters before examining their feature distributions encourages confirmation bias.

Compare four linkages

Use a dataset with one bridge between two dense groups. Fit single, complete, average and Ward linkage.

• Record the cut rule instead of choosing clusters by eye after seeing labels.
• Compare cluster sizes and silhouette for the same requested count.
• Bootstrap rows and note which pairs consistently stay together.
• Explain which linkage’s geometry best matches the task.

Evidence to publish

• Annotated dendrogram with merge height meaning.
• Linkage comparison table.
• Stability note and one ambiguous record.

Official references and further reading

• SciPy hierarchical clustering (Official linkage and dendrogram API)
• scikit-learn hierarchical clustering (Official method comparison and constraints)

Review note for Hierarchical Clustering, Linkage and Dendrograms: recheck the linked documentation after a dependency changes the relevant API, metric or modelling assumption, then record the tested version beside your result.