Choose Evaluation Metrics for Classification and Clustering

Read the confusion matrix before the headline score

You will calculate classification metrics at two thresholds and a silhouette score for clusters, then explain what each number leaves out.

• Derive precision, recall, specificity and F1 from a confusion matrix.
• Distinguish threshold metrics from ranking metrics such as ROC AUC and average precision.
• Evaluate imbalanced problems with baselines, class counts and operational capacity.
• Use internal and external clustering metrics without confusing compactness with usefulness.

A metric is a view of errors

Precision is the share of predicted positives that are correct. Recall is the share of actual positives found. Lowering a decision threshold often raises recall and lowers precision. F1 balances them through a harmonic mean, but it does not include true negatives or business cost.

ROC AUC measures ranking across false-positive and true-positive rates. It can look strong when positives are rare while the useful top of the ranked list has poor precision. Precision-recall curves focus on positive predictions, but their baseline depends on prevalence, so compare results on the intended population.

Calibration asks whether a score of 0.7 corresponds to an observed rate near 70 percent. Ranking and calibration are different qualities. A well-ranked model can have probabilities that need calibration before cost calculations.

Cluster evaluation has no universal ground truth. Silhouette compares within-cluster cohesion with separation from the nearest other cluster. It tends to prefer compact, convex groups. Dunn and Davies-Bouldin use other geometric ratios. External metrics such as adjusted Rand compare to known labels, but those labels may represent a different concept than the discovered structure.

Always report uncertainty and slices. A single metric can hide performance changes over time, source or group. Avoid selecting many slices after seeing results and reporting only the convenient ones.

Move a threshold and watch the errors change

The labels and scores stay fixed while the threshold changes. This isolates the decision policy from the ranking model.

Install the lesson dependencies

python -m pip install numpy scikit-learn

Compare two classification thresholds

import numpy as np
from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score

y_true = np.array([0, 0, 0, 0, 1, 1, 1, 1])
score = np.array([0.05, 0.20, 0.45, 0.55, 0.35, 0.60, 0.75, 0.90])

for threshold in [0.50, 0.30]:
    prediction = (score >= threshold).astype(int)
    tn, fp, fn, tp = confusion_matrix(y_true, prediction).ravel()
    print(f"threshold={threshold:.2f}")
    print(f" tn={tn} fp={fp} fn={fn} tp={tp}")
    print(f" precision={precision_score(y_true, prediction):.3f}")
    print(f" recall={recall_score(y_true, prediction):.3f}")
    print(f" f1={f1_score(y_true, prediction):.3f}")

Expected trade-off

threshold=0.50
 tn=3 fp=1 fn=1 tp=3
 precision=0.750
 recall=0.750
 f1=0.750
threshold=0.30
 tn=2 fp=2 fn=0 tp=4
 precision=0.667
 recall=1.000
 f1=0.800

The lower threshold finds every positive in this sample but sends more false positives to review. Whether that is better depends on the cost and capacity, not F1 alone.

Metric bugs that create confident nonsense

Verify positive-label meaning, score direction and sample population before comparing decimals.

• A label encoded as zero may be the event of interest while a library assumes one is positive.
• Computing metrics after resampling the test set changes prevalence and can distort operational interpretation.
• Choosing the threshold on the final test set uses test information for tuning.
• Silhouette is undefined or unhelpful for one cluster and can reward a solution that has no domain value.

Write a metric decision memo

Choose fraud review, fault detection or ticket routing. Define one primary metric, one safety or capacity metric and one diagnostic plot.

• Create a small cost table for false positives and false negatives.
• Evaluate at a fixed capacity and at two thresholds.
• Report counts beside rates.
• Name a subgroup or time slice that must be monitored and why.

What to save

• Threshold table with confusion-matrix counts.
• Metric rationale tied to action.
• One example where a higher headline metric would be rejected.

Official references and further reading

• scikit-learn classification metrics (Official definitions and limitations)
• scikit-learn clustering performance (Official cluster evaluation guidance)
• scikit-learn calibration (Official probability-calibration guidance)

Review note for Choose Evaluation Metrics for Classification and Clustering: recheck the linked documentation after a dependency changes the relevant API, metric or modelling assumption, then record the tested version beside your result.