Project: Benchmark Classifiers and Diagnose Their Errors

Benchmark acceptance criteria

Use your project brief and leakage-safe split. Include a dummy, one linear model, one single tree and one ensemble.

• Run every candidate through a valid pipeline and identical outer folds.
• Record primary metric, worst fold, fit time, prediction time and artefact size.
• Tune inside training data only and preserve an untouched final test.
• Inspect errors, calibration and relevant subgroup performance before selection.

A fair benchmark controls everything except the candidate

Freeze the split IDs and metric configuration. If each model sees different folds, score differences combine model and sample differences. Nested validation is appropriate when extensive tuning would otherwise reuse outer validation folds.

Include a naive baseline and the current operational rule when possible. A complex model that adds one decimal point but triples latency, monitoring burden or explanation cost may be the wrong choice.

Generate out-of-fold predictions for error analysis. Assign each record to exactly one validation prediction. This lets you build a confusion table, threshold curve and subgroup report without training on the records being explained.

Review false positives and negatives by cause. Some are irreducible ambiguity, some reveal label defects, and some cluster around a source or time period. If errors expose an invalid target, repair the problem definition rather than tuning around it.

Final evaluation comes after the procedure, threshold and checks are fixed. Report it once with the sample period and counts. A disappointing final result should remain visible.

Create a reproducible comparison table

This code provides the benchmark spine. Extend it with your deployment-matched splitter and out-of-fold predictions.

Install the lesson dependencies

python -m pip install pandas scikit-learn

Compare four candidate families

import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_validate
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
models = {
    "dummy": DummyClassifier(strategy="most_frequent"),
    "logistic": make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000)),
    "tree": DecisionTreeClassifier(max_depth=4, min_samples_leaf=8, random_state=42),
    "forest": RandomForestClassifier(n_estimators=300, min_samples_leaf=3,
                                      random_state=42, n_jobs=-1),
}

rows = []
for name, model in models.items():
    result = cross_validate(model, X, y, cv=cv,
                            scoring="balanced_accuracy",
                            return_train_score=True)
    rows.append({
        "model": name,
        "validation_mean": result["test_score"].mean(),
        "validation_min": result["test_score"].min(),
        "train_mean": result["train_score"].mean(),
        "fit_seconds": result["fit_time"].sum(),
    })

report = pd.DataFrame(rows).sort_values("validation_mean", ascending=False)
print(report.round(3).to_string(index=False))

Expected report columns

  model  validation_mean  validation_min  train_mean  fit_seconds
<candidate rows sorted by validation_mean; exact values depend on versions and hardware>

A large train-to-validation gap is one warning, not an automatic disqualification. Use fold behaviour and error evidence to find its cause.

Required stress tests

Do not publish a selected model before these checks have recorded outcomes.

• Repeat the comparison with group or forward splits if deployment requires them.
• Change the random seed and report whether the ranking is stable.
• Evaluate the chosen threshold under realistic class prevalence and capacity.
• Remove one suspicious feature group and explain the performance change.

Complete the classifier checkpoint

Package the benchmark, selected model and decision record so another person can reproduce and challenge them.

• Save the split identifiers, configuration and version information.
• Create a redacted error taxonomy with representative cases.
• Report subgroup counts, metrics and uncertainty where appropriate and lawful.
• State deploy, limited pilot, revise or stop, plus the evidence needed for the next decision.

Portfolio deliverables

• Benchmark table and configuration.
• Error and calibration report.
• Selection memo with rejected alternatives.

Official references and further reading

• scikit-learn model evaluation (Official metric and scoring reference)
• scikit-learn nested versus non-nested CV (Official tuning-bias example)
• scikit-learn calibration (Official calibration evaluation)

Review note for Project: Benchmark Classifiers and Diagnose Their Errors: recheck the linked documentation after a dependency changes the relevant API, metric or modelling assumption, then record the tested version beside your result.