Project: Build and Audit a Leakage-Safe Baseline

What must pass

Use a licensed or synthetic classification dataset. The project must define the deployment boundary, compare a naive baseline and report fold-level evidence.

• Implement a grouped or chronological evaluation that matches the use case.
• Keep every learned transformation inside a pipeline.
• Compare against a dummy strategy and report uncertainty across folds.
• Perform error, subgroup and leakage audits before making a recommendation.

A baseline is an argument about added value

Start with the simplest credible reference. A DummyClassifier can predict the most frequent class or respect observed class proportions. An existing business rule may be a stronger baseline and should be included when available. The new procedure must improve a metric tied to action, not merely beat random guessing on an easy split.

Create a feature availability table with one row per feature. Record source, update time and whether it exists at the intended prediction moment. This audit often finds leakage faster than a statistical test.

Fold-level results reveal instability. Report score, positive count and relevant groups for every fold. A strong mean with one disastrous period may be unacceptable. Keep predictions so errors can be inspected after evaluation without retraining.

Error analysis samples false positives and false negatives with safe identifiers and relevant input fields. Look for definition errors, stale data, subgroup gaps and cases humans also find ambiguous. Do not copy personal data into a public report.

End with a decision. Proceed, revise the data contract or stop. A lower score can still be a successful checkpoint if it exposes an invalid label or deployment mismatch.

Evaluate a real procedure against a dummy

The sample uses stratified folds for a built-in dataset. Replace the splitter with GroupKFold or TimeSeriesSplit when your unit or deployment demands it.

Install the lesson dependencies

python -m pip install scikit-learn

Create out-of-fold baseline evidence

from sklearn.datasets import load_breast_cancer
from sklearn.dummy import DummyClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_validate
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

procedures = {
    "dummy": DummyClassifier(strategy="most_frequent"),
    "logistic": Pipeline([
        ("impute", SimpleImputer(strategy="median")),
        ("scale", StandardScaler()),
        ("model", LogisticRegression(max_iter=5000)),
    ]),
}

for name, procedure in procedures.items():
    result = cross_validate(procedure, X, y, cv=cv,
                            scoring=["balanced_accuracy", "precision", "recall"])
    print(name)
    for metric in ["balanced_accuracy", "precision", "recall"]:
        values = result[f"test_{metric}"]
        print(f" {metric}: mean={values.mean():.3f}, min={values.min():.3f}")

Expected relationship

dummy
 balanced_accuracy: mean=0.500, min=0.500
 precision: <majority-class result>
 recall: <majority-class result>
logistic
 balanced_accuracy: <higher than dummy on this built-in dataset>
 precision: <fold summary>
 recall: <fold summary>

Do not copy these expected relationships to a different dataset. Verify positive-label meaning and select metrics from the real action and error costs.

Audit drills before approval

Treat every surprising success as something to investigate. Keep the audit results beside the model results.

• Remove each suspicious high-signal feature and measure the change.
• Switch from random to group or forward validation and explain any collapse.
• Inspect the worst fold’s dates, sources, class counts and errors.
• Try a permuted target. Performance far above chance can reveal leakage or a broken evaluation.

Complete the checkpoint

Use your chosen dataset and project brief. Do not move to ensembles until this baseline, its failure modes and its rebuild instructions are complete.

• Commit split logic, pipeline and metric configuration.
• Save fold predictions and a redacted error-analysis sample.
• Write feature-availability and subgroup reports.
• Issue a go, revise or stop recommendation with evidence and remaining uncertainty.

Portfolio package

• Baseline comparison table.
• Leakage and error audit.
• One-command rebuild plus model card.

Official references and further reading

• scikit-learn DummyClassifier (Official naive baseline implementation)
• scikit-learn cross_validate (Official multi-metric evaluation API)
• scikit-learn common pitfalls (Official leakage checks)

Review note for Project: Build and Audit a Leakage-Safe Baseline: recheck the linked documentation after a dependency changes the relevant API, metric or modelling assumption, then record the tested version beside your result.