Train, Validation and Test Splits Without Data Leakage

Build a trustworthy evaluation boundary

You will compare random, grouped and chronological splitting, then place preprocessing inside cross-validation so each fold remains genuinely unseen.

• Assign distinct roles to training, validation and final test data.
• Choose random, stratified, grouped or time-based splits from deployment conditions.
• Recognise target, temporal, group and preprocessing leakage.
• Use cross-validation to estimate variability without repeatedly tuning on the final test set.

A split is part of the problem definition

Training data fits parameters. Validation data supports model and threshold choices. A final test set estimates the selected procedure once. Repeatedly checking the test result and changing the model turns the test set into another validation set.

Stratification preserves class proportions but does not solve group dependence. Medical visits from the same patient, purchases from the same customer and crops from the same image can leak identity across a random split. Group-aware splitting keeps related records together.

Time changes the question. When a model predicts next month, training on later records and testing on earlier ones is unrealistic. Use a forward split, preserve a gap when labels mature slowly, and rebuild features as they would have existed at each cutoff.

Preprocessing leaks when statistics are calculated before splitting. Global imputation, scaling, feature selection and target encoding all let evaluation data shape the training representation. Fit them inside a pipeline on each training fold.

Cross-validation estimates performance across several partitions. Its folds must follow the same group or time constraints. Report the distribution of scores, not only the mean, and reserve the final test set for the chosen pipeline.

Keep each customer in one side of the split

The example uses grouped cross-validation and an imputer inside a pipeline. The repeated customer IDs never appear in both train and validation within a fold.

Install the lesson dependencies

python -m pip install numpy scikit-learn

Run grouped, leakage-safe evaluation

import numpy as np
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GroupKFold, cross_val_score
from sklearn.pipeline import Pipeline

X = np.array([
    [2.0, 10], [3.0, 11],
    [8.0, 50], [9.0, 55],
    [1.0, 8],  [np.nan, 9],
    [7.0, 48], [8.0, 52],
])
y = np.array([0, 0, 1, 1, 0, 0, 1, 1])
customer = np.array([101, 101, 102, 102, 103, 103, 104, 104])

model = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("classify", LogisticRegression(max_iter=1000)),
])
cv = GroupKFold(n_splits=4)
scores = cross_val_score(model, X, y, groups=customer, cv=cv, scoring="accuracy")

for fold, (train_index, test_index) in enumerate(cv.split(X, y, customer), 1):
    overlap = set(customer[train_index]) & set(customer[test_index])
    print(f"fold {fold} group overlap: {len(overlap)}")
print("scores:", scores.tolist())

Expected boundary check

fold 1 group overlap: 0
fold 2 group overlap: 0
fold 3 group overlap: 0
fold 4 group overlap: 0
scores: <four scores determined by the tiny folds>

The exact scores are not the lesson. The zero overlaps prove the grouping rule. A realistic dataset needs more independent groups and class representation per fold.

Leakage symptoms worth distrusting

Very high performance is not proof of leakage, but it should trigger a feature and split audit before celebration.

• A feature name includes words such as resolved, cancelled, final or outcome.
• Validation collapses when records are split by customer or time instead of randomly.
• A feature is available in the warehouse now but was not available at prediction time.
• Feature selection was run once on the full dataset before cross-validation.

Design three competing split plans

For one project, draw a random split, a group split and a chronological split. Predict which score will be highest and explain which one best matches deployment.

• List the dependency unit that must not cross the boundary.
• Place every learned transformation inside a pipeline.
• Keep one untouched final period or group set.
• Write a test that asserts no prohibited IDs overlap.

Evaluation evidence

• Split rationale tied to deployment.
• Automated overlap test.
• Fold-level scores with counts and dates.

Official references and further reading

• scikit-learn cross-validation (Official split and cross-validation guidance)
• scikit-learn common pitfalls (Official leakage examples)
• scikit-learn pipelines (Official safe composition pattern)

Review note for Train, Validation and Test Splits Without Data Leakage: recheck the linked documentation after a dependency changes the relevant API, metric or modelling assumption, then record the tested version beside your result.