Treat Missing Values, Noise and Outliers Without Hiding Them

Make a treatment decision you can defend

You will compare simple imputation strategies, add missingness indicators and inspect outliers in context before changing them.

• Profile missingness by source, time and relevant subgroup.
• Compare deletion, constant, median, hot-deck and model-based imputation as assumptions rather than automatic fixes.
• Use robust summaries and domain limits to investigate extreme values.
• Keep cleaning parameters fitted on training data only.

Missingness and rarity carry information

Dropping every incomplete row is reasonable only when the loss is small and unlikely to distort the population. If older devices omit one sensor, complete-case analysis can quietly remove that entire generation. Print the retained share and compare group distributions before accepting deletion.

Mean or median imputation preserves row count but reduces natural variation. Median is usually more stable under skew. Hot-deck methods borrow an observed value from a similar record. Predictive imputation uses other fields. Each method adds assumptions, and none recreates unknown truth.

A missingness indicator can preserve whether a value was absent. This helps when absence reflects a process, such as a test ordered only for severe cases. It can also encode an operational shortcut that later changes, so monitor it like any other feature.

Noise is unwanted variation, not simply variation. Binning and regression smoothing may clarify a signal but can erase short events. Domain constraints are safer for impossible values such as a negative duration. Statistical fences are investigation aids, not universal deletion rules.

Fit imputation, clipping and scaling on the training set. Calculating a median across all records lets evaluation data influence training. A scikit-learn pipeline makes the boundary explicit.

Compare median imputation with an indicator

The example keeps the original age column, applies the learned median and adds a flag. It also prints extreme spend rows for review instead of deleting them.

Install the lesson dependencies

python -m pip install pandas scikit-learn

Fit treatment on training rows

import pandas as pd
from sklearn.impute import SimpleImputer

train = pd.DataFrame({
    "age": [24, 31, None, 46, 38],
    "monthly_spend": [35, 42, 39, 51, 900],
})
future = pd.DataFrame({
    "age": [None, 29],
    "monthly_spend": [47, 44],
})

imputer = SimpleImputer(strategy="median", add_indicator=True)
train_ready = imputer.fit_transform(train[["age"]])
future_ready = imputer.transform(future[["age"]])

q1 = train["monthly_spend"].quantile(0.25)
q3 = train["monthly_spend"].quantile(0.75)
upper_fence = q3 + 1.5 * (q3 - q1)
review = train[train["monthly_spend"] > upper_fence]

print("learned age median:", imputer.statistics_[0])
print("future [age, was_missing]:", future_ready.tolist())
print("spend review rows:", review.index.tolist())

Expected result

learned age median: 34.5
future [age, was_missing]: [[34.5, 1.0], [29.0, 0.0]]
spend review rows: [4]

Row 4 is a review candidate, not a confirmed error. If it is a legitimate high-value customer, deleting it would damage the analysis.

Cleaning failures to investigate first

Compare the raw and transformed distributions and keep a row-level change log when transformations affect decisions.

• A column entirely missing in a training split may need a schema rule rather than a calculated statistic.
• Using zero for missing can confuse absence with a legitimate zero.
• Capping all extremes at an IQR fence can erase the target signal in fraud or fault detection.
• Imputation performed before cross-validation leaks fold information and inflates evaluation.

Create a cleaning decision table

For a synthetic customer or sensor dataset, classify five data-quality issues and test at least two treatments for one numerical field.

• Record the observed issue, suspected cause, chosen action and reversible output column.
• Report missingness before and after the transformation by one meaningful subgroup.
• Keep a boolean indicator for each value changed or imputed.
• Explain one case that must be escalated to a domain owner.

Evidence for review

• Raw-versus-processed summary table.
• Cleaning decision log.
• One test proving the input file remains unchanged.

Official references and further reading

• pandas missing data (Official missing-value semantics)
• scikit-learn imputation (Official imputation methods and indicators)
• scikit-learn outlier detection (Official distinction between outlier and novelty detection)

Review note for Treat Missing Values, Noise and Outliers Without Hiding Them: recheck the linked documentation after a dependency changes the relevant API, metric or modelling assumption, then record the tested version beside your result.