Scale, Discretize and Reduce Features with Evidence

Choose a transformation by its effect

You will compare standardization, min-max scaling, discretization and PCA, then record what each method preserves and loses.

• Explain min-max, z-score and robust scaling assumptions.
• Use equal-width and equal-frequency bins without pretending continuous differences disappear.
• Distinguish feature selection from PCA and SVD feature extraction.
• Fit every transformation on training data and measure its downstream effect.

Transformation should follow the model and meaning

Min-max scaling maps the observed training range to a fixed interval, commonly zero to one. Future values can fall outside that interval. Standardization subtracts the training mean and divides by the training standard deviation. It is useful for many distance and linear methods, but outliers can influence both parameters. Robust scaling uses quantiles and can be steadier under heavy tails.

Discretization turns a continuous field into intervals. Equal-width bins are easy to explain but can leave most records in one bin under skew. Equal-frequency bins balance counts but give intervals different widths. Supervised methods such as ChiMerge repeatedly combine neighbouring intervals whose class distributions are statistically similar. Because ChiMerge uses the target, fit its boundaries only on training data and carry the learned intervals into validation or test data. Fitting them before the split leaks label information.

Feature selection keeps original columns. It may use domain knowledge, low-variance removal, univariate association or model-based importance. PCA creates orthogonal components that capture variance. The largest variance is not necessarily the most predictive or fair signal, so validate the task rather than selecting components solely by a percentage.

SVD can reduce sparse matrices without centring them, which is useful for text and transaction features. Dense PCA can make a sparse matrix consume far more memory. Check the algorithm’s input assumptions before transforming a large sparse table.

Decimal scaling exists in classic data-mining curricula, but standard, robust and min-max scalers are usually clearer in modern Python workflows. Teach it as a concept, not a default production choice.

Compare scaling and two principal components

The sample has features on different scales. PCA is fitted after standardization so neither raw unit dominates merely because of magnitude.

Install the lesson dependencies

python -m pip install numpy scikit-learn

Transform a small training matrix

import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import MinMaxScaler, StandardScaler, KBinsDiscretizer

X = np.array([
    [20, 1],
    [35, 2],
    [50, 4],
    [65, 8],
    [80, 9],
], dtype=float)

standard = StandardScaler().fit_transform(X)
minmax = MinMaxScaler().fit_transform(X)
bins = KBinsDiscretizer(n_bins=3, strategy="quantile", encode="ordinal",
                         quantile_method="averaged_inverted_cdf").fit_transform(X[:, [0]])
pca = PCA(n_components=2).fit(standard)

print("standard means:", np.round(standard.mean(axis=0), 8).tolist())
print("minmax first row:", minmax[0].tolist())
print("age bins:", bins.ravel().astype(int).tolist())
print("explained variance sum:", round(pca.explained_variance_ratio_.sum(), 3))

Expected transformation checks

standard means: [0.0, 0.0]
minmax first row: [0.0, 0.0]
age bins: [0, 0, 1, 2, 2]
explained variance sum: 1.0

Two components retain all variance because the input has two features. Reduction happens only when fewer components than original features are kept.

When transformed data surprises you

Inspect fitted parameters and inverse transforms where available. A silent shape change can misalign feature names and explanations.

• A constant training column has zero variance and needs explicit handling.
• Quantile bins can collapse when many records share the same value.
• PCA component signs can flip without changing the mathematical solution, so do not attach meaning to sign alone.
• Scaling one-hot indicators alongside continuous fields may change their intended weighting.

Build a transformation comparison

Take a classification dataset and compare a linear model under no scaling, standard scaling and robust scaling. Keep the same split.

• Report metric, fit time and coefficient stability.
• Try PCA at two component counts and record validation performance, not only explained variance.
• Name features removed by selection and justify the rule.
• Write a short transformation card describing fitted data and limitations.

Proof of understanding

• A pipeline for every compared transformation.
• A results table from one fixed evaluation plan.
• A paragraph explaining what information the selected transformation loses.

Official references and further reading

• scikit-learn preprocessing (Official scaler and discretizer guidance)
• scikit-learn decomposition (Official PCA and SVD guidance)
• scikit-learn feature selection (Official selection methods and caveats)

Review note for Scale, Discretize and Reduce Features with Evidence: recheck the linked documentation after a dependency changes the relevant API, metric or modelling assumption, then record the tested version beside your result.