Compare Logistic Regression, Naive Bayes and SVM

Match model assumptions to the representation

You will evaluate logistic regression, Gaussian Naive Bayes and an RBF support vector machine under the same stratified folds.

• Interpret logistic coefficients as changes in log-odds, not direct causal effects.
• Explain the conditional-independence assumption behind Naive Bayes.
• Understand margins, kernels and the scaling needs of SVMs.
• Compare ranking, classification and probability calibration separately.

Three different ways to draw a boundary

Logistic regression models a linear relationship between features and log-odds. Regularization limits coefficient magnitude. Standard scaling makes its penalty more comparable across features. Coefficients depend on the other included fields and their encoding, so document transformations before interpreting them.

Naive Bayes estimates class-conditional feature distributions and combines them under a conditional-independence assumption. That assumption is often false, yet the model can work well, especially for high-dimensional text counts with MultinomialNB. GaussianNB instead assumes a Gaussian distribution per numerical feature and class.

A support vector machine maximizes the margin between classes. A linear kernel fits a linear boundary. RBF and other kernels model nonlinear similarity but add tuning and can become expensive on large datasets. Feature scaling is normally important because distance determines the kernel.

Logistic regression produces probabilities through its fitted link, though calibration can still be poor. SVC probabilities require additional calibration work and computation when probability=True. Naive Bayes probabilities can be overconfident when features are correlated. Evaluate calibration rather than assuming it.

Model choice also includes prediction latency, memory, retraining cost, explanation needs and failure recovery. The smallest score gain may not justify a more fragile procedure.

Evaluate three pipelines under the same folds

The built-in dataset keeps the example runnable. Scaling is included only in the models that need it.

Install the lesson dependencies

python -m pip install scikit-learn

Compare balanced accuracy and fit time

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import StratifiedKFold, cross_validate
from sklearn.naive_bayes import GaussianNB
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = load_breast_cancer(return_X_y=True)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
models = {
    "logistic": make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000)),
    "gaussian_nb": GaussianNB(),
    "rbf_svm": make_pipeline(StandardScaler(), SVC(kernel="rbf")),
}

for name, model in models.items():
    result = cross_validate(model, X, y, cv=cv,
                            scoring="balanced_accuracy", return_train_score=True)
    print(name,
          "validation=", round(result["test_score"].mean(), 3),
          "min_fold=", round(result["test_score"].min(), 3),
          "fit_seconds=", round(result["fit_time"].sum(), 3))

Expected output format

logistic validation= <score> min_fold= <score> fit_seconds= <time>
gaussian_nb validation= <score> min_fold= <score> fit_seconds= <time>
rbf_svm validation= <score> min_fold= <score> fit_seconds= <time>

Fit time varies by machine. The comparison is meaningful because all procedures use the same folds and metric. Add confidence intervals or repeated evaluation when differences are small.

Comparison mistakes to avoid

A leaderboard without assumptions and error analysis encourages arbitrary selection.

• Scaling the full matrix before cross-validation leaks fold statistics.
• Applying GaussianNB to sparse word counts mismatches the feature distribution; MultinomialNB is usually the relevant variant.
• Treating SVM decision scores as calibrated probabilities can produce poor risk decisions.
• Interpreting correlated logistic coefficients individually can create unstable stories.

Create a model selection record

Use one dataset and compare the three families under the deployment-matched split from the previous module.

• Tune a small, declared hyperparameter grid inside nested or training-only validation.
• Report fold variability, fit time and prediction time.
• Inspect class-specific errors and calibration if probabilities drive action.
• Choose one model and explain the rejected alternatives.

Selection evidence

• Fixed-fold comparison table.
• Calibration or threshold analysis.
• One-page decision record covering operations as well as score.

Official references and further reading

• scikit-learn linear models (Official logistic regression guidance)
• scikit-learn Naive Bayes (Official variants and assumptions)
• scikit-learn SVM (Official kernel and scaling guidance)

Review note for Compare Logistic Regression, Naive Bayes and SVM: recheck the linked documentation after a dependency changes the relevant API, metric or modelling assumption, then record the tested version beside your result.