Outlier and Novelty Detection with LOF and Isolation Forest

Separate outlier detection from novelty detection

You will compare Local Outlier Factor and Isolation Forest, inspect score direction and turn scores into a review queue rather than an automatic accusation.

• Distinguish global statistical, distance, density and isolation approaches.
• Explain LOF local reachability density and neighbourhood sensitivity.
• Explain why Isolation Forest isolates rare points with shorter paths.
• Choose outlier versus novelty mode from whether training data is assumed clean.

Unusual relative to what?

A z-score or robust deviation works for one-dimensional distributions under stated assumptions. Distance methods compare neighbours globally. Density methods such as LOF compare a point’s local density with its neighbours, which can find a sparse pocket beside a dense group.

LOF depends on the number of neighbours and distance representation. In ordinary outlier mode, fit_predict identifies anomalies in the training data. For novelty detection on new records, fit with novelty=True on data believed to be mostly normal, then score unseen records. Do not use novelty predictions on the fitted samples as if they were independent.

Isolation Forest recursively chooses features and split values. Rare, separated points tend to need fewer splits to isolate. It scales well, but correlated duplicates, categorical encoding and irrelevant features still affect the path structure.

The contamination parameter sets an expected proportion or thresholding policy, not the true anomaly rate. If investigators can review 50 cases, capacity may define a top-k queue. Track precision among reviewed cases after labels mature.

Anomaly systems face drift and adversarial adaptation. Monitor score distributions, feature availability, investigation outcomes and subgroup burden. Never let a score alone trigger a high-impact action without appropriate review and safeguards.

Compare local and isolation scores

The example has eight compact points and one distant point. Both methods flag it under the chosen contamination rate.

Install the lesson dependencies

python -m pip install numpy scikit-learn

Build a review queue

import numpy as np
from sklearn.ensemble import IsolationForest
from sklearn.neighbors import LocalOutlierFactor
from sklearn.preprocessing import StandardScaler

X = np.array([
    [1.0, 1.0], [1.1, 0.9], [0.9, 1.1], [1.2, 1.0],
    [1.0, 1.2], [0.8, 1.0], [1.1, 1.1], [0.9, 0.8],
    [7.0, 7.0],
])
ready = StandardScaler().fit_transform(X)

lof = LocalOutlierFactor(n_neighbors=4, contamination=1/9)
lof_label = lof.fit_predict(ready)
forest = IsolationForest(n_estimators=300, contamination=1/9,
                         random_state=42).fit(ready)
forest_label = forest.predict(ready)

print("LOF outlier indices:", np.where(lof_label == -1)[0].tolist())
print("Isolation Forest outlier indices:", np.where(forest_label == -1)[0].tolist())
print("LOF review score for point 8:", round(float(-lof.negative_outlier_factor_[8]), 3))

Expected detection

LOF outlier indices: [8]
Isolation Forest outlier indices: [8]
LOF review score for point 8: <positive value larger than typical points>

The negative LOF factor is negated only to make larger review scores more intuitive. Document score direction in every output.

Anomaly pipeline failures

A method that flags the right synthetic point can still fail on real operations.

• Fitting a scaler on future records leaks the evaluation distribution.
• One-hot rare categories can appear anomalous merely because they are rare.
• Duplicated anomalies can form their own dense group and escape local methods.
• Using the fitted LOF object incorrectly for novelty scoring can produce misleading comparisons.

Evaluate an investigation queue

Create synthetic transactions with known injected anomalies and a later clean test period.

• Compare a robust univariate rule, LOF and Isolation Forest.
• Rank scores and report precision among the top review capacity.
• Inspect false positives by source and feature contribution.
• Write a human review outcome schema and a feedback delay plan.

Evidence for safe use

• Score and threshold definition.
• Reviewed top-k results.
• Monitoring plan covering drift and investigator feedback.

Official references and further reading

• scikit-learn novelty and outlier detection (Official mode distinction and estimator guidance)
• scikit-learn IsolationForest (Official parameters and score behaviour)
• scikit-learn LocalOutlierFactor (Official LOF and novelty guidance)

Review note for Outlier and Novelty Detection with LOF and Isolation Forest: recheck the linked documentation after a dependency changes the relevant API, metric or modelling assumption, then record the tested version beside your result.