Frame a Data Mining Problem Before Choosing a Model

A question precise enough to test

You will convert an unclear request into a compact problem contract. The contract is deliberately written before model selection, so a fashionable algorithm cannot redefine the problem after the data is seen.

• Define the unit of analysis, eligible population and observation window.
• Separate the feature cutoff from the future outcome window.
• Choose a baseline that exposes whether a model adds useful signal.
• Connect an offline metric to the real cost of false positives and false negatives.

The five clocks hidden inside one prediction

Start with the decision. Who will use the result, and what will they do differently? A retention team may contact at-risk subscribers, but it has a limited weekly capacity. That capacity changes the evaluation. Ranking the most likely 500 customers may matter more than maximizing accuracy across everyone.

Next define the unit. One row might represent a customer, account, order, session or customer-month. Mixing units creates duplicated evidence. If a customer appears twelve times in training and once in testing, a random split may let the model recognize the customer rather than learn a general pattern.

Time needs two boundaries. The feature cutoff is the latest moment from which inputs are allowed. The outcome window is the future period in which churn is measured. A cancellation reason entered after cancellation is an obvious leak. A status field silently updated overnight can be just as damaging.

A useful metric reflects the action. Accuracy is weak when 98 percent of customers stay, because predicting ‘stay’ every time scores 98 percent while finding nobody to help. Precision asks how many contacted customers were truly at risk. Recall asks how many at-risk customers were found. A cost table or capacity-based precision can make the trade-off explicit.

Write a rejection rule too. A project can be technically successful and still unsafe, too expensive or too unstable to deploy. A clear rule such as ‘do not use the model if the error rate differs materially across protected groups’ keeps that decision visible before pressure builds to launch.

Encode a churn brief as testable data

The program below validates the timeline of a project brief and calculates the majority baseline. It does not train a model. That is the point: the problem should survive basic checks before modelling begins.

Validate time and baseline assumptions

from datetime import date

brief = {
    "unit": "customer-month",
    "feature_cutoff": date(2026, 3, 31),
    "outcome_start": date(2026, 4, 1),
    "outcome_end": date(2026, 4, 30),
    "positive_label": "subscription ended during outcome window",
    "weekly_action_capacity": 500,
}

labels = [0, 0, 0, 0, 1, 0, 0, 1, 0, 0]

assert brief["feature_cutoff"] < brief["outcome_start"]
assert brief["outcome_start"] <= brief["outcome_end"]
assert brief["weekly_action_capacity"] > 0

majority = max(labels.count(0), labels.count(1)) / len(labels)
positive_rate = sum(labels) / len(labels)

print(f"Unit: {brief['unit']}")
print(f"Positive rate: {positive_rate:.0%}")
print(f"Majority baseline accuracy: {majority:.0%}")
print("Primary evaluation: precision among the highest-risk 500 eligible customers")

What a valid brief prints

Unit: customer-month
Positive rate: 20%
Majority baseline accuracy: 80%
Primary evaluation: precision among the highest-risk 500 eligible customers

The 80 percent baseline shows why a future 82 percent accuracy would be unimpressive. The action capacity also suggests evaluating a ranked list. Neither decision requires knowing which classifier will be used.

Framing defects that code cannot repair

Most modelling failures here are definition failures. They surface late because the notebook still runs. Review the brief with a domain owner and a data owner.

• An outcome such as ‘inactive’ may have several operational meanings. Translate it into an observable rule and a time window.
• A sampled population may exclude customers who could receive the action. Record inclusion and exclusion rules explicitly.
• Optimizing recall without noting contact cost can overwhelm the team expected to act.
• Using a random split for time-dependent behaviour can train on the future. Plan a chronological evaluation when deployment predicts forward.

Write a fraud-screening contract

Imagine a payment team can manually review 200 transactions per day. Write a one-page contract for a model that ranks suspicious transactions. Keep the contract specific enough that a reviewer could reject invalid data before training.

• Define one row, the eligible transaction population and the exact fraud label.
• Set feature and outcome cutoffs without using chargeback information that arrives later.
• Choose a baseline and one capacity-aware metric.
• Describe the human action and one condition that blocks deployment.

What belongs in your portfolio

• A versioned problem contract with owner and review date.
• A timeline diagram showing observation, cutoff and outcome windows.
• A short metric rationale tied to the 200-review capacity.

Official references and further reading

• scikit-learn model evaluation (Official metric definitions and scoring guidance)
• scikit-learn common pitfalls (Official leakage and consistency warnings)

Review note for Frame a Data Mining Problem Before Choosing a Model: recheck the linked documentation after a dependency changes the relevant API, metric or modelling assumption, then record the tested version beside your result.