What Is Data Mining? A Practical KDD Process

The result you should produce

By the end, you will be able to separate data mining from reporting, database querying and machine learning. You will also run a miniature knowledge discovery process instead of treating a model score as the final answer.

• Explain the stages of knowledge discovery in databases, usually shortened to KDD.
• Identify where selection, cleaning, transformation, pattern discovery and interpretation happen.
• State why a discovered pattern is not automatically useful or causal.
• Create a small, inspectable finding from transaction records.

From stored rows to defensible knowledge

A database answers questions you already know how to ask. A query such as ‘How many orders arrived late last month?’ is valuable reporting. Data mining looks for a structure that was not specified row by row, such as a group of orders that tends to become late under a particular combination of conditions. Machine learning supplies many of the algorithms used for this work, but KDD covers the wider process that makes their output meaningful.

A useful KDD cycle has several linked decisions. Select records that match the real population. Clean errors without erasing legitimate rare cases. Transform fields into representations an algorithm can use. Apply a suitable mining method. Evaluate the pattern against a baseline. Finally, interpret it with someone who understands the domain. A failure in an early decision can make a technically correct algorithm produce a false story.

Prediction and description are different goals. A classifier may estimate whether a customer will cancel. Clustering may describe groups of customers without a target label. Association mining may expose products bought together. Each task asks a different question, uses different evidence and needs a different validation plan.

The word knowledge deserves caution. Correlation is not causation, a frequent pattern may have no business value, and a rare pattern may matter greatly. Keep the wording proportional to the evidence. ‘Orders with fragile items had a higher observed delay rate in this sample’ is supportable. ‘Fragile items cause delays’ needs a stronger design.

Run a five-stage discovery on six orders

The example deliberately uses plain Python. You can see every selection and calculation. The same reasoning should remain visible when a larger library replaces these few lines. Save the file as kdd_walkthrough.py and run it with python kdd_walkthrough.py.

Inspect, transform and compare the groups

orders = [
    {"id": 1, "fragile": "yes", "distance_km": 90, "late": 1},
    {"id": 2, "fragile": "no",  "distance_km": 40, "late": 0},
    {"id": 3, "fragile": "YES", "distance_km": 75, "late": 1},
    {"id": 4, "fragile": "no",  "distance_km": None, "late": 0},
    {"id": 5, "fragile": "yes", "distance_km": 15, "late": 0},
    {"id": 6, "fragile": "no",  "distance_km": 110, "late": 1},
]

# Select the population and standardize a categorical field.
selected = [row.copy() for row in orders if row["id"] >= 1]
for row in selected:
    row["fragile"] = row["fragile"].lower()

# Replace one missing distance with the observed median, 75 km.
for row in selected:
    if row["distance_km"] is None:
        row["distance_km"] = 75

# Transform distance into a readable band.
for row in selected:
    row["long_trip"] = row["distance_km"] >= 70

def late_rate(rows):
    return sum(row["late"] for row in rows) / len(rows)

fragile = [row for row in selected if row["fragile"] == "yes"]
long_trip = [row for row in selected if row["long_trip"]]

print(f"All orders: {late_rate(selected):.1%} late")
print(f"Fragile orders: {late_rate(fragile):.1%} late")
print(f"Long trips: {late_rate(long_trip):.1%} late")
print("Finding: inspect fragile packing and long-distance handling separately.")

Expected console output

All orders: 50.0% late
Fragile orders: 66.7% late
Long trips: 75.0% late
Finding: inspect fragile packing and long-distance handling separately.

The code does not prove why an order was late. It does create two candidate segments whose rates can be checked on a larger period. Notice that the imputed distance affects the long-trip group. That choice belongs in the final report because another reasonable imputation could change the result.

When the pattern looks impressive but is wrong

Small examples make mistakes easy to see. Large datasets can hide the same mistakes behind many decimal places. Check these points before trusting a pattern.

• A group with one or two records can produce a dramatic rate. Always print its count beside its percentage.
• A missing-value rule can create the very pattern you later discover. Compare results before and after imputation.
• A target recorded after the decision point leaks future information. Build features only from what was known when the decision would be made.
• A relationship found in one month may disappear in another. Keep a later period for confirmation.

Your turn: challenge the first finding

Add four believable orders and test a competing explanation. For example, perhaps distance matters while fragile handling does not. Do not change the original records after seeing the answer.

• Print the record count and late rate for every segment you compare.
• Write one sentence that the sample supports and one stronger sentence it does not support.
• Name one new field you would collect before recommending an operational change.

Evidence to keep

• Your runnable kdd_walkthrough.py file.
• A console capture showing counts and rates.
• A short interpretation that distinguishes an observation from a causal claim.

Official references and further reading

• From Data Mining to Knowledge Discovery in Databases (Foundational KDD process and terminology)
• scikit-learn: choosing the right estimator (Official task-oriented estimator map)

Review note for What Is Data Mining? A Practical KDD Process: recheck the linked documentation after a dependency changes the relevant API, metric or modelling assumption, then record the tested version beside your result.