Understand Data with Distributions, Sampling and Robust Statistics

Read a table without telling yourself a false story

This lesson focuses on statistics as diagnostic tools. You will compare mean and median, inspect quantiles and show how one extreme value changes a familiar summary.

• Distinguish categorical, ordinal, discrete and continuous fields.
• Use counts, quantiles, median and interquartile range for an initial profile.
• Explain why a sample may not represent the population even when it contains many rows.
• Identify skew, heavy tails and subgroup mixtures before applying a global rule.

The distribution is the context for every summary

The mean uses every numerical value and is useful for many later methods. It is also pulled toward extremes. The median is the middle ordered value and often describes a skewed centre more faithfully. Neither is automatically better. Report the measure that answers the question and show enough of the distribution to justify it.

Spread matters because two groups can share a mean and behave very differently. Standard deviation is tied to the mean and appears in methods such as z-score scaling. The interquartile range, calculated from the 25th and 75th percentiles, is less sensitive to extremes. Quantiles also translate naturally into operational questions such as the wait experienced by 90 percent of users.

Missingness is part of the distribution. A field can be missing completely at random, related to observed information, or related to the missing value itself. You usually cannot prove the mechanism from a table alone. Compare missingness across time, source and relevant groups, then document the assumptions behind any treatment.

Sampling creates a second layer of uncertainty. A million responses from a voluntary poll may represent the population worse than a smaller probability sample. Selection, non-response, survival and logging systems can all change who appears. Record the population you can actually describe, not the population you wish you had.

Subgroups can reverse an aggregate pattern. Always check whether a total combines regions, product types or time periods with different distributions. This is not permission to search endlessly for a convenient segment. Predefine important slices and report all of them.

Profile a skewed service-time sample

The code compares a robust summary before and after adding one extreme wait. It also profiles missing values by channel. Save it as profile_distribution.py.

Install the lesson dependencies

python -m pip install numpy pandas

Compare centre, spread and missingness

import numpy as np
import pandas as pd

frame = pd.DataFrame({
    "channel": ["chat", "chat", "phone", "phone", "phone", "chat"],
    "wait_minutes": [2, 3, 4, 5, 45, np.nan],
})

wait = frame["wait_minutes"].dropna()
q1, median, q3 = wait.quantile([0.25, 0.50, 0.75])

print(f"count: {wait.size}")
print(f"mean: {wait.mean():.1f}")
print(f"median: {median:.1f}")
print(f"IQR: {q3 - q1:.1f}")
print("missing by channel:")
print(frame.groupby("channel")["wait_minutes"].apply(lambda s: s.isna().mean()))

without_extreme = wait[wait < 40]
print(f"mean without 45-minute case: {without_extreme.mean():.1f}")

Expected profile

count: 5
mean: 11.8
median: 4.0
IQR: 2.0
missing by channel:
channel
chat     0.333333
phone    0.000000
Name: wait_minutes, dtype: float64
mean without 45-minute case: 3.5

The 45-minute wait is not automatically an error. It may be a real service failure and therefore the most actionable record. The mean changes sharply when it is removed, while the median remains close to the common waits.

Why describe output can still mislead

Automated profiles are useful starting points, but they do not know the meaning of the fields. Investigate these common traps.

• An integer customer ID may receive a mean even though arithmetic on it has no meaning.
• A zero may represent a real measurement, a missing-value code or a sensor floor. Check the data dictionary.
• A global median can hide distinct channel or regional distributions.
• Dropping duplicates may erase repeated events that are valid. Define what a duplicate means at the chosen unit of analysis.

Profile a public or synthetic dataset

Choose five useful fields. For each, record its semantic type, missing rate, valid range, centre, spread and one possible collection bias. Use a chart only when it adds information beyond the table.

• Compare mean and median for at least one skewed field.
• Calculate counts and rates for two predefined subgroups.
• Inspect the five smallest and five largest values in their full rows.
• Write one cleaning question rather than silently changing suspicious values.

Portfolio evidence

• A compact profiling table with semantic types.
• One distribution plot with labelled units and sample size.
• A short note on sampling and which population the data can support.

Official references and further reading

• pandas descriptive statistics (Official table-summary operations)
• pandas missing data (Official missing-value behaviour and tools)
• NumPy statistics routines (Official numerical summary reference)

Review note for Understand Data with Distributions, Sampling and Robust Statistics: recheck the linked documentation after a dependency changes the relevant API, metric or modelling assumption, then record the tested version beside your result.