Project: Build a Dataset Profiler in Python

Checkpoint specification

The completed program reads one CSV, produces a machine-readable JSON report and prints a short review queue. It never imputes, deletes or corrects values automatically.

• Profile row count, columns, inferred types, missingness and distinct counts.
• Add numeric min, median and max without treating identifiers as measurements.
• Flag high missingness, constant fields and likely identifiers for human review.
• Write deterministic JSON that can be diffed after a data refresh.

Profiling is observation, not cleaning

The report should distinguish facts from warnings. ‘Column region is 40 percent missing’ is a measured fact. ‘Delete region’ is an unsupported action until you understand the collection process and downstream task. Keep proposed decisions in a separate review file.

Cardinality gives context. A field with one distinct value cannot separate records in the current sample. A field with almost one unique value per row may be an identifier, a timestamp or a genuinely continuous measurement. The name and semantic type decide which interpretation fits.

Inferred data types are guesses based on file contents. A postcode may load as an integer and lose leading zeroes. A mixed date field may load as text. Add an optional schema contract rather than trusting inference for production data.

Deterministic output matters. Sort column names and convert library-specific numeric types to plain Python values before writing JSON. Include an input checksum when the same filename can contain different data.

A production profiler also needs sampling controls and sensitive-data safeguards. This checkpoint stays local, small and explicit. Do not print raw free text, credentials or personal fields into logs.

Implement the reusable profiler

Create a CSV named customers.csv with the fields shown in the code, or change the path to a synthetic dataset you control. Save the script as dataset_profiler.py.

Install the lesson dependencies

python -m pip install pandas

Profile without mutating the input

import hashlib
import json
from pathlib import Path
import pandas as pd

INPUT = Path("customers.csv")
OUTPUT = Path("dataset-profile.json")

frame = pd.read_csv(INPUT)
report = {
    "file": INPUT.name,
    "sha256": hashlib.sha256(INPUT.read_bytes()).hexdigest(),
    "rows": int(len(frame)),
    "columns": {},
}

for name in sorted(frame.columns):
    series = frame[name]
    item = {
        "dtype": str(series.dtype),
        "missing": int(series.isna().sum()),
        "missing_rate": round(float(series.isna().mean()), 4),
        "distinct_non_null": int(series.nunique(dropna=True)),
    }
    if pd.api.types.is_numeric_dtype(series) and not name.lower().endswith("id"):
        non_null = series.dropna()
        if not non_null.empty:
            item["min"] = float(non_null.min())
            item["median"] = float(non_null.median())
            item["max"] = float(non_null.max())
    warnings = []
    if item["missing_rate"] >= 0.25:
        warnings.append("high missingness")
    if item["distinct_non_null"] <= 1:
        warnings.append("constant or empty")
    if name.lower().endswith("id") and item["distinct_non_null"] == len(frame):
        warnings.append("likely row identifier")
    item["review"] = warnings
    report["columns"][name] = item

OUTPUT.write_text(json.dumps(report, indent=2), encoding="utf-8")
print(f"profiled rows: {report['rows']}")
for name, item in report["columns"].items():
    if item["review"]:
        print(f"review {name}: {', '.join(item['review'])}")
print(f"saved: {OUTPUT}")

Example checkpoint output

profiled rows: 6
review customer_id: likely row identifier
review region: high missingness
review source_system: constant or empty
saved: dataset-profile.json

Your exact warnings depend on the CSV. Open the JSON and confirm that numbers are JSON numbers rather than strings. Re-run without changing the input and verify that the output file has no unexplained differences.

Failures your profiler should expose

Test the tool with deliberately awkward files. A profiler that succeeds only on a perfect CSV creates confidence at exactly the wrong time.

• An empty file should stop with a clear message rather than a deep parser traceback.
• Duplicate column names may be renamed automatically by a parser. Detect and document that behaviour before relying on the result.
• Mixed numeric and text values should remain visible instead of being silently coerced to missing.
• Large free-text fields should be counted, not printed into the report.

Finish and test the profiler

Add command-line input and output arguments, then create three test CSVs: normal, empty and mixed-type. Keep the tool read-only with respect to its source file.

• Return a non-zero exit code and a concise explanation for an unreadable or empty file.
• Add a schema override that marks identifier and categorical fields.
• Include duplicate row count without deleting anything.
• Write one automated test for a 50 percent missing column.

Checkpoint evidence

• The profiler, small synthetic fixtures and automated tests.
• A generated JSON report with input checksum.
• A README that separates observed warnings from cleaning decisions.

Official references and further reading

• pandas read_csv (Official parser options and behaviour)
• pandas data types (Official type inspection guidance)
• Python hashlib (Official checksum API used by the project)
• Python json (Official report serialization reference)

Review note for Project: Build a Dataset Profiler in Python: recheck the linked documentation after a dependency changes the relevant API, metric or modelling assumption, then record the tested version beside your result.