Set Up Python for Reproducible Data Mining

A workspace you can rebuild

You will create an isolated environment, install a focused toolset and run a deterministic smoke test. The course uses Python as the main practical language because its data and machine learning libraries fit the topics ahead.

• Create and activate a virtual environment without changing the system Python.
• Understand why dependency versions and random seeds both matter.
• Use a project layout that separates immutable input from generated artefacts.
• Run a pandas and scikit-learn smoke test before starting a larger lesson.

Reproducibility is more than setting a seed

A virtual environment isolates packages for one project. It protects other projects from accidental upgrades and records which interpreter you used. A requirements file or modern lock file then describes the environment, but it should come from a tested setup rather than copying every package installed on the machine.

A random seed makes pseudo-random operations repeatable within the same implementation and inputs. It does not guarantee identical floating-point output on every operating system, processor or future library version. Record the seed, software versions and important hardware details when exact reproduction matters.

Keep raw data read-only in practice. Write cleaned tables, models and reports to separate directories. When a script silently overwrites its source, you lose the evidence needed to explain a surprising result. A small README.md should name the entry command and the expected output.

The course examples use pandas for tables and scikit-learn for preprocessing and models. NumPy supports numerical arrays underneath them. Install only what a lesson needs, and check current official release notes before adopting a new major version in an existing project.

Build the environment and test the stack

Run the commands from a new project directory. On Windows PowerShell, activation is .venv\Scripts\Activate.ps1. On macOS or Linux, use source .venv/bin/activate. Then save the Python sample as smoke_test.py.

Create the isolated environment

python -m venv .venv
python -m pip install --upgrade pip
python -m pip install numpy pandas scikit-learn

Confirm pandas, NumPy and scikit-learn work together

from importlib.metadata import version
import numpy as np
import pandas as pd
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(42)
frame = pd.DataFrame({
    "orders": [2, 5, 9, 4],
    "minutes": rng.integers(10, 60, size=4),
})

scaled = StandardScaler().fit_transform(frame[["orders", "minutes"]])

print("shape:", frame.shape)
print("orders mean:", round(float(frame["orders"].mean()), 2))
print("scaled column means:", np.round(scaled.mean(axis=0), 8).tolist())
print("versions:", {
    "numpy": version("numpy"),
    "pandas": version("pandas"),
    "scikit-learn": version("scikit-learn"),
})

Expected stable parts of the output

shape: (4, 2)
orders mean: 5.0
scaled column means: [0.0, 0.0]
versions: {'numpy': '<installed version>', 'pandas': '<installed version>', 'scikit-learn': '<installed version>'}

The exact random minutes are repeatable for the same NumPy implementation and seed. The scaled column means should be near zero. Version numbers are intentionally reported instead of hard-coded into the expected result.

Fix setup problems without reinstalling everything

Read the first complete error message. Repeatedly installing packages into an unknown interpreter usually makes the environment harder to understand.

• If python and pip point to different environments, run python -m pip so both use the same interpreter.
• If activation is blocked by local PowerShell policy, use the environment’s full Python path rather than changing machine-wide security settings blindly.
• If imports work in a terminal but not a notebook, select the virtual environment as the notebook kernel.
• If output changes, print the seed, input checksum and package versions before blaming the model.

Create your course repository

Make directories named data/raw, data/processed, src, reports and tests. Add a small synthetic CSV under raw data and read it from a script under src.

• Write the environment creation and run commands in the README.
• Add a .gitignore that excludes the virtual environment, credentials and large generated files.
• Print input row count, output row count and package versions from the script.
• Recreate the environment in a second empty directory and run the same smoke test.

Proof that the setup is portable

• The dependency file and Python version.
• A clean terminal transcript from the second environment.
• A project tree in the README with one-line ownership rules for each directory.

Official references and further reading

• Python virtual environments (Official environment creation and activation reference)
• NumPy random generator (Official modern random-generation API)
• scikit-learn install guide (Official supported installation guidance)
• pandas installation (Official pandas environment guidance)

Review note for Set Up Python for Reproducible Data Mining: recheck the linked documentation after a dependency changes the relevant API, metric or modelling assumption, then record the tested version beside your result.