Project: Investigate Transaction Patterns and Anomalies

Project acceptance criteria

Use synthetic data or transactions with clear reuse rights. Keep entity and time boundaries explicit, and remove unnecessary identifiers from analytical outputs.

• Construct baskets and sequences under written session rules.
• Mine association rules with minimum count, support and lift gates.
• Rank anomaly candidates under a declared review capacity.
• Confirm discoveries on a later period and record investigation outcomes.

Discovery and confirmation need separate clocks

Use the first period to choose preprocessing, thresholds and candidate rules. Freeze those decisions before running the confirmation period. Otherwise the later data becomes another source of discoveries rather than an honest check.

Rule mining should include counts and base rates. Compare rules across store, device or customer segments only when those slices were declared and have enough support. A lift change may reflect population mix rather than a changed relationship.

Anomaly detection needs a score-to-review policy. Select the top number investigators can inspect, redact unnecessary fields and capture outcomes such as valid rare event, data defect, confirmed issue and unresolved. Do not force ambiguous cases into a positive label.

Join rule and anomaly evidence carefully. A rare basket is not automatically anomalous in risk. A high-lift rule can be common and benign. Keep discovery tasks separate unless the project brief defines how they interact.

The final report should include rejected patterns and false alarms. They demonstrate that the quality gates worked and help future analysts avoid repeating dead ends.

Create a compact confirmation report

This core function calculates rule metrics for two periods. Extend the project with FP-growth and an Isolation Forest review queue.

Compare discovery and confirmation lift

def rule_metrics(transactions, left, right):
    left, right = set(left), set(right)
    n = len(transactions)
    left_count = sum(left <= basket for basket in transactions)
    right_count = sum(right <= basket for basket in transactions)
    joint_count = sum((left | right) <= basket for basket in transactions)
    support = joint_count / n
    confidence = joint_count / left_count if left_count else 0.0
    lift = confidence / (right_count / n) if right_count else 0.0
    return {"n": n, "joint": joint_count, "support": support,
            "confidence": confidence, "lift": lift}

discovery = [
    {"bread", "milk"}, {"bread", "milk"}, {"bread", "eggs"},
    {"milk", "eggs"}, {"bread", "milk", "eggs"},
]
confirmation = [
    {"bread", "milk"}, {"bread", "eggs"}, {"milk", "eggs"},
    {"bread", "milk"}, {"eggs"},
]

for period, baskets in [("discovery", discovery), ("confirmation", confirmation)]:
    result = rule_metrics(baskets, ["bread"], ["milk"])
    print(period, {key: round(value, 3) if isinstance(value, float) else value
                   for key, value in result.items()})

Expected two-period structure

discovery {'n': 5, 'joint': 3, 'support': 0.6, 'confidence': 0.75, 'lift': 0.938}
confirmation {'n': 5, 'joint': 2, 'support': 0.4, 'confidence': 0.667, 'lift': 1.111}

The small periods give unstable lift and even change which side of one the estimate falls. Counts make that weakness impossible to hide.

Required adversarial checks

Try to make your strongest finding disappear before presenting it.

• Raise minimum support and report which rules vanish.
• Shuffle item membership within relevant strata to create a chance baseline.
• Remove duplicates and compare anomaly rankings.
• Review whether promotions, outages or logging changes explain the period difference.

Complete the investigation

Package the pipeline, thresholds, confirmation and review outcomes into one reproducible report.

• Save basket and sequence construction tests.
• Publish only aggregated rule metrics and redacted anomaly examples.
• Separate observed association, possible explanation and required next evidence.
• State which patterns are confirmed, rejected or unresolved.

Portfolio artefacts

• Rule and anomaly pipeline.
• Two-period confirmation report.
• Investigation log and cautious recommendations.

Official references and further reading

• mlxtend frequent patterns (Primary implementation guidance for itemsets and rules)
• scikit-learn outlier detection (Official anomaly-method guidance)
• Python sets (Official basket representation behaviour)

Review note for Project: Investigate Transaction Patterns and Anomalies: recheck the linked documentation after a dependency changes the relevant API, metric or modelling assumption, then record the tested version beside your result.