Data and model basics in Artificial Intelligence: Build a Reviewable Working Model

Data and model basics becomes useful when the work improves measurable task utility rather than merely producing a polished output. This Artificial Intelligence lesson shows how to separate data uncertainty from model uncertainty in the same experiment log.

It is written for a practitioner who needs inspectable data, fixed evaluation cases and evidence that survives review. You will apply the method to Classify support requests, challenge one assumption deliberately, and retain a frozen test set, traceable sources, latency and cost so the result can be checked without private explanation.

Boundary: the exercise is not complete if it hides unsupported output reaching a consequential decision. Use Jupyter only after writing the expected normal result, the unsafe result and the condition that should stop the work.

The result you should produce

Continue the support-ticket triage project from Lesson 1. This time you will create a tiny labeled dataset, document every field, audit invalid and duplicate records, and design a split that does not leak the same ticket into training and testing.

From an example to a prediction

A supervised classification dataset is a collection of examples. Each example contains input information called features and the expected answer called a label. During training, an algorithm adjusts model parameters so that inputs map to labels with lower error. During inference, the trained model receives a new input and returns a prediction.

Labels describe a decision, not objective truth

The queue stored in an old help desk is not automatically a reliable label. An agent may have chosen the quickest queue, company policy may have changed, or a ticket may legitimately involve billing and technical support. Write labeling instructions and measure disagreements. When reviewers disagree, preserve that fact instead of forcing false certainty.

A direct label matches the prediction you want. A proxy label only approximates it. “Queue that finally resolved the ticket” may be closer to the desired routing decision than “queue first selected,” but even the final queue can reflect transfers or staffing constraints. Google’s current dataset guidance recommends inspecting labels and comparing human ratings rather than assuming collected labels are correct.

Models learn correlations available in the dataset

If all billing tickets in the training data include the word “invoice,” a model may depend heavily on that word. It does not understand the organization’s billing policy in the way an experienced employee does. Useful generalization must be demonstrated with new, representative cases: refunds without “invoice,” spelling variations, short messages and wording from supported regions.

Training, validation and test data have different jobs

A model evaluated on its training examples can appear excellent because it has already used those examples to adjust itself. Separate data supports a more honest question: does the approach work on examples it did not learn from?

• Training set: examples used to learn parameters.
• Validation set: separate examples used while choosing features, prompts, thresholds or model settings.
• Test set: held-back examples used after design decisions to estimate final quality.

There is no universal percentage that makes a split correct. The test set must be large and representative enough for the decisions you will make. For a small learning project, the important lesson is separation. For a production system, design the split around how data arises and how the system will be used.

Three leakage traps

1. Duplicate leakage: a copied ticket lands in training and test. The model appears to generalize while recognizing repeated text.
2. Future information: the feature set contains the final queue, resolution note or response time:values unavailable when routing happens.
3. Preprocessing leakage: statistics are calculated using the entire dataset before the split. Information from test examples influences training preparation.

Split first, then fit transformations using training data and apply the learned transformation to validation, test and production inputs. If tickets from the same customer or incident are closely related, keep the group in one split. If the project predicts future tickets, a time-based split may better match deployment than random shuffling.

A data-quality checklist tied to the use case

“Clean data” is not a universal state. A dataset is useful when it supports the project’s intended outcome and evaluation. Track each decision in a data preparation log.

Run a small dataset audit

This program checks allowed labels, counts valid rows and detects exact duplicate text after lowercasing and trimming. It does not “clean” anything automatically. The output gives a reviewer evidence to decide what to correct, merge or exclude.

const rows = [
  [1, "I was charged twice", "billing"], [2, "Refund my invoice", "billing"],
  [3, "The app crashes after login", "technical"], [4, "Login shows an error", "technical"],
  [5, "Change my email", "account"], [6, "Update my profile", "account"],
  [7, "I was charged twice", "billing"], [8, "Reset my password", "password"]
];
const allowed = ["account", "billing", "technical"];
const counts = Object.fromEntries(allowed.map(label => [label, 0]));
const seen = new Set();
const issues = [];
let valid = 0, duplicates = 0;
for (const [id, text, label] of rows) {
  const normalized = text.trim().toLowerCase();
  if (seen.has(normalized)) duplicates++;
  seen.add(normalized);
  if (!allowed.includes(label)) { issues.push(`row ${id} unknown label: ${label}`); continue; }
  valid++; counts[label]++;
}
console.log(`rows=${rows.length}`);
console.log(`valid_rows=${valid}`);
console.log(`invalid_rows=${rows.length - valid}`);
console.log(`duplicate_texts=${duplicates}`);
console.log(`labels=${allowed.map(x => `${x}:${counts[x]}`).join(",")}`);
for (const issue of issues) console.log(`issue=${issue}`);

rows = [
    (1, "I was charged twice", "billing"), (2, "Refund my invoice", "billing"),
    (3, "The app crashes after login", "technical"), (4, "Login shows an error", "technical"),
    (5, "Change my email", "account"), (6, "Update my profile", "account"),
    (7, "I was charged twice", "billing"), (8, "Reset my password", "password"),
]
allowed = ("account", "billing", "technical")
counts = {label: 0 for label in allowed}
seen, issues = set(), []
valid = duplicates = 0
for row_id, text, label in rows:
    normalized = text.strip().lower()
    duplicates += normalized in seen
    seen.add(normalized)
    if label not in allowed:
        issues.append(f"row {row_id} unknown label: {label}")
        continue
    valid += 1
    counts[label] += 1
print(f"rows={len(rows)}")
print(f"valid_rows={valid}")
print(f"invalid_rows={len(rows) - valid}")
print(f"duplicate_texts={duplicates}")
print("labels=" + ",".join(f"{label}:{counts[label]}" for label in allowed))
for issue in issues: print(f"issue={issue}")

<?php
$rows = [
 [1,"I was charged twice","billing"],[2,"Refund my invoice","billing"],
 [3,"The app crashes after login","technical"],[4,"Login shows an error","technical"],
 [5,"Change my email","account"],[6,"Update my profile","account"],
 [7,"I was charged twice","billing"],[8,"Reset my password","password"]
];
$allowed = ["account", "billing", "technical"];
$counts = array_fill_keys($allowed, 0); $seen = []; $issues = [];
$valid = 0; $duplicates = 0;
foreach ($rows as [$id, $text, $label]) {
    $normalized = strtolower(trim($text));
    if (isset($seen[$normalized])) $duplicates++;
    $seen[$normalized] = true;
    if (!in_array($label, $allowed, true)) { $issues[] = "row $id unknown label: $label"; continue; }
    $valid++; $counts[$label]++;
}
echo "rows=" . count($rows) . PHP_EOL;
echo "valid_rows=$valid" . PHP_EOL;
echo "invalid_rows=" . (count($rows) - $valid) . PHP_EOL;
echo "duplicate_texts=$duplicates" . PHP_EOL;
echo "labels=" . implode(",", array_map(fn($x) => "$x:" . $counts[$x], $allowed)) . PHP_EOL;
foreach ($issues as $issue) echo "issue=$issue" . PHP_EOL;

import java.util.*;
public class Main {
  record Row(int id, String text, String label) {}
  public static void main(String[] args) {
    List<Row> rows = List.of(
      new Row(1,"I was charged twice","billing"), new Row(2,"Refund my invoice","billing"),
      new Row(3,"The app crashes after login","technical"), new Row(4,"Login shows an error","technical"),
      new Row(5,"Change my email","account"), new Row(6,"Update my profile","account"),
      new Row(7,"I was charged twice","billing"), new Row(8,"Reset my password","password"));
    List<String> allowed = List.of("account", "billing", "technical");
    Map<String,Integer> counts = new LinkedHashMap<>(); allowed.forEach(x -> counts.put(x, 0));
    Set<String> seen = new HashSet<>(); List<String> issues = new ArrayList<>();
    int valid = 0, duplicates = 0;
    for (Row row : rows) {
      String normalized = row.text().trim().toLowerCase(Locale.ROOT);
      if (!seen.add(normalized)) duplicates++;
      if (!allowed.contains(row.label())) { issues.add("row " + row.id() + " unknown label: " + row.label()); continue; }
      valid++; counts.put(row.label(), counts.get(row.label()) + 1);
    }
    System.out.println("rows=" + rows.size()); System.out.println("valid_rows=" + valid);
    System.out.println("invalid_rows=" + (rows.size() - valid)); System.out.println("duplicate_texts=" + duplicates);
    System.out.println("labels=" + String.join(",", allowed.stream().map(x -> x + ":" + counts.get(x)).toList()));
    issues.forEach(x -> System.out.println("issue=" + x));
  }
}

var rows = new (int Id, string Text, string Label)[] {
 (1,"I was charged twice","billing"),(2,"Refund my invoice","billing"),
 (3,"The app crashes after login","technical"),(4,"Login shows an error","technical"),
 (5,"Change my email","account"),(6,"Update my profile","account"),
 (7,"I was charged twice","billing"),(8,"Reset my password","password")
};
string[] allowed = ["account", "billing", "technical"];
var counts = allowed.ToDictionary(x => x, _ => 0);
var seen = new HashSet<string>(); var issues = new List<string>();
int valid = 0, duplicates = 0;
foreach (var row in rows) {
    var normalized = row.Text.Trim().ToLowerInvariant();
    if (!seen.Add(normalized)) duplicates++;
    if (!allowed.Contains(row.Label)) { issues.Add($"row {row.Id} unknown label: {row.Label}"); continue; }
    valid++; counts[row.Label]++;
}
Console.WriteLine($"rows={rows.Length}"); Console.WriteLine($"valid_rows={valid}");
Console.WriteLine($"invalid_rows={rows.Length - valid}"); Console.WriteLine($"duplicate_texts={duplicates}");
Console.WriteLine("labels=" + string.Join(",", allowed.Select(x => $"{x}:{counts[x]}")));
foreach (var issue in issues) Console.WriteLine($"issue={issue}");

Expected output:

rows=8
valid_rows=7
invalid_rows=1
duplicate_texts=1
labels=account:2,billing:3,technical:2
issue=row 8 unknown label: password

The program reports; it does not guess that password means account. A domain owner must decide whether the row is mislabeled, whether the allowed schema is incomplete, or whether the example needs multi-label treatment.

Classify support requests: isolate the Data and model basics decision

Data dictionary template

FIELD: ticket_id
TYPE: string
REQUIRED: yes
MEANING: stable synthetic example identifier
SOURCE: lesson author
ALLOWED VALUES: unique non-empty value
AVAILABLE AT ROUTING TIME: yes
SENSITIVE: no
TRANSFORMATIONS: none
KNOWN LIMITATIONS: not linked to a real help-desk record

Create one entry for every field. “Text” is not enough: state encoding, language assumptions, maximum size, whether formatting is preserved and which redaction happens before the model sees it.

Knowledge check

1. A ticket’s final resolution note predicts its queue extremely well. Why might it be an invalid feature?
2. Why can a duplicate in both training and test inflate confidence in the result?
3. When could a time-based split be more honest than a random split?
4. What is the difference between an invalid label and a rare but valid label?
5. If 90% of examples are billing, why is overall accuracy insufficient?
6. What evidence would show that Roman Urdu tickets are represented?

Portfolio package

• data-dictionary.md : definitions, ownership and availability for every field;
• tickets-synthetic.csv : clearly marked synthetic learning data;
• audit program in your chosen language with run instructions;
• quality-report.md : counts, problems found and correction decisions;
• split-plan.md : grouping, time assumptions and leakage controls;
• limitations.md : unsupported users, languages, products and risks.

Primary references

• Google ML Crash Course: Data Characteristics
• Google ML Crash Course: Dividing Datasets
• Google ML Crash Course: Labels
• Google ML Universal Guides: Data Quality and Interpretation
• scikit-learn: Common Pitfalls and Data Leakage

Dataset suitability changes when the collection process, label policy, supported users or deployment environment changes. Re-run the audit and review the data dictionary whenever one of those conditions changes.

Created and reviewed by Muhammad Azhar, Lead Software Engineer with more than 16 years of software-development experience.