AI problem framing in Artificial Intelligence: Define the Decision and Evidence

AI problem framing becomes useful when the work improves measurable task utility rather than merely producing a polished output. This Artificial Intelligence lesson shows how to write a decision-first AI brief with an explicit non-AI baseline.

It is written for a practitioner who needs inspectable data, fixed evaluation cases and evidence that survives review. You will apply the method to Classify support requests, challenge one assumption deliberately, and retain a frozen test set, traceable sources, latency and cost so the result can be checked without private explanation.

What you will be able to do

By the end, you will produce a one-page AI problem brief for a support-ticket triage system. The brief will state what should improve for a user, what decision the system supports, which inputs are allowed, what happens after an output, how success is measured and where a person must remain in control.

AI problem framing starts before the model

AI problem framing is the work of deciding whether an AI system is appropriate and, if it is, defining the job precisely enough to evaluate. “Add a chatbot,” “automate support” and “use our data” are solution ideas, not problem statements. They omit the affected user, current pain, desired action and cost of failure.

Google’s current ML problem-framing guidance begins with a non-ML goal, asks whether predictive ML, generative AI or a simpler approach is suitable, and then defines output and success measures. NIST’s AI Risk Management Framework adds context: intended use, users, deployment setting, benefits, limitations and potential impacts should be understood and documented. Those two views belong together. A technically possible model can still be the wrong product decision.

The seven-part problem brief

Choose the simplest approach that can meet the goal

AI is useful when relevant patterns cannot be captured reliably with a small set of explicit rules, when sufficient representative examples exist, and when the output can drive a helpful action. It is a poor default when exact behavior is required, the data is unavailable, errors cannot be recovered, or a normal software rule already solves the task.

A baseline is not throwaway work. It tells you whether the added cost, latency and maintenance of AI buy a meaningful improvement. If a dozen transparent rules perform well enough, shipping a model may be an engineering downgrade.

Product success is not the same as model performance

A classification score describes behavior on a test set; it does not prove that the workflow improved. Keep two scorecards:

• Product measures: assignment time, manual-review workload, reassignment rate and missed urgent cases.
• Model measures: precision and recall for each queue, confusion matrix, abstention rate and performance on new language or ticket types.

Suppose overall accuracy rises because billing tickets dominate the dataset while rare security tickets are still missed. The model score can look better while the operational risk becomes worse. Evaluate important slices separately and connect each measure to an action.

Worked example: frame support-ticket triage

Start with the vague request: “Build an AI system that automatically handles support.” It combines routing, answering and case resolution, provides no failure boundary, and encourages automation before anyone has measured the current workflow.

A reviewable first version is narrower:

Inputs, outputs and decisions

• Allowed input: ticket subject and message after secrets and unnecessary personal data are removed.
• Output: one allowed queue, a confidence/abstention signal and a short routing reason.
• Decision: a coordinator confirms or changes the suggestion during the pilot.
• Non-goals: diagnosing customer intent, measuring employee performance, drafting final replies or resolving safety/legal cases.
• Failure response: ambiguous, unsupported or high-risk messages go to manual review.

Create a small evaluation set before building

Collect representative, permissioned examples and have qualified reviewers assign expected queues. Keep this evaluation set separate from examples used to design rules or prompts. Include short messages, spelling mistakes, multiple issues in one ticket and messages that belong nowhere. A four-row demo proves only that code runs; it does not establish production quality.

Build the non-AI baseline first

The following program routes four synthetic tickets with transparent keyword rules. It deliberately returns manual-review when no rule matches or when two queues tie. That abstention behavior is part of the product design, not a coding accident.

Choose the language you already use. Every tab implements the same behavior with standard libraries and requires no API key.

const rules = {
  billing: ["charged", "refund", "invoice"],
  technical: ["crash", "error", "login"],
  account: ["email", "password", "profile"]
};

const tickets = [
  "I was charged twice for order 1042",
  "The app crashes after login",
  "Please change the email on my profile",
  "I have a suggestion about your pricing"
];

function classify(text) {
  const value = text.toLowerCase();
  const scores = Object.entries(rules)
    .map(([label, words]) => [label, words.filter(w => value.includes(w)).length])
    .sort((a, b) => b[1] - a[1]);
  if (scores[0][1] === 0 || scores[0][1] === scores[1][1]) return "manual-review";
  return scores[0][0];
}

for (const ticket of tickets) console.log(`${classify(ticket)} | ${ticket}`);

RULES = {
    "billing": ("charged", "refund", "invoice"),
    "technical": ("crash", "error", "login"),
    "account": ("email", "password", "profile"),
}

TICKETS = [
    "I was charged twice for order 1042",
    "The app crashes after login",
    "Please change the email on my profile",
    "I have a suggestion about your pricing",
]

def classify(text: str) -> str:
    value = text.lower()
    scores = sorted(
        ((sum(word in value for word in words), label) for label, words in RULES.items()),
        reverse=True,
    )
    if scores[0][0] == 0 or scores[0][0] == scores[1][0]:
        return "manual-review"
    return scores[0][1]

for ticket in TICKETS:
    print(f"{classify(ticket)} | {ticket}")

<?php
$rules = [
    "billing" => ["charged", "refund", "invoice"],
    "technical" => ["crash", "error", "login"],
    "account" => ["email", "password", "profile"],
];

$tickets = [
    "I was charged twice for order 1042",
    "The app crashes after login",
    "Please change the email on my profile",
    "I have a suggestion about your pricing",
];

function classify(string $text, array $rules): string {
    $value = strtolower($text);
    $scores = [];
    foreach ($rules as $label => $words) {
        $scores[$label] = count(array_filter($words, fn($word) => str_contains($value, $word)));
    }
    arsort($scores);
    $values = array_values($scores);
    if ($values[0] === 0 || $values[0] === $values[1]) return "manual-review";
    return array_key_first($scores);
}

foreach ($tickets as $ticket) echo classify($ticket, $rules) . " | " . $ticket . PHP_EOL;

import java.util.*;

public class Main {
    static final Map<String, List<String>> RULES = Map.of(
        "billing", List.of("charged", "refund", "invoice"),
        "technical", List.of("crash", "error", "login"),
        "account", List.of("email", "password", "profile")
    );

    static String classify(String text) {
        String value = text.toLowerCase(Locale.ROOT);
        List<Map.Entry<String, Integer>> scores = new ArrayList<>();
        RULES.forEach((label, words) ->
            scores.add(Map.entry(label, (int) words.stream().filter(value::contains).count()))
        );
        scores.sort((a, b) -> Integer.compare(b.getValue(), a.getValue()));
        if (scores.get(0).getValue() == 0 || scores.get(0).getValue().equals(scores.get(1).getValue()))
            return "manual-review";
        return scores.get(0).getKey();
    }

    public static void main(String[] args) {
        List<String> tickets = List.of(
            "I was charged twice for order 1042",
            "The app crashes after login",
            "Please change the email on my profile",
            "I have a suggestion about your pricing"
        );
        tickets.forEach(ticket -> System.out.println(classify(ticket) + " | " + ticket));
    }
}

var rules = new Dictionary<string, string[]>
{
    ["billing"] = ["charged", "refund", "invoice"],
    ["technical"] = ["crash", "error", "login"],
    ["account"] = ["email", "password", "profile"]
};

string[] tickets =
[
    "I was charged twice for order 1042",
    "The app crashes after login",
    "Please change the email on my profile",
    "I have a suggestion about your pricing"
];

string Classify(string text)
{
    var value = text.ToLowerInvariant();
    var scores = rules
        .Select(pair => new { pair.Key, Score = pair.Value.Count(value.Contains) })
        .OrderByDescending(item => item.Score)
        .ToArray();
    if (scores[0].Score == 0 || scores[0].Score == scores[1].Score) return "manual-review";
    return scores[0].Key;
}

foreach (var ticket in tickets) Console.WriteLine($"{Classify(ticket)} | {ticket}");

All five versions should print:

billing | I was charged twice for order 1042
technical | The app crashes after login
account | Please change the email on my profile
manual-review | I have a suggestion about your pricing

Make the baseline fail on purpose

Try “I cannot sign in and need a refund.” It matches both technical and billing terms. The program abstains because the top scores tie. That is safer than silently picking a queue, but it also shows why keyword rules may create too much manual work. Record this case; it becomes part of the evaluation set for a later AI prototype.

Next, try spelling variation, Urdu or Roman Urdu, a product-specific term and a message containing two unrelated issues. Do not immediately add more keywords. First decide whether each case belongs within the project’s intended scope.

Classify support requests: isolate the AI problem framing decision

Use this brief template

USER AND MOMENT:
CURRENT WORKFLOW:
OBSERVED PROBLEM:
DESIRED OUTCOME (no AI terms):
ALLOWED INPUTS:
REQUIRED OUTPUT:
ACTION AFTER OUTPUT:
NON-GOALS:
SIMPLE BASELINE:
PRODUCT SUCCESS MEASURES:
MODEL/OUTPUT QUALITY MEASURES:
UNACCEPTABLE FAILURE:
HUMAN REVIEW AND OVERRIDE:
PRIVACY/SECURITY BOUNDARIES:
DECISION AFTER BASELINE:

Common framing mistakes

• Choosing a model first: model names change; the user problem should remain understandable.
• Using “accuracy” as the only goal: averages can hide expensive or harmful failures.
• Predicting without an action: a score that changes no decision creates no user value.
• Automating the whole workflow: start with a narrow assistive step and an override.
• Ignoring the baseline: without it, you cannot show that AI is worth its cost.
• Testing on design examples: memorized examples exaggerate quality. Keep evaluation cases separate.
• Treating abstention as failure: refusing uncertain cases can be a deliberate safety feature.

Checkpoint and scoring rubric

Answer these using your own project, not the support example:

1. What user decision becomes easier if the project succeeds?
2. Which simpler approach is your baseline, and what would make it sufficient?
3. What output will the system return when it is uncertain?
4. Which failure matters more than the average score suggests?
5. What data is unavailable, unreliable or inappropriate to use?
6. Who can override the system, and how will corrections be recorded?

Pass standard: score at least 8/10 with no zero in Risk. A lower score does not mean “add more AI.” It means narrow the project or gather better evidence.

Portfolio evidence worth publishing

Create a small repository or case-study page with:

• problem-brief.md containing the completed template;
• your baseline program and exact run instructions;
• test-cases.csv with synthetic inputs, expected queue and actual result;
• a short failure analysis, including ambiguous and out-of-scope cases;
• a decision memo explaining whether AI is justified and what you would test next.

This is stronger than claiming you “built an AI app.” It demonstrates product judgment, reproducible engineering and awareness of failure:all before expensive implementation begins.

Primary references

• Google for Developers: Machine Learning Problem Framing
• Google for Developers: Understand the Problem
• Google for Developers: Frame an ML Problem
• NIST AI RMF Core: Map
• People + AI Guidebook: User Needs and Defining Success

These sources describe a method, not a guarantee that a particular system is safe or production-ready. Recheck requirements when the users, data, model, deployment setting or potential impact changes.

Created and reviewed by Muhammad Azhar, Lead Software Engineer with more than 16 years of software-development experience.