Capstone: Build a Multilingual Knowledge Discovery System

Learning outcomes

• Fit classification, clustering, anomaly-detection and retrieval components on one corpus.
• Interpret each output without converting a score into an unsupported claim.
• Package evaluation, security, deployment and rollback evidence in a portfolio project.

Problem and deliverables

The scenario is support-ticket triage across English, Urdu script and Roman Urdu. The system classifies known intents, clusters messages to reveal recurring themes, flags unusual text for review and retrieves a cited example. This combines supervised NLP, unsupervised data mining, anomaly detection and evidence retrieval.

Your final project must include a lawful source inventory, label guide, grouped splits, reproducible environment, baseline comparison, per-language evaluation, threat model, API contract, monitoring plan and rollback drill. Do not claim business impact from the tiny learning fixture below.

Run the integrated local pipeline

Save as capstone_pipeline.py. It performs real model fitting and returns real outputs from the fixture corpus.

python -m pip install scikit-learn numpy
python capstone_pipeline.py

# capstone_pipeline.py
from dataclasses import dataclass
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.cluster import KMeans
from sklearn.ensemble import IsolationForest
from sklearn.metrics.pairwise import cosine_similarity

@dataclass(frozen=True)
class Ticket:
    ticket_id: str
    language: str
    text: str
    label: str

tickets = [
    Ticket("t1", "en", "refund my payment", "refund"),
    Ticket("t2", "en", "money back for the order", "refund"),
    Ticket("t3", "ur", "میری رقم واپس کریں", "refund"),
    Ticket("t4", "roman-ur", "payment wapas chahiye", "refund"),
    Ticket("t5", "en", "parcel has not arrived", "delivery"),
    Ticket("t6", "en", "where is my delivery", "delivery"),
    Ticket("t7", "ur", "میرا پارسل نہیں آیا", "delivery"),
    Ticket("t8", "roman-ur", "order abhi tak nahi mila", "delivery"),
    Ticket("t9", "en", "cannot sign in to account", "account"),
    Ticket("t10", "en", "password reset failed", "account"),
    Ticket("t11", "ur", "اکاؤنٹ نہیں کھل رہا", "account"),
    Ticket("t12", "roman-ur", "login kaam nahi kar raha", "account"),
]

vectorizer = TfidfVectorizer(analyzer="char_wb", ngram_range=(3, 5))
x = vectorizer.fit_transform([t.text for t in tickets])
y = [t.label for t in tickets]

classifier = LogisticRegression(max_iter=1000, random_state=7).fit(x, y)
clusters = KMeans(n_clusters=3, n_init=20, random_state=7).fit_predict(x)
anomalies = IsolationForest(contamination=0.17, random_state=7).fit_predict(x.toarray())

query = "mera refund kab milega"
query_vector = vectorizer.transform([query])
query_label = classifier.predict(query_vector)[0]
similarity = cosine_similarity(query_vector, x).ravel()
winner_index = int(np.argmax(similarity))
winner = tickets[winner_index]

print("query_label", query_label)
print("citation", winner.ticket_id, winner.language, winner.text)
print("similarity", round(float(similarity[winner_index]), 3))
print("discovery")
for ticket, cluster, anomaly in zip(tickets, clusters, anomalies):
    print(ticket.ticket_id, ticket.label, "cluster", cluster,
          "review" if anomaly == -1 else "normal")

assert query_label in {"refund", "delivery", "account"}
assert winner.ticket_id in {t.ticket_id for t in tickets}
assert len(clusters) == len(tickets) == len(anomalies)

Expected output and validation

Interpret, do not overstate

The query should normally classify as refund and retrieve a related refund example, although similarity and exact cluster numbering can vary by library version. Cluster IDs have no inherent meaning. Inspect their members before naming a theme. Isolation Forest flags statistical outliers in this representation; it does not prove fraud, abuse or bad data.

The classifier is fitted and inspected on the same tiny corpus, so this code does not report accuracy. Your real capstone must reserve a grouped test set and compare against majority and lexical baselines.

Turn the lab into a portfolio system

1. Govern data: record licences, consent, retention, language and access policy.
2. Prevent leakage: group messages from the same conversation or author in one split.
3. Benchmark: report macro F1 and per-language errors for classification, silhouette and reviewed themes for clustering, precision at a review budget for anomaly flags, and recall at 5 for retrieval.
4. Add evidence: store immutable document versions and reconstruct cited excerpts.
5. Secure generation: make any LLM answer optional, evidence-bound and unable to execute tools.
6. Serve: expose validated single and batch endpoints with model versions.
7. Operate: monitor errors, latency, language mix, drift and reviewed outcomes.
8. Release: canary the complete bundle and rehearse rollback.

Architecture decisions to defend

Failure investigation

• If Urdu performance falls, inspect support, labels, tokenizer coverage and review quality separately.
• If clusters merely separate scripts, compare representations and decide whether that grouping helps the user.
• If every rare phrase becomes an anomaly, tune on a reviewed validation set rather than lowering the threshold blindly.
• If citations change after reindexing, restore immutable source versions and stable offsets.
• If rollback changes output shape, the deployment bundle was incomplete.

Knowledge check

Capstone references

• Dataset cards
• Model cards
• OWASP Top 10 for LLM Applications
• NIST Generative AI Profile
• Original RAG paper
• FastAPI documentation