Multilingual and Low-Resource NLP with Urdu and Code-Switching

Define the language reality

Language labels such as “Urdu” hide important variation. Users may write Urdu script, Roman Urdu, English terms inside Urdu sentences, regional vocabulary or inconsistent spellings. A low-resource setting also affects annotation guides, pretrained model coverage and the availability of fluent reviewers.

Do not translate every test example into English and call the system multilingual. Preserve original text, record script and code-switching attributes, and evaluate each meaningful slice. Translation can be a baseline, but it introduces another model and another source of error.

Data design

• Obtain lawful data with clear consent and retention rules.
• Write labels with bilingual examples and hard boundary cases.
• Keep near-duplicates, authors and conversations in one split to prevent leakage.
• Ask fluent reviewers to inspect errors, not only translations.
• Report support counts beside every score so a tiny slice is visible.

Evaluate one task across three slices

This runnable baseline classifies support messages as delivery or refund using character n-grams. Character features often tolerate spelling variation better than word-only features. Save as multilingual_baseline.py. The miniature data teaches the workflow, not a deployable model.

python -m pip install scikit-learn
python multilingual_baseline.py

# multilingual_baseline.py
from collections import defaultdict
from sklearn.pipeline import make_pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

train = [
    ("refund my payment please", "refund"),
    ("I need my money back", "refund"),
    ("میری رقم واپس کریں", "refund"),
    ("ادائیگی واپس چاہیے", "refund"),
    ("parcel has not arrived", "delivery"),
    ("where is my order", "delivery"),
    ("میرا پارسل نہیں آیا", "delivery"),
    ("order abhi tak deliver nahi hua", "delivery"),
]
test = [
    ("please return the payment", "refund", "English"),
    ("delivery is still missing", "delivery", "English"),
    ("رقم واپس کب ملے گی", "refund", "Urdu script"),
    ("آرڈر ابھی تک نہیں ملا", "delivery", "Urdu script"),
    ("payment wapas chahiye", "refund", "Roman Urdu"),
    ("mera parcel kahan hai", "delivery", "Roman Urdu"),
]

model = make_pipeline(
    TfidfVectorizer(analyzer="char_wb", ngram_range=(3, 5), min_df=1),
    LogisticRegression(max_iter=1000, random_state=7),
)
model.fit([x[0] for x in train], [x[1] for x in train])
predictions = model.predict([x[0] for x in test])

by_slice = defaultdict(lambda: [[], []])
for (_, expected, language), predicted in zip(test, predictions):
    by_slice[language][0].append(expected)
    by_slice[language][1].append(predicted)
    print(language, "expected", expected, "predicted", predicted)

for language, (expected, predicted) in by_slice.items():
    print(language, "n", len(expected),
          "accuracy", accuracy_score(expected, predicted))

Expected output and review

Your predictions may reveal weak Roman Urdu or Urdu-script performance. That is useful evidence, not a reason to hide the slice. With only two examples per slice, accuracy is unstable. Expand the test set, calculate macro F1 and confidence intervals, and review mistakes with fluent speakers.

Modern multilingual model comparison

Compare the character baseline with a multilingual encoder such as XLM-R only after confirming its tokenizer and model card cover the target language. Track tokens per message, truncation, latency and per-slice errors. For generative systems, add meaning preservation, cultural appropriateness and unsupported-claim review.

Code-switching deserves its own slice because language identification can be ambiguous at sentence level. Store the observed script and language mix rather than forcing one label when the evidence is uncertain.

Failures a single average conceals

Review code-switching as communication

Code-switching is not noise to remove automatically. A user may choose English product names because no familiar Urdu equivalent exists, or switch scripts to type faster on a phone. Preserve the original message beside any normalized form. Ask reviewers whether normalization changed politeness, urgency or meaning. For voice or transliteration systems, document the conversion model separately and test names, numbers and borrowed technical terms.

Knowledge check

Primary references

• XLM-R paper
• Universal Dependencies Urdu resources
• Tokenizers documentation
• Dataset cards