Normalization, Stemming and Lemmatization

Choose Normalisation, Stemming or Lemmatization for the Task

• Compare surface forms, stems and lemmas rather than assuming they are interchangeable.
• Create a preprocessing policy that names preserved features such as negation and identifiers.
• Keep raw text and compare alternatives on the same split with the same metric.
• Compare the alternatives on one fixed split, then document which linguistic signals each transformation removes.

Three transformations, three purposes

Case folding supports caseless comparison. Stemming reduces spelling variants cheaply. Lemmatization produces a more readable base form when the language model recognises the word.

None guarantees better accuracy. Their value depends on dataset size, language, model and the errors the product can tolerate.

Context changes the lemma

Saw can be a noun or the past tense of see. A lemmatiser that lacks part-of-speech information can select the wrong base form.

When less cleaning is the better baseline

Character n-grams and modern subword models already handle many surface variations. Extra stemming can damage proper names, hashtags or morphology.

Stopword removal can erase not, without or before, which may reverse meaning. Begin with a minimal transform and add rules only when error analysis supports them.

Make transformations observable

Log a sample of before-and-after token pairs and count how often different words collapse to the same form.

Review the largest collision groups before accepting the policy.

Build the Normalization, Stemming and Lemmatization example

Install NLTK and download the WordNet resource once with nltk.download(“wordnet”) before running the example. The explicit verb tag shows that lemmatisation depends on grammatical assumptions.

The stem studi is useful as an internal feature but poor display text. The lemma study is readable, while connection remains unchanged as a verb because the supplied part of speech is inappropriate.

Install:

python -m pip install nltk

from nltk.stem import PorterStemmer, WordNetLemmatizer

stemmer = PorterStemmer()
lemmatizer = WordNetLemmatizer()
words = ['studies', 'studying', 'better', 'denied', 'connection']
for word in words:
    stem = stemmer.stem(word)
    lemma = lemmatizer.lemmatize(word, pos='v')
    print(f'{word:12} stem={stem:10} verb_lemma={lemma}')

Results to Compare Before Choosing a Policy

studies      stem=studi      verb_lemma=study
studying     stem=studi      verb_lemma=study
better       stem=better     verb_lemma=better
denied       stem=deni       verb_lemma=deny
connection   stem=connect    verb_lemma=connection

Do not show stems to users as corrected language. Treat them as model features.

For multilingual data, use a language-specific tokenizer and lemmatiser, then evaluate each language separately.

Diagnose failures in Normalization, Stemming and Lemmatization

Inspect collision examples and downstream errors, not only a shorter vocabulary.

Check your Normalization, Stemming and Lemmatization reasoning

Decide which transformation belongs in each scenario before opening the explanations. Pay attention to search recall and lost distinctions.

Primary references for Normalization, Stemming and Lemmatization

• NLTK documentation: official language-processing toolkit guidance.
• spaCy linguistic features: official tags, dependencies and entity guidance.

Repeat the policy comparison after a stemmer, lemmatiser or language model update. Archive the exceptions and tested package versions.