Language Foundations and Text Preprocessing

Why this module belongs in the course

Text preprocessing should preserve meaning by default. Every normalisation step needs a reason because punctuation, case, spelling and word form can carry useful information.

Before you begin

The concepts and project evidence from Module 8. You should also be able to create a Python virtual environment and keep private or employer data out of the exercise.

Four lessons, one connected result

1. Lesson 1Unicode, Encodings and Real-World Text55 min · Beginner
2. Lesson 2Regex and Tokenization Without Losing Meaning65 min · Beginner
3. Lesson 3Normalization, Stemming and Lemmatization60 min · Beginner
4. Lesson 4Project: Audit a Reproducible Text Preprocessing Pipeline90 min · Intermediate

How to know you are ready to continue

Complete the checkpoint without copying the worked example. Keep the code, output and a short decision note. Your note should explain one choice, one failure you observed and one limitation a reviewer should know.

Primary references for this module

The lessons explain the ideas in original wording. Use these primary or official sources when a library interface, standard or research claim needs verification.

• https://unicode.org/reports/tr29/
• https://docs.python.org/3/howto/unicode.html
• https://docs.python.org/3/library/re.html
• https://www.nltk.org/
• https://spacy.io/usage/linguistic-features
• https://huggingface.co/docs/hub/datasets-cards