Self-Attention, Subword Tokenization and Transformer Families

Learning objective

By the end, you can calculate a small attention result, explain why token counts differ from word counts, and choose an encoder, decoder or encoder-decoder family for a stated NLP task.

From text to contextual vectors

A tokenizer first maps text to subword units. A familiar word may become one token, while a name, spelling variation or Urdu-English code-switched phrase may split into several. This matters because model limits and usage costs are measured in tokens. Always inspect the tokenizer used by the exact checkpoint rather than estimating from spaces.

For one attention head, the model projects each input vector into a query, key and value. It scores every allowed query-key pair, divides by the square root of the key width, applies the mask, then normalizes with softmax. The weighted sum of value vectors is the contextual output. Multi-head attention repeats this operation with different learned projections and combines the results.

Context windows are an engineering limit

A larger context window does not guarantee that every supplied fact will influence the answer equally. Long inputs increase memory and latency, and important evidence can be diluted. Count tokens, reserve room for output, place trusted instructions separately from untrusted documents, and test answer quality by evidence position. For long corpora, retrieval and structured summaries are usually more controllable than sending everything on every request.

Calculate one attention head

Save this as attention_demo.py. The matrices are deliberately tiny so you can inspect the scores rather than trusting a framework.

python -m pip install numpy
python attention_demo.py

# attention_demo.py
import numpy as np

tokens = ["refund", "request", "today"]
q = np.array([[1.0, 0.0], [0.8, 0.2], [0.0, 1.0]])
k = np.array([[1.0, 0.0], [0.5, 0.5], [0.0, 1.0]])
v = np.array([[1.0, 0.0], [0.0, 2.0], [3.0, 1.0]])

scores = q @ k.T / np.sqrt(k.shape[1])
weights = np.exp(scores - scores.max(axis=1, keepdims=True))
weights = weights / weights.sum(axis=1, keepdims=True)
context = weights @ v

for token, row, vector in zip(tokens, weights, context):
    print(token, "weights", np.round(row, 3), "context", np.round(vector, 3))

Expected output

refund weights [0.456 0.32  0.225] context [1.129 0.864]
request weights [0.406 0.328 0.266] context [1.203 0.922]
today weights [0.225 0.32  0.456] context [1.591 1.095]

The exact rows sum to one. The last query gives the largest weight to the last key because their directions align most strongly. A real model learns the projections, adds positional information and runs many layers. This script proves the calculation only, not linguistic quality.

Debug the representation before the model

• Unexpected truncation: print token IDs, token count, special tokens and the truncation side.
• Padding changes predictions: confirm the attention mask marks padding as unavailable.
• Decoder leaks future text: inspect the causal mask rather than assuming the architecture created it.
• Names fragment badly: compare tokenizer coverage on the target language and preserve the checkpoint revision.

Knowledge check

Authoritative references

• Attention Is All You Need
• Hugging Face Tokenizers documentation
• Hugging Face Transformers documentation
• Hugging Face model cards