Project: Compare Five Transformer NLP Tasks

Project outcome

You will run classification, named entity recognition, extractive question answering, summarization and translation on fixed examples. The deliverable is a task matrix containing model revision, input, raw output, latency, evidence type, failure notes and the metric you would use on a larger test set.

The five checkpoints download several models. Run them one at a time if memory or bandwidth is limited. Public model IDs can change, so record the resolved revision in your report.

Build the five-task lab

Save as compare_transformer_tasks.py. The code executes every task instead of printing a prewritten description.

python -m pip install "transformers>=4.45" torch sentencepiece sacremoses
python compare_transformer_tasks.py

# compare_transformer_tasks.py
from time import perf_counter
from transformers import pipeline

jobs = [
    ("classification", "text-classification",
     "distilbert/distilbert-base-uncased-finetuned-sst-2-english",
     {"text_inputs": "The setup was clear, but search is unreliable."}),
    ("ner", "token-classification", "dslim/bert-base-NER",
     {"inputs": "Sara joined MetaCyberGuru in Berlin."}),
    ("qa", "question-answering",
     "distilbert/distilbert-base-cased-distilled-squad",
     {"question": "Where did Sara join?",
      "context": "Sara joined MetaCyberGuru in Berlin."}),
    ("summarization", "summarization", "sshleifer/distilbart-cnn-12-6",
     {"text_inputs": ("The release adds search filters and clearer error messages. "
                      "It also fixes mobile navigation and reduces image size. "
                      "The team measured the changes on three phone widths before launch.")}),
    ("translation", "translation", "Helsinki-NLP/opus-mt-en-de",
     {"text_inputs": "The course is free and practical."}),
]

for name, task, model_id, kwargs in jobs:
    started = perf_counter()
    if name == "ner":
        runner = pipeline(task, model=model_id, aggregation_strategy="simple")
    else:
        runner = pipeline(task, model=model_id)
    output = runner(**kwargs)
    elapsed = perf_counter() - started
    print("\nTASK", name)
    print("MODEL", model_id)
    print("SECONDS_WITH_LOAD", round(elapsed, 2))
    print("OUTPUT", output)

If your Transformers version rejects aggregation_strategy=None for a non-NER task, create the NER pipeline in its own branch and omit that argument for the others. The first run includes download and model-loading time, so it is not an inference benchmark. Time a warm second call when latency matters.

Compare outputs honestly

Do not rank tasks by their returned score. A classifier score, token score and QA span score have different meanings. Instead, create ten to thirty reviewed examples per task and use a suitable rubric or metric. Record abstentions and human disagreements.

Context budget check

Before summarization or QA, count tokens and reserve output capacity. Test whether truncation removes evidence. If the document exceeds the model window, split by structure and evaluate a map-reduce or retrieval approach instead of silently dropping the end.

Required project files

• compare_transformer_tasks.py with pinned model revisions
• fixtures.jsonl containing licensed or original examples
• results.jsonl preserving raw outputs and warm latency
• task-matrix.md explaining evidence, metrics and observed failures
• README.md with setup, hardware, versions and limitations

Definition of done: all five tasks run, outputs are preserved, and each task has at least one manually checked failure case. A failed model download should be reported, not replaced with invented output.

Troubleshooting

• If a translation tokenizer fails, install SentencePiece and restart the environment.
• If the process runs out of memory, delete each pipeline after its result and run the jobs sequentially.
• If NER returns fragments, enable aggregation and still verify the original offsets.
• If a short summary copies the input, increase source length before changing decoding parameters.

Knowledge check

Primary references

• Transformers documentation
• Evaluate documentation
• Model cards
• Dataset cards