Fine-Tuning, PEFT, LoRA and Quantization

Learning outcomes

• Distinguish full fine-tuning, LoRA, QLoRA, distillation and preference optimization.
• Run a genuine LoRA gradient update and verify which parameters changed.
• Design an adaptation experiment that measures quality, memory and latency.

Choose the adaptation method

Full fine-tuning updates all trainable weights. LoRA keeps the base model frozen and learns low-rank updates in selected linear layers. PEFT is the wider family of parameter-efficient methods that includes LoRA. QLoRA loads the frozen base at low precision while training LoRA adapters, reducing memory use on supported hardware.

Quantization changes numerical representation. Weight-only or weight-and-activation approaches can reduce memory, but hardware support, latency and quality must be measured on the deployment device. It is not a substitute for pruning bad training examples.

Instruction tuning and preference alignment

Instruction tuning uses prompt-response examples to teach a generative model to follow task instructions. Preference optimization uses comparisons such as a chosen response and a rejected response. RLHF commonly trains a reward model from human preferences and then optimizes a policy. Direct Preference Optimization, or DPO, learns from preference pairs without a separate reward-model training loop. These methods solve different problems from supervised classification. A preference label is not automatically evidence that an answer is factual, safe or culturally appropriate.

Start with supervised fine-tuning when the required output is explicit and verifiable. Consider preference optimization only when relative response quality is the real target and reviewers can apply a reliable rubric. Keep a held-out safety and factuality set outside the optimization data.

Use knowledge distillation when the deployed model must be smaller

Quantization changes how one model stores or computes its weights. Knowledge distillation trains a separate student model to reproduce useful behaviour from a larger teacher. The teacher is needed during training, but the deployed student runs independently. A smaller student may reduce memory or latency, yet the benefit must be measured on the target hardware rather than inferred from parameter count.

Combine soft targets with the verified label

The teacher’s full probability distribution contains more information than its winning class. Temperature scaling softens that distribution so the student can see which alternatives the teacher considers plausible. A common training objective combines cross-entropy on the true label with a temperature-scaled divergence between teacher and student probabilities. Record the teacher revision, temperature and loss weights with the dataset and split.

import torch
import torch.nn.functional as F

teacher_logits = torch.tensor([[5.0, 2.0, -1.0]])
student_logits = torch.tensor([[2.5, 1.0, 0.0]], requires_grad=True)
label = torch.tensor([0])
temperature = 2.0
soft_weight = 0.7

teacher_soft = F.softmax(teacher_logits / temperature, dim=-1)
student_log_soft = F.log_softmax(student_logits / temperature, dim=-1)
soft_loss = F.kl_div(
    student_log_soft, teacher_soft, reduction="batchmean"
) * temperature**2
hard_loss = F.cross_entropy(student_logits, label)
loss = soft_weight * soft_loss + (1 - soft_weight) * hard_loss

print("teacher_soft", teacher_soft.detach().round(decimals=3).tolist())
print("soft_loss", round(soft_loss.item(), 3))
print("hard_loss", round(hard_loss.item(), 3))
print("combined_loss", round(loss.item(), 3))

Expected calculation

teacher_soft [[0.786, 0.175, 0.039]]
soft_loss 0.494
hard_loss 0.266
combined_loss 0.426

This calculation demonstrates the loss, not a quality gain. Train the student without touching the final test set, then compare task metrics, per-class errors, calibration, peak memory and latency. Test rare labels and multilingual or code-switched examples separately because compression can remove the behaviour that justified the teacher.

Distillation can transfer teacher mistakes, bias and overconfidence. Use data you are allowed to process, preserve the teacher and dataset licences, and never generate training targets from final test examples. Reject the student when its measured efficiency gain does not justify its quality or safety loss.

Run a real LoRA update

This small CPU-friendly lab performs actual gradient updates on LoRA adapters for sentiment classification. Save it as train_lora_demo.py. It is a mechanism demonstration, not a meaningful benchmark because the dataset is intentionally tiny.

python -m pip install "transformers>=4.45" "peft>=0.13" torch
python train_lora_demo.py

# train_lora_demo.py
import torch
from peft import LoraConfig, TaskType, get_peft_model
from transformers import AutoModelForSequenceClassification, AutoTokenizer

model_id = "distilbert/distilbert-base-uncased"
texts = ["clear and helpful", "worked perfectly", "easy setup",
         "confusing instructions", "failed twice", "very slow"]
labels = torch.tensor([1, 1, 1, 0, 0, 0])

tokenizer = AutoTokenizer.from_pretrained(model_id)
batch = tokenizer(texts, padding=True, truncation=True, return_tensors="pt")
base = AutoModelForSequenceClassification.from_pretrained(model_id, num_labels=2)
config = LoraConfig(
    task_type=TaskType.SEQ_CLS,
    r=4,
    lora_alpha=8,
    lora_dropout=0.05,
    target_modules=["q_lin", "v_lin"],
)
model = get_peft_model(base, config)
optimizer = torch.optim.AdamW(
    [p for p in model.parameters() if p.requires_grad], lr=5e-4
)

model.train()
before = None
for step in range(4):
    optimizer.zero_grad()
    loss = model(**batch, labels=labels).loss
    before = loss.item() if before is None else before
    loss.backward()
    optimizer.step()
    print(step, round(loss.item(), 4))

trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
total = sum(p.numel() for p in model.parameters())
print("trainable_percent", round(100 * trainable / total, 3))
model.save_pretrained("sentiment-lora-demo")
assert loss.item() != before
assert trainable < total

The loss usually changes and the trainable percentage remains small. A falling training loss on six examples proves only that updates occurred. A real experiment needs a leakage-safe train, validation and test split, a frozen baseline, per-class error analysis and repeated runs.

Bounded low-bit loading example

The following configuration requires a CUDA device and a supported bitsandbytes build. It is shown separately so CPU learners do not confuse an unavailable backend with a modelling error.

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch

quant = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained(
    "your-reviewed-model-id",
    quantization_config=quant,
    device_map="auto",
)
print(model.get_memory_footprint())

Replace the placeholder with a licensed model that supports the task. Compare memory, latency and held-out quality against the non-quantized checkpoint on the same machine.

Failure patterns worth catching early

Knowledge check

Authoritative references

• PEFT documentation
• bitsandbytes integration
• Transformers sequence classification guide
• Direct Preference Optimization paper
• Distilling the Knowledge in a Neural Network