LoRA & QLoRA: Parameter-Efficient Fine-Tuning

Why Parameter-Efficient Fine-Tuning?

Full fine-tuning of a 7B-parameter model requires storing a complete copy of all weights plus optimizer states. With Adam, that means 3× the model size in GPU memory: the parameters themselves, plus first and second moment estimates. For a 7B model in FP16 that's roughly 7B × 2 bytes × 3 ≈ 42 GB — far exceeding a single A100 40 GB.

Parameter-efficient fine-tuning (PEFT) methods address this by updating only a small subset of parameters while keeping the pretrained base frozen. The key insight is that the weight updates during fine-tuning live in a low-dimensional subspace, so we can parameterize them compactly.

The major PEFT approaches include:

• Adapters: Small bottleneck layers inserted between transformer blocks (Houlsby et al., 2019)
• Prefix Tuning: Learnable virtual tokens prepended to keys/values (Li & Liang, 2021)
• LoRA: Low-rank decomposition of weight update matrices (Hu et al., 2021)
• QLoRA: LoRA with 4-bit quantized base model (Dettmers et al., 2023)

LoRA has become the dominant method because it adds zero inference latency — the low-rank matrices can be merged back into the base weights at deployment time.

LoRA: Low-Rank Adaptation

LoRA (Low-Rank Adaptation) works by freezing all pretrained weights and injecting trainable rank-decomposition matrices into each target layer. Instead of updating the full weight matrix W ∈ ℝ^d×k, we learn a low-rank update ΔW = BA where:

• B ∈ ℝ^d×r — the "up-projection" matrix
• A ∈ ℝ^r×k — the "down-projection" matrix
• r ≪ min(d, k) — the rank, typically 4–64

During training, only B and A receive gradients. The base weight W₀ remains frozen. At initialization:

• A is initialized with Kaiming uniform (random Gaussian)
• B is initialized to zero — so ΔW = BA = 0 at the start

The scaling factor α/r controls the magnitude of the LoRA update. The output becomes:

Typically α = 2r (e.g., α = 16 when r = 8). This scaling ensures the learning rate doesn't need to change much across different rank choices.

Mathematical Foundation

Consider a pretrained weight matrix W₀ ∈ ℝ^d×d (e.g., a query/key/value projection in a transformer with d = 4096). Full fine-tuning learns:

where ΔW ∈ ℝ^d×d has d² = 16.7M trainable parameters per layer. LoRA constrains ΔW to rank r:

The number of trainable parameters drops to 2 × d × r:

Which layers get LoRA adapters? The original paper applies LoRA to query (W_q) and value (W_v) projections in each attention block. Subsequent work shows applying to all linear layers (Q, K, V, O, and MLP up/down/gate) often yields better results:

# Target modules for different architectures

# LLaMA / Mistral
target_modules = [
    "q_proj", "k_proj", "v_proj", "o_proj",
    "gate_proj", "up_proj", "down_proj"
]

# GPT-2 / GPT-NeoX
target_modules = ["c_attn", "c_proj", "c_fc"]

# Falcon
target_modules = ["query_key_value", "dense", "dense_h_to_4h", "dense_4h_to_h"]

Rank Selection Tradeoffs

The rank r is the most important hyperparameter in LoRA. It controls the expressiveness of the update and the memory/compute budget:

Empirical guidelines from community experience:

• Chat/instruction tuning: r = 8, α = 16 (the "safe default")
• Code generation: r = 16–32, α = 32–64 (code requires more expressiveness)
• Domain-specific QA: r = 4–8, α = 8–16 (narrow domain, less capacity needed)
• Mathematical reasoning: r = 16–64, α = 32–128 (complex skill acquisition)

QLoRA: Quantized LoRA

QLoRA (Dettmers et al., 2023) pushes memory efficiency further by combining LoRA with aggressive base-model quantization. The three key innovations are:

1. 4-bit NormalFloat (NF4) Quantization

The base model weights are quantized to 4-bit using a new data type called NF4 — an information-theoretically optimal quantization for normally distributed weights. Each weight is stored as a 4-bit index into a shared codebook per block of 64 weights.

2. Double Quantization

The quantization constants themselves (one FP32 scale per 64-weight block) are further quantized to 8-bit, saving an additional ~0.37 bits per parameter. For a 65B model, this saves ~3 GB.

3. Paged Optimizers

Optimizer states are offloaded to CPU RAM using unified memory (CUDA unified memory / paging), preventing OOM errors during gradient spikes from long sequences.

The forward pass for a QLoRA layer:

Dequantization happens block-by-block during matrix multiplication — the full FP16 weight is never materialized in memory at once. This is why QLoRA achieves near-4-bit memory usage while maintaining BF16 training precision for the LoRA parameters.

Memory & Compute Comparison

The following table shows GPU memory requirements for fine-tuning LLaMA-2 models with different methods (including optimizer states and activations for batch size 1, sequence length 2048):

Interactive: Weight Decomposition

Step through the LoRA decomposition process. See how a large weight update matrix is decomposed into two small low-rank matrices:

PEFT Library Implementation

The Hugging Face PEFT library provides a clean API for applying LoRA and QLoRA to any Hugging Face model. Here's a complete training setup:

Basic LoRA Configuration

from peft import LoraConfig, get_peft_model, TaskType
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load base model
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    torch_dtype=torch.float16,
    device_map="auto"
)

# Define LoRA config
lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=8,                           # rank
    lora_alpha=16,                 # scaling factor (α)
    lora_dropout=0.05,             # dropout on adapter
    target_modules=[               # which layers to adapt
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj"
    ],
    bias="none",                    # don't train biases
)

# Wrap model with LoRA
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Output: trainable params: 4,194,304 || all params: 6,742,609,920 || trainable%: 0.0622%

QLoRA Configuration (4-bit Training)

from transformers import BitsAndBytesConfig
import torch

# 4-bit quantization config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",           # NormalFloat4
    bnb_4bit_compute_dtype=torch.bfloat16, # compute in BF16
    bnb_4bit_use_double_quant=True,      # double quantization
)

# Load model in 4-bit
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    quantization_config=bnb_config,
    device_map="auto",
)

# Prepare model for k-bit training
from peft import prepare_model_for_kbit_training
model = prepare_model_for_kbit_training(model)

# Apply LoRA on top of quantized model
model = get_peft_model(model, lora_config)
# Now: base weights in NF4 (~3.5 GB), LoRA adapters in BF16 (~20 MB)

Training with SFTTrainer

from trl import SFTTrainer
from transformers import TrainingArguments

training_args = TrainingArguments(
    output_dir="./lora-llama2-7b",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,     # effective batch = 16
    learning_rate=2e-4,                # higher LR for LoRA
    weight_decay=0.01,
    warmup_ratio=0.03,
    lr_scheduler_type="cosine",
    logging_steps=10,
    save_strategy="epoch",
    bf16=True,                         # BF16 mixed precision
    gradient_checkpointing=True,       # save activation memory
    optim="paged_adamw_8bit",           # paged optimizer for QLoRA
)

trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    args=training_args,
    max_seq_length=2048,
    dataset_text_field="text",
)

trainer.train()

Merging and Saving

# After training — merge LoRA weights into base
merged_model = model.merge_and_unload()

# Save merged model (same architecture as original)
merged_model.save_pretrained("./llama2-7b-merged")
tokenizer.save_pretrained("./llama2-7b-merged")

# Or save just the adapter (~20MB vs 14GB)
model.save_pretrained("./llama2-7b-lora-adapter")

# Load adapter later on any compatible base
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
model = PeftModel.from_pretrained(base, "./llama2-7b-lora-adapter")

Best Practices & Pitfalls

Hyperparameter Guidelines

• Learning rate: Use 2–5× higher LR than full fine-tuning (typically 1e-4 to 3e-4 vs 2e-5)
• Alpha/rank ratio: Keep α/r ≈ 2. This is the effective scaling of the adapter contribution
• Dropout: 0.05 is a safe default. Increase to 0.1 for small datasets (<10K samples)
• Epochs: 1–3 epochs usually suffice. LoRA overfits faster than full FT due to fewer params
• Target modules: Apply to all linear layers for best results, not just Q/V

Multi-Adapter Serving

One of LoRA's most powerful production features: serve multiple task-specific adapters on a single base model.

# Load base model once in GPU memory
base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")

# Hot-swap adapters per request (each ~20MB)
model = PeftModel.from_pretrained(base_model, "./chat-adapter")
model.load_adapter("./code-adapter", adapter_name="code")
model.load_adapter("./medical-adapter", adapter_name="medical")

# Switch at inference time
model.set_adapter("code")        # ~0ms switch, no model reload
output = model.generate(input_ids)

Common Pitfalls