Small LMs: Phi, Gemma, On-Device

MLOps Series SOTA & Frontiers

Small LM Family Landscape

Not every workload needs a 70 B-parameter model behind an API paywall. A new generation of small language models (≤ 7 B parameters, often ≤ 3 B) achieves surprisingly competitive quality at a fraction of the cost, latency, and energy. These models are purpose-built for on-device, edge, and cost-sensitive cloud deployment.

Each family brings a different philosophy to small-model design:

Microsoft Phi Series

Phi-1 (1.3 B) pioneered "textbook-quality" data curation — training on carefully filtered, synthetically augmented code data. Phi-2 (2.7 B) scaled the recipe to general NLP, matching 7 B models on many benchmarks. Phi-3 mini (3.8 B) added long-context (128 k) and achieved 69 MMLU — within striking distance of Llama-3-8B. The key insight: data quality > model size.

Google Gemma 1 & 2

Gemma 1 (2 B / 7 B) used decoder-only Transformer with RoPE, GeLU, and RMSNorm — a distilled variant of Gemini's architecture. Gemma 2 introduced sliding-window + global attention alternation and knowledge distillation from a larger Gemma teacher. The 2B variant scores 58 on MMLU, impressive for its parameter count. Released with JAX, PyTorch, and Keras weights.

Meta Llama 3.2 1B / 3B

Meta pruned and distilled from the full Llama 3.1 8B to create 1B and 3B variants. The 3B model retains grouped-query attention and scores ~63 on MMLU. Released with quantized GGUF variants explicitly targeting mobile (iOS via MLX, Android via ExecuTorch). The 1B model fits in under 1 GB quantized — viable for browser-side inference.

Alibaba Qwen 2.5 Series

Qwen 2.5 spans 0.5 B to 72 B with consistent architecture. The 1.5 B and 7 B variants lead many multilingual benchmarks. Notable features: 128 k context by default, SwiGLU activation, and strong tool-calling ability even at small sizes. The 7B variant achieves 71+ MMLU — competitive with models 10× its size from 18 months ago.

Interview insight: When asked "why not just use GPT-4?", frame the answer around latency (p99 < 50 ms for mobile), cost (100× cheaper per token), privacy (data never leaves device), and offline capability.

Distillation & Training Recipes

Small models rarely train from scratch on raw internet text alone. Modern recipes combine knowledge distillation, data curation, and structured training phases to pack maximum capability into minimal parameters.

Core Distillation Techniques

Logit distillation trains the student to match the teacher's full output distribution (soft labels), not just the argmax token. This transfers the teacher's "uncertainty" — the student learns that "happy" and "glad" are both plausible, not just the top-1 choice.

# Knowledge distillation loss — soft label matching import torch import torch.nn.functional as F def distillation_loss(student_logits, teacher_logits, labels, T=4.0, alpha=0.7): """Combine soft-label KD loss with hard-label CE loss.""" # Soft targets: temperature-scaled KL divergence soft_student = F.log_softmax(student_logits / T, dim=-1) soft_teacher = F.softmax(teacher_logits / T, dim=-1) kd_loss = F.kl_div(soft_student, soft_teacher, reduction="batchmean") * (T ** 2) # Hard targets: standard cross-entropy on ground truth ce_loss = F.cross_entropy(student_logits, labels) return alpha * kd_loss + (1 - alpha) * ce_loss

Structured Pruning + Distillation (Llama 3.2 Recipe)

Meta's approach for Llama 3.2 1B / 3B combines depth pruning (removing entire Transformer layers) with width pruning (reducing hidden dimensions), followed by distillation from the 8B teacher. The key steps:

Importance scoring — Rank layers by their gradient-weighted contribution to the loss. Remove the least important 40-60% of layers.
Width reduction — Apply low-rank factorization to attention heads and FFN blocks. GQA head count drops from 8 → 4 (for 3B).
Distillation fine-tuning — Train the pruned model on the teacher's logits for ~100 B tokens. Learning rate is 10× lower than pre-training.
Alignment — Standard RLHF / DPO alignment pass to restore instruction-following quality.

Common pitfall: Naively pruning without distillation typically degrades MMLU by 8-15 points. The distillation step recovers 70-90% of the lost quality. Always pair pruning with a distillation phase.

Data-Centric Training (Phi Recipe)

Microsoft's Phi series demonstrates that a curated 1.4T-token dataset can outperform 10T-token web crawls for small models. The recipe:

Synthetic textbook generation — GPT-4 generates "textbook-quality" explanations of concepts, which the small model trains on.
Web data filtering — A classifier trained on GPT-4 scores filters Common Crawl, keeping only the top 5-10% by educational value.
Code-heavy curriculum — Up to 30% of training data is high-quality code, which transfers reasoning ability to non-code tasks.
Multi-phase training — Phase 1: general pre-training. Phase 2: "textbook" data up-weighting. Phase 3: instruction tuning on curated demonstrations.

Quantization for the Edge

Even a 3 B-parameter model at FP16 consumes ~6 GB — too large for most phones and edge devices. Quantization compresses weights (and optionally activations) to 4-bit or lower, cutting memory by 4× and often improving throughput.

Quantization Formats Comparison

GGUF (llama.cpp native)

Block quantization with per-block scale factors. Supports Q2_K through Q8_0 formats. Q4_K_M (4.5 bits effective) is the sweet spot for quality/size. Single-file format with embedded tokenizer. CPU-optimized with AVX2/NEON SIMD kernels. The de facto standard for local inference.

GPTQ (GPU post-training)

Layer-wise quantization using second-order Hessian information (OBQ). Produces 4-bit models with minimal perplexity degradation. Requires a calibration dataset (~128 samples). GPU inference via AutoGPTQ or ExLlama. Best for server-side GPU deployment where you want maximum batch throughput.

AWQ (Activation-Aware)

Identifies salient weight channels (those multiplied by large activations) and preserves them at higher precision. Typically 0.1-0.3 perplexity better than GPTQ at 4-bit. Faster quantization process. Supported by vLLM and TGI for production serving.

QLoRA (Training-time)

Quantizes the base model to 4-bit NormalFloat (NF4) and trains LoRA adapters in FP16 on top. Enables fine-tuning a 7B model on a single 24 GB GPU. The adapters can be merged back and then re-quantized to GGUF for deployment. Essential for cost-effective fine-tuning of small LMs.

# Quantize a model to GGUF Q4_K_M with llama.cpp # Step 1: Convert HuggingFace model to GGUF $ python convert_hf_to_gguf.py \ --model microsoft/Phi-3-mini-4k-instruct \ --outfile phi3-mini-f16.gguf # Step 2: Quantize to Q4_K_M (4.5 bits effective) $ ./llama-quantize phi3-mini-f16.gguf phi3-mini-q4km.gguf Q4_K_M # Result: 3.8B model → ~2.2 GB (from ~7.6 GB FP16) # MMLU drop: typically 1-2 points from FP16 baseline

Quantization rule of thumb: Q4_K_M preserves ~98% of FP16 quality for most tasks. Q3_K_M saves another 15% memory but degrades reasoning tasks noticeably. Q2_K is only viable for simple classification or extraction — avoid for generation.

Mixed-Precision and Importance-Based Quantization

Not all layers tolerate quantization equally. The first and last Transformer layers, plus attention projection matrices, are typically more sensitive. Advanced approaches like AQLM and QuIP# use learned codebooks to achieve 2-bit quantization with less quality loss than naive round-to-nearest:

# Sensitivity-aware mixed precision (conceptual) def assign_bit_widths(model, calibration_data, target_avg_bits=4.0): sensitivities = {} for name, layer in model.named_modules(): if is_linear(layer): # Measure perplexity increase when quantizing this layer to 2-bit sensitivities[name] = measure_layer_sensitivity(layer, calibration_data) # Allocate more bits to sensitive layers, fewer to robust ones bit_assignment = optimize_allocation( sensitivities, target_avg=target_avg_bits, min_bits=2, max_bits=8 ) return bit_assignment # e.g., {"attn.q_proj": 6, "ffn.gate": 3, ...}

On-Device Inference Runtimes

Getting a quantized model onto a phone or IoT device requires specialized inference engines that bypass Python, exploit hardware-specific SIMD/GPU instructions, and manage memory carefully. Three major runtimes dominate.

llama.cpp

The most widely adopted local inference engine. Written in C/C++ with zero dependencies. Key characteristics:

GGUF format — single-file models with embedded metadata and tokenizer.
CPU kernels — Hand-tuned AVX2/AVX-512 (x86), NEON (ARM), and WASM SIMD (browser) kernels.
GPU offloading — Partial or full layer offload to CUDA, Metal, Vulkan, or SYCL backends.
Memory mapping — mmap-based weight loading avoids copying the entire model into RAM.
Server mode — Built-in HTTP API compatible with OpenAI's chat completions format.

# Run Phi-3 mini locally with llama.cpp server $ ./llama-server \ --model phi3-mini-q4km.gguf \ --ctx-size 4096 \ --n-gpu-layers 99 \ # offload all layers to GPU --port 8080 # Client usage — OpenAI-compatible API import openai client = openai.OpenAI(base_url="http://localhost:8080/v1", api_key="none") resp = client.chat.completions.create( model="phi3-mini", messages=[{"role": "user", "content": "Explain quantization in 3 sentences."}], temperature=0.7, max_tokens=256 )

MLC-LLM (Machine Learning Compilation)

Built on Apache TVM, MLC-LLM compiles models into optimized native code for each target platform:

Compile-once, run-anywhere — Generates Metal shaders (iOS/macOS), Vulkan SPIR-V (Android), CUDA kernels (NVIDIA), and WebGPU shaders (browser).
PagedKVCache — Efficient KV-cache paging adapted from vLLM, critical for limited-memory devices.
Structured generation — Built-in JSON mode / grammar-constrained decoding at the engine level.
Quantization-aware compilation — Fuses dequantize + matmul into a single kernel, eliminating memory round-trips.

ExecuTorch (Meta)

PyTorch's official on-device runtime, designed specifically for Llama 3.2 1B/3B deployment:

Export pipeline — torch.export → Edge dialect → Backend delegation (XNNPACK for CPU, CoreML for Apple Neural Engine, QNN for Qualcomm Hexagon DSP).
Operator fusion — Fuses quantized ops (e.g., int4 linear + RMSNorm) for minimal kernel launch overhead.
Memory planning — Static memory allocation avoids malloc at inference time. Peak RAM for Llama 3.2 1B Q4: ~600 MB.
iOS/Android SDKs — Native Swift and Kotlin wrappers with streaming token output.

Runtime selection guide: Use llama.cpp for maximum compatibility and community support. Use MLC-LLM when you need WebGPU/browser deployment or TVM-level kernel optimization. Use ExecuTorch for deep PyTorch ecosystem integration and Apple Neural Engine / Qualcomm DSP delegation.

Mobile & Browser Deployment

Deploying small LMs beyond the server introduces unique constraints: thermal throttling, battery drain, intermittent connectivity, and strict app-size limits. Here's how production deployments handle these challenges.

Mobile Deployment Architecture

# ExecuTorch iOS deployment pipeline (Swift) # 1. Export model with quantization import torch from executorch.exir import to_edge from executorch.backends.apple.coreml import CoreMLBackend model = load_llama32_1b() example_input = torch.randint(0, 32000, (1, 512)) # Export → Edge → CoreML delegation exported = torch.export(model, (example_input,)) edge_prog = to_edge(exported) edge_prog = edge_prog.to_backend(CoreMLBackend) edge_prog.save("llama32_1b_coreml.pte") # .pte file size: ~550 MB (Q4 quantized) # Loads in ~1.2s on iPhone 15 Pro, ~2.8s on iPhone 13

Key Mobile Constraints

App store size limits — iOS: 4 GB max. Android: 150 MB APK + on-demand asset download. Solution: download model weights on first launch, not bundled in app binary.
Thermal throttling — Sustained generation at 30+ tok/s causes thermal throttling within 60 seconds on most phones. Solution: batch processing with cooldown intervals, or cap generation at 20 tok/s.
Memory pressure — iOS kills background apps aggressively when an LM consumes >1.5 GB. Solution: use Q4 quantization (keeps 3B models under 2 GB) and release KV-cache between conversations.
Battery — A 3B model generating 500 tokens uses ~0.3% battery on iPhone 15 Pro (Neural Engine) vs ~1.2% on CPU-only path.

Browser Deployment (WebGPU / WebAssembly)

Running LLMs in the browser enables fully private, zero-install AI experiences. Two approaches:

WebGPU (via MLC-LLM)

Compiles model into WebGPU compute shaders. Requires Chrome 113+ or Edge 113+. Achieves 15-25 tok/s for Phi-3 mini Q4 on desktop GPUs (RTX 3060+). Mobile browser support still limited. Model weights cached in Origin Private File System (OPFS) — persists across sessions.

WASM (via llama.cpp)

Compiles llama.cpp to WebAssembly with SIMD. Works in all modern browsers. Slower than WebGPU (~3-8 tok/s for 1B models) but universally compatible. Good fallback when WebGPU is unavailable. Memory limited to 4 GB in 32-bit WASM; use 1B models.

Production reality: Browser-based LLM inference is impressive for demos but faces real friction: multi-GB downloads on first visit, WebGPU fragmentation across browsers/OSes, and no background execution. Best suited for progressive-enhancement UIs where the LM is optional, not required.

Benchmarks & Latency Profiles

Numbers matter more than marketing. Below are real-world benchmarks measured on representative hardware, not cherry-picked configurations.

Quality Benchmarks (Quantized Models)

Model	Params	Quant	Size	MMLU	HumanEval	GSM8K
Phi-3 mini	3.8B	Q4_K_M	2.2 GB	67.5	58.5	82.1
Gemma 2 2B	2.6B	Q4_K_M	1.6 GB	56.3	36.6	52.4
Llama 3.2 3B	3.2B	Q4_K_M	1.9 GB	61.8	48.2	72.3
Llama 3.2 1B	1.2B	Q4_K_M	0.7 GB	44.6	22.0	33.8
Qwen 2.5 1.5B	1.5B	Q4_K_M	0.9 GB	55.8	42.7	58.2
Qwen 2.5 7B	7.6B	Q4_K_M	4.4 GB	70.6	67.1	85.4

Latency Benchmarks (Tokens/Second)

Model (Q4_K_M)	M2 MacBook	RTX 4090	iPhone 15 Pro	Pixel 8 Pro	WebGPU (3060)
Phi-3 mini 3.8B	38 tok/s	120 tok/s	18 tok/s	12 tok/s	22 tok/s
Llama 3.2 3B	42 tok/s	135 tok/s	21 tok/s	14 tok/s	25 tok/s
Llama 3.2 1B	78 tok/s	210 tok/s	45 tok/s	28 tok/s	40 tok/s
Gemma 2 2B	48 tok/s	145 tok/s	24 tok/s	16 tok/s	28 tok/s
Qwen 2.5 1.5B	65 tok/s	185 tok/s	36 tok/s	22 tok/s	34 tok/s

The 20 tok/s rule: Human reading speed is ~4 words/s ≈ 5 tokens/s. For a satisfying streaming UX, aim for at least 15-20 tok/s on the target device. Below 10 tok/s feels sluggish. All 3B Q4 models clear this bar on modern flagship phones.

Memory Footprint Breakdown

# Memory equation for inference (single request) # Total RAM = Model Weights + KV Cache + Activations + Runtime Overhead # Example: Llama 3.2 3B Q4_K_M, context = 2048 tokens model_weights = 1.9 # GB (quantized weights) kv_cache = 2 * num_layers * 2 * head_dim * n_kv_heads * seq_len * 2 # bytes (FP16) = 2 * 28 * 2 * 128 * 4 * 2048 * 2 / (1024**3) ≈ 0.11 # GB activations ≈ 0.05 # GB (transient, batch_size=1) runtime_overhead= 0.15 # GB (llama.cpp runtime, tokenizer, buffers) total ≈ 2.21 # GB — fits comfortably on 4 GB phone

Interview Checklist

Key questions and crisp answers for MLOps and ML systems design interviews:

Q: When would you choose a small LM over an API?

A: Four scenarios: (1) Latency-critical — p99 < 100 ms, no network round-trip. (2) Privacy-sensitive — PII never leaves the device (healthcare, finance). (3) Offline-required — aviation, field devices, rural connectivity. (4) Cost-sensitive at scale — millions of requests/day where API costs dominate.

Q: How does distillation differ from fine-tuning?

A: Fine-tuning adapts a model to a new task using labeled data. Distillation transfers knowledge from a larger teacher to a smaller student — the student learns the teacher's output distribution, not just ground-truth labels. Distillation can be combined with fine-tuning (task-specific distillation).

Q: What's the quality/size tradeoff at 4-bit?

A: Q4_K_M typically loses 1-2 MMLU points vs FP16 for models ≥ 3B. Below 3B, quality degradation is sharper (3-5 points). The inflection point is around 2B params — below this, consider Q5 or Q6 to preserve quality. Always benchmark on your specific task, not just MMLU.

Q: How do you deploy an LM on iOS?

A: Three paths: (1) ExecuTorch with CoreML backend — best for Apple Neural Engine delegation. (2) llama.cpp with Metal GPU offloading — best community support, GGUF models. (3) MLX (Apple's framework) — native Swift, optimized for Apple Silicon unified memory. Choose based on hardware delegation needs and team expertise.

Parting insight: The small LM space is evolving at breakneck speed. Models that were SOTA 6 months ago are now baseline. The durable skills are understanding distillation mechanics, quantization tradeoffs, and inference optimization principles — these transfer across every new model release.