GGUF and llama.cpp

GGUF Format Overview

GGUF (GPT-Generated Unified Format) is a binary file format designed by Georgi Gerganov for efficient local inference with llama.cpp. It replaced the older GGML format in August 2023, adding a structured metadata system and extensible architecture. GGUF is now the de facto standard for running quantized LLMs on consumer hardware — CPUs, Apple Silicon, and consumer GPUs.

Key design decisions in GGUF:

• Single-file format: Metadata, tokenizer, and all tensor data in one file. No sidecar files needed.
• Memory-mapped loading: Tensor data is aligned for direct mmap(), enabling ~1 second load times regardless of model size.
• Per-tensor quantization type: Different layers can use different quantization levels. Attention layers might use Q6_K while MLP layers use Q4_K — this is the foundation of k-quant "mixed" strategies.
• Extensible metadata: Arbitrary key-value pairs store architecture, tokenizer, training hyperparameters, and quantization details.

Quantization Types Explained

GGUF supports a rich set of quantization types. Each type defines how a block of weights (typically 32 or 256 values) is stored. Every block has a scale factor (and sometimes a minimum/offset) stored in higher precision.

K-Quants vs Legacy Quants

The "K" in Q4_K, Q5_K, etc. stands for k-quantization, introduced by @ikawrakow in June 2023. K-quants represent a major improvement over legacy quants (Q4_0, Q4_1, Q5_0, Q5_1) through two innovations:

The k-quant improvement is most visible at 4-bit precision. At 8-bit (Q8_0), legacy quants are already near-lossless, so k-quants provide diminishing returns.

Memory Requirements by Model Size

GGUF models need memory for: (1) model weights, (2) KV cache, and (3) compute buffers. The table below shows weight-only memory for popular models and quant types:

Add approximately 1–2 GB for KV cache at 2K context length, or 4–8 GB at 8K context length (varies by model architecture and GQA ratio). A practical rule of thumb: total RAM needed ≈ model weight size × 1.2 for comfortable headroom.

Converting & Quantizing Models

The typical workflow for creating GGUF files from HuggingFace models:

# Step 1: Clone llama.cpp and build
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j$(nproc) GGML_CUDA=1   # GPU support (optional)
# On macOS: make -j$(sysctl -n hw.ncpu) GGML_METAL=1

# Step 2: Convert HuggingFace model to GGUF (FP16)
python convert_hf_to_gguf.py \
    --outfile ./models/llama-3-8b-f16.gguf \
    --outtype f16 \
    /path/to/Meta-Llama-3-8B/

# Step 3: Quantize to desired type
./llama-quantize \
    ./models/llama-3-8b-f16.gguf \
    ./models/llama-3-8b-Q4_K_M.gguf \
    Q4_K_M

# Available quantization types (most common):
#   Q4_0    - legacy 4-bit (fastest, lowest quality)
#   Q4_K_S  - k-quant 4-bit small
#   Q4_K_M  - k-quant 4-bit medium (recommended default)
#   Q5_K_S  - k-quant 5-bit small
#   Q5_K_M  - k-quant 5-bit medium (recommended for quality)
#   Q6_K    - k-quant 6-bit (near-lossless)
#   Q8_0    - 8-bit (highest quality quantized)

# Step 4: Test the quantized model
./llama-cli \
    -m ./models/llama-3-8b-Q4_K_M.gguf \
    -p "The meaning of life is" \
    -n 128 \
    --temp 0.7 \
    --top-p 0.9 \
    -ngl 99   # offload all layers to GPU

For Python users, the llama-cpp-python package provides a clean API:

from llama_cpp import Llama

# Load quantized model
llm = Llama(
    model_path="./models/llama-3-8b-Q4_K_M.gguf",
    n_ctx=4096,           # context window
    n_gpu_layers=-1,      # -1 = offload all layers to GPU
    n_threads=8,          # CPU threads for non-offloaded layers
    verbose=False,
)

# Chat completion (OpenAI-compatible API)
response = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain quantum computing in 3 sentences."}
    ],
    max_tokens=256,
    temperature=0.7,
)
print(response["choices"][0]["message"]["content"])

llama.cpp Server Setup

llama.cpp includes a built-in HTTP server that provides an OpenAI-compatible API, making it a drop-in replacement for production deployments:

# Start the server with GPU offloading
./llama-server \
    -m ./models/llama-3-8b-Q4_K_M.gguf \
    --host 0.0.0.0 \
    --port 8080 \
    -ngl 99 \                   # GPU layers
    -c 4096 \                   # context size
    --parallel 4 \              # concurrent request slots
    --cont-batching \           # continuous batching
    --metrics \                 # Prometheus metrics endpoint
    --flash-attn                # Flash Attention (if supported)

# Test with curl (OpenAI-compatible endpoint)
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3-8b",
    "messages": [
      {"role": "user", "content": "Hello, how are you?"}
    ],
    "max_tokens": 128,
    "temperature": 0.7
  }'

# Or use the OpenAI Python client directly
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")
response = client.chat.completions.create(
    model="llama-3-8b",
    messages=[{"role": "user", "content": "Tell me about GGUF"}]
)

Hardware Benchmarks

Real-world performance numbers for LLaMA-3-8B Q4_K_M (4.9 GB) across consumer hardware. All measurements at 2048 context length, single request:

For LLaMA-3-70B Q4_K_M (42 GB), the options narrow:

llama.cpp + GGUF has democratized LLM inference, making it possible to run capable models on commodity hardware. The combination of efficient quantization (k-quants), optimized CPU/GPU kernels (SIMD, Metal, CUDA), and a robust serving stack makes it the go-to solution for local LLM deployment.