Prompt Caching: Reducing Latency and Cost

KV cache reuse across Anthropic, OpenAI, and self-hosted engines — mechanics, pricing models, and strategies for maximizing cache hit rates in production.

Why Cache Prompts

Modern LLM applications repeatedly send near-identical prompts. A customer-support chatbot prepends the same 4,000-token system prompt to every user query. A RAG pipeline prefixes the same retrieval context across a batch of follow-up questions. A coding assistant re-sends the same repository context on each edit. Without caching, the model recomputes the KV projections for these shared tokens on every single request — wasting both compute and money.

The three main motivations for prompt caching:

• Cost reduction: Cached input tokens cost 50–90% less than fresh tokens across major API providers. For high-volume workloads, this translates to thousands of dollars in monthly savings.
• Latency reduction: Prefill is the dominant phase for long-context requests. Skipping prefill on cached tokens reduces time-to-first-token (TTFT) proportional to the cached prefix length.
• Throughput improvement: Fewer FLOPs per request means the same GPU fleet handles more concurrent requests.

KV Cache Reuse Mechanics

Prompt caching fundamentally reuses the Key-Value projections computed during the prefill phase. In a transformer decoder, each layer produces K and V matrices for every input token. These matrices are stored in GPU memory and reused during autoregressive generation. Prompt caching extends this principle across requests — if two requests share an identical token prefix, the KV entries for that prefix are computed once and shared.

Prefix Matching Rules

Caching works via exact prefix matching. The token sequence from position 0 up to some boundary must be byte-identical between requests. A single different token at position k invalidates the cache for all positions ≥ k. This has practical implications:

• Timestamps, request IDs, or per-request metadata injected into the system prompt destroy cache hits.
• Ordering of few-shot examples must be deterministic — shuffling examples means different token prefixes.
• Even whitespace differences (trailing newline vs. no newline) break the prefix match.

Cache Invalidation

Providers use time-based TTLs. Anthropic caches last 5 minutes (extended on each hit). OpenAI caches persist 5–60 minutes depending on traffic. Self-hosted engines evict via LRU when GPU memory pressure rises. There is no explicit invalidation API — you rely on TTL expiry and natural eviction.

Anthropic Prompt Caching

Anthropic provides explicit, developer-controlled prompt caching via the cache_control parameter. You mark specific message blocks as cache breakpoints, and the API caches KV state up to those breakpoints. This gives fine-grained control over what gets cached and when.

Setting Cache Breakpoints

# Anthropic prompt caching — explicit breakpoints
import anthropic

client = anthropic.Anthropic()

# The system prompt is cached at the breakpoint
response = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": LARGE_SYSTEM_PROMPT,   # 3,000+ tokens
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[
        {"role": "user", "content": user_query}
    ]
)

# Check cache performance in response headers
usage = response.usage
print(f"Cache read tokens:  {usage.cache_read_input_tokens}")
print(f"Cache write tokens: {usage.cache_creation_input_tokens}")
print(f"Uncached tokens:    {usage.input_tokens}")

Multi-Turn Caching with Multiple Breakpoints

# Cache system prompt + few-shot examples + conversation history
response = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": SYSTEM_PROMPT,
            "cache_control": {"type": "ephemeral"}  # Breakpoint 1
        }
    ],
    messages=[
        {"role": "user", "content": [
            {"type": "text", "text": FEW_SHOT_EXAMPLES,
             "cache_control": {"type": "ephemeral"}}  # Breakpoint 2
        ]},
        {"role": "assistant", "content": assistant_reply_1},
        {"role": "user", "content": [
            {"type": "text", "text": conversation_history,
             "cache_control": {"type": "ephemeral"}}  # Breakpoint 3
        ]},
        {"role": "user", "content": current_question}
    ]
)

Anthropic Pricing & TTL

OpenAI Prompt Caching

OpenAI implements automatic prompt caching — no code changes required. The API detects matching prefixes and serves cached KV state transparently. This simplifies adoption but gives less control over caching behavior.

How Automatic Caching Works

# OpenAI automatic caching — no special parameters needed
from openai import OpenAI

client = OpenAI()

# First request — computes and caches KV
response_1 = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": LARGE_SYSTEM_PROMPT},
        {"role": "user", "content": "What is prompt caching?"}
    ]
)

# Second request — same prefix triggers cache hit
response_2 = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": LARGE_SYSTEM_PROMPT},  # identical
        {"role": "user", "content": "How does it reduce costs?"}
    ]
)

# Check cache usage in response
usage = response_2.usage
print(f"Total tokens:         {usage.total_tokens}")
print(f"Cached input tokens:  {usage.prompt_tokens_details.cached_tokens}")

OpenAI Caching Rules

Anthropic vs OpenAI Comparison

Self-Hosted Caching

When running models on your own infrastructure, you control the caching layer directly. Two major engines support prefix caching: vLLM (automatic prefix caching) and SGLang (RadixAttention). Both eliminate redundant prefill computation for shared prefixes.

vLLM Prefix Caching

# Enable prefix caching in vLLM
# Command-line flag — no code changes needed
#   python -m vllm.entrypoints.openai.api_server \
#       --model meta-llama/Llama-3-70B \
#       --enable-prefix-caching \
#       --gpu-memory-utilization 0.90

# Programmatic usage with the LLM engine
from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3-70B",
    enable_prefix_caching=True,
    gpu_memory_utilization=0.90
)

system_prompt = "You are a helpful coding assistant..."  # 3,000 tokens

# All requests sharing this prefix reuse cached KV blocks
prompts = [
    f"{system_prompt}\nUser: Explain decorators\nAssistant:",
    f"{system_prompt}\nUser: What is asyncio?\nAssistant:",
    f"{system_prompt}\nUser: Explain generators\nAssistant:",
]

params = SamplingParams(temperature=0.7, max_tokens=512)
outputs = llm.generate(prompts, params)

# First request computes prefix KV (cache miss)
# Subsequent requests skip prefix prefill (cache hit)

SGLang RadixAttention

# SGLang uses RadixAttention for prefix caching
# It stores prefixes in a radix tree for efficient matching

# Launch with prefix caching enabled (default in SGLang)
#   python -m sglang.launch_server \
#       --model meta-llama/Llama-3-70B \
#       --port 30000

import sglang as sgl

@sgl.function
def cached_qa(s, system_prompt, question):
    s += sgl.system(system_prompt)     # Cached via radix tree
    s += sgl.user(question)
    s += sgl.assistant(sgl.gen("answer", max_tokens=512))

# RadixAttention automatically detects shared prefixes
# across requests and reuses KV cache entries
# Key difference from vLLM: uses radix tree (trie) data structure
# — supports partial prefix matching, not just full prefix

questions = ["What is RLHF?", "Explain DPO", "Define PPO"]
states = cached_qa.run_batch(
    [{"system_prompt": SYSTEM, "question": q} for q in questions],
    num_threads=3
)

Monitoring Cache Performance

# Monitor vLLM prefix cache hit rates via metrics endpoint
import requests

metrics = requests.get("http://localhost:8000/metrics").text

# Key metrics to track
# vllm:prefix_cache_hit_rate        — fraction of blocks served from cache
# vllm:prefix_cache_total_blocks     — total blocks in cache
# vllm:prefix_cache_used_blocks      — blocks currently referenced
# vllm:prefix_cache_evictions_total  — cumulative evictions

def parse_cache_metrics(metrics_text):
    hit_rate = None
    for line in metrics_text.splitlines():
        if "prefix_cache_hit_rate" in line and not line.startswith("#"):
            hit_rate = float(line.split()[-1])
    return hit_rate

hit_rate = parse_cache_metrics(metrics)
print(f"Cache hit rate: {hit_rate:.1%}")

# Alert if hit rate drops below threshold
if hit_rate is not None and hit_rate < 0.70:
    print("⚠ Low cache hit rate — check prompt ordering")

Cost Optimization Strategies

Maximizing the return from prompt caching requires deliberate prompt design. The goal is to maximize the ratio of cached-to-total tokens while maintaining prompt quality.

Strategy 1: Prompt Ordering

Place static, shared content at the beginning of the prompt. Variable, per-request content goes at the end. This maximizes the cacheable prefix length.

Strategy 2: Maximize Cache Hit Rates

# Build prompts with a stable, cacheable prefix
class CacheOptimizedPromptBuilder:
    def __init__(self, system_prompt, few_shot_examples):
        # Static components — deterministic order, no timestamps
        self.system_prompt = system_prompt
        self.few_shot_examples = sorted(few_shot_examples, key=lambda x: x["id"])

    def build_messages(self, user_query, conversation_history=None):
        messages = []

        # Layer 1: System prompt (always first, always identical)
        messages.append({
            "role": "system",
            "content": self.system_prompt
        })

        # Layer 2: Few-shot examples (sorted, deterministic)
        for ex in self.few_shot_examples:
            messages.append({"role": "user", "content": ex["input"]})
            messages.append({"role": "assistant", "content": ex["output"]})

        # Layer 3: Conversation history (grows per turn)
        if conversation_history:
            messages.extend(conversation_history)

        # Layer 4: Current query (variable, uncached)
        messages.append({"role": "user", "content": user_query})

        return messages

Strategy 3: Monitor and Tune

# Track caching efficiency across requests
class CacheMonitor:
    def __init__(self):
        self.total_input_tokens = 0
        self.cached_tokens = 0
        self.cache_write_tokens = 0
        self.request_count = 0

    def record_anthropic(self, usage):
        self.request_count += 1
        self.total_input_tokens += usage.input_tokens
        self.cached_tokens += getattr(usage, "cache_read_input_tokens", 0)
        self.cache_write_tokens += getattr(usage, "cache_creation_input_tokens", 0)

    def record_openai(self, usage):
        self.request_count += 1
        self.total_input_tokens += usage.prompt_tokens
        details = getattr(usage, "prompt_tokens_details", None)
        if details:
            self.cached_tokens += getattr(details, "cached_tokens", 0)

    @property
    def hit_rate(self):
        if self.total_input_tokens == 0:
            return 0.0
        return self.cached_tokens / self.total_input_tokens

    def report(self):
        print(f"Requests:       {self.request_count}")
        print(f"Cache hit rate: {self.hit_rate:.1%}")
        print(f"Cached tokens:  {self.cached_tokens:,}")
        print(f"Total input:    {self.total_input_tokens:,}")
        # Estimate savings (Anthropic Sonnet pricing)
        base_cost = self.total_input_tokens * 3.0 / 1_000_000
        actual_cost = (
            (self.total_input_tokens - self.cached_tokens) * 3.0 / 1_000_000
            + self.cached_tokens * 0.30 / 1_000_000
            + self.cache_write_tokens * 3.75 / 1_000_000
        )
        print(f"Est. savings:   ${base_cost - actual_cost:.2f}")

Common Anti-Patterns