RAG Fundamentals

MLOps Series RAG Systems

What Is RAG?

Retrieval-Augmented Generation (RAG) is an architecture pattern that grounds a large language model's output in external, up-to-date knowledge. Instead of relying solely on the parametric memory baked into model weights during pre-training, RAG introduces a non-parametric memory — typically a vector store — that the model queries at inference time. The seminal paper by Lewis et al. (2020) formalized this as a latent-variable model where the retriever p(z|x) fetches documents z conditioned on the input x, and the generator p(y|x, z) produces the output conditioned on both.

In practical terms, a RAG system accepts a user query, converts it into a dense vector via an embedding model, performs approximate nearest-neighbor (ANN) search against a pre-indexed corpus, and feeds the top-k retrieved passages alongside the original query into the LLM prompt. This simple but powerful pattern dramatically reduces hallucination, enables citation of sources, and allows the knowledge base to be updated without retraining the model.

Key insight: RAG decouples knowledge storage (the vector DB) from reasoning (the LLM), giving you independent scaling and update cadences for each component.

Why RAG Matters

Pure generative models suffer from three well-documented failure modes that RAG directly addresses:

Knowledge cut-off: Model weights are frozen at a point in time. RAG lets you inject information that postdates training without any weight update.
Hallucination: Without grounding, models confidently fabricate plausible-sounding content. By supplying retrieved evidence, you constrain generation to verifiable claims.
Domain specificity: General-purpose LLMs lack depth in proprietary or niche domains. RAG surfaces your internal knowledge base — PDFs, Confluence pages, code repositories — at inference time.

Beyond correctness, RAG offers operational advantages. Updating a vector index is orders of magnitude cheaper than fine-tuning a 70B-parameter model. You can A/B test different retrieval strategies independently of the generator. And because retrieved passages are visible in the prompt, you get built-in explainability — every answer can cite the chunks that informed it.

Caveat: RAG does not eliminate hallucination entirely. The model can still misinterpret or ignore retrieved context, especially when passages are contradictory or poorly chunked.

RAG Architecture

A production RAG system comprises four logical layers: Ingestion, Indexing, Retrieval, and Generation. During ingestion, raw documents are loaded, cleaned, and split into chunks (see the Chunking Strategies post for details). Each chunk is then embedded into a dense vector and stored in a vector database alongside its metadata (source URL, page number, timestamp).

At query time the user's question is embedded with the same model, and an ANN search returns the top-k most similar chunks. These chunks are injected into the LLM's context window via a system prompt template. The generator produces a response grounded in those passages, optionally returning inline citations.

Advanced variants add a re-ranker (cross-encoder) between retrieval and generation to refine relevance ordering, a query rewriter that transforms ambiguous queries into retrieval-friendly forms, and a guardrails layer that validates the output before returning it to the user.

Interactive RAG Pipeline

Step through the five core stages of a RAG pipeline. Use the controls below to advance or rewind.

Step 1 / 5

The user submits a natural-language query.

RAG vs Fine-Tuning

The most common question when scoping an LLM project is whether to use RAG or fine-tuning. They are complementary rather than mutually exclusive, but the trade-offs differ sharply:

RAG

Knowledge is external and easily updated
No GPU-intensive training required
Latency cost from retrieval hop (~50-200 ms)
Prompt length scales with retrieved context
Best for factual QA over evolving corpora

Fine-Tuning

Knowledge is baked into weights
Requires labeled data + training compute
No retrieval latency at inference
Prompt stays short — knowledge is implicit
Best for style, tone, or domain-specific reasoning

In practice, many production systems combine both: fine-tune a smaller model to follow a specific output format and reasoning style, then augment it with RAG to supply fresh domain knowledge at inference time. This hybrid approach yields the best accuracy-per-dollar trade-off for most enterprise use cases.

Rule of thumb: Start with RAG. Fine-tune only when you need to change how the model reasons or responds — not what it knows.

Python Implementation

Below is a minimal but complete RAG pipeline using OpenAI embeddings and ChromaDB as the vector store. This covers document loading, chunking, indexing, retrieval, and generation in roughly 60 lines of production-ready Python.

import openai import chromadb from chromadb.utils import embedding_functions # ── 1. Initialise clients ── oai = openai.OpenAI() ef = embedding_functions.OpenAIEmbeddingFunction( api_key=oai.api_key, model_name="text-embedding-3-small" ) chroma = chromadb.PersistentClient(path="./chroma_store") collection = chroma.get_or_create_collection( name="docs", embedding_function=ef ) # ── 2. Ingest documents ── def ingest(texts, ids, metadatas=None): """Chunk, embed, and upsert into the collection.""" collection.upsert( documents=texts, ids=ids, metadatas=metadatas or [{} for _ in texts] ) # ── 3. Retrieve top-k chunks ── def retrieve(query: str, k: int = 5): results = collection.query( query_texts=[query], n_results=k ) return results["documents"][0], results["metadatas"][0] # ── 4. Build augmented prompt ── def build_prompt(query, passages): context = "\n---\n".join(passages) return ( f"Answer the question based ONLY on the context below.\n\n" f"Context:\n{context}\n\nQuestion: {query}\nAnswer:" ) # ── 5. Generate grounded answer ── def generate(query: str, k: int = 5) -> str: passages, metas = retrieve(query, k) prompt = build_prompt(query, passages) resp = oai.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": prompt}], temperature=0.2, max_tokens=1024 ) return resp.choices[0].message.content # ── Usage ── ingest( texts=["RAG combines retrieval with generation...", "Vector databases store dense embeddings..."], ids=["doc-1", "doc-2"] ) answer = generate("What is RAG and how does it work?") print(answer)

This skeleton handles the happy path. For production, you would add: retry logic with exponential back-off, token-budget management (truncate retrieved context if it exceeds the model's window), a re-ranker between retrieve and build_prompt, and observability hooks (latency, retrieval recall, answer faithfulness) via a framework like LangSmith or Phoenix.

Next steps: Explore Embedding Models for a deep comparison of encoders, and Chunking Strategies for optimal document splitting.