RAG Pipelines: LangChain, LlamaIndex, Haystack
The three dominant frameworks for building RAG pipelines — LangChain, LlamaIndex, and Haystack — each take fundamentally different approaches to the same problem. LangChain prioritizes composability through chains and agents, LlamaIndex focuses on data indexing and retrieval abstractions, and Haystack emphasizes production-grade pipeline orchestration. This post builds the same RAG pipeline in all three so you can compare directly.
Framework Overview
Before diving into code, it helps to understand each framework’s philosophy and where it excels:
- LangChain (~85k GitHub stars): Broadest ecosystem with 700+ integrations. The LangChain Expression Language (LCEL) enables declarative pipeline composition. Best for prototyping and when you need extensive tool/agent support.
- LlamaIndex (~35k stars): Purpose-built for RAG and data augmentation. Provides the most sophisticated indexing strategies (tree, keyword-table, knowledge graph). Best when your primary use case is document Q&A.
- Haystack (~18k stars): Originally built by deepset for enterprise search. Strong typing, explicit pipeline graphs, and built-in evaluation. Best for production deployments with strict reliability requirements.
LangChain RAG Pipeline
LangChain’s LCEL (LangChain Expression Language) lets you compose retrieval and generation steps using the pipe (|) operator. Each step is a Runnable that transforms input to output, and the pipeline supports streaming, batching, and async natively.
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
from langchain_community.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser
# 1. Load and chunk documents
loader = PyPDFLoader("technical_report.pdf")
docs = loader.load()
splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200,
separators=["\n\n", "\n", ". ", " "]
)
chunks = splitter.split_documents(docs)
# 2. Create vector store and retriever
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma.from_documents(chunks, embeddings)
retriever = vectorstore.as_retriever(
search_type="mmr", # Maximal Marginal Relevance
search_kwargs={"k": 5, "fetch_k": 20}
)
# 3. Define prompt template
prompt = ChatPromptTemplate.from_template("""Answer the question based only on the context below.
If the context doesn't contain the answer, say "I don't know."
Context: {context}
Question: {question}""")
# 4. Build LCEL chain
llm = ChatOpenAI(model="gpt-4o", temperature=0)
def format_docs(docs):
return "\n\n".join(d.page_content for d in docs)
chain = (
{"context": retriever | format_docs, "question": RunnablePassthrough()}
| prompt
| llm
| StrOutputParser()
)
# 5. Query with streaming
for chunk in chain.stream("What were the key findings?"):
print(chunk, end="")
LangChain strength: LCEL’s pipe operator makes it trivial to swap components. Replace
Chroma with Pinecone, or ChatOpenAI with ChatAnthropic, and the rest of the pipeline stays identical.
LlamaIndex RAG Pipeline
LlamaIndex takes a data-centric approach. Instead of composing chains, you build an Index from your documents and query it through a QueryEngine. LlamaIndex manages chunking, embedding, storage, and retrieval behind high-level abstractions while allowing full customization at every layer.
from llama_index.core import (
VectorStoreIndex, SimpleDirectoryReader, Settings
)
from llama_index.core.node_parser import SentenceSplitter
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.core.postprocessor import (
SimilarityPostprocessor, KeywordNodePostprocessor
)
# 1. Configure global settings
Settings.llm = OpenAI(model="gpt-4o", temperature=0)
Settings.embed_model = OpenAIEmbedding(model_name="text-embedding-3-small")
Settings.node_parser = SentenceSplitter(chunk_size=1024, chunk_overlap=200)
# 2. Load documents and build index
documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)
# 3. Create query engine with post-processors
query_engine = index.as_query_engine(
similarity_top_k=5,
node_postprocessors=[
SimilarityPostprocessor(similarity_cutoff=0.7),
KeywordNodePostprocessor(required_keywords=["revenue"]),
],
response_mode="compact", # Stuff all chunks into one prompt
)
# 4. Query
response = query_engine.query("What were the Q3 revenue figures?")
print(response)
print(f"Sources: {[n.node.metadata['file_name'] for n in response.source_nodes]}")
# 5. Advanced: Custom retriever with hybrid search
from llama_index.core.retrievers import VectorIndexRetriever
from llama_index.retrievers.bm25 import BM25Retriever
from llama_index.core.retrievers import QueryFusionRetriever
vector_retriever = VectorIndexRetriever(index=index, similarity_top_k=5)
bm25_retriever = BM25Retriever.from_defaults(index=index, similarity_top_k=5)
# Reciprocal Rank Fusion combines both retrievers
hybrid_retriever = QueryFusionRetriever(
[vector_retriever, bm25_retriever],
similarity_top_k=5,
num_queries=1,
mode="reciprocal_rerank",
)
LlamaIndex strength: Built-in support for 10+ index types (vector, tree, keyword-table, knowledge graph, SQL) and sophisticated response synthesis modes (refine, compact, tree_summarize).
Haystack RAG Pipeline
Haystack 2.x uses a strongly-typed pipeline architecture where components declare their inputs and outputs as typed slots. The pipeline validates the graph at build time, catching wiring errors before runtime. This makes Haystack the most production-friendly framework.
from haystack import Pipeline
from haystack.components.preprocessors import DocumentCleaner, DocumentSplitter
from haystack.components.writers import DocumentWriter
from haystack.components.embedders import (
OpenAIDocumentEmbedder, OpenAITextEmbedder
)
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
from haystack.components.builders import PromptBuilder
from haystack.components.generators import OpenAIGenerator
from haystack.document_stores.in_memory import InMemoryDocumentStore
# 1. Build indexing pipeline
doc_store = InMemoryDocumentStore()
indexing = Pipeline()
indexing.add_component("cleaner", DocumentCleaner())
indexing.add_component("splitter", DocumentSplitter(
split_by="sentence", split_length=5, split_overlap=1
))
indexing.add_component("embedder", OpenAIDocumentEmbedder())
indexing.add_component("writer", DocumentWriter(document_store=doc_store))
# Connect components explicitly
indexing.connect("cleaner", "splitter")
indexing.connect("splitter", "embedder")
indexing.connect("embedder", "writer")
# 2. Build query pipeline
template = """Answer the question based on the context.
Context:
{% for doc in documents %}
{{ doc.content }}
{% endfor %}
Question: {{ question }}"""
query_pipe = Pipeline()
query_pipe.add_component("text_embedder", OpenAITextEmbedder())
query_pipe.add_component("retriever", InMemoryEmbeddingRetriever(
document_store=doc_store, top_k=5
))
query_pipe.add_component("prompt", PromptBuilder(template=template))
query_pipe.add_component("llm", OpenAIGenerator(model="gpt-4o"))
query_pipe.connect("text_embedder.embedding", "retriever.query_embedding")
query_pipe.connect("retriever.documents", "prompt.documents")
query_pipe.connect("prompt", "llm")
# 3. Run query
result = query_pipe.run({
"text_embedder": {"text": "What were the key findings?"},
"prompt": {"question": "What were the key findings?"}
})
print(result["llm"]["replies"][0])
Haystack strength: The explicit
connect() API catches mismatched component interfaces at pipeline build time, not at query time. Combined with Jinja2 prompt templates and built-in pipeline serialization (YAML), it’s the most ops-friendly framework.
Feature Comparison
The following comparison covers the dimensions that matter most in production RAG deployments:
LangChain
- Composability: LCEL pipe operator, Runnables
- Streaming: Native async & streaming support
- Integrations: 700+ (broadest ecosystem)
- Agent support: Excellent (ReAct, OpenAI tools)
- Learning curve: Moderate (many abstractions)
- Debugging: LangSmith tracing platform
LlamaIndex
- Composability: Index + QueryEngine pattern
- Streaming: Supported via callbacks
- Integrations: 300+ (data-focused)
- Agent support: Good (data agents)
- Learning curve: Low (intuitive API)
- Debugging: Built-in observability hooks
Haystack
- Composability: Pipeline graph with typed slots
- Streaming: Component-level streaming
- Integrations: 50+ (curated, stable)
- Agent support: Basic (pipeline routing)
- Learning curve: Low–moderate
- Debugging: Pipeline visualization, YAML export
Performance Notes
- LangChain adds ~15ms overhead per chain step
- LlamaIndex’s index build is one-time; queries are fast
- Haystack’s typed pipelines have near-zero runtime overhead
- All three are I/O-bound (LLM calls dominate latency)
- Async support: LangChain > LlamaIndex > Haystack
- Memory: LlamaIndex uses most (index structures in RAM)
Choosing the Right Framework
The best framework depends on your team, use case, and stage of development. Use this decision matrix:
- Rapid prototyping & experimentation: Start with LangChain. Broadest integrations, fastest time to first query, and LCEL makes it easy to iterate on pipeline structure.
- Document-heavy Q&A (primary use case): Choose LlamaIndex. Its index abstractions (tree, knowledge graph, SQL) are purpose-built for document retrieval and give you the most control over how data is organized.
- Production deployment with strict requirements: Use Haystack. Typed pipeline graphs, build-time validation, YAML serialization, and explicit component wiring minimize runtime surprises.
- Agent-heavy systems with tools: LangChain’s agent framework is the most mature, with support for OpenAI function calling, ReAct, and custom tool chains.
- Hybrid approach: Use LlamaIndex for indexing and retrieval, LangChain for agent orchestration, and Haystack for the production deployment layer. They can interoperate through shared vector stores.
# Decision helper: which framework for your use case?
def recommend_framework(
use_case: str,
team_size: int,
needs_agents: bool,
production_ready: bool
) -> str:
if needs_agents and not production_ready:
return "LangChain" # Best agent ecosystem
if use_case == "document_qa" and not needs_agents:
return "LlamaIndex" # Purpose-built for RAG
if production_ready and team_size > 3:
return "Haystack" # Production-grade pipelines
if team_size <= 2:
return "LlamaIndex" # Lowest learning curve
return "LangChain" # Default: broadest ecosystem
Migration warning: All three frameworks have frequent breaking changes between major versions. Pin exact versions in your requirements and test thoroughly before upgrading. LangChain’s v0.1→v0.2 migration was particularly disruptive.