Multi-Modal RAG: Images, Tables, Documents

MLOps Series RAG Systems

Real-world documents are not purely text. Financial reports contain charts, research papers include figures and tables, and technical manuals mix diagrams with prose. A production RAG system must ingest, embed, retrieve, and reason over all modalities in a unified pipeline. This post covers the techniques and architectures that make multi-modal RAG practical.

The Multi-Modal Challenge

Text-only RAG discards the majority of information in document-heavy domains. Consider a quarterly earnings report: the narrative text accounts for perhaps 30% of the information, while tables, charts, and footnotes carry the remaining 70%. Three core challenges arise:

Representation gap: Text embeddings cannot encode visual layout, chart trends, or table structure. A bar chart showing revenue growth has no useful text representation.
Alignment problem: Images, tables, and text must be embedded into a shared vector space so a text query can retrieve visual content and vice versa.
Generation integration: The LLM must receive multi-modal context in a format it can reason over — either as linearized text or through a vision-language model.

Image Retrieval in RAG

There are three dominant strategies for handling images in RAG pipelines, each with different trade-offs between accuracy, latency, and complexity:

Caption & Embed

Use a vision-language model (GPT-4V, LLaVA) to generate text captions for each image. Embed the captions with standard text embedders. Simple but lossy — captions cannot capture all visual nuance.

CLIP-Based Retrieval

Embed images directly with CLIP into a shared text-image vector space. Text queries retrieve images by cosine similarity. Preserves visual information but retrieval quality depends on CLIP’s alignment training.

import base64 from openai import OpenAI from PIL import Image class ImageRAGProcessor: def __init__(self, vision_model="gpt-4o"): self.client = OpenAI() self.vision_model = vision_model def caption_image(self, image_path: str) -> str: # Encode image as base64 for the vision API with open(image_path, "rb") as f: img_b64 = base64.b64encode(f.read()).decode() response = self.client.chat.completions.create( model=self.vision_model, messages=[{ "role": "user", "content": [ {"type": "text", "text": "Describe this image in detail for RAG indexing."}, {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{img_b64}"}} ] }] ) return response.choices[0].message.content def embed_with_clip(self, image_path: str): # Direct CLIP embedding for multimodal retrieval from transformers import CLIPModel, CLIPProcessor model = CLIPModel.from_pretrained("openai/clip-vit-large-patch14") processor = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14") image = Image.open(image_path) inputs = processor(images=image, return_tensors="pt") return model.get_image_features(**inputs)

Practical tip: Use the caption-and-embed approach for charts and diagrams (where textual description captures meaning), but prefer CLIP embeddings for photographs and screenshots (where visual detail matters more).

Table Extraction for RAG

Tables are the hardest modality for RAG. They encode relational data in a 2D grid that loses all structure when naively converted to text. Effective table RAG requires: (1) accurate extraction, (2) structure-preserving serialization, and (3) embedding that captures row-column relationships.

The main extraction strategies:

Rule-based parsing (Camelot, Tabula): Works on well-formatted PDFs with clear grid lines. Fails on borderless tables and complex merges.
Deep learning detection (Table Transformer, DETR): Uses object detection to localize tables, then structure recognition to identify rows and columns. Handles complex layouts.
Vision-LLM extraction: Send table screenshots to GPT-4V or similar models and ask for structured output (Markdown or JSON). Most flexible but highest latency.

import camelot import pandas as pd from transformers import TableTransformerForDetection class TableExtractor: def extract_with_camelot(self, pdf_path: str, page: int) -> list[pd.DataFrame]: # Rule-based extraction for clean PDFs tables = camelot.read_pdf(pdf_path, pages=str(page), flavor="lattice") return [table.df for table in tables] def serialize_for_embedding(self, df: pd.DataFrame) -> str: # Convert table to structured text preserving relationships rows = [] headers = df.columns.tolist() for _, row in df.iterrows(): pairs = [f"{h}: {v}" for h, v in zip(headers, row)] rows.append(" | ".join(pairs)) return "\n".join(rows) def detect_with_transformer(self, image): # Deep learning table detection for complex layouts model = TableTransformerForDetection.from_pretrained( "microsoft/table-transformer-detection" ) # Process image and detect table bounding boxes outputs = model(image) return self.post_process_detections(outputs)

ColPali: Late-Interaction Vision Retrieval

ColPali (Faysse et al., 2024) represents a paradigm shift in document retrieval. Instead of extracting text from documents and embedding it, ColPali embeds document page images directly using a vision-language model with late interaction scoring. This eliminates the entire OCR/parsing pipeline.

The architecture combines PaliGemma (a vision-language model) with ColBERT-style late interaction:

Document encoding: Each page is rendered as an image and passed through PaliGemma to produce a set of patch embeddings (one per image patch).
Query encoding: The text query is tokenized and embedded through the same model’s language tower, producing one embedding per token.
Late interaction scoring: For each query token embedding, compute maximum similarity against all document patch embeddings. Sum these maxima for the final relevance score.

import torch from colpali_engine.models import ColPali, ColPaliProcessor class ColPaliRetriever: def __init__(self, model_name="vidore/colpali-v1.2"): self.model = ColPali.from_pretrained(model_name) self.processor = ColPaliProcessor.from_pretrained(model_name) def index_pages(self, page_images: list) -> torch.Tensor: # Encode each page image into patch embeddings all_embeddings = [] for img in page_images: inputs = self.processor.process_images([img]) with torch.no_grad(): embeddings = self.model(**inputs) # [1, N_patches, 128] all_embeddings.append(embeddings) return torch.cat(all_embeddings, dim=0) def retrieve(self, query: str, page_embeds: torch.Tensor, top_k=5): # Encode query into token embeddings q_inputs = self.processor.process_queries([query]) with torch.no_grad(): q_embeds = self.model(**q_inputs) # [1, M_tokens, 128] # Late interaction: MaxSim scoring scores = self.maxsim_score(q_embeds, page_embeds) top_indices = scores.topk(top_k).indices return top_indices def maxsim_score(self, q_embeds, doc_embeds): # For each query token, find max similarity with doc patches sim = torch.einsum("qd,bpd->bqp", q_embeds[0], doc_embeds) return sim.max(dim=-1).values.sum(dim=-1)

Why ColPali matters: It eliminates the entire OCR + text extraction + chunking pipeline. One vision model handles text, tables, charts, and layout simultaneously. Benchmarks show it outperforms text-only RAG on document-heavy tasks by 15–25%.

Unified Multi-Modal Pipeline

A production multi-modal RAG pipeline needs to handle heterogeneous document types and route each modality through the appropriate processing path. The key architectural decision is whether to use a modality-specific or modality-agnostic embedding strategy:

Modality-Specific

Separate embedding models per modality (text embedder, CLIP for images, table embedder). Each modality gets its own index. Query-time fusion combines results from all indices. More control but higher complexity.

Modality-Agnostic (ColPali)

Single vision model embeds entire page images regardless of content type. One unified index. Simpler architecture but less fine-grained control. Works best when documents are page-oriented.

In many production systems, a hybrid approach works best: use ColPali for initial page-level retrieval, then apply modality-specific processors for fine-grained extraction from retrieved pages.

Implementation: End-to-End Multi-Modal RAG

from dataclasses import dataclass, field from enum import Enum from pathlib import Path class ModalityType(Enum): TEXT = "text" IMAGE = "image" TABLE = "table" @dataclass class MultiModalChunk: content: str modality: ModalityType source_page: int embedding: list[float] = field(default_factory=list) metadata: dict = field(default_factory=dict) class MultiModalRAG: def __init__(self, text_embedder, clip_model, table_extractor, vlm, vector_store): self.text_embedder = text_embedder self.clip_model = clip_model self.table_extractor = table_extractor self.vlm = vlm # Vision-language model for generation self.vector_store = vector_store def ingest_document(self, doc_path: Path): # Extract all modalities from document pages = self.render_pages(doc_path) chunks = [] for i, page in enumerate(pages): # Extract text chunks text_chunks = self.extract_text(page) for tc in text_chunks: emb = self.text_embedder.embed(tc) chunks.append(MultiModalChunk(tc, ModalityType.TEXT, i, emb)) # Extract and caption images images = self.extract_images(page) for img in images: caption = self.vlm.caption(img) emb = self.clip_model.encode_image(img) chunks.append(MultiModalChunk(caption, ModalityType.IMAGE, i, emb)) # Extract and serialize tables tables = self.table_extractor.extract(page) for tbl in tables: serialized = self.serialize_table(tbl) emb = self.text_embedder.embed(serialized) chunks.append(MultiModalChunk(serialized, ModalityType.TABLE, i, emb)) self.vector_store.upsert(chunks) def query(self, question: str, top_k=5) -> str: # Retrieve from unified index results = self.vector_store.search(question, top_k=top_k) # Build multi-modal context context_parts = [] for r in results: if r.modality == ModalityType.TABLE: context_parts.append(f"[TABLE]\n{r.content}") elif r.modality == ModalityType.IMAGE: context_parts.append(f"[IMAGE DESCRIPTION]\n{r.content}") else: context_parts.append(r.content) context = "\n\n".join(context_parts) return self.vlm.generate(f"{context}\n\nQuestion: {question}")

Cost consideration: Vision-language model calls for captioning are 5–10× more expensive than text embedding. For large document corpora, batch caption generation during off-peak hours and cache results.