Multivector Search: Efficient Document Retrieval with ColPali and LanceDB¶

Modern documents—PDFs, scans, forms, invoices, or scientific diagrams—rely heavily on visual elements like tables, figures, and spatial layouts to convey meaning. Retrieving context from these documents poses unique challenges:

• 🖼️Loss of Context: Plain-text extraction destroys critical visual relationships (e.g., a table's structure or a diagram's annotations).
• 🧩Multi-Modal Complexity: Layouts combine text, images, and structured elements that require joint understanding.
• 📏Scale vs. Precision: Balancing pixel-perfect accuracy with efficient search across millions of documents.

Why Traditional Methods Fail¶

The traditional method is a brittle, multi-stage pipeline where visual context is eroded at every step. Retrieval becomes a "best guess" based on partial text. Usually, it will involve the following steps:

1. OCR Text Extraction - extract raw text from scanned PDFs/images.
2. Layout Detection - use models like LayoutLM or rule-based tools to segment pages into regions (titles, tables, figures).
3. Structure Reconstruction - use heuristic rules or ML models try to infer reading order and hierarchy.
4. Optional: Image/Table Captioning - apply vision-language models (e.g., GPT-4V) to describe figures/tables in natural language.
5. Text Chunking - split text into fixed-size chunks or "semantic" passages (e.g., by paragraphs).
6. Embedding & Indexing- use text-based embeddings (e.g., BERT) and store in a vector DB (e.g., LanceDB).

Our Approach: ColPali with XTR for performant retrieval¶

ColPali (Contextualized Late Interaction Over PaliGemma) enhances document retrieval by combining a vision-language model (VLM) with a multi-vector late interaction framework inspired by ColBERT. In this framework, documents and queries are encoded as collections of contextualized vectors—precomputed for documents and indexed for queries. Unlike traditional methods, late interaction defers complex similarity computations between query and document vectors until the final retrieval stage, enabling nuanced semantic matching while maintaining efficiency.

To further accelerate retrieval, we integrate XTR (ConteXtualized Token Retriever), which prioritizes critical document tokens during initial retrieval stage and removes the gathering stage to significantly improve the performance. By focusing on the most semantically salient tokens early in the process, XTR reduces computational complexity with improved recall, ensuring rapid identification of candidate documents.

We used theUFO dataset, a dataset with rich tables, images and text, to demonstrate how to efficiently retrieve documents with ColPali and LanceDB.

Step 1: Install Required Libraries¶

Step 2: Load the UFO dataset¶

The UFO dataset has 2243 rows in total with an embedding of 128 dimension each. We show an example of the document to show how complicated it is with text and images blended in the document.

Step 3: Load the ColPali model¶

Note: select "cuda" if you are using a Nvidia GPU or "cpu" if there is no GPU available. Mac users, please use "mps". This step can take a few minutes.

Step 4: Connect to LanceDB¶

⚠️ WARNING: LONG EMBEDDING & INGESTION STEP
❗ Skip this cell unless you want to re-run the full embedding process.

Why?
Embedding the UFO dataset and ingesting it into LanceDB takes~2 hours on a T4 GPU. To save time:

• Use the pre-prepared table with index created (provided below) to proceed directly toStep 7: search.
• Step 5a contains the full ingestion code for reference (run it only if necessary).
• Step 6 contains the details on creating the index on the multivector column

Step 5: Open the LanceDB table¶

Step 7: Retrieve documents from a query¶

Let's see the retrieved documents!¶

Step 5a: Embed the UFO dataset and ingest data into LanceDB¶

Note: This step will take up to 2h when running with T4 GPU with abatch_size=4. You can increase thebatch_size to accelerate the process if there is more memory available, e.g.batch_size=32 requires 60GB of memory.

Step 6: Create an index on the multivector column¶

Note: LanceDB Cloud automatically infers the multivector column directly from the schema. If your dataset contains only one column with a list of vectors, no manual specification is required when building the vector index—the system handles this implicitly.