Multimodal RAG
A bank's product brochure is a PDF. The interest rates are in a table, the eligibility rules are in a flowchart, and the fees are in a footnote under a bar chart. Run it through a basic text extractor and you get the paragraphs, a table flattened into a jumble of numbers with no column headers, and nothing at all from the flowchart. The RAG system built on that text will confidently answer questions about rates with numbers from the wrong column.
Real documents are not text files. Multimodal RAG is the set of techniques for retrieving from PDFs, slides, scans, screenshots and images, where the meaning lives partly in the layout and the pictures.
A table in a PDF lists home loan rates by tenure and loan amount. After text extraction, the RAG system quotes the wrong rate. What went wrong, and what would you try?
Three ways to index a page#
The answer to the question at the top: the extractor flattened the table, so the numbers lost their row and column headers and the model matched a rate to the wrong tenure. A layout aware parser that keeps the table as markdown fixes it, and so does retrieving the page as an image and letting a vision model read the table itself.
Sometimes you do not need retrieval at all#
Current models accept PDFs and images directly, and context windows are large. If a user uploads one 40 page contract and asks about it, send the document itself rather than building an index. Retrieval earns its place when the collection is too big for one prompt, changes often, or needs per user permissions.
Making answers checkable#
Whatever you index, keep the link back to the page and region a chunk came from. Then the answer can cite "page 7, table 2", and the interface can show the user the actual table. In documents full of numbers, letting a person see the source is worth more than any accuracy metric.
| Choice | What you gain | What you pay | Pick it when |
|---|---|---|---|
| Layout aware parsing | Reuses your text pipeline, and tables keep their structure. | Charts and diagrams are lost, and parser quality varies by document type. | Text heavy documents with some tables: policies, reports, manuals. |
| Captioning visuals | Charts and diagrams become searchable, and the image is still there for answering. | A vision model call per figure at index time. | Documents where key facts live in charts or diagrams. |
| Page image embeddings | Nothing is lost to extraction, layout and tables survive. | Bigger indexes, and image tokens in every answer prompt. | Visually rich documents: brochures, slides, forms, scanned records. |
Parsers and vision embeddings are usually demonstrated on clean digital PDFs. Your corpus has scans at an angle, two column layouts, tables split across pages and stamps over text. Build your evaluation set from the worst ten percent of real documents before choosing a tool.
Checkpoint#
1. Why does basic text extraction break tables?
2. What does a ColPali style approach index?
3. A user uploads a single 25 page PDF and asks questions about it. What is simplest?
- Three ways in: parse to text, caption the visuals, or embed page images.
- Keep tables as tables. Markdown or HTML, never flattened text.
- One document that fits in context needs no index.
- Cite the page and region, so people can check the numbers.
Real documents carry meaning in tables, charts and layout, and basic text extraction loses it, which is how a RAG system quotes a rate from the wrong column. I would choose between three approaches by document type: a layout aware parser that keeps tables as markdown for text heavy documents, captioning charts and diagrams with a vision model so they become searchable, or embedding whole page images, ColPali style, and sending the retrieved pages to a vision model so nothing is lost. If a single document fits in the context window I would skip retrieval and send it directly. And every answer cites the page and region, so people can check the numbers themselves.