Imagine you have set up a RAG system for a global company. A staff member in London types, in English, “what is the company leave policy?” The bot returns a clear answer from the right document. A few minutes later someone in Delhi asks the identical question in Hindi. Same LLM, same documents — and the system stumbles, pulling an unrelated file or inventing an answer.
To the user, the AI looks like it lost its mind. To anyone who has debugged one of these, the picture is different: the retrieval layer is bottlenecking the model. The generator is fine. The search system feeding it only sees the world in one dimension.
This is worth separating clearly, because teams routinely respond to it by swapping the LLM — which changes nothing. If you want to stop these quiet breakdowns, the pipeline has to be built for real-world complexity: multilingual phrasing, and visual data like charts, tables and images. If you have not built a retrieval pipeline before, the complete RAG pipeline guide covers the foundations this article builds on.
It is not the LLM, it is the map
In a standard RAG setup, documents and queries are converted into vector embeddings — coordinates in a high-dimensional space. A good embedding model places a question and its answer close together in that space. Most widely used embedding models were trained overwhelmingly on English, so they place a Hindi question and its English answer far apart, even when the two mean exactly the same thing.
The retrieval step has already failed before the generator sees anything. The vector database returns irrelevant passages, the generator does its job faithfully on bad input, and the user gets confident nonsense.
A two-minute test for this exact failure
Take the failing Hindi query and translate it into English by hand, then run it again. If the English version retrieves the correct document, your embedding model cannot handle cross-lingual search. That single test separates an embedding problem from a data problem, and it costs nothing to run.
There is a second failure that hides behind the first: the reranker. Teams upgrade the embedding model, see partial improvement, and stop there. But if an English-only reranker sits after retrieval, it can score a correctly retrieved Hindi document so low that it never reaches the prompt window. Both stages have to be multilingual, or the weaker one silently sets your ceiling.
Why keyword search is only a patch
The usual first response is hybrid search — combining vector similarity with classic BM25 keyword matching. It genuinely helps with Hinglish, where Hindi is typed in Latin characters, because tokens like “policy” and “leave” still overlap with the English index.
It does nothing for native Devanagari script. There are no shared character sequences between the query and an English document, so the keyword half of the hybrid contributes zero. Hybrid search does not fix cross-lingual retrieval; it just masks how bad it is for the subset of users who happen to type in Latin characters.
| Query translation | Multilingual embeddings | |
|---|---|---|
| Method | Translate incoming queries to English before searching | Re-index with a natively multilingual model such as multilingual-E5 or BGE-M3 |
| Accuracy | Decent for English-only corpora; loses nuance and domain terms in translation | Maps languages into shared coordinates; handles Hinglish far better |
| Overhead | Low. No database changes. Adds latency per query. | High. Requires re-indexing every document you hold. |
| Best as | A stopgap, or a fallback path | The production answer |
Translation is attractive because it ships in an afternoon. It is still a patch: every query pays a translation hop, and mistranslated domain vocabulary fails invisibly. For a system people depend on, re-index with a model that maps languages into a shared space.
When text-only search hits a wall
Language is one instance of a broader problem. An English-only index misses Hindi text for the same structural reason a text-only index misses a financial chart: the information was never encoded in a form the retriever can search.
Ask “which product grew fastest according to this report?” and answering requires two sources at once — the numbers inside a chart image, and the explanatory prose beside it. Different file types need different parsing:
| Content type | What the pipeline must do |
|---|---|
| Scanned pages | High-accuracy OCR that preserves text placement, not just character extraction |
| Tables | Extract structured rows and columns; flattening a table into one line destroys the relationships that made it a table |
| Charts and diagrams | Generate descriptive text, extract the underlying figures, and index the image itself with a visual retrieval model such as ColPali |
| Audio and video | Store transcripts alongside frames and timestamps so answers can cite a moment, not just a file |
The trap of text-only image summaries
The most common shortcut in multimodal RAG is to caption every image with a short text summary and index that. It feels like it solves the problem. It quietly discards the data.
Consider a chart showing Product A at 42% growth and Product B at 38%. A generated summary might read: “Product A showed strong growth.” Retrieval will now find this chart for a question about growth — and then fail completely when the user asks for the exact difference, because 42 and 38 no longer exist anywhere in your index. The system does not know the numbers are missing, so it estimates. That is where hallucinated figures come from.
Keep both: the extracted text and a reference to the original image region. When a query needs precision, pass the relevant crop straight to a multimodal model so it reads the actual figures rather than reconstructing them from a sentence.
How to evaluate a pipeline like this
Every capability added here introduces a new way to be wrong. OCR misreads digits. Table extraction scrambles column alignment. Visual models describe things that are not in the image. You cannot catch any of it with end-to-end “does the answer look right” testing.
Test retrieval and generation separately. This is the single most useful discipline in RAG work. Measure retrieval with Recall@K (did the correct chunk appear in the top K at all?) and MRR (how high did it rank?). If Recall@K is poor, no amount of prompt engineering will save the answer — the right information never arrived.
A test set for a multilingual, multimodal system needs to cover:
- Query variants — the same question in Devanagari Hindi, Romanized Hindi, mixed Hinglish and English. Same expected document for all four.
- Media variants — questions answerable only from prose, only from an image, and from both together.
- Grounding checks — verify the model cites a page, image region or timestamp. An answer that cannot point at its source is a guess that happened to be right.
Building for the data you actually have
Moving from plain-text RAG to a multilingual, multimodal pipeline is less a feature upgrade than a change in what the system is for. Enterprise knowledge does not live in tidy English paragraphs. It lives in scanned contracts, quarterly decks, recorded calls and messages typed in three scripts.
The practical order matters. Fix embeddings and reranking first, because a language gap breaks retrieval for entire user populations at once. Then extend to visual content, where the failures are narrower but harder to notice. And instrument retrieval separately throughout, so you can tell the difference between a model that reasoned badly and a model that was handed the wrong page.
Evaluating AI systems is no longer a text comprehension exercise. It is a question of whether your infrastructure can find the right information — in whatever language and whatever format it happens to exist.