Will your assistant find the same evidence in every language?

Where a second language breaks retrieval

A company working across Germany and Poland rarely has a clean split. Policies exist in German, some were translated into Polish and English, and engineering notes mix English terms into Polish sentences. People ask in whatever language they think in. The common breaks are specific:

• German compounds, where Kündigungsfrist has to match a question about the notice period
• Polish inflection, where umowa, umowy and umowie are the same word to a reader but not to a naive keyword index
• Internal terms and abbreviations that differ by language, or exist only in one
• Answers that exist only in one language, so a Polish question must reach a German document

A parallel test set

1. 1

   Collect real questions in each language

   Take them from tickets and chats in all three languages, not only from the language the project team speaks.

2. 2

   Translate them with domain knowledge

   Have people who use the terms daily produce the other versions, and keep the wording a real user would choose.

3. 3

   Label the same expected passages

   All three versions of a question point to the same passage IDs, plus any translated copies that are equally valid.

4. 4

   Record languages on both sides

   Store the query language and the document language with every result, so that results can be grouped into a matrix.

Questions the documents cannot answer belong in every language as well. An assistant that declines correctly in English and invents an answer in Polish has a language problem that a single-language test never shows.

Reading the language matrix

Expected passage found in the top 10, by query and document language
(hypothetical: 30 questions per cell, same expected passages in each row)

                    documents in EN   documents in DE   documents in PL
query in EN              28                24                22
query in DE              25                27                19
query in PL              23                17                26

Largest gap: PL questions about DE documents (17 / 30)
Failure review: 9 of 13 misses involve German compounds or internal
abbreviations that appear in the glossary but not in the query

The numbers are invented, but the pattern is worth looking for: the diagonal, where query and document share a language, scores best, and one cross-language cell lags. The failure review matters more than the score, because it tells you whether to fix analyzers, terminology or the embedding model.

Keyword search needs language-aware analysis

Keyword search is what finds exact terms such as part numbers and legal names, so it should stay in a hybrid setup. It needs the right analyzer for each language. Azure AI Search, for example, supports German and Polish with both Lucene and Microsoft analyzers; Microsoft documents that its own analyzers add lemmatization and decompounding for languages including German, while indexing two to three times slower on average. A field takes one language analyzer, so multilingual content usually goes into separate fields per language, and a query searches all of them when its language is uncertain.

Embeddings, translation and a shared glossary

Multilingual embedding models map questions and passages from different languages into one space. The M3-Embedding paper, for example, describes a model supporting more than 100 languages with dense, sparse and multi-vector retrieval, and its authors report strong multilingual and cross-lingual benchmark results. Benchmarks rarely contain your contract vocabulary, so compare two or three candidate models on your parallel set. Translating the query into each document language and searching every version is a fallback that costs latency and can mistranslate internal terms.

Internal terminology needs a glossary with the agreed English, German and Polish name for each concept, owned by someone who can settle disputes. Use it to expand queries and to add equivalent terms to chunk context. Anthropic suggests that a domain-specific contextualisation prompt, possibly including a glossary, may improve retrieval. Answer in the language of the question and cite the original passage. If you show a translated quotation, label it as a translation and link it to the original or to an approved translated version, so readers can check it.