Ghazi Ben Dahmane
Contact
Back to blog

Enterprise RAG at Scale · Part 2 of 7

RAG Retrieval: Query Rewriting, Hybrid Search, and Fusion

Design the online retrieval path with language and intent detection, safe rewrites, BM25 plus vector search, filtering, RRF fusion, and candidate budgets.

Enterprise RAGInformation RetrievalSearch

Retrieval should maximize the probability that the evidence needed for an answer enters a manageable candidate set. It should not try to choose the final ten chunks in one step.

Build a query envelope

Represent each request explicitly:

{
  "original": "Can I expense it there next month?",
  "standalone": "Can a France employee expense meals during a business trip to Japan in September 2026?",
  "language": "en",
  "intent": "policy_lookup",
  "entities": { "country": "JP", "date": "2026-09" },
  "filters": { "region": ["FR"], "status": ["effective"] },
  "principal": { "tenant": "acme", "groups": ["employees-fr"] }
}

Keep the original, every transformation, and the model or rule version that produced it. Rewrites are hypotheses, not truth.

Classify before retrieving

Some requests should not enter RAG at all: greetings, calculator operations, account actions, unsafe requests, or questions outside the product scope. A fast rule/model cascade can determine intent, language, likely domains, and whether conversation resolution is required.

Do not infer authorization filters from free text. Region or product filters may come from the question, but tenant and ACL filters must come from trusted identity context.

Rewrite conservatively

Conversation makes questions ambiguous. A standalone rewrite resolves pronouns and adds relevant context while preserving identifiers and constraints.

System: Rewrite the final user message as a standalone search query.
Preserve names, numbers, dates, quoted text, negations, and product codes.
Use conversation context only to resolve references. Do not answer.
Return JSON with query, preserved_terms, and ambiguities.

Reject or fall back to the original when a rewrite drops a required entity, changes a number, removes negation, or introduces a name absent from the conversation. Search both original and rewritten versions when uncertainty is high.

Expand only when it improves a measured slice

Expansion can add acronyms, aliases, spelling variants, and translated queries. Multi-query retrieval generates several perspectives, while decomposition breaks a compound request into answerable subquestions.

Example decomposition:

Question: Compare our parental-leave policy in France and Germany.
Subqueries:
1. Current parental-leave policy for France employees
2. Current parental-leave policy for Germany employees
3. Effective dates and eligibility definitions for both policies

Cap expansion count. Three high-quality queries are often better than ten correlated queries that inflate latency and duplicate candidates. Do not decompose simple fact lookups.

Why hybrid retrieval is the default

Dense vector search handles paraphrases and conceptual similarity. Lexical retrieval such as BM25 handles product codes, names, error messages, exact clauses, and rare terms. Enterprise queries contain both.

Run the branches in parallel:

lexical: BM25(query, title^3 + headings^2 + body)
semantic: ANN(embed(query), chunk_vector)
optional: entity/graph or structured lookup

Apply tenant, ACL, validity, and hard business filters inside every branch. Over-filtering on uncertain inferred metadata reduces recall, so distinguish hard trusted filters from soft boosts.

Retrieve generously before reranking. A starting budget might be 50 lexical plus 50 vector candidates per query variant, deduplicated to 80–150 items. Tune it against Recall@K and latency.

Fuse incomparable scores

BM25 and cosine scores have different distributions. A weighted sum requires calibration. Reciprocal Rank Fusion (RRF) is a strong baseline:

RRF(document) = sum over result lists of 1 / (k + rank(document))

With k = 60, top positions matter without allowing one branch’s raw score to dominate. Add weights only when evaluation shows a consistent benefit:

score = 1.0 / (60 + lexical_rank) + 1.2 / (60 + vector_rank)

Here is implementation-shaped pseudocode:

async function retrieve(request: QueryEnvelope) {
  const variants = await buildQueryVariants(request);
  const filters = authorizationFilters(request.principal);

  const lists = await Promise.all(
    variants.flatMap((query) => [
      lexical.search(query.text, { filters, limit: 50 }),
      vectors.search(embed(query.text), { filters, limit: 50 })
    ])
  );

  return reciprocalRankFuse(lists, { k: 60 })
    .filter(dedupeByDocumentVersion)
    .slice(0, 120);
}

Production code also needs deadlines, partial failure behavior, trace IDs, query and index versions, and deterministic tie-breaking.

Parent-child and multi-field retrieval

Index child chunks for precise matching, then expand selected children to their parent sections. This avoids embedding large sections while giving the generator enough surrounding context.

Use separate fields for title, heading path, body, keywords, and entities. In lexical search, titles and headings usually deserve boosts. For dense retrieval, test separate title and body vectors or a combined representation. More vectors increase storage and write cost, so retain them only if ablation tests prove value.

Handle time and authority

Similarity is not authority. Boost current, approved, canonical sources and demote drafts, duplicates, and superseded policies. For time-sensitive questions, parse the requested date and filter by effective interval.

When two authoritative documents conflict, retrieve both and expose the conflict. Do not let a ranking score silently decide company policy.

Measure retrieval without generation

For every labeled question, identify all acceptable evidence chunks or documents. Report:

  • Recall@5, @10, @20, and @50
  • Mean Reciprocal Rank for first relevant evidence
  • nDCG@K when relevance is graded
  • zero-result and no-relevant-result rates
  • ACL-filtered candidate counts
  • latency by branch, language, tenant, and query class

Slice results for exact identifiers, acronyms, multilingual questions, dates, long conversational questions, tables, and rare domains. Overall recall can rise while a critical minority language gets worse.

Useful ablations compare lexical only, vector only, hybrid, hybrid plus rewrite, and hybrid plus expansion. Keep candidate budget and test set constant. The next part takes that high-recall candidate pool and turns it into compact, defensible evidence.