Building Production RAG: Reference Architecture and Rollout Playbook

The final design should be replaceable at every model and database boundary. This reference architecture is deliberately vendor-neutral and can begin as a modular monolith before traffic or organizational ownership justifies separate services.

A practical component map

                         CONTROL PLANE
 configs | prompts | experiments | evaluation | index aliases | audit
                              |
Sources -> queue -> ingest workers -> object store -> search indexes
                              |                lexical + vector
                              v
Client -> API gateway -> RAG orchestrator -> query service
              |              |        |-> hybrid retrieval
              |              |        |-> reranker / context packer
              |              |        |-> model gateway
              |              |        `-> citation validator
              |              v
              `---------- telemetry -> traces / metrics / quality samples

Start with clear modules and queues. Split services only when scaling, reliability, security, or ownership requires it. A distributed system is not a prerequisite for good boundaries.

Define versioned interfaces

An answer request can look like:

type AnswerRequest = {
  requestId: string;
  tenantId: string;
  principal: { subject: string; groups: string[]; entitlementVersion: string };
  question: string;
  conversation?: Array<{ role: "user" | "assistant"; content: string }>;
  locale?: string;
  filters?: { domain?: string[]; effectiveAt?: string };
};

type AnswerResponse = {
  status: "answered" | "insufficient_evidence" | "denied" | "degraded";
  answer?: string;
  citations: Array<{
    sourceId: string;
    title: string;
    section?: string;
    url: string;
    quote?: string;
  }>;
  traceId: string;
  configurationVersion: string;
};

Do not expose internal search scores as confidence percentages. If the product needs confidence, calibrate a decision model against observed correctness and describe what the value means.

Orchestrate with deadlines and evidence

async function answer(req: AnswerRequest): Promise<AnswerResponse> {
  const identity = await authorize(req.principal, req.tenantId);
  const query = await understandQuery(req, { deadlineMs: 250 });
  const filters = buildTrustedFilters(identity, req.filters);

  const candidateLists = await withDeadline(350, Promise.all([
    lexicalSearch(query.variants, filters, 60),
    vectorSearch(await embed(query.variants), filters, 60)
  ]));

  const fused = reciprocalRankFusion(candidateLists, 60);
  const ranked = await rerank(query.standalone, fused.slice(0, 120));
  const evidence = packEvidence(ranked, {
    maxTokens: 6000,
    maxPerDocument: 3,
    requiredFacets: query.subquestions
  });

  if (!isAnswerable(query, evidence)) return insufficientEvidence();

  const draft = await generateGroundedAnswer(query, evidence);
  const validation = await validateClaimsAndCitations(draft, evidence);
  return validation.supported ? publish(draft, evidence) : insufficientEvidence();
}

In production, wrap every dependency with timeouts, circuit breakers, typed errors, and trace spans. Propagate cancellation. Define whether a partial branch is acceptable instead of letting Promise.all decide policy accidentally.

Keep configuration immutable

Deploy a named configuration that binds compatible versions:

id: rag-prod-2026-08-25.1
corpus: enterprise-docs-184
chunker: semantic-v4
embedding: multilingual-1024-v2
index: docs-hnsw-v17
rewrite_prompt: rewrite-v6
fusion: { method: rrf, k: 60 }
reranker: cross-encoder-multi-v3
context_packer: evidence-pack-v5
answer_prompt: grounded-answer-v12
generator: approved-model-route-v8

Store this ID on every trace and evaluation result. Promote configuration artifacts from development to staging and production. Roll back an alias or configuration pointer rather than editing live values by hand.

Choose a storage stack proportionate to scale

For a first production system, a relational database for metadata and jobs, object storage for raw and parsed documents, a search engine that supports lexical plus vector retrieval, and a queue are often enough.

PostgreSQL plus pgvector is attractive when the corpus and query load fit comfortably and transactional metadata matters. OpenSearch or Elasticsearch provides mature lexical search, filters, aggregations, and vector capabilities. Dedicated vector systems can offer stronger vector scale or operational ergonomics. Benchmark with real filters and concurrency before deciding.

Keep source text outside proprietary index-only storage when possible. The ability to rebuild into a different engine limits lock-in and improves disaster recovery.

Use caching at stable boundaries

Safe candidates include:

• normalized query embeddings keyed by embedding version
• language and intent classifications keyed by classifier version
• parsed document artifacts keyed by content hash and parser version
• source metadata keyed by document version
• public or identically authorized retrieval results keyed by entitlement and index versions

Generated-answer caching is riskier because permissions, freshness, conversation, and non-determinism all matter. Prefer caching deterministic intermediate work first.

Make ingestion and serving independently scalable

Ingestion is throughput-oriented and bursty; online retrieval is latency-sensitive. Give them separate queues, compute pools, quotas, and autoscaling signals. Bulk index builds should not starve interactive queries.

Use backpressure when downstream indexing slows. Preserve ordering for versions of the same document or use compare-and-swap publication so an older job cannot overwrite a newer version.

Roll out in stages

Stage 1: measurable baseline

• one approved corpus and user group
• lexical and vector baselines
• 100–300 adjudicated questions including no-answer cases
• citations and retrieval-time permissions
• end-to-end traces and cost attribution

Exit when the system can be reproduced and its main failures are categorized.

Stage 2: hybrid quality

• hybrid retrieval with RRF
• safe conversation rewrite and language handling
• reranking, deduplication, and context budgets
• offline gates for recall, citations, latency, and ACL isolation

Exit when gains hold across critical slices and failure behavior is acceptable.

Stage 3: production resilience

• SLOs, alerts, canaries, fallbacks, quotas, and runbooks
• index aliases and rollback
• deletion and permission-propagation tests
• load, soak, dependency-failure, and disaster-recovery exercises

Exit when an on-call team can operate the service without its original authors.

Stage 4: controlled expansion

• onboard tenants and connectors through readiness checklists
• add difficult languages and formats with dedicated evaluations
• introduce model routing, query decomposition, or graph retrieval only for proven gaps
• review value, risk, and unit economics by use case

Release gates

A configuration is releasable only when:

• retrieval and answer metrics meet thresholds on the locked set
• no critical slice crosses its regression budget
• permission, injection, deletion, and tenant-isolation tests pass
• p95/p99 latency and saturation remain within budget at target load
• cost per grounded answer is acceptable
• dashboards, alerts, owners, rollback, and runbooks exist
• corpus, index, prompt, model, and configuration versions are recorded

The most useful debugging sequence

When an answer is wrong, ask in order:

1. Was the required source ingested, current, parsed correctly, and authorized?
2. Did language detection, rewriting, filters, or decomposition preserve the question?
3. Did the relevant evidence enter the candidate set?
4. Did fusion or reranking push it out?
5. Did deduplication or context packing remove it?
6. Did the generator ignore or contradict good evidence?
7. Did citation validation fail to catch the unsupported claim?

This sequence prevents teams from changing the prompt when the actual problem is stale ingestion or low retrieval recall.

What “enterprise scale” really means

Scale is not only vector count or queries per second. It is the number of languages, source systems, permission models, owners, regulatory zones, configurations, experiments, and incidents the team can handle without losing control.

A good enterprise RAG platform makes evidence traceable, changes reversible, quality measurable, access enforceable, and failure safe. Once those properties exist, models, indexes, and vendors can evolve without rebuilding the entire product around them.