Operating Enterprise RAG: Security, Observability, SLOs, and Cost

A RAG service handles two untrusted inputs: user messages and retrieved documents. It also sits between sensitive data and an external or internal model. Its threat model must be designed before launch, not after the first prompt-injection demonstration.

Define trust boundaries

Map the actors and stores:

• users, service accounts, administrators, and document owners
• source connectors and their credentials
• raw document store, processing queues, lexical/vector indexes, caches, and traces
• embedding, reranking, generation, OCR, and evaluation model providers
• administrative configuration and evaluation datasets

For each boundary, document authentication, authorization, encryption, retention, residency, logging, and failure behavior. Minimize the data sent to each model. An embedding provider does not need user identity; a generator does not need chunks that lost authorization filtering.

Enforce access during retrieval

Convert the authenticated principal into trusted filters and apply them in lexical and vector search. Never ask the language model to decide whether a user may see a document.

Protect against:

• cross-tenant retrieval
• stale group memberships and revoked users
• cache entries reused across principals
• unauthorized content in traces or evaluation logs
• “existence leaks” through result counts, titles, or timing
• deleted content remaining in replicas or snapshots

Include negative permission fixtures in every release test. Create similarly named documents in different tenants and verify that neither results nor citations cross the boundary.

Cache keys must include tenant, authorization scope or entitlement version, query transform version, corpus/index version, and relevant filters. Avoid caching generated answers for broad identity groups unless the risk is understood.

Treat retrieved instructions as data

A document can contain “ignore previous instructions,” malicious markup, or text planted to exfiltrate neighboring context. Defenses are layered:

1. label source boundaries and state that source instructions are untrusted
2. strip active content, hidden HTML, scripts, and dangerous file constructs during parsing
3. detect suspicious instruction patterns and quarantine or annotate documents
4. give the answer model no tools or credentials it does not require
5. use allowlisted tool schemas and validate all tool arguments server-side
6. validate output for citations, secrets, and policy violations
7. rate-limit and alert on repeated extraction patterns

Prompt wording alone is not a reliable defense. Keep control instructions outside retrieved content and use application authorization for every action.

Protect sensitive data

Classify data before indexing. Some content should remain in a restricted deployment, use an approved private model endpoint, be redacted, or never enter RAG.

Encrypt data in transit and at rest, rotate connector and provider credentials, isolate tenants as required, and apply retention to raw content, vectors, caches, prompts, traces, and backups. Embeddings can leak information and should be treated as derived sensitive data.

Redact secrets and unnecessary personal data before observability export. Use stable hashes or synthetic IDs for correlation. Keep a tightly controlled debug path for authorized investigators rather than logging complete prompts everywhere.

Trace the request as a pipeline

A useful trace connects:

request -> classification -> rewrite -> each retrieval branch -> fusion
        -> reranking -> context assembly -> model call -> validation -> response

Record durations, status, candidate counts, score summaries, cache state, token counts, and configuration versions. Retain document IDs and safe metadata by default; capture content only under an explicit, access-controlled policy.

Core dimensions include tenant tier, language, intent, index version, model, result status, and experiment—not raw user text. High-cardinality labels can make telemetry expensive or unusable.

Define SLOs by user outcome

Example service-level indicators:

• availability: valid responses plus intentional abstentions divided by eligible requests
• latency: time to first token and complete response at p50/p95/p99
• freshness: time from approved source change to searchable version
• security freshness: time from permission revocation to enforcement
• quality: sampled grounded-answer and citation-precision rates
• retrieval coverage: share of requests with relevant evidence above threshold

Set separate objectives where use cases differ. Report degraded responses separately from fully healthy ones. A lexical fallback may preserve availability but reduce quality.

Allocate the latency budget:

Stage	Example p95 budget
Classification and rewrite	250 ms
Hybrid retrieval	300 ms
Reranking and packing	400 ms
Generation to first token	1,200 ms
Validation and overhead	250 ms

These are starting examples. Deadlines should propagate to dependencies. Cancel work after a request is abandoned.

Design graceful degradation

Declare fallback behavior before incidents:

• vector timeout -> lexical search with a visible degraded trace status
• reranker timeout -> use fused order and a stricter answer threshold
• generator outage -> return ranked source links when appropriate
• stale index -> warn or disable time-sensitive answers
• authorization service failure -> fail closed

Use bounded retries with jitter only for transient, idempotent work. Retries can turn a slow dependency into a full outage.

Control cost per successful task

Track cost per request and per grounded answer, not only monthly spend. Attribute embedding, search, reranking, generation, evaluation, and storage costs by tenant and use case.

High-leverage controls include:

• route simple classification and rewriting to smaller models
• skip rewriting for clear standalone queries
• batch offline embeddings and reranking where supported
• cap query variants, candidates, context tokens, answer tokens, and retries
• cache embeddings for normalized repeated queries
• reuse document embeddings across approved applications
• send fewer, better passages to the generator
• use model routing based on measured difficulty

Budgets should fail predictably: reject, queue, downgrade, or require approval—never produce an ungrounded answer merely because a quota was reached.

Detect drift and regressions

Monitor corpus, query, retrieval, and answer distributions:

• languages, intents, query length, and new terminology
• chunk counts, sizes, source mix, and index growth
• score and rank distributions by retrieval branch
• abstention, citation, reformulation, and escalation rates
• token use, latency, error rate, and cost

Drift alerts should trigger evaluation on recent samples, not automatic model changes. Correlate metric shifts with deployments, corpus updates, connector failures, and organizational events.

Prepare incident playbooks

Create runbooks for cross-tenant leakage, stale permissions, poisoned documents, index corruption, quality regression, provider outage, latency saturation, and runaway spend. A high-severity data leak procedure should identify how to disable retrieval, revoke credentials, locate affected traces and caches, delete derived data, notify owners, and preserve audit evidence.

Every response should carry enough version information to reconstruct its path. Operational maturity is the ability to detect a problem quickly, constrain its blast radius, explain it, and restore a known-good configuration.