RAG Ingestion: Multilingual Data, Chunking, Metadata, and Freshness

Retrieval quality cannot exceed corpus quality. Before choosing a vector database, build an ingestion pipeline that can reproduce every indexed chunk and explain why it exists.

Use an immutable document envelope

Convert every source into a common envelope while retaining the raw original:

{
  "tenant_id": "acme-fr",
  "source_id": "sharepoint://policies/travel-v7.docx",
  "document_id": "travel-policy",
  "version": "7",
  "content_hash": "sha256:...",
  "mime_type": "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
  "source_updated_at": "2026-08-18T13:04:00Z",
  "acl": { "groups": ["employees-fr"], "classification": "internal" },
  "connector_version": "sharepoint-3.2.0"
}

The pair (tenant_id, source_id, version) should be idempotent. Replaying the same event must not create duplicate chunks. Content hashes avoid recomputing parsing and embeddings when only irrelevant metadata changed.

Use a durable queue between connector, parser, enrichment, embedding, and indexing stages. Dead-letter failures with the source identifier and stage; never silently skip a document.

Parse structure, not only text

Preserve headings, paragraphs, lists, tables, page numbers, slide numbers, code blocks, links, and footnotes. Those fields improve chunking, citation display, and reranking.

For scanned documents, OCR confidence belongs in metadata. Route low-confidence pages to a better OCR model or human review. For tables, store both a structured representation and a readable linearization. A flattened table without headers often retrieves well but gives the model unusable evidence.

Useful extraction checks include:

• non-empty text ratio by MIME type
• replacement-character and encoding error rate
• OCR confidence distribution
• page and section count anomalies
• repeated header/footer ratio
• parser failure rate by connector and parser version

Detect language at document and chunk level

Enterprise corpora are commonly multilingual inside a single document. Detect language on the title, document body, and each final chunk. Libraries such as fastText, CLD3, or compact language detectors are good first stages; a small model can handle ambiguous cases.

Store a distribution, not just a label:

{
  "language": "fr",
  "language_confidence": 0.94,
  "language_scores": { "fr": 0.94, "en": 0.04, "de": 0.02 },
  "script": "Latin"
}

Short strings, product codes, names, and source code are difficult to classify. Below a confidence threshold, inherit the surrounding section language or use und rather than inventing certainty.

Choose one of three retrieval strategies after testing on language slices:

1. Cross-lingual embeddings: index original text once and retrieve it with questions in another language. This is operationally simple.
2. Query translation: retain original chunks, translate the query into one or more corpus languages, and merge results. This helps lexical retrieval.
3. Document translation: index original and translated fields. It can improve recall but doubles storage and introduces translation-version governance.

Always return citations in the original language. If the answer is translated, make that transformation explicit.

Chunk for semantic units

Fixed 500-token windows are a baseline, not a design. Prefer boundaries such as heading plus paragraphs, FAQ question plus answer, table plus title, or function plus docstring.

A practical hierarchical strategy is:

• build child chunks of roughly 200–500 tokens for precise retrieval
• retain a parent section of roughly 800–1,500 tokens for context expansion
• overlap only where a sentence or list crosses a boundary
• attach the heading path, such as Policy > Travel > International meals

Measure chunk sizes in the tokenizer used by the downstream model. Remove headers repeated on every page before chunking. Do not split identifiers, legal clauses, list items, or table rows arbitrarily.

Every chunk needs stable provenance:

chunk_id = hash(tenant + document_id + version + structural_path + offsets + chunker_version)

Store text, parent_text_id, byte or character offsets, page, heading path, language, ACL, timestamps, content hash, parser version, chunker version, and embedding version.

Enrich cautiously

Generated titles, summaries, keywords, entities, and hypothetical questions can improve retrieval. Keep generated fields separate from source text and record the model and prompt version. Never present synthetic enrichment as cited evidence.

Metadata with direct retrieval value usually wins first:

• business domain, document type, region, product, and effective date
• owner and source system
• confidentiality and retention class
• validity status: draft, effective, superseded, or expired
• named entities and normalized acronyms

Select and version embeddings

Benchmark candidate models on your languages and domains. Evaluate cross-language retrieval, acronym-heavy questions, numerical content, and short exact queries—not only generic semantic similarity.

Normalize vectors when the database expects cosine similarity implemented as an inner product. Record model name, dimensionality, normalization, truncation policy, and input prefix. A change to any of them requires a new embedding version.

Publish new embeddings into a shadow index. Compare them offline and with mirrored traffic, then atomically move an alias. Never overwrite an active vector field in place if rollback matters.

Preserve permissions end to end

Resolve source ACLs into searchable attributes during ingestion. Keep both the raw ACL and a normalized representation. At query time, intersect user entitlements with chunk permissions inside the retrieval engine.

Test group removal, nested groups, tenant isolation, document deletion, and cache eviction. Permission changes may require faster propagation than content changes, so give ACL events a high-priority path.

Delete and update correctly

Use versioned publication:

1. Parse and index all chunks for document version 8 as inactive.
2. Verify expected chunk counts and embeddings.
3. Atomically activate version 8 and deactivate version 7.
4. Remove version 7 after a rollback window.

Tombstones must flow through every lexical index, vector index, cache, and evaluation snapshot. A source deletion is not complete while an old chunk can still be returned.

Ingestion metrics

Track these by tenant, connector, MIME type, language, and version:

Metric	Why it matters
Source-to-searchable lag p50/p95/p99	Freshness experienced by users
Parse, OCR, enrichment, and embedding failure rates	Locates pipeline defects
Documents and chunks quarantined	Reveals silent coverage loss
Duplicate chunk ratio	Finds connector and versioning bugs
Empty or very short chunk ratio	Detects poor extraction/chunking
Language confidence and und rate	Exposes multilingual gaps
ACL propagation lag	Measures security freshness
Orphan and stale-version count	Verifies deletion correctness

Sample a fixed number of documents from every major source weekly and inspect the rendered chunks. Automated metrics find shifts; human review finds unusable structure.

The output of ingestion is not “some vectors.” It is a versioned, permission-aware, observable evidence corpus that retrieval can trust.