Case study · 02 · 2025 — Now
Enterprise LLM Gateway
A high-throughput Go gateway for multi-provider routing, streaming, quotas, cost accountability, and operational visibility.
Architect & R&D Lead
GoS3Delta LakeRedis SentinelSSE
Impact
- 200 GB of logs processed daily
- Database bottleneck eliminated
- Budgets managed by team and user
Context
- Model traffic outgrew the original Django/PostgreSQL gateway, which became a bottleneck under logging and quota load.
- Multiple providers (Azure OpenAI, AWS Bedrock) needed unified routing, streaming, and policy enforcement.
- Finance and engineering needed per-team and per-user cost accountability.
Approach
- Rebuilt the gateway in Go for high-throughput routing and streaming (SSE).
- Moved observability off the transactional database to S3 and Delta Lake, handling 200 GB of logs per day.
- Implemented quotas and budgets by team and user with operational dashboards.
- Kept provider boundaries replaceable so models and vendors can change without rewriting callers.
Lessons
- Log and analytics traffic must never share the transactional path.
- Multi-provider routing is an organizational decision as much as a technical one.
- Budgets and quotas are what make enterprise AI governable.