← All posts
·2 min read

Designing Scalable RAG Architectures

RAGAI ArchitectureLLM

Most retrieval-augmented generation demos work. Most RAG systems don't — not at first, and usually not for the reason teams expect. The gap between "works in a notebook" and "works for 50,000 users with SLAs" is almost never the LLM. It's the architecture around it.

The three failure modes

1. Retrieval quality is treated as a solved problem. It isn't. Chunking strategy, embedding model choice, and re-ranking are where most of the actual product quality lives — and they're the parts teams tend to bolt on last.

2. There's no evaluation harness. Without a way to measure retrieval and generation quality on a fixed test set, every change is a guess. I won't sign off on a RAG architecture that doesn't ship with an eval suite from day one.

3. Cost and latency are an afterthought. Naive RAG calls an embedding model, a vector store, and an LLM in sequence for every request. At scale, that's three round trips you're paying for on every single query — and the first thing to optimize once traffic grows.

A reference architecture

The pattern I default to for enterprise RAG:

  1. Ingestion pipeline — versioned, idempotent, with document-level lineage so you always know which model version indexed which chunk.
  2. Retrieval layer as an abstraction, not a direct vector-store call — this is what let one platform I designed swap vector stores without touching a single downstream service.
  3. Evaluation harness running against a golden dataset on every pipeline or prompt change, gating deploys.
  4. Model routing — smaller/cheaper models for classification and re-ranking, reserving the expensive model calls for final generation.
  5. Observability on every hop — retrieval hit rate, re-ranker scores, generation latency, and token cost, all tied to a request ID.

None of this is exotic. It's the same systems-architecture discipline you'd apply to any distributed system — RAG just adds a retrieval and a generation hop to reason about.

The trade-off that matters most

Teams ask me whether to build a custom retrieval stack or use a managed RAG platform. My honest answer: it depends on whether retrieval quality is your product. If it is, own that layer. If it's a supporting feature, buy it and spend your engineering time elsewhere. Getting this trade-off wrong is the single most expensive architecture mistake I see in this space.