The Solution Architect's Guide to LLMOps
When teams ask me to "add MLOps" to their LLM stack, I usually stop them — because half of classical MLOps doesn't apply, and half of what LLM systems actually need isn't in the MLOps playbook at all.
What's different
Classical MLOps assumes you own the model: you can retrain it, inspect its weights, and version it like an artifact. Most LLM-based systems don't have that luxury — the model is a black box behind an API, and it changes underneath you on the vendor's schedule, not yours.
That shifts the operational surface from model management to three other things:
- Prompt and context versioning. Your prompt template is now a deployable artifact with its own version history, just like code — because a one-line prompt change can silently change behavior across your entire user base.
- Evaluation as a continuous process, not a pre-deploy gate. Because you don't control the underlying model version, you need evals running in production, not just in CI.
- Cost governance. Token spend is a new operational metric that didn't exist in classical ML serving, and it needs the same dashboards and alerting you'd put on infrastructure cost.
The architecture pattern
For every LLM-backed system I design, three components are non-negotiable regardless of use case:
- A prompt/config registry — decoupled from application code, with rollback.
- An observability layer capturing every prompt, response, latency, and cost — tied to a trace ID that spans the whole request.
- A guardrail layer sitting between the model and the user — output validation, PII redaction, and a fallback path for when the model's response fails a check.
Where teams over-invest
I regularly see teams build elaborate fine-tuning pipelines before they've built basic prompt observability. Fine-tuning is expensive to operate and rarely the highest-leverage lever early on. Get visibility into what's actually happening in production first — you'll usually find the fix is a retrieval or prompt problem, not a model problem.