The Eval Pyramid: Most Teams Only Built the Top Layer

Mike Cohn's test pyramid made one argument: put your volume where it's cheap. Lots of fast unit tests at the bottom, fewer integration tests in the middle, a handful of slow end-to-end tests at the top — because the alternative, an "ice cream cone" of mostly slow UI tests, gives you a test suite nobody wants to run and everybody works around. AI teams building evaluation suites keep rediscovering this shape the hard way, except most of them build only the top of it: a slow, expensive "does the agent complete the task" eval, run occasionally, and nothing structured underneath it at all.

Same shape, different question at each layer

Unit evals are assertion-style checks on a single function or a single turn: did the classifier return one of the allowed labels, did the extraction produce valid JSON, did the retrieval step return at least one document above a similarity threshold. They're cheap enough to run in the thousands, in seconds, on every commit — which means they're the only layer fast enough to actually stop a bad change before it merges. Integration evals test a chain: does the agent call the right tool given this input, does a multi-step retrieval-then-generate flow hold state correctly across turns, does the system recover when a tool call fails. Fewer cases, minutes not seconds, run on every pull request. End-to-end evals judge whether the full task actually succeeded against a real user goal — usually with an LLM-as-judge or a human rubric, because "did this help the user" isn't an assertion you can write in one line. Slow, expensive, and necessarily small in number, which is exactly why it can't be your only layer.

Why AI teams keep building the pyramid upside down

Writing a good end-to-end eval feels like the real work — it's the one that answers the question stakeholders actually ask, "does the agent work." Unit evals feel like busywork by comparison, easy to skip when the deadline is close. So teams ship with one LLM-as-judge eval running against fifty examples, no unit-level coverage underneath it, and then are confused when every regression takes a full end-to-end run — ten or twenty minutes — to even detect, and even longer to localize to the specific step that broke.

The tell that a team has an inverted pyramid: a prompt change or a retrieval-config tweak takes half a day to validate, because the only signal available is "did the slow eval's score move," with no cheap layer underneath to say which of the twelve steps in the chain actually regressed. A healthy pyramid catches that same regression in the unit layer, in seconds, with a stack trace pointing at the exact function.

What each layer actually catches that the others don't

The layers aren't redundant — they catch structurally different failures. A unit eval catches "the extraction step returns malformed JSON on this edge case," something an end-to-end eval would report only as a vague quality drop, if it noticed at all. An integration eval catches "the agent calls the refund tool with the wrong currency field," a wiring bug that a unit test on the tool itself would never see, because the tool works fine in isolation. An end-to-end eval catches "the agent technically did everything right and the user still didn't get what they needed," the only layer that can see task success as a human would judge it, and the one thing no amount of unit coverage can substitute for.

Skip the bottom two layers and you don't just get a slower suite — you lose the ability to know why the top layer's score moved. Skip the top layer and you get a system where every unit test passes and the agent still fails at the one thing it's for.

Build it bottom-up, not top-down

The instinct to start with the end-to-end eval is right — it's the one that defines what "working" means. But building only that one is where the mistake happens. Once you know what success looks like at the top, work backward: what's the smallest assertion that would have caught each failure this eval surfaces, and what's the chain-level check that would catch the wiring bugs before they reach the top. Every incident is a missing layer, not a reason to add more end-to-end cases — those were already too slow to run as often as you needed them to.