Why AI Evals Are Necessary but Insufficient: The Reality Gap
My reflections on how AI feature development has evolved, what evals get right, where they break down, and why designing for emergence matters more than prediction
If you’re a software builder in 2026, there’s a good chance you’ve heard the word evals more times than you can count. With the rise of LLMs in product development, evals have quickly become shorthand for how teams are supposed to reason about quality when building AI-powered features.
For most of software history, quality assurance rested on a relatively stable assumption: that software was deterministic. You designed user flows, anticipated the actions users could take, and tested whether the system behaved correctly. Even as systems grew complex, the space of possible behaviors was still bounded by finite constraints: buttons, screens, predefined paths.
With LLMs, that assumption no longer holds.
When users interact with AI-driven features through freeform text, intent is no longer constrained by interface affordances. A single input can fan out into many plausible interpretations, each producing a different response depending on the context, phrasing, and prior state. Under the hood, these features rely on probabilistic next-token prediction which means even the same prompt can yield different outcomes across runs. Software is more malleable, more expressive, and much harder to reason about.
So product teams faced a new problem: how do you ensure quality when you can no longer anticipate behavior exhaustively?
Evals emerged as the industry’s answer. We define test prompts, grade outputs, and track scores over time as a way to restore rigor and confidence to AI development.
As I spent more time understanding these new ways of building and shipping AI features, I’ve thought deeply about how building software is changing. While I believe evals are a meaningful step forward, I also believe we need something more.
This essay is my attempt to unpack why evals are not enough and why borrowing ideas from other disciplines — especially game design — may point toward a more durable way of thinking about quality in AI-first products & agents.
How Quality Assurance Evolved to Manage Uncertainty
Over the entire history of software development, we’ve sought to build software that works through the practice of QA:
A proactive, systematic process designed to prevent defects and ensure products or services meet specified requirements, standards, and customer expectations
Though the definition hasn’t changed much over time: the “how” has evolved. To see why AI evals exist and how they fall short, it’s worth tracing how QA adapted across different eras of software.
Phase 1: Manual testing with early software (1950s - 1970s)
In the earliest days of software, programs ran on mainframes and prohibitively expensive to build. Software was reserved for mission-critical use-cases (e.g. financial systems, military applications, scientific computation). If something broke, it could be catastrophic.
Teams were small, highly specialized, and worked on long release cycles. Quality assurance itself was fairly manual, with step-by-step verification of program behavior. This worked because software programs were relatively linear and tightly scoped.
Quality was focused on correctness under high stakes.
Phase 2: Automated testing & unit tests (late 1970s - 1990s)
Software spread beyond mainframes to workstations and PCs. It began powering enterprise & internal business workflows like accounting, payroll, inventory systems, databases, and so on. As codebases grew larger, teams became more distributed. Failures were still costly but they were no longer as catastrophic given the use cases.
QA adapted by shifting towards automation. Developers wrote unit and regression tests that asserted expected inputs and outputs, so tests could be re-run repeatedly and regressions could be avoided. QA moved from manual inspection to executable specifications directly written into the codebase.
Quality still focused on correctness, with more complexity.
Phase 3: Integration, QA teams, and process (late 1990s - 2000s):
With the rise of the Internet, software increasingly ran on networked systems: client-server architectures, web apps, early distributed services. Software was being built for consumer-facing use cases at scale, with graphical interfaces designed for a much broader audience. Release cycles got faster and teams needed to coordinate even more closely to avoid shipping bugs.
QA evolved to become an even more formal discipline of its own. Dedicated QA teams emerged alongside engineering: focusing on end-to-end testing, formal test plans, coverage matrices, and bug tracking systems.
Quality focused on managing complexity across interconnected systems.
(Author’s note: I have fond memories of this phase, as it’s how my dad made his career in software as a QA automation architect at companies like Informix, Oracle, HP, Apple, Google, and so on).
Phase 4: Speed, SaaS, and product-led quality (late 2000s-2010s)
As software moved to the cloud and SaaS delivery became the prevailing commercial model, release cycles got even faster. Products shipped continuously to global user bases across web and now mobile too.
Quality issues expanded beyond obvious crashes and into regressions, degraded experiences across different clients, subtle behavioral bugs that could erode user trust.
Again, QA evolved to embrace continuous integration and delivery (CI/CD pipelines), automated regression suites, feature flags, staged rollouts, canary releases, production monitoring, and so on. “Dogfooding” became a thing: a way to catch issues that tests couldn’t always catch.
Quality shifted towards balancing speed with safety, relying on both quantitative signals and also qualitative judgment.
Phase 5: Early ML-augmented software (late 2010s - pre-GenAI)
Before GenAI entered mainstream product development, machine learning models were starting to embed into otherwise deterministic systems. These models powered very narrow, well-scoped use cases like ranking, recommendation, classification, search relevance, and personalization.
The failures were bounded and fairly interpretable: a ranking might be suboptimal, a recommendation could feel irrelevant. But the surrounding system was still predictable.
QA extended existing best practices: online A/B testing, guardrails, offline training data set evaluation. Since most user interactions could be anticipated, the overall QA paradigm stayed intact.
Enter Phase 6 (late 2022 - present): AI Evals
In November 2022, OpenAI released ChatGPT with GPT-3.5. Within a few months, public APIs made LLMs easy to integrate directly into software products. So teams began shipping AI-powered features at a pace the industry had not seen before.
Very quickly, existing QA practices started to strain. Teams no longer had reliable ways to know whether an AI-driven feature was behaving well across the range of ways users might interact with it. Traditional tests assumed stable inputs, predictable paths, and repeatable outcomes.
To fill that gap, the industry converged on a new quality assurance primitive: evaluations, or evals.
What are evals, at a high-level? According to Anthropic:
An evaluation (“eval”) is a test for an AI system: give an AI an input, then apply grading logic to its output to measure success.
In practice, evals turn fuzzy questions like “is this response good?” into something that can be repeated, measured, and tracked over time through a “hillclimbing” approach. They promise a way to reason about quality in systems where behavior can’t be exhaustively specified ahead of time.
AI Evals 101
To make this more tangible, let’s take a hypothetical example. Suppose you are a PM at Wayfair responsible for improving conversion on product listing pages. Your team is working on a new feature: AI-powered chat that lets users ask questions about a specific SKU.
It’s exactly the kind of personalized, contextual experience LLMs are well-suited for. After you work with your design and engineering teams to implement the feature mechanically, you now face the real question: how do we know the feature is high quality enough to ship?
In practice, you’ll reach for evals as a way to build this confidence. The resulting workflow usually unfolds as a few confidence-building stages and a reactive loop:
Stage 1: Offline evals on known behavior with ground truth data
Construct an eval set of example prompts/ideal outputs from something you understand well. This becomes your golden set: “a curated, high-quality collection of input-output pairs that represent the ground truth or ideal behavior of an AI model.” As the PM, you reach for what you already understand: data sources like common customer questions from support tickets, FAQs that humans already answer today, known product attributes on the listing page, and so on. This is the closest analog to defining traditional test cases in earlier QA methodologies.
Then set up a scoring rubric for grading the quality of the AI outputs, an eval pipeline to run the test cases, and an actual grading mechanism (human graders or LLM-as-a-judge, assertion tests).
These evals are offline: because they are static, easily repeatable, and disconnected from live product usage.
The goal is to isolate the problem and establish baseline scores to hillclimb on: experiment with changes to your product levers and see if they maximize the quality scores. This kicks off the experimentation loop.
Stage 2: Offline evals on extrapolated behavior
Once the feature seems to work well on known examples, it’s time to expand the eval set. You know that the reality is users will throw different types of queries at you. So you expand your eval set to involve more ambiguous phrases, hypothetical scenarios the team expects may occur, or synthetic examples as edge cases.
These evals are still offline.
The goal is make the coverage of your eval cases more representative, closing the gap to real-world behavior.
Stage 3: Online E2E evals
At this point, we need to wire the feature into the real product and test things end-to-end. The evals go online: running on the production stack, with real infra dependencies. A single user prompt will fully execute against our production workflow.
The goal of this stage is to catch any issues that were invisible in stage 1 or 2.
Stage 4: Early rollouts via feature flags, alphas/betas
If everything is going well, you may decide to slowly release the feature to real customers. Using feature flags, you can expose it to small cohorts of users and carefully monitor use with rollback mechanisms in place. You can control for risk by releasing the feature in certain markets or geographies with lower traffic.
You start seeing users ask questions you didn’t anticipate, or combine context in unexpected ways. You get the benefit of time, seeing how repeat usage unfolds organically.
The goal is to learn what emerges: subtle inaccuracies, incorrect assumptions we made in development, initial behaviors we did not anticipate earlier. With these insights, we continue tweaking our product to address observed gaps.
Stage 5: Iterative calibration through progressive release
As you gradually release the feature to more customers, you continue the learning loop from stage 4. There is a back and forth process as rubrics are adjusted, prompts and guardrails are tweaked, and you may even swap out models. You spend time looking through more failure modes from the real world, just as you did in stage 4.
The goal is to keep improving quality with observed real-world behavior, by fixing our eval pipeline to be as representative as possible. At some point, you may decide you’ve hit diminishing returns on hillclimbing and you can call it successfully launched!
Where AI Evals Break Down
When I take a step back and look at these stages, I see two fundamental states of the world:
Proactive, anticipatory development (stages 1-3): either you are working on a feature before it launches to the real world. Your aim is to anticipate how it might behave with real users
Reactive, corrective development (stages 4-5): or you are working after the feature is out there, reacting to how it actually behaves in the wild
These states existed even before AI feature development. The goal of thoughtful product design and testing was to minimize the loss between both states. I refer to this as the reality gap: the gap between our imagined behavior in software features and real, observed behavior. Quality assurance evolved to help us bridge this gap as software grew more complex.
In traditional software, the reality gap was smaller because the behavioral space was combinatorially smaller: user actions were constrained by buttons, screens, and predefined flows. The possibilities were daunting, but still finite.
With AI-driven features, especially those that put natural language as the primary input paradigm to users, the behavior space expands dramatically. Text is open-ended. This is the beauty of AI in its ability to deliver personalized experiences. Users can express intent in their own way.
But this makes the reality gap even larger, as the distance between imagined behavior and real behavior can grow faster than evals can keep up.
The brittleness of evals can be distilled to two optimization objectives:
Before launch, you are lacking information. No matter how much effort you go into designing evals, you cannot encode every possible way users might interact with an AI system. Some behavior is always left out of the imagined world. So you seek to maximize information to improve the anticipatory state’s coverage. This is expensive: acquiring more data sets takes time, has high human cost, and engineering dependencies to set up evals on a mature pipeline at scale. This cost does not pay for itself, as eventually the investment hits diminishing returns.
After launch, you are losing time. Users bring their messy intent and now you’ve started the process of making a feature real, the clock has started. You learn, encode, and feed the learnings back into evals. You seek to minimize the time it takes to correct poor behaviors for the feature. This is also expensive, not just in human time and effort but also in opportunity cost. Everything moves faster in AI-driven development, and you can quickly fall behind to your competitors.
Teams are forced into an impossible tradeoff or may often end up doing both things in parallel. Either way, the reality gap remains.
New Ways of Working with the Reality Gap
Once you see eval-centric development as a way of managing the reality gap, the underlying problem becomes clear.
The gap itself cannot be eliminated. We can only focus on where to shift the cost of managing it. Let’s consider 3 approaches on how to improve managing this gap:
Option 1: Reduce information loss by augmenting coverage with synthetic data generation
On the before lens, constructing representative eval sets with wide enough coverage is expensive and laborious. You need to acquire data sets (which may not already exist), synthesize them, wire up the prompts, define the outputs. This work is slow and difficult to estimate for planning purposes. Furthermore, the effort can hit diminishing returns faster than you anticipated.
An immediate and obvious instinct is: what if we could use AI to help close this gap? Instead of hand-authoring eval cases, teams could provide an AI with rich context about the intended feature (PRDs, design docs, example evals) and ask it to generate diverse, adversarial, and long-tail interaction scenarios. You could iteratively refine coverage based on where you see the poorest quality outputs.
This approach would not eliminate information loss, but it can shift the cost curve. Teams could explore a much larger behavior space before launch than would have been feasible if done entirely by hand.
Option 2: Reduce time loss after launch through agent-driven automation & continuous learning
On the after lens, the main bottleneck is time. Today, correcting issues after you launch something requires a mix of tools and processes to notice failures, diagnose root causes, coordinate across teams, and manually implement fixes. We’re already seeing some early infrastructure startups emerge in this space: whether with AI coding agents like Codex and Claude Code or AI observability tools/platforms like Braintrust and Raindrop.
Observability is important, but even more important is actionability to compress the time loss in the reality gap. We need systems that surface emerging gaps automatically, no longer relying on humans to passively monitor and diagnose issues. We need agents to propose concrete fixes proactively, and in some cases, even implement them directly with the correct guardrails. Braintrust offers Loop, as an early iteration of this direction.
And last week, OpenAI released Frontier: “a new platform that helps enterprises build, deploy, and manage AI agents that can do real work.”
A core offering in Frontier is the ability to configure and manage your evals over time. In addition to defining golden sets and an eval rubric, the platform seeks to help you go one step further in monitoring live data and programmatically updating your evals and your agents. I expect we’ll see much more investment in tools and products in this direction, as it’s necessary to bridge the gap between manual effort before and after launching AI features and agents.
You can also imagine reinforcement learning flows where live data more continuously feeds into the reward model loop. Before launch, teams can train a reward model to approximate the eval rubric using the ground truth eval set. Then as you get production data with real users, you can run the reward model to predict quality scores for those prompts. Agents can automatically inspect low quality scores and update both the ground truth evals and the reward model itself with the findings. This could kick off a recursive loop where the live production data stream is directly integrated into core development workflows that were used to launch the feature. The best part is that refining the reward model can have compounding returns when you want to launch another feature in the same business context in the future.
When discussing this general approach with my good friend Chris Wang, he likened it to treating software as a living organism. Long gone are the days of shrink-wrapped software. But just because release cycles got shorter and we can publish updates seamlessly over the air, we haven’t fully transitioned from treating software something that gets “packaged” and then “shipped”.
With AI, we should view the entire product as living even before it goes out to users. This mindset will help us think about the feedback loop as one continuous process instead of a clear “before” and “after”. When we delegate time-constrained and error-prone steps to agents, instead of humans, then the whole loop can really take off.
Option 3: Change the frame — design for emergence, not exhaustiveness
The last approach is the one I’m most excited about, because it requires a deeper shift in how we think about quality in software.
At the heart of the reality gap is a flawed assumption: that we can anticipate enough of a system’s behavior ahead of time to guarantee quality. I believe this assumption no longer holds with AI-driven features. The behavior space is simply too large.
So instead of better prediction, we should focus on a different goal. We should design for emergence, instead of exhaustiveness or representativeness.
I believe we can take heavy inspiration from the world of video games. Video games have faced this problem for decades. In game testing, designers don’t bother trying to enumerate every possible player action. The interaction space is too vast and adversarial to capture.
Game QA instead focuses on strengthening system-level behavior under pressure.
Designers define the physics of a game world. They shape incentives and constraints for the players to navigate. And then they playtest aggressively to see how the system behaves when those assumptions are stressed.
AI feature development is increasingly similar to this kind of problem space:
Users can take many paths to complete actions: too many to exhaustively predict
Their intent is emergent: they discover new use cases when interacting with the feature and learning they can do new things
Their interactions compound over time: as these learnings emerge, they come back and have more complex interactions
Once you see AI systems with this lens, the limits of evals are clear. So, how can we take inspiration from game design to improve how we build AI features in addition to evals as a practice?
Reframe “coverage” to stress conditions: instead of asking whether we tested a specific use case, ask under what conditions does the system break? Product teams should explicitly define stress conditions where AI features degrade:
Long, multi-turn interactions
Looping usage
Ambiguous / adversarial prompting
Compounding context over multiple sessions
Partial failures in tool calls
Context compaction
Model swaps
Turn dogfooding into structured playtesting: we already have dogfooding, but it can often look like box-checking or persona-driven scripts. What if we approached software testing more like playtesting:
Inherently goal-driven
Rewards creative misuse
Assumes testers will try to break the system
In practice, this means scheduling explicit playtesting sessions where participants are encouraged to explore freely and follow their curiosity. We should consistently rotate testers to bring on fresh perspectives and maximize novelty.
Make agent trajectories first-class QA artifacts: we should make studying agent trajectories a core discipline for testing AI features. The goal is to build intuition for model behavior and failure modes. Studying agent trajectories would involve stepping through the reasoning steps, tool invocations, interrogating the model about key decision points, and studying how context windows evolve. This should not be an ad-hoc activity. And this should not live solely with engineers. PMs and designers need this intuition too, because it shapes how products are designed and constrained in practice. This is one of the most important skills for product builders to master.
The real truth about the reality gap
Ultimately, whether you are building a game or an AI product…I believe the reality gap cannot be closed.
No amount of anticipation, coverage, or eval sophistication will fully bridge the distance between imagined behavior and real behavior in AI systems. At some point, every team must confront the moment where prediction runs out.
I worry the mistake we’re making immediately today in the industry is over-optimizing for confidence in a way that never fully arrives. Evals are powerful but when taken too far, can pull teams deeper into the world of prediction: expanding coverage, refining rubrics, chasing scores.
Instead, the actual work of building great AI products begins when a system meets the real world and behaves in ways that no one fully anticipated. This is why I love the video game parallel, and treating building software as an emergent learning process. We are building open-ended systems — it is impossible to predict everything in advance.
Crossing the reality gap with AI features requires a different mindset for product teams: one that prioritizes learning over certainty, resilience over exhaustiveness, and speed of adaptation over confidence before launch. We should celebrate shipping early: in controlled, low-risk conditions. We should focus on building intuition, surfacing dynamics under surprise.
Evals will remain a part of the game, because they help us prepare. But as any good athlete knows, preparation itself is not readiness. As AI systems become more open-ended, especially with agents, quality will depend on how well our systems survive contact with reality.
This is what it means to design for emergence.