- Coverage matters more than a large, repetitive golden dataset.
- Metrics need context; discrete outcomes can be easier to turn into product decisions than false precision.
- Use deterministic checks where possible and LLM judges where interpretation is required.
- Production traces, human review and judge disagreement show what to test next.
- The payoff is a shorter path from a production signal to an evidence-backed product improvement.
Over the last 12 months, I’ve spent weeks building evaluations, refining prompts and monitoring how production AI features are actually used. Evals have become a key part of the AI PM and AI Engineer job stack. They take time up front, but they pay dividends by answering questions that otherwise become slow, subjective debates: did the prompt change help, is this model better for this use case, what is failing in production, and did the fix work?
Coverage beats repetitive volume
A bigger golden dataset is not automatically a better one. Before launch, I want a versioned set that covers happy paths, edge cases, known regressions, safety scenarios and important customer or workflow variations. Synthetic data is useful for exploring scenarios we have not seen yet, but hundreds of near-duplicates can add cost and noise.
The useful question is: What new behaviour, edge case or risk does this test cover? Keep the dataset stable enough to compare releases, then evolve it deliberately as production reveals new failure modes.
Generic metrics need context
Conciseness, helpfulness and relevance only become useful when tied to the question and the decision. A two-line answer can be perfect for a simple request and poor for a complex one. I increasingly prefer outcomes such as Good / Okay / Bad or true / false when deciding whether an interaction succeeded.
A score of0.83looks precise, but someone still has to decide whether0.83is acceptable.
Continuous scores still help with ranking, anomaly detection and threshold tuning. But an opinionated evaluator can be easier to operationalise when the question is “should we ship this?”
Use the right kind of evaluator
LLM-as-a-Judge is powerful, but it should not be the default for every problem. Use deterministic checks for schema validation, exact matches, required fields and structural rules. Use an LLM judge where interpretation is required: was the answer appropriate, sufficiently complete, policy-consistent or a reasonable course of action?
A judge can only evaluate the context it receives. For agentic features, that may be closer to request + context + available tools + tool calls + outcome than simply prompt → response. That distinction matters if PMs, QA, domain experts and governance teams are going to help define what “good” means rather than leaving evaluation entirely to engineering.
Production traces are where the loop gets real
Pre-launch tests cover what you already know to test. Production shows what you forgot: strange combinations of context, unusual tool paths, borderline outcomes and failures that matter to important customers.
Automated checks and judges should surface the highest-value traces for human review. The judge is not replacing the reviewer; it is helping decide where human attention is worth spending. When review finds a useful failure mode, that trace can become a candidate for the next version of the golden dataset.
Track disagreement, then turn signal into decisions
If the judge says Good and a reviewer says Bad, the disagreement is a calibration signal. The rubric may be unclear, the judge may be missing context, or the human may be applying a criterion that was never encoded. Persistent disagreement tells you the evaluator needs refinement.
Tools such as Langfuse and Braintrust help compare prompts and models, inspect traces and run judges. The next bottleneck is product workflow: reviewing interactions quickly, separating review data from aggregate analytics, filtering by release, customer, timezone and model, and turning a trace into a Jira idea or bug with its evidence intact.
The workflow becomes:
Golden dataset → evaluate and refine → launch → production traces → evals → human review → decisions → improvements → test
That is the real payoff. Evals are work, but they should shorten the distance between something happened in production and we understand it, improved the product and measured whether the improvement worked.
At Propel Ventures, we’re helping product teams adopt evaluation practices drawn from lessons across industries, so they can build higher-quality AI features that more consistently match their intended outcomes.