AI Evals in Production: The Error-Analysis-First Playbook (2026)

Product managers evaluate AI and LLM features by starting with error analysis, not tooling: read a sample of real production traces, note what went wrong in plain language, group the failures into recurring categories, and count them to get your actual failure distribution. In Dexity's analysis of live job descriptions, evals now appear in 56% of AI-engineer and 32% of product-manager postings — up from near-zero in 2024. Only then do you build measurement — LLM-as-a-judge scorers, assertion checks, and monitoring — around the failures that matter. Evals, not model choice or prompt cleverness, are the difference between an AI feature that works in production and a demo that falls over. As models commoditize, the durable advantage is knowing whether your system actually works — and why.

Why did AI evals become a hiring requirement?

Two years ago "evals" was a research word. Today it's on the job description.

The reason is simple: AI systems are probabilistic. You can't ship them the way you ship deterministic code — you have to measure whether they produce correct, useful output, on your data, for your users.

Employers are now writing this into the job itself:

"Help customers develop evaluation frameworks to measure Claude's performance for their specific use cases." — Anthropic, Applied AI Architect job description (2026)

How do you run error analysis on your AI outputs?

The most common mistake is reaching for an eval tool first. The teams that ship reliable AI invert that — they start by looking at their own data:

1. Read real traces. After any significant change, manually review a sample of your system's outputs (a few dozen is enough to see patterns).
2. Note what's wrong, in open-ended language — don't force categories yet.
3. Categorize the failures into recurring buckets (wrong retrieval, hallucinated field, tone, format, broken tool call).
4. Count them. Now you know your actual failure distribution, not your imagined one.

Only then do you build measurement — because now you know what to measure. This error-analysis-first approach is the method practitioners like Hamel Husain and Shreya Shankar have made the standard for applied LLM evals (Hamel Husain, "Your AI Product Needs Evals"), and it's the same discipline Aman Khan lays out in Lenny's Newsletter's guide to evals for PMs.

What kinds of AI evals are there?

Once error analysis tells you what to measure, you choose how to measure it. There are three canonical approaches, and mature systems use all three in combination rather than picking one:

The practical ordering follows from that: humans first (to see failures and create labels), assertion-based checks wherever correctness is deterministic (they're cheap, fast, and never drift), and LLM-as-a-judge to scale the subjective judgments once a human-labeled set exists to keep the judge honest.

How do you scale from manual review to automated evals?

Once error analysis reveals the failure modes, you scale from manual review to automated measurement:

• LLM-as-a-judge — encode each recurring failure as a judge prompt, validated against your human labels, so every new output gets scored.
• Synthetic data — manufacture edge cases you can't yet collect in production.
• Production monitoring — keep sampling live traces; the failure distribution drifts as usage grows.
• A data flywheel — each round of analysis → judges → fixes → new traces compounds into a system that improves over time.

Open frameworks from the major AI labs make the plumbing straightforward (OpenAI Evals, Anthropic evaluation docs) — but the frameworks are the easy part. The judgment about what to measure comes from error analysis.

How do you design a good eval?

How do you write an eval rubric?

A rubric turns a vague "is this good?" into specific, independently checkable dimensions. Break "good" apart into the qualities that actually matter for your feature — for example correctness, retrieval relevance, format, tone, and safety — and score each one separately instead of collapsing everything into a single blurry number. For each dimension, write down what a pass and a fail look like, anchored to concrete examples pulled straight from your error analysis. Prefer a binary pass/fail or a short ordinal scale per dimension, and tie every criterion back to a real trace so two reviewers reading the same output land on the same label.

How big should your eval dataset be?

There's no universal number, and bigger isn't automatically better. Start small: as the error-analysis step above notes, a few dozen traces is often enough to surface your recurring failure patterns. Size the set for coverage of the failure modes you found — a dataset that contains real examples of each category beats a larger random sample that misses the cases you care about. Then grow it over time as production surfaces new failure modes, so the set keeps reflecting how the system actually behaves.

What are evals not?

• Not public benchmarks. An MMLU score doesn't tell you whether your RAG bot answers your customers correctly. Evals are application-specific.
• Not vibes. "It looks good in the demo" is not an eval — the point is a repeatable, counted failure distribution.
• Not an infra project first. Tooling comes after error analysis; the reverse is the most common failure.
• Not one-and-done. The failure distribution drifts; evals are a continuous loop, not a launch gate.

Why is the window to learn evals closing?

Evals went from research topic to hiring requirement in under two years — 56% of AI-engineer and 32% of PM JDs now call for them. The builders who learn error-analysis-first evals now are scarce; the ones who wait will be competing against teams whose products measurably improve every week.

Related reading

• AI Engineer Career Path in 2026 — evals appear in 56% of these roles.
• What Does a Product Manager Career Look Like in 2026? — 32% of PM JDs demand evals.
• How to Become an AI Product Manager in 2026.

Build a real eval system for your AI product

Reading about evals isn't the same as running them on your own product. Dexity's AI Evals for PMs course walks you through the full loop — data collection, error analysis, architecture-specific eval strategies, and regression detection — so you ship AI you can actually stand behind.