LLM Evaluation in Production: How to Test AI Features Before They Break

LLM Evaluation in Production: How to Test AI Features Before They Break

The standard approach to testing LLM-powered features before shipping is still, in most teams, a manual review by whoever built the feature, a few spot checks against inputs the developer already expects to work, and a go/no-go decision based on gut feel. That process fails silently. Prompt changes, model version bumps, and retrieval layer edits all introduce regressions that manual spot checks will not catch systematically. The result is quality degradation in production that users notice before your engineering team does.

This post is a practical framework for LLM evaluation in production: how to build the data foundation, how to automate scoring, how to wire it into CI/CD, and how to define thresholds that are actually worth enforcing. This is not a survey of evaluation tooling. It is a decision architecture you can begin implementing this sprint.

Why Traditional Testing Logic Breaks for LLM Features : 

Unit tests work by asserting that a given input produces a specific, deterministic output. LLMs do not produce deterministic outputs. The same prompt, passed to the same model at the same temperature, can return subtly different responses across calls. Increase temperature, change the system prompt by a single sentence, or switch from gpt-4o to a newer model version, and output quality can shift materially across your entire feature surface.

This means the conventional test assertion, "output equals expected string," is useless for most LLM features. What you need instead is a scoring layer that evaluates outputs against criteria: is the answer grounded in the provided context? Does it follow the required format? Is the tone appropriate for the product? Is the factual content accurate relative to a reference? These are not pass/fail assertions. They are scored dimensions, and your evaluation pipeline needs to treat them that way.

The practical consequence is that LLM evaluation in production is a measurement and monitoring problem, not a testing problem in the traditional software engineering sense. Teams that try to force it into a conventional QA model end up with either trivially weak tests (regex checks on output structure) or no automated checks at all. Neither is acceptable once an AI feature is serving real users at scale.

Building a Golden Dataset That Is Actually Useful : 

A golden dataset is the foundation of any repeatable evaluation pipeline. It is a curated collection of input prompts paired with reference outputs or quality labels, held fixed as a benchmark against which all future changes are measured. Without it, you have no baseline and no way to detect regression objectively.

Building a useful golden dataset requires deliberate sampling, not convenience sampling. Most teams default to logging inputs they already know the model handles well. That produces a dataset that confirms capability rather than exposing fragility. Instead, sample across:

• Typical high-frequency inputs from real production logs : 

1. Edge cases where the feature has previously produced poor outputs.
2. Adversarial inputs designed to stress the boundaries of the prompt.
3. Inputs from underrepresented user segments or query patterns.

For most SaaS AI features, 50 to 150 examples is sufficient to detect meaningful regressions without making evaluation runtime prohibitive in a CI/CD context. Below 50, your signal is too noisy. Above 200, you are adding evaluation latency without proportionate sensitivity gain, unless your feature surface is genuinely broad.

Label each example with either a reference output (for grounded tasks like summarisation or extraction) or a set of quality criteria with expected scores (for generative tasks where there is no single correct answer). Treat the golden dataset as a versioned artefact in your repository, not a spreadsheet someone owns locally.

LLM-as-Judge: How to Automate Output Scoring

Once you have a golden dataset, you need a scoring mechanism that does not require a human reviewer for every evaluation run. LLM-as-Judge is the most practical approach for most teams: you use a separate, typically more capable, model to evaluate the outputs of your production model against defined criteria.

A basic LLM-as-Judge prompt structure for a summarisation feature looks like this:

System: "You are an evaluator assessing summary quality. Score the following summary on three dimensions, each from 1 to 5: (1) Faithfulness: does the summary contain only information present in the source? (2) Completeness: does it cover the key points? (3) Conciseness: is it appropriately brief? Return a JSON object with keys faithfulness, completeness, conciseness, and a brief rationale for each score."

User: "Source: [original document]. Summary: [model output]."

The evaluator returns structured scores you can aggregate across your golden dataset, compute a weighted composite, and compare against your threshold. The key design decisions are:

1. Use a more capable model as judge than the model being evaluated, or at minimum a model from a different provider to reduce correlated failure modes.
2. Keep the scoring rubric stable across evaluation runs so scores are comparable over time.
3. Run each example through the judge at least twice and average scores to reduce evaluator variance.
4. Log full judge rationale alongside scores so you can audit unexpected failures.

This approach pairs well with the observability instrumentation described in AI agent observability in production, where intermediate outputs and scoring signals should be traced as first-class spans rather than treated as debug logs.

Wiring LLM Evaluation into CI/CD as a Quality Gate : 

LLM testing in CI/CD means running your golden dataset evaluation automatically on every pull request that touches a prompt, a retrieval configuration, or a model version pin. The gate logic is straightforward: if the composite quality score drops more than a defined threshold below your baseline, the build fails and the change cannot merge.

A minimal pipeline structure looks like this:

1. PR opened: trigger evaluation workflow
2. Load golden dataset from versioned store
3. Run production prompt (or changed prompt) against each input using the target model
4. Pass each output to LLM-as-Judge for scoring
5. Aggregate scores, compute composite, compare to baseline
6. Post score summary as PR comment, fail build if composite drops below threshold

Setting the threshold requires running a baseline evaluation first. Use your current production prompt and model to score the full golden dataset, then treat that composite score as your floor. A regression threshold of 5 to 10 percent below baseline is a practical starting point. Tighten it for customer-facing features where quality directly affects retention. Loosen it slightly for internal automation steps where minor output variation is tolerable.

One common failure mode is treating the quality gate as a binary pass/fail without logging the per-dimension breakdown. If your gate fails because faithfulness dropped but completeness improved, you need that signal to debug the regression. Always persist the full scoring output, not just the composite, as part of the CI artefact.

Offline vs Online Evaluation: When to Use Each

Offline evaluation, running your golden dataset before deployment, is your pre-production gate. Online evaluation is your post-deployment monitoring layer. They are not interchangeable and both are necessary for mature LLM evaluation in production.

Offline evaluation catches regressions you can detect before users see them: prompt changes, model swaps, retrieval layer modifications. It is fast, controlled, and produces a binary deployment signal. Its limitation is that your golden dataset is a fixed sample. It cannot anticipate every input pattern that real users generate, and it cannot detect distribution shift as your user base or their query patterns evolve.

Online evaluation addresses exactly those gaps. The implementation involves:

1. Sampling a percentage of live production requests (typically 5 to 15 percent) for automated scoring.
2. Collecting explicit user feedback signals where the product UI supports it (thumbs up/down, edit actions, regeneration requests).
3. Running automated LLM-as-Judge scoring on the sampled outputs asynchronously.
4. Alerting when rolling quality scores drop below a defined production threshold . 

If you are building RAG-based features, online evaluation is especially important because retrieval quality can degrade independently of prompt quality, and that degradation will not show up in offline evaluation unless your golden dataset includes retrieval as a variable. The decision framework for RAG versus other retrieval approaches is covered in more depth in RAG vs fine-tuning vs prompt engineering, but the evaluation implication is consistent: any component that can change independently needs its own monitoring signal.

How ZycoSoft Builds LLM Evaluation into Production AI Pipelines : 

The framework above is architecture, not theory. Implementing it in a production SaaS product requires decisions about where evaluation state lives, how to version golden datasets alongside prompt changes, how to instrument LLM calls for scoring without adding latency to the critical path, and how to surface evaluation signals to engineering leads in a way that is actionable rather than noise.

These are the problems our team works through in every AI automation and custom SaaS development engagement. Our AI automation practice covers end-to-end LLM pipeline design including evaluation architecture, so that quality gates are a first-class part of the system design rather than a retrofit after the first production regression. We build with n8n for orchestration, wire LLM-as-Judge scoring into CI workflows, and design golden datasets in collaboration with product teams who understand the feature's quality requirements from the user's perspective, not just the model's perspective.

For teams building AI features into custom SaaS products, we apply the same scoping discipline we use across all our engagements: we avoid over-engineering evaluation at MVP stage (a 50-example golden dataset and a basic composite gate is enough to start) while architecting the evaluation layer to extend cleanly as the product scales. That means the evaluation pipeline does not become a bottleneck when your golden dataset grows to 500 examples or when you add a second model provider. If you are also thinking about how model swappability affects your evaluation baseline, the architectural considerations in LLM vendor lock-in and model-swappable architecture are directly relevant to how you version and re-baseline your evaluation pipeline when models change.

We work as a dedicated team extension, embedded in your engineering workflow, not as a black-box delivery shop. Evaluation architecture is one of the areas where that model matters most, because the decisions compound: a golden dataset built without understanding the product's real failure modes will generate false confidence at every evaluation run from that point forward.

If you are shipping LLM features into a production SaaS product and your current evaluation process is still manual spot checks, the fastest path to a repeatable gate is to start with the golden dataset. Everything else in this framework builds on that foundation.

Ready to build an evaluation pipeline that catches regressions before your users do? Talk to the ZycoSoft team about your AI feature architecture.