A QA lead at a 200-person SaaS company ran their full regression suite before shipping a new AI-powered customer support feature. Every test passed. Green across the board. Three days into production, users started filing complaints: the assistant was confidently citing policies that had changed six months ago. No test had caught it because every test was checking whether the system responded, not whether the response was right.
That's the problem in one story. Traditional QA is built on a simple contract: given input X, the system must return output Y. Every assertion, every snapshot, every mock in your test suite encodes that assumption. Generative AI breaks it completely. Same input. Different output every time. None of it is technically "wrong." And yet some of it will sink your product.
This guide walks through a production-grade framework for testing non-deterministic AI features, covering AI observability, LLM evaluation metrics, failure classification, and the tooling QA teams can actually operationalise without a machine learning background.
Why Deterministic Test Assertions Fail on Generative AI Features
QA teams didn't choose the wrong tools. The tools were right for the problem they were built to solve. Deterministic systems are testable because they're predictable. Non-deterministic AI systems are not, and the mismatch runs deeper than most teams realise until they're already in production.
The fundamental issue: your test suite asserts exact outputs. Large language models produce probabilistically valid responses. These two things cannot coexist in the same assertion logic. Every test case you've written for a traditional API assumes the function is pure (same input, same output, always), similar to deterministic utilities like an ml to grams converter, where identical inputs always return identical results.
The Determinism Assumption Baked Into Your Test Suite
Standard CI/CD pipeline quality gates assume a deterministic function. Pytest fixtures mock an API response and assert the string you get back. Snapshot tests capture an exact output and fail on any deviation. Recorded API playbacks replay a fixed interaction, but once the model is live, that recording is fiction.
Every one of these breaks when the model response is real. Not because the tooling is bad, but because it was designed to test functions, not judgments. Writing test cases for non-deterministic AI-driven features requires a fundamentally different mental model, and most teams don't realise this until they've already shipped something that failed silently.
How Output Variance Differs From a Bug
Here's where most QA teams get stuck. A traditional bug is binary: the system either returns the right value or it doesn't. With LLMs, output variance exists on a spectrum:
- Acceptable variance: Different phrasing, same semantic meaning. The conversational AI assistant, including an AI Call Assistant, says, "I can help you reset your password," vs "Let me walk you through resetting your password." Both are correct. Neither is a bug.
- Defect-level variance: Factual errors, hallucinated policy details, off-topic responses, or outputs that violate safety rules, even if they're grammatically fluent and structurally coherent.
That distinction is exactly what forces QA teams to adopt LLM evaluation metrics rather than binary pass/fail assertions. The question stops being "did the system respond?" and becomes "was the response good?"
Building a Non-Deterministic Test Strategy: The Four-Layer Framework
Across our AI quality assurance engagements, the teams that ship generative AI features with confidence aren't running fewer tests; they're running a different kind of test structure. We use a four-layer framework: Functional Boundary Testing, Semantic Evaluation, Behavioural Regression, and Production Observability.
Each layer targets a different failure surface. Each maps to a different toolset. And critically, the first layer is still deterministic, which means it belongs in your existing CI pipeline right now.
Layer 1: Functional Boundary Testing
This is where traditional software testing still works. Functional boundary tests check what the model is and is not allowed to do, not whether its output is good, but whether the system around it behaves correctly.
What belongs here:
- Input validation: Malformed inputs, oversized payloads, unsupported languages
- Prompt injection guards: Adversarial prompts designed to hijack the system prompt or extract sensitive context. This is where your test scenarios for security edge cases live.
- Output schema enforcement: If the LLM is supposed to return structured JSON, assert that it does, every time. API responses must conform to the contract; this is a hard, deterministic gate.
- Token limit edge cases: What happens at the boundary of your context window?
- Refusal-rate benchmarks: For safety-critical features, test that the model refuses out-of-scope or harmful requests at the expected rate
These test cases ARE deterministic. They test the guardrails around the model, not the model's judgment. They run in CI. They pass or fail cleanly. Build them first.
Layer 2: Semantic Evaluation with LLM Evaluation Metrics
This is where most teams have no test coverage at all, and it's the layer that catches the class of defect that makes it into production anyway.
Semantic evaluation replaces "did the output match?" with "was the output correct?" The core LLL evaluation metrics your team should instrument:
Each metric maps to a CI assertion: faithfulness score below 0.85 fails the build; toxicity score above 0.05 routes to human evaluation before release. The thresholds are calibrated against your golden dataset, more on that in Layer 3.
For evaluation frameworks, RAGAS is the most practical starting point for RAG pipelines. LangSmith works well for LangChain-based stacks. Arize AI covers the commercial end with stronger enterprise support and SOC 2 compliance. None of them is plug-and-play; expect a week of setup to get a meaningful signal.
Layer 3: Behavioural Regression Testing
Most teams think about regression tests as "did this output change?" For LLMs, that framing is wrong. The output will change. The question is whether the behaviour changed.
A behavioural regression suite is built from a golden dataset: a fixed set of prompts paired with human-validated expected-behaviour profiles, not exact expected outputs, but descriptions of what a correct response looks like. "The assistant should acknowledge the cancellation request, confirm the policy, and offer an alternative." That's a behaviour profile. You evaluate against it using LLM graders, not string matching.
Human raters build the ground truth. LLM graders run the evaluation at scale. Human-in-the-loop evaluation kicks in when the grading logic produces a low-confidence score, typically flagging outputs in the 40th to 60th percentile range for human graders to review before a deploy goes ahead. That routing logic is what separates a production-grade regression suite from a prototype.
Detecting behavioural drift matters particularly across:
- Model version updates (your LLM provider ships a new model)
- Prompt template changes (your engineering team edits the system prompt)
- RAG corpus updates (your knowledge base gets refreshed)
The detection mechanism is a "soft regression gate," a statistical threshold on semantic similarity scores. If the mean faithfulness score across your golden dataset drops by more than 8% relative to your baseline, you flag for review rather than hard-blocking the deploy. Hard blocks are appropriate for toxicity and safety metrics. Quality drift needs human review, not automation.
Layer 4: Production AI Observability
Pre-deployment testing is necessary. It is not sufficient. No test dataset anticipates every edge case that real users will surface, and for LLMs, the edge cases are where the interesting failures live. Continuous validation in production is what closes that gap.
AI observability is the production-side counterpart to pre-deployment evaluation. It monitors output quality in real time, across real traffic, and feeds signals back into your evaluation pipeline for autonomous testing improvement.
What a production AI observability stack needs to instrument:
- Request/response logging: Full capture of every prompt and completion, with metadata (user ID, session context, model version, latency, token count)
- Semantic scoring at inference time: Running lightweight automated evals against live responses, not all of them, but enough for statistical sampling
- User feedback signals: Explicit (thumbs down, flagged response) and implicit (immediate follow-up questions, session abandonment after a response)
- Error taxonomy: Distinguishing hallucinated responses, refusals, format failures, and latency issues, because they have different remediation paths
- Production monitoring dashboards: System stability metrics alongside output quality metrics, in the same view
Selecting the Right AI Observability Tools for Your Stack
The tool selection question comes up in almost every engagement we run. Teams either default to whatever observability vendor they already pay for, or they spend three weeks evaluating platforms and ship nothing. Neither approach works.
Here's the evaluation framework we recommend:
Five criteria that actually matter:
- Integration surface: does it plug into your existing APM stack (Datadog, Grafana, New Relic) or does it require a parallel observability layer?
- Scoring methodology: Which LLM evaluation metrics does it surface natively, and can you add custom metrics?
- Alerting granularity: Can you alert on per-feature semantic drift, not just system-level errors?
- Data retention: Do you have enough history to run regression analysis across model updates?
- Compliance: for enterprise environments, SOC 2 Type II is non-negotiable
Open-Source vs Commercial AI Observability Platforms
Datadog AI Observability: When It Fits and When It Doesn't
If your team is already running Datadog for APM, the LLM Observability module is the lowest-friction path to production monitoring. You get LLM traces, token cost tracking, and semantic anomaly detection surfaced in dashboards you already know. The onboarding is a few hours, not a few weeks.
The honest caveat: Datadog's native LLM evaluation metrics are shallower than purpose-built platforms. Faithfulness scoring, context recall, and hallucination detection require custom instrumentation or integration with an external evaluation layer. It is excellent for latency monitoring, error rates, and cost attribution. Less strong for output quality without additional work.
Our Take: For teams already on Datadog Enterprise with moderate LLM traffic, the AI Observability add-on is worth it. For teams evaluating purpose-built platforms, Arize AI and LangSmith go deeper on semantic quality scoring. The right choice depends on whether you prioritise operational metrics or quality metrics. Most production systems eventually need both.
LLM Evaluation Frameworks: What QA Engineers Actually Need
There are three evaluation approaches in practical use. Knowing which to apply is more important than knowing the tools. A well-designed evaluation harness combines all three, routing each output type to the right scoring method based on task structure and confidence signals.
Reference-Based Evaluation for Structured AI Outputs
When a correct answer exists, factual responses, structured data extraction, classification, and reference-based evaluation work well. These tasks have ground truth you can compare against.
BLEU and ROUGE scores measure n-gram overlap between the model output and a reference answer. Exact-match metrics check whether specific facts appear in the response. These are reliable for structured tasks and actively misleading for open-ended generation: a response can score zero on BLEU while being completely correct. Context sensitivity matters here too: a metric that works for code generation will give you noise on a conversational AI feature.
Model-Graded Evaluation for Open-Ended LLM Responses
For open-ended generative tasks, the most practical approach uses a judge LLM: a separate, typically more powerful model that scores responses against a success criteria rubric.
The rubric typically covers accuracy, helpfulness, safety, and relevancy, each scored on a 0 to 1 scale, with per-metric weights you calibrate to your feature's requirements. The judge model approach scales to production traffic volume in a way that human evaluation cannot. Multi-turn evaluations require the full conversation history passed to the judge, not just the final response. Grading logic that ignores context sensitivity will miscategorise a lot of valid outputs.
The key risk: judge model bias. Your judge model has its own failure modes, and its scores are only as good as your rubric prompt. Before relying on judge scores in CI, validate them against human raters on at least 50 examples. If the correlation is below 0.75, your rubric needs work.
Setting Evaluation Thresholds Your CI Pipeline Can Enforce
Threshold calibration is the step most teams skip, and it's why their automated evals pipeline produces noise instead of signal.
The process:
- Run your evaluation metrics across your golden dataset (your human-validated examples)
- Identify the score distribution for each metric on examples that humans rated as acceptable
- Set your hard fail threshold at the 10th percentile of acceptable scores. Below that, something is genuinely wrong
- Set your review flag threshold at the 25th percentile. Outputs in this range warrant a human look before they ship
Statistical validity requires at minimum 100 samples per test run for your thresholds to hold at p < 0.05. Below that, you're reading noise. A/B testing machine learning evaluation configurations, comparing two rubric variants or two judge models on the same dataset, is the most reliable way to validate that your grading logic is stable before you lock it into CI.
Testing AI Agents and Agentic Pipelines in Production
Single-turn LLM evaluation is hard. AI agent observability is a different problem category entirely.
Agents make multi-step decisions. They call them external tools. They accumulate context across turns and use prior outputs to inform future actions. Agent development introduces failure modes that don't exist in single-turn systems: tool hallucination (calling a tool that doesn't exist), reasoning loop drift (the agent loses track of the original goal across a long Multi-Turn Conversation), cascading context errors (a wrong assumption in turn 3 corrupts every subsequent decision), and goal misalignment in long sequences. Multi-agent orchestration adds another layer: failures can propagate across agent boundaries in ways that are genuinely hard to trace without the right instrumentation.
Standard LLM evaluation metrics score a single response. They don't capture whether the agent reached the right end state via a valid reasoning path. You need trace instrumentation for that.
Tracing Agent Decision Paths for QA Review
The instrumentation approach: use OpenTelemetry spans to make every agent action a traceable event. Each tool call, each reasoning step, each API response becomes a span in a distributed trace you can query, score, and aggregate.
The OpenTelemetry GenAI Semantic Conventions provide the standard attribute schema for LLM traces. Use this rather than rolling a custom schema. LangSmith and Arize AI both support multi-turn agent trace evaluation natively, which is a meaningful difference from platforms built primarily for single-turn use cases. For teams building conversational AI features with complex agent development requirements, this native support is worth prioritising in your platform evaluation.
Defining Pass/Fail Criteria for Multi-Step Agent Tasks
For agentic systems, the evaluation unit is the task, not the turn. The agent passed if it reached the correct end state via any valid reasoning path, not if it used a specific sequence of steps. Success criteria are defined at the task level.
This requires task-level outcome definitions: "the agent successfully retrieved the correct order status and communicated it to the user" is an outcome definition. "The agent called the get_order_status tool in step 2" is a path constraint, and path constraints break as soon as your agent's reasoning patterns evolve.
Path coverage testing ensures your test scenarios cover the critical branch points in the agent's decision tree, particularly the branches where tool selection is ambiguous or context is noisy. Autonomous testing of agent decision paths at scale requires a trace-aware evaluation harness, not just a prompt-response pair.
How Frugal Testing Builds Your AI Observability and LLM Evaluation Stack
The teams we work with on AI in test automation share a common starting point: generative AI features are in production, the QA team is responsible for quality outcomes, and there's no existing LLM evaluation infrastructure. Manual spot-checking is the only quality gate. It doesn't scale, and it doesn't give you the test coverage you need for AI-driven features.
We scope and build the full pipeline: evaluation framework design, golden dataset construction, metric selection, threshold calibration, observability instrumentation, and CI gate integration. Teams we've worked with have moved from zero LLM test coverage to a production-grade evaluation pipeline in three to four weeks, with measurable quality baselines established before the next major model update.
What our AI QA engagement delivers:
- Weeks 1 to 2: Audit of your AI feature surface: which generative features are live, what evaluation exists, and where the highest-risk test coverage gaps are
- Weeks 3 to 4: Evaluation pipeline build: golden dataset construction, metric selection, judge model calibration, threshold setting
- Weeks 5 to 6: Observability instrumentation: wiring evaluation scores into your CI pipeline as soft and hard regression gates, configuring production monitoring dashboards
- Week 7 onwards: Ongoing maintenance as your model, prompts, or RAG corpus evolve
What you own at close: a fully documented AI quality assurance framework, instrumented dashboards, and CI gates your team can maintain independently.
This engagement fits engineering teams at US software companies (50 to 2,000 employees) shipping customer-facing generative AI features, chatbots, copilots, AI-assisted workflows, or RAG-powered search, where QA is responsible for AI quality but has no ML background and no existing evaluation infrastructure.
"The teams that ship AI features with confidence aren't testing less. They're testing differently. Functional boundaries in CI, semantic quality in evaluation, behavioural regression in staging, and continuous validation in production."
Key Takeaways
Conclusion
QA teams didn't fall behind on AI features because they weren't paying attention. They fell behind because the tooling conversation moved faster than the testing methodology conversation. Everyone talked about which LLM to use. Nobody talked about how to know whether it was working.
The framework here isn't theoretical. It comes from working with teams that shipped first and scrambled to build quality infrastructure second, usually after the first production incident made the absence of that infrastructure visible. The four layers exist because each one catches a class of failure that the others don't, and skipping any layer leaves a gap that real user traffic will eventually find.
Generative AI testing tools are maturing fast, but the tools don't tell you how to think about the problem. That thinking still requires a team that's done it before and knows where the gaps are.
The gap between "our AI feature is in production" and "we know our AI feature is working" is narrower than most teams assume. It usually comes down to about four weeks of focused instrumentation work. The question is whether you close that gap before or after your users find it for you.
People Also Ask (FAQs)
Q1.What's the difference between AI observability and traditional APM monitoring?
Ans: APM monitors system health: latency, error rates, and uptime. AI observability monitors output quality and semantic correctness of what the model actually returns. Both are necessary; they're not substitutes for each other.
Q2.How do you set a pass/fail threshold for a non-deterministic LLM test?
Ans: Run your evaluation metrics across human-validated examples to establish baseline score distributions, then set your hard fail threshold at the 10th percentile of acceptable scores. Everything below that boundary represents output quality worse than any human-validated acceptable response.
Q3.Can generative AI testing be fully automated, or does it always require human review?
Ans: Most evaluations can be automated with model-graded scoring. Human-in-the-loop evaluation is reserved for low-confidence outputs flagged by the evaluation pipeline, typically five to fifteen per cent of production traffic, not every inference.
Q4. Which LLM evaluation framework should a QA team start with?
Start with RAGAS if you have a RAG pipeline. Use LangSmith if you're on LangChain. Use MLflow if you're already on a model registry workflow. The framework matters less than the metric selection and threshold calibration; those decisions are yours regardless of the tool.
Q5. How do we test AI agents differently from single-turn LLM features?
Single-turn: evaluate each response independently against relevancy and faithfulness metrics. Agents: evaluate the full decision path and final task completion state using trace instrumentation. You need task-level success criteria, multi-turn evaluations, and OpenTelemetry spans; per-turn scoring alone misses the failure modes that matter in agentic systems.
