How-To

How to Test AI Features: Reliable Tests for Non-Deterministic Output

An LLM feature never returns the same string twice. Here are the assertion patterns that make it testable in CI, and the judge problem nobody warns you about.

Adithya Aggarwal

CTO & Co-founder at Pie

11 min read Posted Sep 22, 2026

An LLM feature does not return the same string twice. Sampling from a probability distribution is the entire point of the model, so two runs of the same input give you two different answers, and both of them can be right.

Now go write a test for it. Every framework you have ever used is built on one assertion, “actual equals expected”, and it is the first thing you have to throw out. What goes in its place is a set of checks on the properties the output has to hold no matter how it is worded. There is also a trap sitting inside the most popular replacement, and almost every guide on this topic walks you straight into it.

Why AI Features Break Your Normal Tests

AI features break your normal tests because the output is non-deterministic. The same input produces different, still-valid results across runs, so any assertion that checks for one exact answer will pass and fail at random. Retries do not fix it. The variance lives in the model, not in your test.

The instinct is to reach for the temperature knob. Set it to zero and you get repeatability, the reasoning goes. You mostly do not. Anthropic puts it in its own API reference, noting that even with a temperature of 0.0 the results will not be fully deterministic, and OpenAI’s guidance on the seed parameter carries the same caveat.

The cause sits below the API. Thinking Machines Lab traced it to batch size, where your request gets grouped with whatever else is in flight, the floating-point reductions run in a different order, and the numbers come out slightly different. Add silent model updates and load-based routing on top of that. A suite built on frozen strings goes red on a Tuesday because a provider rebalanced traffic, not because your code changed, which is the same shape as any other non-deterministic test you cannot simply delete.

There is a second reason, and it decides how you test. Your AI feature is not the model. It is the model plus the prompt you wrapped around it, the context you retrieved, the tools it can call, and the UI that renders the answer. Any of those four can break while the model call still returns a perfectly valid response, which is exactly the failure a benchmark score will never catch.

So the model is a dependency, not the thing under test. AI testing at the feature layer means testing that whole surface and treating the stochastic part as one component inside it.

What to Assert When Output Changes Every Run

When the output changes every run, assert on the properties that must hold regardless of wording, not on the wording itself. A correct answer to “summarize this refund policy” can be phrased a hundred ways. What makes it correct is that it names the refund window, cites the real policy, and invents nothing. Those are checkable. The exact sentence is not, and does not need to be.

There are five assertion types, and mature AI test suites use most of them at once. Pick the strictest one the output shape allows, because strict assertions catch more and flake less.

Forcing structured output is the highest-leverage move on that list and the one most teams skip. If the feature can return JSON instead of prose, most of the test problem goes away, because the shape is deterministic even when the values are not. Validate the shape first, then assert on the values:

// The feature classifies a support ticket. Force JSON, not prose.
// toMatchSchema is not built into Jest. It comes from jest-json-schema.
import { matchers } from 'jest-json-schema';
expect.extend(matchers);

const out = await classifyTicket(input);        // { category, priority, confidence }

expect(out).toMatchSchema(ticketSchema);         // shape is deterministic
expect(VALID_CATEGORIES).toContain(out.category);
expect(out.confidence).toBeGreaterThanOrEqual(0);
expect(out.confidence).toBeLessThanOrEqual(1);
// No exact-match on any generated string. Nothing here flakes on wording.

For free text you cannot schema-constrain, lead with invariants. They are cheap, deterministic, and they catch what actually reaches production. An answer that leaks the system prompt, blows a length budget, or fabricates a citation fails on a rule, with no model in the loop to second-guess it.

Semantic similarity and a judge come in only for the residue, the is-this-answer-actually-good question that no rule can express. Start strict and loosen only when the output shape forces you to. It is the same discipline that keeps ordinary flaky tests from accumulating on a normal suite.

Your AI Judge Is Non-Deterministic Too

The judge is a model too. It carries its own sampling variance and its own biases, which means a naive grader can mark an output as passing when the output was wrong, and you will believe it because the run came back green. Almost every guide on testing AI features leaves this out. It is the failure mode that quietly rots a suite.

LLM-as-a-judge is a real technique with real evidence behind it. Zheng et al. introduced and benchmarked it in Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena at NeurIPS 2023, where a GPT-4 judge reached over 80% agreement with human preference, roughly the level at which two humans agree with each other.

The same paper names the biases you inherit along with it.

• Position bias: The judge favors an answer for sitting in a particular slot in the prompt.
• Verbosity bias: It favors the longer answer over the shorter, clearer, more accurate one.
• Self-enhancement bias: A judge prefers answers it generated itself, which the authors flag as suggestive rather than settled given the data they had.

Hand a judge with a verbosity lean a confident, wrong, well-padded answer and it will often wave it through.

We run a judge in production, on Pie’s own testing agents, and ours skews the opposite way from what people expect. When those agents propose a fix for a bug they found, a separate model reviews the evidence, and it rejects fixes it should pass because it wants more proof than the run gave it. Same problem, other direction. Once a model is grading a model, the grade is an estimate, and you need to know which way yours leans before you let it gate anything.

Dropping the judge is not the answer. For open-ended quality it is often the only scorer you have. Constrain it the way you would constrain any unreliable dependency. Give it an explicit rubric instead of asking whether the answer is good. Force a structured verdict so you can read its reasoning. Run it more than once, with the answer order swapped between runs so position bias cannot cast the deciding vote, and require agreement.

Before it gates anything, validate it against a set of human-labeled examples and measure how often it agrees with people. If the judge and a human disagree a third of the time, the judge is not ready to hold your release. The same trust gap shows up when you are testing AI-generated code, because more AI in the loop means more places a confident wrong answer passes unchallenged.

// Constrain the judge: rubric in, structured verdict out, two runs with the
// answer order swapped. Same input twice only samples variance; the swap is
// what takes position bias out of the vote.
const rubric = "Pick the answer that (1) directly addresses the question, " +
               "(2) cites a source present in the context, and (3) invents no facts. " +
               "Reply with A, B, or tie.";

async function judge(candidate, reference, context) {
  const [first, second] = await Promise.all([
    gradeWithRubric({ A: candidate, B: reference }, context, rubric), // { winner: 'A' | 'B' | 'tie', reason }
    gradeWithRubric({ A: reference, B: candidate }, context, rubric),
  ]);
  const ok = (v, slot) => v.winner === slot || v.winner === 'tie'; // at least as good as the reference
  const votes = [ok(first, 'A'), ok(second, 'B')];
  // Require agreement across both orders. A split vote is a manual-review
  // signal, so it comes back as 'review' rather than a silent fail.
  if (votes.every(Boolean)) return 'pass';
  if (!votes.some(Boolean)) return 'fail';
  return 'review';
}

How to Test an AI Feature, Step by Step

Testing an AI feature end to end is a repeatable process. Pin the model, build a golden set, assert on properties, measure consistency across runs, and gate CI on an aggregate pass rate rather than a single result. Each step exists to convert a probabilistic feature into a signal you can put in a pipeline.

1. Pin the model and its version: Record the exact model and version your feature calls, and set it explicitly in the test environment. Providers ship silent updates, so an unpinned model means a passing suite can break with zero code changes, and you will lose a day proving it was not you.
2. Build a golden set: Assemble 30 to 100 representative inputs covering the happy path, the edge cases, the adversarial prompts from someone trying to jailbreak your feature, and past production failures. The golden set is your regression suite. It grows every time a real bug escapes.
3. Pick an assertion per output shape: For each case, choose the strictest assertion the output allows from the table above. Structured output gets schema plus value checks. Free text gets invariants first, then similarity or a judge only where quality is genuinely subjective.
4. Run each case N times and measure consistency: Because a single run is a coin flip, run each case several times and record the pass rate. A case that passes 10 of 10 is solid. One that passes 6 of 10 is a real reliability problem in your feature, surfaced instead of hidden.
5. Gate CI on the aggregate, not the run: Fail the build when the suite-wide pass rate drops below a threshold you set, for example 95%, rather than on any individual run going red. Treat a falling pass rate exactly like a latency regression, a trend to investigate rather than a flake to re-run.
6. Alert on drift: Re-run the golden set on a schedule, not only on deploys. Because the model can change under you, a scheduled run is how you catch a provider update that quietly degraded your feature before a customer does.

TLDR on the process. Pin what you can, sample what you cannot, and gate on the trend rather than the run.

Testing the AI Feature Inside the Real App

Unit-testing the model call is necessary and nowhere near sufficient. The AI feature your user touches is the model plus the prompt, the retrieval, the tools, and the UI that renders it, and any of those can break while the model call still returns a valid response. An end-to-end test exercises that whole path. Open the app, trigger the AI action, and verify the resulting screen state is actually correct.

Traditional automation struggles most right here. Selector-based end-to-end tests assert on the DOM, so they either check for an exact string, which non-deterministic output breaks on every run, or check that some element merely exists, which passes even when the rendered answer is nonsense. Neither one verifies the thing that matters, which is whether the feature behaved correctly for the user. That gap is precisely where a false pass lives, and where a suite starts lying to you.

The principle underneath the tooling matters more than the tooling. At the end-to-end layer, assert on behavior and intent, not on markup. Ask whether the feature did the right thing for the person using it, the way a human tester would, and leave the deterministic schema-level checks lower in the stack where they belong. An intent-level check like that is the whole premise of autonomous QA, and it is the one thing a DOM assertion cannot express.

Pairing the two layers is how you cover an AI feature without drowning in flake. Strict property checks on the model call, intent-level verification in the running app. The second layer has to watch the end-to-end flow and judge the end state against what the step was supposed to achieve, which is a different job from confirming that a div appeared.

Why There Is No 30% Rule for AI Testing

People search for a “30% rule” hoping there is a clean threshold hiding somewhere in here. There is not one. No formal 30% rule exists in AI, and nothing resembling one exists in testing. The phrase circulates as a loose human-in-the-loop split, roughly 70% of the repetitive execution to the machine and the 30% that needs judgment to a person.

As a boundary for testing AI features, that split is worth keeping. AI can generate test cases, expand your golden set, and grade open-ended output. What stays human-owned is the definition of correct, the acceptance criteria, the rubric the grader follows, and the call on whether a pass rate is good enough to ship. Automation scales the labor. A person still owns the meaning, and audits the grader that enforces it.

Test the Behavior, Not the Wording

Every hard part of testing an AI feature comes back to one shift. Stop asserting on the words and start asserting on the behavior. The output will never be identical twice, so a correct feature is one that returns valid structure, holds its invariants, and does the right thing for the user, however it phrases the answer.

And watch the judge. The most dangerous test on an AI feature is the one that passes while the feature is broken, and an unconstrained model grading another model produces exactly that. At the end-to-end layer, where the model and the prompt and the UI all meet, the check you want is behavior against intent.

That is the check Pie is built to run. It exercises the real flow and verifies the feature did what it was supposed to do, which is the one assertion a false pass cannot fake.

Adithya Aggarwal

CTO & Co-founder at Pie

Eight years building search and delivery systems at Amazon. The kind of scale where flaky tests block billion-dollar releases. Now CTO at Pie, building AI agents that adapt when your UI changes. LinkedIn →