Ask a team how they test their AI feature and you get one of three answers. Some have evals: a golden set of inputs, scored offline, owned by whoever tunes the prompt. Some have contract tests: JSON schema validation, tool-call assertions, a token ceiling. And almost all of them, when you ask about the actual feature in the actual browser, say some version of "you cannot really test that, the output is different every time."
That sentence is doing an enormous amount of work, and most of it is wrong.
The output is different every time. The feature is not. Open your AI panel and count what is on screen: a prompt input, a submit control that disables while in flight, a streaming indicator, a generated block, a copy button, a regenerate button, a feedback control, a token or cost readout, an error state for rate limits, an error state for refusals, an empty state before first use. Exactly one of those things is nondeterministic. The rest is a normal UI that would get a normal E2E test if it lived anywhere else in your product.
So we have a strange situation in 2026. The login form, which changes twice a year, has six tests. The feature that spends real money per keystroke, ships weekly, and is the entire reason half your users showed up has a screenshot in the PR description and a vibe check from whoever built it.
Why the string assertion poisoned the well
Most teams did try, once. The first test somebody writes against an AI feature looks like this:
await page.getByRole('button', { name: 'Summarize' }).click()
await expect(page.getByTestId('summary'))
.toContainText('Here is a summary of') // <- the mistakeIt passes on Tuesday. On Wednesday somebody adjusts the system prompt and the model starts with "This thread covers" instead. The test fails. Nothing is broken. Somebody loosens the assertion. Then a model version bumps, the phrasing shifts again, and the test fails again. Nothing is broken again.
Two or three rounds of that and the test gets deleted, not because anyone decided AI features do not need tests but because a test that cries wolf is worse than no test. The team then generalises from the specific failure ("asserting on generated text is hopeless") to the general conclusion ("testing AI features is hopeless"), and the panel goes uncovered for the next two years.
The fix is the one the evals world figured out years ago and the E2E world has been slow to import: assert properties, not strings. You are not checking that the model said a particular thing. You are checking that the product did its job around whatever the model said.
The three layers, and which one you are missing
It helps to be precise about what each kind of test is for, because these get conflated constantly and the conflation is how the gap opens.
Evals answer "is the answer good?" They run offline against a curated set, or sample live traces, and they score faithfulness, relevance, and format with deterministic checks, rubrics, or a judge model. They belong to whoever owns the prompt. They are essential and they are not tests of your application.
Contract tests answer "is the shape right?" Does the response parse. Does the tool call carry the arguments it claims. Does a refusal come back as a refusal rather than an unhandled exception. Do we stay under the cost ceiling. These are cheap, deterministic, and mostly live at the API layer.
Product E2E answers "does the feature work?" And this is the missing one. Not "is the summary accurate," which evals cover, but: does the panel render, does the submit control disable during the request, does the stream settle into a stable layout, does the copy button copy, does a 429 from the provider surface as a human-readable message rather than a blank card, does the whole thing still look right in dark mode at 1280 and at 390.
None of those questions involve the content of the generated text. All of them are questions your users hit daily. And all of them are answerable with the same tooling you already use for the rest of the product.
Six assertions that hold across model swaps
Concretely, here is what a durable E2E test of an AI feature actually asserts. Every one of these survives a reworded prompt, a temperature change, and a provider migration.
- The scaffolding renders. Panel present, input enabled, controls labelled correctly, usage readout visible. Pure structural assertions on a UI that does not change when the model does.
- The in-flight state is correct. Submit disables, a busy indicator appears, and the previous answer either clears or stays visibly stale. This is where most real AI-feature bugs live: double-submit, a spinner that never resolves, a stale answer presented as fresh.
- The output region is non-empty and bounded. Not what it says. That it exists, that it is not blank, that it did not blow past its container and push the controls off screen. A 400-word answer where you designed for 80 is a layout regression, and it is one a pixel diff sees instantly.
- The controls act on the output. Copy puts something on the clipboard. Regenerate produces a second request. Feedback posts. These are ordinary interaction tests that happen to sit next to generated text.
- Failure paths are real UI. Stub the provider to return a rate limit, a timeout, a refusal, and a malformed payload, then assert that each one produces a specific, readable, recoverable state. This is the single highest-value test on the list and almost nobody has it, because it never comes up in manual testing where the provider is usually healthy.
- The layout holds under content variance. Feed it a one-line answer and a wall of text and confirm both look deliberate.
Notice how much of that is about stubbing the provider rather than calling it. Testing your error handling against a live model is both expensive and unreliable; you cannot summon a 429 on demand. Intercepting the request at the network layer, the same way you would intercept any other API in a data-heavy app, turns the nondeterministic dependency into a fixture you control. Most of your AI feature's test suite should never touch a model at all.
Masking: the setting that makes the panel testable
Visual testing an AI feature sounds absurd until you separate the generated region from everything else. A pixel diff over a page with regenerated text is 100% noise. A perceptual diff is better but still fires on every reword. So you mask the generated block and diff the rest.
Lastest ships three diff engines and auto-masking of dynamic content, and the combination is what makes this practical. Mask the generated block. Let the perceptual engine handle the rest, since it models human vision and ignores the antialiasing and font-rendering noise that makes pixel diffs unusable across machines. What survives that filter is signal: a control that disappeared, a panel that lost its border, a usage row that stopped rendering, a dark-mode variant where the masked region's background no longer matches its container.
The same stabilization machinery that exists for timestamps and rotating avatars applies directly here. Frozen clocks, network-idle waits, DOM stability detection, and auto-masking were built because every real app has nondeterministic regions. An LLM response is just a larger, more interesting one.
Then there is cost, which is what actually kills most attempts at this. If every test run hits a model, your CI bill scales with your test count and somebody eventually caps the suite. Lastest's replays are zero-token: AI runs when you author or heal a test, and every replay after that is plain Playwright execution. Combine stubbed providers with zero-token replays and testing an AI feature costs exactly what testing a settings page costs, which is the only way this survives contact with a budget.
The failure mode nobody writes tests for
One last category, and it is the one I would write first if I were adding a single test to an existing AI feature today.
Streaming. Almost every AI UI streams, and streaming introduces a class of bug that has nothing to do with model quality and everything to do with your rendering: layout thrash as tokens arrive, a scroll container that fights the user's scroll, a copy button that becomes active before the response finishes, markdown that renders half-parsed mid-stream (an unclosed code fence is a spectacular one), and a cancel control that leaves the UI wedged. None of that shows up in an eval. All of it shows up for users, constantly.
Testing it is unglamorous and entirely mechanical: stub the stream, drive it in controlled chunks, assert the intermediate states, then assert the settled state. Capture a screenshot mid-stream and at rest. That is two visual baselines and maybe fifteen lines of interception, and it covers a category of bug that currently reaches production every single time.
Frequently asked
How do you test something whose output changes every run? You do not assert the output. You assert the properties around it: that the panel renders, that the submit control disables in flight, that the output region is non-empty and stays inside its container, that the controls act on it, that provider errors surface as readable UI. Every one of those assertions survives a reworded prompt, a temperature change, and a provider swap, because none of them depends on what the model said.
Do our evals not already cover this? Evals answer whether the answer is good. They say nothing about whether the panel rendered, whether a rate limit produced a readable message, or whether the streaming layout holds. Those are product questions, and they fail independently of model quality. Keep the evals; they are not a substitute for testing the feature.
Should our tests call the real model? Mostly no. Stub the provider at the network layer so you can summon rate limits, timeouts, refusals, and malformed payloads on demand, which is where most real bugs live and which you cannot trigger reliably against a healthy provider. Keep a small number of live-model smoke tests if you want integration confidence, and run the rest against fixtures.
Can visual regression testing work on a page with generated text? Yes, once you mask the generated region. Pixel diffing is useless there because every reword is a change, but masking the block and running a perceptual diff over the rest catches the failures that matter: a missing control, a broken panel, a layout that collapses when the answer runs long. Lastest auto-masks dynamic content and lets you pick the diff engine per test.
What is the single highest-value test to add first? The provider failure path. Stub a 429, a timeout, and a malformed response, and assert each produces a specific, readable, recoverable state. Manual testing almost never covers it because the provider is usually healthy during development, so it is both the least-tested and the most-hit path in production.
Start here
Pick your most-used AI panel and write the failure-path test this afternoon. Stub a rate limit, assert the message, screenshot the state. If it passes on the first try, you have learned something good. If it does not, you have found a bug your users have been quietly hitting for months.
Self-host Lastest for free to generate and replay those tests: AI writes the Playwright, a human reviews it once, and every replay afterward is zero-token and unlimited on your own hardware. If you would rather skip the ops, Lastest Cloud is a flat $299 a month with no per-seat or per-screenshot fees. Source is at github.com/las-team/lastest.
The nondeterminism is real, and it is a rounding error on the surface area. Stop letting 5% of the pixels excuse 100% of the feature.