The real cost of LLM-driven test runs is not the model. It is the meter. Your AI bill is about to become your testing bill, and you will notice it the first time a finance partner forwards you an API invoice with a subject line that just says "what is this".
Over the last 18 months the industry quietly slid from one bad pricing model to a worse one. Percy and Applitools charged per screenshot, which was annoying but at least deterministic. The new wave of LLM-driven test runners charges per token, per run, per commit. Nobody has done the token math out loud.
So let's do it, then show the one structural fix that takes the recurring cost to zero.
The new testing cost model
In the old world, your testing line item was a fixed SaaS seat plus a variable per-screenshot fee. You could forecast it. You could cap it. You could stare at a dashboard and know, roughly, what a 30% increase in test coverage would cost you next quarter.
In the new world, an LLM agent drives the browser on every run. It reads the DOM, reasons about state, retries when things look off, and occasionally decides to patch your test. Every one of those steps is tokens. Every token is money. And critically: every replay is a fresh bill. Not once per test authored. Once per test executed. Per commit. Per PR. Per CI retry. Per nightly. Per preview branch.
That is the structural shift. Your test suite stopped being an asset you built once and started being a metered API call you pay for forever.
The math for a small team
Let's make the assumptions explicit so you can argue with them. These are plausible mid-2026 numbers, not measurements from a specific vendor.
- Team: 5 engineers pushing roughly 50 commits/day between them
- Suite: 25 E2E tests
- Per test attempt, a browser-driving agent consumes roughly 15k to 40k input tokens (DOM snapshots, prior turns, tool schemas, screenshots if multimodal) and 3k to 8k output tokens (tool calls, reasoning, assertions).
- Retries and self-correction loops push the effective per-test cost up by roughly 1.5x on average.
- API pricing, as of early 2026, for a frontier-class model (Claude Opus / GPT-5 tier): input in the range of ~$15 per million tokens, output in the range of ~$75 per million. Cheaper models exist, but "cheap" models also retry more, so the per-run cost converges.
Crunching the middle of each range: call it 25k input + 5k output per attempt, times 1.5 for retries = ~37.5k input + ~7.5k output per test per run. That is ~$0.56 + ~$0.56 = roughly $1.10 per test per run on the expensive end, and around $0.20 per test per run if you're lucky with a smaller model and low retry rates.
Take a middle estimate of ~$0.40 per test per run. At 25 tests, that is ~$10 per full suite run. At 50 commits/day with CI triggered on each, that is ~$500/day, or ~$10,500/month for 21 working days. For 5 engineers.
Scaled out:
| Team size | Tests | Full suite runs/day | Est. monthly token cost |
| Small (5 eng) | 25 | 50 | ~$10,500 |
| Medium (20 eng) | 80 | 200 | ~$134,000 |
| Large (100 eng) | 300 | 800 | ~$2,000,000 |
Adjust those by 2x in either direction depending on model choice, retry discipline, and how aggressive your CI triggers are. The absolute number isn't the point. The point is that a testing line item measured in seven figures per year is trivially reachable without anyone making a bad decision.
And that's just the happy path
The numbers above assume well-behaved runs. Reality has a few extra meters running:
- Retries on flaky runs. LLMs are non-deterministic. A test that passes 9 times out of 10 will, at scale, cost you 10 to 30% more than the baseline just from re-attempts. Flake becomes a direct P&L item.
- Context inflation. As your app grows, so does the DOM. So does the number of user flows. So does the codebase context the agent pulls in to "understand" a failure. The token cost per test is not constant - it grows with your product. You are paying a tax on your own success.
- Maintenance loops. When a test breaks, an LLM-driven runner will typically try to fix it. That means multi-turn reasoning, additional tool calls, sometimes a full re-read of surrounding code. A single stubborn failing test can quietly burn $5 to $10 in a single CI run before giving up.
None of this shows up in a product screenshot. All of it shows up on the invoice.
Why this is structurally worse than the old pricing
Percy's per-screenshot model was bad. It punished teams for testing more. But it had one redeeming property: the cost was deterministic and visible. You knew, before you ran anything, what it would cost. You could budget it. You could argue with it.
LLM token costs are variable, invisible until the month closes, and - here's the worst part - they scale with problems, not with value delivered. A flaky test costs more than a stable one. A broken build costs more than a green one. A growing codebase costs more than a small one. Every structural pain point in your engineering org is now also a line item on your API bill. This is the wrong direction for an incentive gradient.
The fix is a single design decision: spend tokens only when the AI creates or fixes a test, never when it replays one. That is exactly the model Lastest is built on - zero-token replays. Every subsequent execution - a CI commit, a manual re-run, a scheduled cron - is plain Playwright at $0 in tokens. The recurring, per-run bill that the math above projects doesn't shrink; it collapses into a one-time authoring cost. And if you author against a local Ollama model on your own hardware, even that authoring cost is $0 tokens. On self-hosted infra, screenshots and replays are unlimited regardless of volume, so the horizontal axis of every table above - run frequency - stops mattering entirely.
The escape hatch: generate once, replay forever
There is a clean structural fix, and it is almost embarrassingly simple: use the LLM exactly once.
Use AI to generate the test. Capture the output as plain deterministic Playwright code. Commit that code. Run it on CI with zero model calls. The AI did its job - it turned "log in, add item to cart, check out, verify receipt" into a real script - and then it goes home. Your CI pipeline never pings an API again.
The math changes shape entirely. Instead of a recurring cost per run, you pay a one-time generation cost of roughly $1 to $3 per test authored, and then $0 per run forever. Compare the cumulative 12-month cost for the small team from above:
| Month | LLM-driven re-runs (cumulative) | Generate-once, replay forever (cumulative) |
| 1 | ~$10,500 | ~$50 |
| 3 | ~$31,500 | ~$80 |
| 6 | ~$63,000 | ~$130 |
| 12 | ~$126,000 | ~$250 |
One line climbs linearly. The other flattens almost immediately, with a tiny slope that only moves when you add or rewrite tests. Over a year, the difference is two orders of magnitude for the same coverage.
Want to see the flat line in practice? Browse real generate-once replay runs - each was authored by AI once and has replayed at $0 tokens ever since - or spin up your own in a couple of minutes on Lastest Cloud. Bring your own AI for authoring: Claude CLI, the Anthropic API, OpenRouter, OpenAI, or a local Ollama model for $0 local tokens.
What about AI test fixing?
Honest answer: yes, when a test legitimately breaks, AI is useful, and you should use it. The difference is when and how bounded.
In the LLM-driven-run model, AI touches every execution, whether or not anything is wrong. In the generate-once model, AI is only invoked when a human or the system flags a real breakage - and even then, a well-designed tool classifies the failure before burning tokens on a fix. Lastest auto-classifies every failure as real regression, flaky, environment, or test-maintenance, with a confidence score and reasoning, so a flaky run never triggers a paid fix loop. When a fix is warranted, attempts are capped and control returns to the human if it can't resolve cleanly. The cost is bounded, visible, and tied to actual signal instead of noise.
This is the model Lastest is built around: AI writes the test once with a 7-layer self-healing selector stack, deterministic Playwright code runs it forever, and AI only re-enters the loop on a flagged failure with strict attempt limits. Self-hosted via docker-compose, FSL-1.1-ALv2 licensed and free forever on your own infra, with zero tokens per replay by design.
The one-line takeaway
Use AI where it creates leverage (writing the test). Don't use AI where it creates a subscription (running the test).
If you're a solo builder or a small team shipping fast and watching every dollar of your API spend, the generate-once model is not a nice-to-have - it is the only version of AI testing that doesn't quietly become your largest infrastructure line item by end of year. More on that philosophy over on the vibe coders page, and the broader pattern is in the flake tax and the 2026 AI testing scale gap.
Do the math for your own team before your next renewal cycle. The assumptions above are rough on purpose - plug in your own commit volume, your own suite size, your own model choice. The conclusion tends to hold regardless of which numbers you pick: paying per run for something you can pay for once is a structural mistake, and it gets more expensive the more your team grows.
Stop renting your test suite back from an API. Self-host Lastest for free (open source, FSL-1.1-ALv2, unlimited screenshots and replays on your infra), or skip the ops and run on Lastest Cloud at $299/month flat - no per-seat and no per-screenshot fees, with zero-token replays either way. The full source is on GitHub.
The three-step path from this page: see it working in the demos gallery → start a run on Lastest Cloud or self-host in minutes → watch every replay land at $0 tokens on the test management dashboard. Do the math once, then never pay it per run again.