Testing telemetry, like restaurant kitchens and parliamentary procedure, is something almost everyone in the industry has opinions about and almost no one publishes. We have a strong intuition that visual regression suites are flaky. We have stronger intuitions about why. We have surprisingly little data of the form: here is a single suite, run for ten weeks, with every retry and every diff and every selector attempt logged - and here is what it actually says. This post is an attempt to publish exactly that artefact, with six charts, a frank discussion of the dataset's limits, and the entire ~13MB of CSV/JSONL bundled for download at the end.
The headline number is that we have 8,308 test runs, 12,894 visual diffs, and 499 selector-fragility records, captured between 2026-03-01 and 2026-05-09 from a single test runner operating against a small but representative set of internal apps. We did not write tests for this study. We did not sequence runs to manufacture a result. The data is whatever ten weeks of an internal team running its visual test suite happened to produce. It is messier than a lab-run benchmark, and that is the point.
I. Why nobody publishes telemetry like this
The first thing to understand about test-suite telemetry is that publishing it is institutionally awkward. The numbers tell on you. They expose the suite's flakiness, your team's tolerance for that flakiness, and - most uncomfortably - the gap between your testing playbook (the document) and your testing practice (the reality). The honest version of any sufficiently old test suite contains pockets of waste, neglect, and quiet acquiescence to chronic failure that nobody is proud of and nobody has time to fix. Releasing a dataset of how it actually behaves means publishing the receipts.
The second thing is that without telemetry like this, the literature on testing is forced to argue from anecdote. Best-practice guides and tooling vendors confidently rank selector strategies, recommend stabilization techniques, and prescribe pixel thresholds - and they do this overwhelmingly without reference to ground-truth data, because that data is not available in public form. The result is a discourse where strongly-held positions float free of empirical anchoring. We can do a small thing about that by anchoring our own discourse: by publishing the rough record of one suite's behaviour, in a form that anyone can re-derive every claim from.
So that is the spirit of the dataset. Not a benchmark - a field recording. Not "here is what visual testing should look like" - "here is what ours looked like, and here is the file."
II. Shape of the data
Three tabular files form the core of the dataset, with a fourth JSON file containing pre-computed summary statistics:
flake_runs- one row per run. 8,308 rows × 21 columns. Status, duration, error category, error signature, retry count, console / soft / assertion error counts, browser, date bucket, hashed run/test/team identifiers. The unit is "the runner attempted to execute a test."visual_diffs- one row per diff produced. 12,894 rows × 17 columns. Pixel difference (count), percentage difference (0–100), classification (unchanged/changed/flaky), region count, page-shift detection, DOM-diff presence, status (auto_approved/pending/approved), hashed identifiers. The unit is "the diff engine compared one capture to one baseline."selector_fragility- one row per (test, selector kind) pair. 499 rows × 8 columns. Selector kind, failure count, success count, total attempts, mean response time. The unit is "across all this test's runs, how reliably did this kind of locator find what it was supposed to find."
One disclosure up front: retry_count is uniformly 0 across flake_runs. We deliberately captured this dataset with retries disabled, because retries hide flake - a test that "passes after one retry" registers as a pass, not as the disturbance it actually is. With retries off, every flaky outcome lands in the data as either a fail or as a diff classified flaky. The cost is that is_flaky on flake_runs (which the runner sets only when retries succeed) is uniformly false. Use the flaky classification on visual_diffs for diff-level flake. This is the first of several places where the dataset's instrumentation does not say what its column names suggest.
III. The bimodal failure curve
Here is the observation that, more than any other in the dataset, has changed how we think about the suite. The overall failure rate - 870 of 8,308 runs, or 10.5% - is a useless number. It is the average of two completely different failure modes that share almost no causes, almost no fixes, and almost no temperament.
The U is the story. The <1s bucket - 294 runs - fails 91.8% of the time. These are not tests; these are crashes. The runner started, hit a network or environment problem within a fraction of a second, and quit. Looking at the error categories in this bucket: 153 "other" (mostly setup-time exceptions), 127 "network" (DNS / connection refused before the page even rendered), 11 "selector_not_found" (selectors evaluated against a page that never loaded). There is no "test" being run in any meaningful sense. These are environment failures masquerading as test failures.
The >60s bucket - 1,432 runs - fails 17.4% of the time, almost entirely from timeouts: 201 of the 222 timeouts in the entire dataset live here, and a further 11 navigation errors are timeouts wearing trench coats. Long-running tests fail because they ran too long, not because they discovered something wrong with the application. The fix here is structural - break the test up, parallelise its setup, replace the polling-loop assertions with event-driven ones - and almost never "make the assertion more permissive."
The 5–15s bucket - 2,998 runs, the largest in the suite - fails just 3.0% of the time. This is the band where the suite is doing its job. Tests start, navigate, perform actions, take screenshots, finish, and pass. If the entire suite lived here, we would have a 97% pass rate and a different blog post.
The implication is uncomfortable: the suite's overall pass rate is being dragged down almost entirely by the two ends. The healthy middle is doing fine. Aggregating across duration buckets is hiding the structure of the problem, and the standard intervention - "improve the suite's stability" - is the wrong abstraction. The two extremes need different fixes. Lumping them under one number is how teams end up doing neither.
IV. The category honesty test
If failure-rate-by-duration is the geometric story, error categories are the linguistic one. They tell you what the runner thought happened.
The single largest failure category is "other" - 351 of 870 failures, or 40.3% of the total. Look at any production test suite long enough and you will find that "other" is the largest category, because the rule-based categorizer that decides which bucket each error goes into is built by a small team during a two-week project that ends just before the team learns what the actual error landscape looks like. The categorizer recognises timeouts because timeouts are obvious, network errors because Playwright tags them with net::, and navigation errors because the framework names them - and then everything else lands in "other" and stays there.
This is not a criticism of categorizers. It is a criticism of treating any category named "other" as a signal of anything other than "we are not paying attention to half of what's happening." A useful exercise we ran on this dataset was to grep for the most common error_signature strings inside the "other" bucket, and we found three patterns we wish had their own category - page.evaluate: Execution context was destroyed, most likely because of a navigation alone accounts for a couple of dozen rows. This is a categorizable failure that is hiding in "other" because nobody told the categorizer about it. The dataset preserves the raw error_signature column precisely so anyone can do the same exercise.
The next thing the chart tells you is that real assertion failures are vanishingly rare. One. There is one assertion failure in the entire dataset of 870 failures. A test suite, in the platonic sense, is supposed to fail because an assertion caught a regression. In practice, that almost never happens. What happens is environmental noise (timeouts, networks), structural noise (navigations and JS errors), and bot-detection noise from the categorizer ("other"). Visual-diff regressions are tracked separately in visual_diffs and don't show up in this column at all. If you wanted to use this suite as evidence that "tests catch bugs", you would have to look very carefully at where the bugs are being caught, because it isn't in the assertion column.
V. Selector fragility, or: the role-locator scandal
Of all the patterns in the dataset, none caused us more discomfort than the selector-fragility story. We have repeated, in our own writing, the now-canonical advice that role-based selectors - page.getByRole('button', { name: 'Save' }) and friends - are the most reliable form of locator, the most semantic, the one most aligned with accessibility, and the one a well-written test should reach for first. Then we looked at the data.
Of 345 selector attempts where the locator strategy was role-based, 328 failed. That is a 95.1% failure rate. CSS selectors, the alleged worst-practice, failed 61.4% of the time. Test-IDs - the gold standard pattern recommended by every testing guide we have ever written - failed 60.6% of the time. The most reliable category in the table, with a 0% failure rate across 16 attempts, is "other" - which in our schema means a fallback bucket that includes XPath and a handful of opaque JS-driven locator strategies. Read that sentence again and decide how much you want to update your priors.
Some honesty about what this does and doesn't say. The dataset's "selector kind" is the kind reported when the selector failed or succeeded - meaning that a test that uses a role selector first and falls back to CSS will register two attempts, one of each kind. So the role row over-counts attempts that were always destined to fail because the role selector was speculative ("try the semantic name, fall back if not found"). Sample sizes are also wildly unequal: CSS at 1,910 attempts, role at 345, testid at 66. Treat the smaller-sample numbers as suggestive, not conclusive.
With those caveats stated, the chart still says something. It says that in our suite, role-based selectors are speculative. They are tried first, they often fail to match because the pages we test do not consistently expose their controls with the accessibility names the tests expect, and the runner falls back to something else. The naive reading - "role selectors are bad" - is wrong. The accurate reading is uglier: the apps under test are not as accessible as we believed they were, and our tests are quietly working around that. The dataset surfaces this as a 95% failure rate on role; the underlying truth is a remediation backlog on the apps themselves, not a failure of the locator strategy.
This is the kind of finding that only a field-recording dataset can produce. A benchmark suite, where the tests are written for the benchmark, will always be perfectly accessible and will always register role selectors as the most reliable. A real suite, against real apps, with real codebases at real points in their accessibility journey, will tell you the truth.
VI. Visual diffs are bimodal too
If failure rates by duration form a U, percentage-difference distributions form something stranger. The mass is at exactly two points. Almost everywhere else is empty.
Of 5,332 diffs classified as "changed" (i.e., the engine decided the new capture differed from the baseline), 4,014 had a percentage_difference of 0%. Read that twice. The diff was classified as a change, but the percentage-difference metric reported zero. This is not a contradiction once you understand the engine: the classification can fire from a structural diff, a region-bounded diff, or a non-pixel signal (page-shift, DOM-diff), and the percentage-difference field reflects pixel-level accumulation only. So the engine can - and frequently does - say "this changed, but not in a way that pixel arithmetic captures."
The 1,074 diffs at ≥25% are the opposite story. These are massive changes - entire layouts moved, a redesign landed, a screenshot captured before vs. after a viewport-doubling resize. They are the screenshots where the test was, in some sense, comparing apples to oranges. They are obvious enough that no human reviewer needs more than a glance.
The interesting cell is the empty middle. Between 0.1% and 25%, the count is zero. There is no medium-sized visual change in this dataset. The diff engine almost never reports a delta that you would naively call "a few pixels different" or "a small layout shift" or "an animation caught mid-frame." It either reports near-perfect match (with the differences expressed in the structural / region channels) or it reports overwhelming difference. The middle band - the "1%-of-pixels-changed" zone where naive pixel-threshold tools spend most of their time - is empty in our data.
We have a hypothesis about why. The diff engine has aggressive stabilization at the front (frozen timestamps, font fallback handling, animation pinning, page-shift tolerance) which collapses small noise to zero. Anything left after stabilization is either definitively the same or definitively different; the noisy middle has been engineered out. This is a satisfying explanation if it's true, and it would mean the bimodal distribution is the desired outcome of stabilization, not a measurement artefact. We don't have causal evidence - the dataset doesn't include the un-stabilized runs as a counterfactual - but the shape is consistent with the hypothesis.
Two more observations from the same file, both worth more attention than they got: the flaky classification covers 153 diffs, and their median percentage_difference is 1.74% - squarely in the empty middle. So the middle band exists, but only as flake. And only 46 diffs were tagged with page-shift detection, all in the "changed" classification, which is a very small fraction of even the "changed" pool - a hint that the page-shift detector is conservative, and that the layout-shift class of bug is rarer than its discourse share would suggest.
VII. The test populations split too
Bimodality, it turns out, is the dataset's recurring shape. Here it is again at the per-test granularity. Of 1,167 unique tests in the suite, 279 had at least 5 runs (the threshold below which per-test rates are uninterpretable noise). Their failure-rate distribution looks like this:
The tests cluster at the ends. 85 tests (30.5% of the population) failed exactly zero times across all their runs - including 69 tests with at least 10 runs and a perfect record. They are the suite's load-bearing wall. A further 64 tests failed less than 5% of the time; these are also broadly fine, with the occasional environmental burp. So roughly half the suite's tests are stable.
The other half tells a different story. 74 tests sit in the 5–20% failure-rate band - a range we would call "starting to be a problem"; the team can ignore them but probably shouldn't. 29 tests sit at 20–50% - the "everyone has a workaround" band. And the most uncomfortable cell, 27 tests with a failure rate of 50% or higher, is what we'd call the chronic failure cluster: a small set of tests that fail more than they pass, and yet remain in the suite.
The chronic failure cluster is the dataset's most actionable finding. Twenty-seven tests are responsible for an outsized share of the suite's pain. They consume CI minutes, they generate notifications, they trigger investigations that arrive at the same root cause they arrived at last week, and they erode trust in the rest of the suite by training engineers to ignore failures by default. The fix is unromantic: identify them, decide whether to repair them or remove them, and stop pretending they're going to self-heal. The dataset surfaces them by their test_id_hash; in practice, of course, this means a one-week project for a small team to do the boring work of triage.
VIII. The triage backlog as cultural artefact
The last chart is the one that the human-process people will recognise immediately. Visual diffs go into a queue. Some are auto-approved (engine confident the change is noise), some are pending review (engine flagged them, no human has decided), some are approved (a human has stamped them as intentional). We logged the status of every diff at the moment of capture export.
The funnel reveals an attention budget more than a technical pipeline. 7,409 of 12,894 diffs (57.5%) were auto-approved as unchanged - the engine was confident, the diff didn't fire as a change. 5,332 (41.4%) were classified as changed, and joined the human queue. Of those, 1,436 were approved by a reviewer (26.9% of the changed pool, 11.1% of the total) and 4,048 are still pending - they have not been reviewed at the time of dataset export.
That ratio of pending-to-approved (4,048 to 1,436, or roughly 3:1) is the artefact worth studying. It is not a technical artefact. It is a human-attention artefact. It says that the team's appetite for reviewing diffs is roughly one-third of the rate at which the engine produces them. The backlog is structural, and a faster diff engine would only deepen the imbalance unless paired with either (a) better auto-classification (the engine confidently approves more of the noise) or (b) a more selective firing threshold (fewer non-actionable diffs hit the queue).
The 153 diffs in the flaky classification are the smaller, weirder cell. These are diffs where the engine determined the same comparison fired differently across consecutive captures - an actual flake, in the strict sense: same baseline, same target, different verdict. They are 1.2% of total diffs, which is small in the gross but disproportionately costly: each one consumes a triage cycle, generates a notification, and contributes to the "I can't trust the suite" feeling that, more than any technical bug, kills visual testing programs.
IX. What this dataset doesn't show
It is more honest to be specific about the dataset's limits than to leave you to discover them after a Saturday's worth of analysis.
One browser. Everything is Chromium. The capture window predates our Firefox/WebKit rollout. Cross-browser effects - the very real font-rendering and viewport-rounding differences between engines - are not in this data, and you should not generalise any of these findings to "browsers" as a category.
One runner family. The execution mode is uniformly procedural (test-script-driven), as opposed to AI-agentic. Lastest's agentic mode produces a different failure profile, with different error signatures, different selector usage, and different diff characteristics. We would expect a separate field-recording from agentic runs to look noticeably different and we encourage someone - possibly us, eventually - to publish one.
Three teams. Only three workspaces (team_id_hash values) appear in the data, and they are weighted heavily toward one of them. Generalising to "all visual testing" from this base is a mistake. Generalising to "a single team's suite over ten weeks" is appropriate.
No retries. As discussed, retries were off. This is a deliberate choice for clean failure measurement, but it means the dataset cannot speak to the question of "how much of the failure budget does retry hide?" because retry is uniformly absent. A complementary dataset with retries enabled would let you measure that, and we don't have one.
No baseline drift information. The dataset records the diff at capture time but not the baseline's version history. We have evidence elsewhere (see the baseline-management piece) that baseline rot drives a meaningful share of false positives, but you cannot measure that from this file alone.
No test source code. We do not publish the tests themselves (they are internal) or the apps they test. The dataset is telemetry, not corpus. If you want to reproduce a test's behaviour, you cannot; you can only reproduce its statistical profile.
X. Patterns worth taking away
If you read nothing else, these are the takeaways we'd back with money:
Bimodality is everywhere, and it kills averages. Failure rate by duration is bimodal. Diff percentage is bimodal. Per-test failure rate is bimodal. The single largest mistake we found in our own pre-dataset thinking was reasoning about averages instead of distributions. A suite with a 10.5% overall failure rate is not "10.5% flaky everywhere"; it is fine in the middle and broken at the edges. Treat it that way and the interventions get specific. Treat it as one number and you'll get nothing for your money.
"Other" is a measurement gap, not a category. If your largest failure bucket is the one called "other," you are not being told what's failing - you're being told what your categorizer doesn't know about. Mining error signatures for the most common strings inside "other" is the highest-ROI hour you can spend on test-suite analysis. Add categories for what you find. Do it again in three months.
Selector strategy advice should be empirically anchored. The role-locator advice that everyone (us included) has been confidently dispensing assumes the apps under test are accessibly named. In practice, lots of them aren't. The selector chosen by the test is downstream of the application's accessibility posture. Until you've measured selector failure rates against your own suite, your selector advice is theological.
The chronic-failure cluster is small and obvious. Twenty-seven tests in our suite fail more than half the time and are responsible for a wildly outsized share of the suite's noise. Triaging them is a one-week project that pays back forever. Most teams don't do it because they're waiting for a tool to magically identify them. The tool is a SQL query.
Triage is an attention budget. The pending-to-approved ratio in your visual-diff queue is the most under-discussed number in your testing infrastructure. It tells you the rate at which the engine produces diffs versus the rate at which humans process them. Faster engines without faster triage just deepen the queue. The fix lives on the human side - clearer auto-classification, sharper firing thresholds, better presentation of the diff so the reviewer can decide in under five seconds.
XI. Download the dataset
The full dataset - three CSV files, three matching JSONL files for streaming use, a precomputed summary.json, and a README documenting the schema and the anonymization - is available as a single ~1.1 MB zip:
⬇ Download flake-dataset.zip · 1.1 MB
Or grab the individual files directly:
flake_runs.csv· 1.1 MB · 8,308 rows · one per test runvisual_diffs.csv· 1.4 MB · 12,894 rows · one per visual diffselector_fragility.csv· 27 KB · 499 rows · one per (test, selector kind)summary.json· 1.6 KB · pre-computed top-level statisticsREADME.md· schema, anonymization, license notes- JSONL versions of all three tables for streaming-friendly use are in the zip.
Released under CC BY 4.0. Use it in writing, in benchmarking, in tooling, in classes - commercial included - provided you cite the source. Suggested citation:
Lastest Team. Flake Dataset: A Field Study of 8,308 End-to-End Test Runs (2026). https://lastest.cloud/blog/flake-dataset-field-study
Every chart above derives from one or two columns of one of these files, with no hidden steps. The zip contains a one-line provenance comment in the README mapping each chart to its file/columns. Re-deriving a chart is a pandas.read_csv and a groupby away. Disagree with our interpretation, run your own.
XII. The honest version
We had assumptions about our suite before we built this dataset, and most of them were wrong in interesting ways. We thought we knew where the flake came from, and the duration U-curve revealed a structural split we hadn't noticed. We thought role selectors were the foundation of robust testing, and the data exposed a 95% failure rate that turned out to be downstream of an accessibility-debt finding nobody had ever framed that way. We thought visual diffs would distribute roughly normally across the percentage-difference axis, and the bimodal valley revealed something specific about the stabilization pipeline that we now want to study in more detail. We thought 870 failures was a single phenomenon to be reduced; the data revealed two phenomena (crashes and timeouts) that share almost no fixes.
What we hope a reader takes from the dataset is permission. Permission to disaggregate. Permission to mistrust averages. Permission to grep "other" until it stops being the largest category. Permission to publish the receipts even when they are not flattering. Test-suite telemetry is one of the rare engineering surfaces where the data is plentiful, the publishing tradition is thin, and the discourse is correspondingly under-evidenced. Ten weeks of one suite's behaviour will not solve that. But it is a row in a table that we hope, eventually, will have many rows.
The file is here. The charts above are derivable from it. The interpretation we offered is one read; yours will probably differ in places, and we're interested in where. If you publish your own analysis or your own dataset, send it over - there is room for many of these.