Insights

What Do Flaky Tests Really Cost? More Than the Reruns You're Counting

The rerun button feels free. The real cost of flaky tests is the trust it burns and the coverage holes it hides, and AI-written code is raising the bill.

Jinoo Jain

CPO & Co-founder at Pie

8 min read Posted Oct 2, 2026

What Flaky Tests Cost on Paper

Ask a team what flaky tests cost and you get the visible number, wasted CI minutes plus the hours engineers burn investigating failures that turn out to be noise. A five-year industrial case study of a commercial project with around 30 developers, presented at ICST 2024 by researchers from TU Munich and CQSE, found that handling flaky tests consumed at least 2.5% of productive developer time. A single manual investigation cost about $5.67. Automatically rerunning the test cost 0.02 cents, roughly 28,000 times less per incident.

That gap is why “just hit rerun” wins in the moment, and the rational engineer reruns. The study’s own outcome, after all that measuring, was a decision to shift effort away from investigation and repair and toward automatic reruns. On per-incident math, that decision is right. It is also how a reflex becomes policy, and policy is where the interest starts compounding.

The cheap choice per incident is the expensive choice per quarter.

The usual suspects behind flaky tests are timing and async waits, shared state, test-order dependencies, and the brittle selectors that snap on a restyle. None of that is new. What is new is how much code is flowing through the pipeline.

The Cost That Never Hits the Invoice

The cost no dashboard tracks is the slow erosion of trust that turns a test suite into noise you route around. Anyone who has lived with flaky tests in a CI pipeline has watched the slide:

1. Tests fail randomly. Developers shrug and rerun them.
2. They fail often. Developers stop reading the failures at all.
3. The suite is always red. Nobody believes it. A real break sits in the noise, looking like the false ones.

The third step is where flakiness stops being an annoyance and starts shipping bugs. A February 2026 study of 1,960 open-source Java projects on GitHub Actions found that 3.2% of builds get rerun, and that 67.73% of those rerun builds were flaky, meaning the outcome changed with no code change.

Flip it around. About a third of rerun builds gave the same outcome the second time. A rerun treats the flake and the real failure the same way until someone reads the output, and “it’s just flaky” is sometimes a lie your pipeline tells you about a real bug.

The reflex is not a small-team habit. In 2020, Meta’s engineering team wrote that engineers trust a passing test, retry a failing one on the same code, and call a fail-then-pass flaky, and that the team had no good theory for why the habit prevails. A September 2025 study of 142,387 CI jobs across 81 industrial projects found that 11% of successful jobs were rerun as well. Red gets second-guessed, and so does green.

Quarantine Is the Same Bet in a Nicer Suit

Quarantine looks like the grown-up fix, and unlike a blind rerun it records the debt. Discourse’s Sam Saffron makes a strong case for it, and his 2019 write-up advises teams to be more aggressive about quarantine.

The bet underneath is the same, though. Say a checkout test flakes and you quarantine it to unblock the pipeline. Three weeks later a real checkout regression ships, because the one test that would have caught it was muted and nobody ever went back to put coverage on the flow.

Quarantine only pays off if the fix half happens, and the teams that make it work treat it as a system. Reddit’s iOS engineers first retried failing tests up to three times. Their 2024 write-up says CI looked stable, but they could not tell which tests were flaky or how often. They built an automated quarantine service that opens a ticket for each test’s owner, and reported 65.7% fewer flaky test runs when they compared the six months before and after.

Even so, they wrote that quarantining costs some coverage while a test is disabled. Two years on, they had moved from a static quarantine list to a dynamic service, one pain point being that developers had to rebase just to pick up each updated list.

Every quarantined test is a coverage hole with your name on it, and the hole stays invisible until something falls through it. Muting is faster than learning how to fix flaky tests at the root. It also leaves the debt where it was.

Why the Bill Is Compounding Now

Flaky-test cost is compounding in 2026 because AI now writes a large share of shipped code. Google said in its Q3 2024 earnings remarks that more than a quarter of its new code was AI-generated and then reviewed by engineers. By April 2026, Sundar Pichai put the figure at 75%, up from 50% the previous fall. More code means more tests and more UI churn, and with them more surface area for flakes. The detect-and-quarantine playbook was designed for a world where test volume grew at human speed.

The volume comes with a quality problem attached. In a December 2025 CodeRabbit analysis of 470 open-source pull requests, sorted into AI-co-authored and human-only by co-authorship signals, the AI-co-authored changes averaged 10.83 issues each against 6.45. The pile of failures to sort grows right along with the pile of tests, and a flaky suite is the wrong instrument for testing AI-generated code at this speed.

Why Triage Headcount Can’t Catch Up

Hire the QA engineer if you need one. Just do not hand them a triage queue that grows faster than anyone can clear it. You can model the maintenance cost on your own suite, and the direction will not surprise you.

Triage scales with the people you can hire. The flake surface scales with how fast code gets written, and those two lines no longer meet.

Manage the Flake, or Remove It

There are two responses to flaky tests, managing the symptom or removing the cause. The three usual moves, retry, quarantine, and locator auto-healing, all manage the symptom. Each one buys a green build today and sends the cost to later.

Engineers have argued over these moves on Hacker News for a decade without settling it. Some run a zero-tolerance policy and remove a flaky test on sight. Others retry fast tests and want each test’s flake rate measured. In 2015, one commenter describing Chromium’s test infrastructure called a test with a known flake rate hugely valuable. Both camps are describing a way to live with the flake. Neither removes the cause.

The first three keep you inside the selector model and get better at managing its failures. The fourth changes the model. You can spend forever muting one class of flake at a time, or you can remove the class.

How Pie Removes the Flake Class

Pie removes a whole class of flakiness by taking the selector out of your hands. Nobody on your team writes or maintains one. The agents find elements the way a person does, by context, position, and function rather than an ID. A class rename, a restyle, or a refactor does not turn the suite red. Pie’s self-healing tests replace the brittle model instead of patching a healer on top of it.

What Happens to a Real Failure

We don’t want self-healing to get so good at healing that it hides an error. A product bug surfaces as a finding, deduplicated across runs so near-identical failures collapse into one issue. Red starts meaning broken again.

Slower Per Run, Faster Per Release

A run on our platform takes longer than a Playwright run, because an agent reading the screen is slower than a script replaying a locator. We mitigate that with parallel runs, and what the trade helps prevent is the week you would have spent fixing flaky tests before a release.

What to Do Before the Next Rerun

You start cutting the cost of flaky tests by measuring it, then attacking the cause instead of the symptom. The reflex to rerun is the enemy here, because a rerun can tell you a build is flaky but never why. The order that works:

1. Measure the real rate. Track how often each test fails and passes on unchanged code. You cannot manage a cost you have never sized.
2. Separate flaky from broken. Before you quarantine anything, confirm the intermittent failure isn’t a real race condition or timeout hiding in the noise. Reruns skip this step.
3. Fix the top offenders at the root. Start with the tests that fail most often. Fix their timing, isolation, or state, and delete the ones guarding flows that no longer exist.
4. Stop quarantining into blind spots. If a test is muted, the flow it covered needs coverage from somewhere else, or you have traded a red build for an unwatched regression.
5. Remove the class, don’t just manage it. The deepest fix is ending the selector maintenance that produces a whole class of flakes. Then there is nothing to rerun, quarantine, or heal.

Pay for Flakiness Once, Not Twice

The rerun button was never free. You pay once in the CI minutes and the investigation hours, and again in the trust it burns and the real bugs it hides. AI-written code is raising both charges at once. No amount of triage headcount catches a flake surface that grows every time your team ships.

Managing flakiness faster gets you a greener build. Removing the cause gets you a suite you can believe, which beats a green build you can’t.

We built Pie to take the selector out of the flake equation, so red means broken again. When your build went red this morning, did anyone read the failure before reaching for rerun?

Jinoo Jain

CPO & Co-founder at Pie

Spent a decade in B2B SaaS sales before building Pie. Now obsessed with helping engineering teams ship without the fear of breaking things. LinkedIn →