Insights

Why Your Team Stopped Trusting the Test Suite (and What It's Costing You)

A test suite is only worth what your team believes about a red build. How flaky tests train engineers to ignore failures, and how to rebuild the signal.

Jinoo Jain

CPO & Co-founder at Pie

10 min read Posted Sep 11, 2026

A Test Suite Is Only Worth What Your Team Believes About It

A test suite’s entire value is the trust your team places in a red build. The moment engineers stop believing that red means broken, the suite stops doing its job, no matter how many tests it holds or how much coverage it reports. A suite is a signal, and a signal nobody acts on is noise.

A passing suite is supposed to buy you the confidence to merge without re-reading every line, to deploy on a Friday, to let a junior ship without a senior babysitting the release. All of that rests on one assumption, that green means safe and red means stop. Break the assumption and every one of those conveniences quietly disappears while the dashboard keeps reporting a healthy coverage percentage.

A suite nobody believes is a smoke alarm with the battery pulled out. Still mounted on the ceiling. Still in the compliance doc.

Coverage becomes a vanity metric the moment trust is gone, because it counts the volume of tests rather than whether their verdict changes anyone’s behavior. A 90%-covered suite that everyone reruns past is worth less than a 40%-covered suite whose red builds actually stop the line.

How a Flaky Suite Trains Your Team to Ignore It

A flaky suite erodes trust in three predictable stages. Engineers rerun failures, then ignore them, then delete them. Each step is a rational response to a suite that fails for reasons unrelated to the code, and each step moves the team closer to shipping with no real safety net. Nobody decides to abandon the tests. They just follow the incentives the suite created.

It starts small. A test fails. The engineer knows their change was unrelated, so they rerun it and it passes. Reasonable. But now the rerun is the default response to red, and the log goes unread. Next, failures pile up faster than anyone triages them, so red becomes background noise. Finally, someone doing a “cleanup” disables the worst offenders, and the suite that was supposed to catch regressions is quietly smaller than anyone realizes.

The trend is not in your favour either. Bitrise tracked more than ten million CI builds between January 2022 and June 2025 and found the share of teams hitting flaky tests climbed from one in ten to one in four, while workflow complexity grew by more than twenty percent (Bitrise Mobile Insights, 2025). The suite your team stopped believing is getting harder to trust, not easier.

The stages are worth naming, because most teams sit further down this ladder than they think, and the last rung is the one that shows up in a postmortem.

What a Suite Nobody Trusts Actually Costs You

Wasted CI minutes are the cheap half of the bill. The expensive half is lost signal. Once the team stops believing red, the suite can no longer separate a genuine regression from noise, so real bugs slip through while engineers burn time triaging false failures. You keep paying to run a suite that has stopped protecting you, which is the worst deal in engineering.

Researchers have put a real number on the expensive half. A 2024 study tracked five years of CI history at a commercial project of roughly thirty developers and a million lines of code (Leinen et al., ICST 2024).

Run that on a team of thirty. At 2.5% of productive time you are losing about three quarters of an engineer, permanently, to failures that were never real, and it never lands in a planning doc because it leaves in ten-minute pieces.

The other two rows explain the behavior. Investigating a red build by hand costs $5.67 of engineering time. Clicking rerun costs two cents. Reruns are roughly 280 times cheaper than looking, so your team is not cutting corners when they retry. They are doing that arithmetic and getting the right answer, on every build, until the one where the failure was real.

The waste compounds in three places at once.

• Focus: Engineers context-switch out of deep work to babysit a red build, and that time never comes back.
• Review: Reviewers can no longer treat a green check as a proxy for correctness, so they re-scrutinize everything by hand.
• Signal: When flaky failures and real ones look identical in the dashboard, the real one is just another red you have trained yourself to skip.

Then there is the cost you only see in a postmortem. A regression reaches production, and the retro reveals a test did catch it, three days earlier, in a build someone reran to green. The suite worked. The trust did not. A customer found the bug before an engineer did, which is the exact outcome the suite existed to prevent.

Why a Green Build Stopped Meaning “Safe to Ship”

Trust breaks in both directions, and the quieter break is the false green, a test that passes without verifying anything real. It asserts nothing, it was silently skipped, or it got disabled during a flaky-test cleanup and never came back. Flaky reds waste your time. False greens hand you unearned confidence and let a regression ship while the dashboard stays a calm, reassuring green.

False greens are more dangerous precisely because nobody looks at a passing build. A test that renders a component and never asserts on the output will pass forever, including the day the component breaks. A suite with a rising count of skipped tests reports the same green it always did while covering less every sprint. At that point you are measuring the absence of complaints and calling it safety. The full anatomy of a suite that passes while the app is broken is its own subject, and it drains the same trust from the opposite direction.

So “our tests pass” is a weak claim on its own. The strong version is “our tests pass, and when the product breaks, they fail.” Most teams have never checked the second half. Mutation testing exists for exactly this reason. It deliberately breaks your code to see whether any test notices, and it routinely finds green suites that catch far less than their coverage number implies.

Five Moves That Rebuild Trust in Your Test Suite

You rebuild trust by making red rare and meaningful again, then keeping it that way. The goal is easy to state and slow to reach. A red build should reliably mean something is broken, and a green build should reliably mean it is safe to ship. Getting there is a sequence rather than one heroic cleanup, and the order matters.

1. Quarantine flaky tests immediately. The instant a test is known-flaky, move it out of the blocking path so it stops polluting the signal for everyone else. One flaky test in the blocking suite discredits every honest one next to it. Google automated this years ago, with tooling that quarantines a test once its flakiness crosses a threshold and files a bug against it.
2. Make the remaining suite blocking again. Once the known flakes are quarantined, red has to mean stop. If people can still merge past failures, you are back to stage two by next week.
3. Measure trust, do not assume it. Track a flaky rate, the share of failures that pass on rerun with no code change. When that number is visible on a dashboard, trust becomes something you manage instead of something you argue about in retro.
4. Fix root causes, not symptoms. Most flakiness is a handful of patterns. The foundational study by Luo and colleagues (2014) traced 45% of the flaky tests they analyzed to async waits, 20% to concurrency, and 12% to test-order dependencies. A retry hides all three. Fixing the flakiness is slower and it is the only thing that holds.
5. Hunt the false greens. Audit the tests that have never failed. Break the code they cover on purpose and confirm they turn red. The ones that stay green are theater, and they are the reason a passing build stopped meaning safe.

What a Test Suite You Can Actually Trust Looks Like

Most flaky failures have nothing to do with your app breaking. A selector moves, a wait fires a beat early, a locator snaps on a redesign, and the build goes red while the product works exactly as intended.

That is the largest single source of false alarms in a normal suite, and it is the only one you can engineer away completely. Remove it and most of that 2.5% goes with it.

What Pie Takes Off the Board

Pie is an autonomous QA platform built to remove that category rather than help you cope with it.

1. It reads the screen, not the selector: Pie works from what is rendered, the way a person does. On native mobile there is no DOM in the loop at all. On web the driver still reaches for a locator where that is more reliable, but nobody on your team writes or maintains one, so there is no locator of yours left to snap.
2. Redesigns stop producing red: when your interface changes, the tests repair themselves instead of firing off a wave of failures that have nothing to do with your product.
3. One bug arrives as one bug: a single broken button can throw identical findings across a dozen test cases. Pie clusters them into one issue with a root cause instead of fifteen tickets that each need triaging.
4. The noise is filtered before it reaches you: flaky and test-data findings are filtered out automatically, so what lands in front of an engineer is what a real user would actually hit.

The point of all four is the same. Not fewer failures. Fewer failures that were never worth reading.

What Pie Does Not Fix

A slow staging environment will still time out on you. Thin or drifting test data will still break a flow that works fine in production. A genuinely nondeterministic backend will still be nondeterministic, and no amount of vision changes that.

What goes away is the largest and most demoralizing category, the one where nothing was broken except the locator. Claiming the rest would be the same unearned confidence this post is arguing against.

What It Looks Like When That Category Is Gone

The change is not that the dashboard turns green more often. It is that a red build gets read instead of retried, because the last several were real.

That is the whole return on the work. A suite becomes worth something the moment its verdict stops depending on which run you happened to catch.

Red Should Mean Red

A test suite makes one promise. Green is safe, red is stop. Every flaky failure and every false green chips at it, and once the promise is gone, the coverage number on your dashboard measures something nobody uses.

Every team I know that ships fearlessly on a Friday afternoon has the same thing going for it. They still believe a red build. Get the flakiness out, hunt the false greens, and make red mean red again.

We built Pie so the suite stops lying to your team. When tests repair themselves instead of breaking on every UI change, your engineers get their instinct back. Red means red, and everything else on that dashboard is decoration.

Jinoo Jain

CPO & Co-founder at Pie

Spent a decade in B2B SaaS sales before building Pie. Now obsessed with helping engineering teams ship without the fear of breaking things. LinkedIn →