Insights

Why Your Regression Suite Slows You Down (and How to Stop It)

Every bug fix ends with 'add a regression test.' Nobody ever deletes one, and that append-only habit is what's slowing your pipeline down.

Dhaval Shreyas

CEO & Co-founder at Pie

10 min read Posted Aug 4, 2026

Why Regression Suites Grow Without Bound

Suites bloat because adding a test is a local decision with an obvious owner, and removing one is a global decision with no owner at all. When a bug ships, the engineer who fixes it writes a test to guard against its return, and that is correct incentive design. But nobody is assigned to ask, a quarter later, whether that test still guards a live risk or whether the flow it covers even exists anymore.

The Reflex Is Reasonable in Isolation, Destructive in Aggregate

So tests only accumulate, and the suite becomes a museum of every bug your team has ever fixed. Look at what is actually sitting in yours:

• One-off race condition: Surfaced once during a bad deploy two years ago, now guarded by a permanent, forever-running test.
• Twice-redesigned form: Rebuilt twice, still carrying three generations of assertions stacked on top of each other.
• Retired feature: Pulled from the product last quarter, its happy-path cases still running on every commit.

Each of those was the right call on the day it was made. Together they are the reason your pipeline takes 40 minutes.

Flakiness Is a Symptom, Not the Disease

Google’s engineers wrote openly in 2016 about flaky tests becoming a tax on CI at scale, and the bill hasn’t shrunk since. A 2024 industrial case study tracking one company’s pipeline over five years found flaky-test handling alone consumed at least 2.5% of productive developer time, with a single manual investigation costing about $5.67 against 0.02 cents to just auto-rerun the test.

Chasing flakes treats the symptom. The root cause is structural. Your process has an accelerator and no brakes.

What an Append-Only Suite Actually Costs

An append-only regression suite costs you the one thing testing is supposed to protect, which is the speed to ship with confidence. The cost shows up as CI run time that climbs every quarter, longer feedback loops that push developers to batch changes and context-switch, and a growing temptation to skip the suite entirely under deadline pressure. When the safety net takes 40 minutes to run, engineers stop trusting it and start routing around it, which is the exact failure mode the suite was built to prevent.

The scale of this is not hypothetical. Kim Herzig and colleagues at Microsoft, in their ICSE 2015 study on testing less without sacrificing quality, replayed past development periods across three major Microsoft products. Their model skipped a test whenever running it was expected to cost more than leaving it out, which cut test executions by half and saved millions of dollars a year while product quality held.

The lesson generalizes. Past a certain size, “run everything” stops being a safety guarantee and starts being a bottleneck you pay for on every commit.

Three Ways to Run Regression

There are three fundamentally different ways to decide what regression tests to run, and most teams are stuck on the most expensive one by default. Retest-all runs the entire suite on every change, which is simple and exhaustive right up until the suite is too large to finish quickly.

Test-impact selection runs only the tests affected by a given change. Risk-based regression prioritizes by where failure hurts most, running high-stakes flows constantly and low-stakes ones rarely. The table below maps when each earns its place.

Test-impact selection is not exotic anymore. Microsoft ships Test Impact Analysis in Azure Pipelines, which automatically selects only the tests exercising the code a commit changed and falls back to a full run when it cannot map the change with confidence.

The catch is that selection is only as good as the mapping it depends on, and it can miss the indirect effects that end-to-end flows are specifically meant to catch. Risk-based regression sidesteps that by asking a blunter question, which is not “what did this change touch” but “what would it cost us if this broke,” and running accordingly.

Why Nobody Wants to Touch the Suite

Teams hoard tests instead of pruning them out of fear, and the fear is rational. When your cases are anchored to CSS selectors and XPath, deleting one and rebuilding coverage later means paying the authoring cost twice, so keeping it is always the safe move.

That produces a loop which turns a suite rotten:

• Selectors break: A cosmetic UI edit that broke nothing real turns the suite red.
• Engineers patch, not prune: Waits and workarounds pile up, because a repair is cheaper than a rewrite.
• Brittle and bloated at once: Touching the suite costs even more than it did, so nobody does.

Frameworks have chipped at pieces of this. Playwright’s auto-waiting removed a common class of timing flakiness, and low-code platforms added locator auto-healing. Those reduce the maintenance tax inside the selector model. They do not remove the reason the suite is afraid to shrink.

How Pie Automates Regression End to End

Every fix above is a discipline you have to install and then keep installing, sprint after sprint, against a deadline that never cooperates. Pie’s autonomous QA platform runs that entire loop instead, from the first line of coverage to a merged fix in your repo.

Here is the full path a regression cycle takes:

1. Discovery: Pie’s agents explore your app and generate cases from what they actually find, so nobody hand-writes the first hundred. See autonomous test discovery for how the exploration works.
2. Risk weighting: High-stakes flows like authentication and checkout get prioritized automatically, so coverage tracks where failure would hurt instead of piling up as an append-only archive.
3. Self-healing execution: Vision-based execution identifies elements by what a user sees rather than by CSS selector, so renaming a class does not turn the suite red.
4. Deduplicated findings: Repeat failures collapse into one app-level issue instead of forty near-identical reports, so triage is a short list rather than an afternoon.
5. Fixes as pull requests: Pie Loop opens PRs against your repo with the fix already drafted. Your engineers review them, push back, and merge, exactly as they would a teammate’s.
6. In your IDE: Pie connects through MCP inside Claude Code, Cursor, VS Code, or Windsurf, so none of this asks your team to live in another dashboard.

The pruning ritual stops being something you have to schedule. Coverage tracks risk on its own, the maintenance cost that made deletion feel dangerous is gone, and your engineers spend their judgment on whether a fix is correct rather than on repairing selectors a redesign broke.

How to Get the Suite Back Under Control

You get a bloated regression suite back under control by treating it like code that needs refactoring, not like an archive that only ever grows. The goal is not fewer tests for their own sake. It is a suite where every test that runs still earns its execution time. Here is the sequence that works:

1. Measure first: You cannot prune what you cannot see. Pull the last year of results and rank every test by run time and by how often it has actually caught a real failure.
2. Delete dead weight: A case that has not failed in a year and does not cover a revenue-critical or high-churn flow is costing you run time for almost no protection. Remove it deliberately.
3. Tag by flow: Group what remains by feature and risk level, so high-risk flows can run on every change and lower-risk ones on a schedule.
4. Adopt selection: Wire up impact analysis or risk-based selection so a one-line copy change does not trigger a 40-minute suite.
5. Fix the flakes: Flakiness forces reflexive retries that multiply run time. Fix or quarantine them, and see our guide to fixing flaky tests for the triage framework.
6. Kill the maintenance tax: The deepest fix is a suite you are not afraid to cut, which means ending the selector-repair cost that makes every test feel too expensive to lose.

Stop Growing the Suite. Start Pruning It.

Adding a regression test for every bug was never the mistake. Never removing one is. A suite that only grows is a suite that only gets slower, until the safety net you built to protect velocity becomes the slowest and most-avoided step in your pipeline.

Teams that stay fast prune as deliberately as they add, and they run regression by risk rather than by reflex. That habit is worth building whether or not you ever hand the loop to a platform.

Hand it to Pie and the habit stops depending on anyone remembering to keep it.

Dhaval Shreyas

CEO & Co-founder at Pie

13 years building mobile infrastructure at Square, Facebook, and Instacart. Now building the QA platform he wished existed the whole time. LinkedIn →