Insights

Why Your Tests Pass While Your App Is Broken

A green test suite is not proof your app works. Why tests run, pass, and still miss real bugs, why AI-written code makes it worse, and how to catch false passes.

Dhaval Shreyas

CEO & Co-founder at Pie

10 min read Posted Sep 11, 2026

What a False Pass Actually Is

A false pass is a test that reports success while the behavior it was supposed to protect is actually broken. In formal QA terms it is a false negative, meaning the test failed to detect a defect that is really there. The result is honest about what it measured. It measured the wrong thing, then reported green, and green is the color that tells your team to stop looking.

Line this up against its opposite and the danger gets obvious. A flaky test is a false positive. It fails when the code is fine, so it costs you a rerun and a slow bleed in your team’s trust in a red build. A false pass is a false negative. It succeeds when the code is broken, so it costs you nothing up front and everything later. Nobody reruns a green test. Nobody opens the log on a passing build. The bug rides the green straight to production.

False passes are easy to ignore precisely because flakiness is loud and they are not. A flaky test interrupts you. A false pass is silent by construction, so teams over-invest in the noise and under-invest in the silence. A suite you cannot trust is expensive and at least visible. A suite you trust too much is invisible until it isn’t.

Five Reasons a Test Runs, Passes, and Still Misses the Bug

A test produces a false pass when it verifies something narrower than what the user actually does. It executes real code and returns green. The gap sits between what it checked and what shipping the feature requires.

Five patterns cause nearly all of them, and every one is a normal, well-intentioned practice taken slightly too far.

1. The Dependency Was Mocked

This is the biggest single source. A mock swaps a real dependency for a canned response so the test runs fast and in isolation, which proves your code handles that canned response correctly. Genuinely useful, right up until the real service changes shape. Then the mock keeps returning the old contract, the test keeps passing, and production keeps failing.

The gap: you verified that your code talks to the fake. The real API changed its contract, or went down.

2. Only the Happy Path Was Asserted

The test feeds in the input the feature was designed for and confirms nothing exploded. Empty states, malformed input, expired sessions and the second tap on a slow button never get exercised, because nobody wrote the case. Green here means the one path somebody imagined still works.

The gap: you verified the function returned a value. The value is wrong, or the edge case throws.

3. The Units Were Tested in Isolation

Unit tests check components one at a time, which is exactly what they are for. But a suite made only of green unit tests tells you every brick is solid and nothing about whether the wall stands. The frontend calls an API that calls a database, all three pass their own tests, and the seams between them were never exercised.

The gap: you verified each piece works alone. The pieces were never wired together.

4. The Assertion Checked Existence, Not Outcome

The assertion is the part that decides what “working” means, and a weak one quietly redefines working as “did not crash.” A test that confirms a return value exists but not that it is correct will pass on garbage. An end-to-end test that asserts the checkout button is on the page, rather than that the purchase completed, passes on a checkout that charges nobody.

The gap: you verified an element is present. Clicking it does nothing, or the wrong thing.

5. The Environment Drifted

The test environment has clean data, a small dataset, permissive config and no real concurrency. Production has none of those things. A query that returns in milliseconds against a thousand seeded rows behaves differently against ten million real ones, and the test never had a chance to notice.

The gap: you verified it works in the test environment. Production has different data, config or scale.

AI Writes the Code and the Test, So Both Agree

What makes 2026 different from 2019 is who writes the test. The same model that writes your code now writes its tests, and a test is only as good as its assumptions. When one model produces both sides, they share the assumptions. The test asserts the behavior the model intended. The code implements the behavior the model intended. They agree perfectly, and both can be wrong about what the user needs. You have automated the production of code and tests that validate each other instead of the requirement.

The data on AI code quality points one direction. A 2024 Uplevel study of nearly 800 developers found the group using GitHub Copilot introduced 41% more bugs, with no matching gain in pull-request throughput. GitClear’s analysis of 211 million changed lines found copy-pasted code overtaking moved code, its proxy for refactoring, for the first time on record. More code, more duplication, more bugs, and a test suite generated by the same tool that generated the bugs.

“The AI wrote tests too” isn’t the reassurance it sounds like. Volume of tests was never the constraint. Testing AI-generated code is hard because the failure mode is subtle correctness rather than crashes, and that is exactly what a shallow assertion misses.

Developers can already feel it. In the same survey, 45% named debugging AI-generated code as a top frustration, and 46% now distrust the accuracy of AI output against 33% who trust it. Generate ten thousand passing tests from a model that is confidently almost-right and you haven’t verified your product. You have industrialized the false pass.

How to Catch a False Pass Before Your Users Do

You catch false passes by closing the gap between what the test checks and what the user does. Every fix below moves verification closer to real behavior and further from convenient proxies. None of them require more tests. They require tests that cannot go green unless the feature actually works.

1. Assert on outcomes, not existence: Stop checking that an element is present or that a function returned something. Check that the purchase completed, that the email arrived, that the balance updated. A test’s assertions are its definition of working, so make that definition match what a user would call success.
2. Cut mocking at integration boundaries: Mock third-party systems you cannot control, and stop mocking your own API, your own database, your own services. Every mock at an internal boundary is a place a real contract can drift while the test stays green. Exercise the real wiring wherever you reasonably can.
3. Test the running product, not the code: End-to-end testing on the real, deployed app is the strongest structural defense, because it exercises the system a user touches instead of a lattice of isolated units. It only works if the assertions check user-visible outcomes rather than the presence of a selector.
4. Use mutation testing to find dead assertions: Mutation testing injects small faults into your code and checks whether any test fails. A surviving mutant is a test that runs the line and never verifies it, which is a false pass waiting to happen. It measures the one thing a coverage number cannot, which is whether your assertions actually bite.
5. Track escaped defects, not green rate: Count the bugs that reached production per release. If your green rate climbs while escaped defects hold flat, the suite isn’t getting safer. It’s getting better at passing.

How Pie Kills False Passes

Every fix above moves verification closer to what the user actually does. That is the whole design premise here. Do not instrument the code and assert on proxies. Run the product and check what a person would check.

Here is what that looks like against the five reasons.

It Runs the Product, Not a Model of It

Closes reason 1, mocked dependencies, and reason 3, units tested in isolation.

• No internal mocks: there is no stub layer inside a run, so there is nothing to drift out of sync when a real service changes its contract. The integration either works when it is exercised or it does not.
• The seams get exercised by default: a run moves through the built app against the backend it is pointed at, so the joins between frontend, API and data layer are covered as a matter of course rather than as an integration test somebody had to remember to write.
• One definition across surfaces: the same test logic covers native iOS, Android and web from one configuration, so a flow cannot pass on one surface because it was only ever really tested on another.

It Judges the Screen, Not the Selector

Closes reason 4, asserting existence instead of outcome.

This is the one that matters most, because “the element was there” is the single most common way a test reports success on a broken feature.

• Vision-based execution: Pie reads the rendered screen and acts on what is displayed, the way a person does. On native mobile there is no DOM in the loop at all. On web the driver still reaches for a locator where that is more reliable, but nobody on your team writes or maintains one.
• The verdict is the outcome: because the judgement is read off what the screen shows, “a button exists” is not a conclusion a run can reach. The check is whether the result a user would recognise as success actually appeared.
• Redesigns do not manufacture red: when the interface moves, the tests survive it, and a suite that stops screaming on every redesign is a suite whose reds get read.

It Tests the Flows Nobody Thought to Script

Closes reason 2, the happy path being the only path anyone asserted.

A test can only produce a false pass on a flow somebody wrote a test for. The flows with no test at all are the bigger hole, and coverage tooling cannot see them.

• Discovery runs in four phases: Scout, Multiplication, Parallel Exploration, Synthesis. A scout agent opens the app and maps the navigation like a first-time user, then spawns specialist agents at every branch, guarded section and complex feature.
• The agents work in parallel: hundreds of them explore different paths at once, probing boundaries and following flows to completion rather than stopping where the happy path ends.
• Synthesis does the editing: it merges duplicates, fills coverage gaps and ranks what it found by criticality, so the output is a ranked suite rather than a pile.
• The baseline lands before anyone writes a test: 60 to 80 percent of core flows covered on day one, built from the app as it actually is rather than from what a team remembered to specify.

What It Does Not Fix

Reason 5, environment drift, is still yours.

Different data, different config and different scale in production will still produce failures nothing caught in a test environment, and no amount of vision changes that. A flow nobody has ever exercised is also still untested, which is exactly why the agents map the journeys before they test them.

Claiming otherwise would be the same move this whole post argues against. Confidence that outruns verification.

Four of the five stop being your problem. The fifth is honest work, and at least you know which one it is.

Green Should Mean It Works

We built Pie because a green build should be a promise, not a hope. For most teams it is a hope. The suite passed, so the app probably works, and “probably” is carrying far more weight than anyone admits on the way to a deploy.

Flaky tests get all the attention because they are loud. False passes deserve more, because a test that fails when nothing is wrong costs you a rerun, and a test that passes when something is wrong costs you the incident, the rollback, and the trust. The answer is not more tests but tests that verify the product a user touches, so nothing shorter than a working feature can turn the build green.

Stop shipping on “the tests passed.” Start shipping on “the product works, and we checked.”

Dhaval Shreyas

CEO & Co-founder at Pie

13 years building mobile infrastructure at Square, Facebook, and Instacart. Now building the QA platform he wished existed the whole time. LinkedIn →