Insights

AI That Writes Tests vs AI That Tests: What Are You Actually Buying?

Vendors use one label for two products that do different jobs. The question that separates them is who fixes the test when a button moves.

Dhaval Shreyas

CEO & Co-founder at Pie

9 min read Posted Sep 11, 2026

Two Products, One Label

AI testing splits cleanly into two products. The first uses AI to generate test code faster, so you describe a test in plain English or record a click-through, and a model writes the script. The second uses AI to run the testing itself, so the model navigates your app, decides what to check, executes, and repairs its own tests when the interface moves. The first makes a person faster at writing tests. The second takes the writing and the upkeep off that person entirely and leaves them the judgment.

Both are legitimately “AI,” which is exactly why the label is useless for buying decisions. A 2022 GitHub study of Copilot users found developers completed a coding task 55% faster with AI assistance. Real gain, real product. But faster authoring answers a question most QA teams aren’t actually stuck on. They can already write tests. What they can’t do is keep hundreds of them green while the product ships every day.

The tell is what happens next. When a tool leads with “generate tests in seconds,” ask what it does on day 30, when the tests exist and the app has changed twice. The answer sorts the category.

Three Levels of AI Testing, From Autocomplete to Autonomous

There are three levels of AI in testing, and they are not degrees of the same thing. They are different amounts of the loop handed to the model. At Level 1 the model writes code you own. At Level 2 it patches the code you own. At Level 3 it owns the loop end to end. Most tools marketed as “AI testing” live at Level 1 or 2 and let the word imply Level 3.

The jump that matters is Level 2 to Level 3. Self-healing is a real feature and a genuinely useful one, but repairing a broken locator only rescues tests that already exist. Someone still has to decide what to test, write the first version, and expand coverage as the product grows.

Level 3 is the only rung where that authoring-and-upkeep job leaves your team’s plate, which is why autonomous QA gets treated as a separate category rather than a better feature. Everything below it is a faster way to keep doing the work.

Why Writing Tests Isn’t Testing

I spent 13 years building mobile infrastructure at Square, Facebook, and Instacart, and the test suites on those teams all got expensive the same way. Not in the week we wrote them. Months later, when the app moved and the suite didn’t.

The Bill Arrives in Maintenance, Not Authoring

Writing a test is the cheap part. Keeping it alive is the bill.

The reason is structural, and you can work it out without a survey. A test is written once, against one version of a screen. That screen then changes every time the product does, and each change can break the test without anything being wrong with the product.

So authoring is paid once per test, and maintenance is paid once per test per release, for as long as the test exists. Multiply that across a few hundred tests and a team shipping weekly, and the second number swamps the first. A tool that only speeds up authoring is optimizing the line item that was never the problem.

Tests also fail for reasons that have nothing to do with your product breaking. A selector changes, a class gets renamed, an async call lands a hair late, and a green test goes red with no bug behind it.

Martin Fowler named the problem fifteen years ago. “Non-deterministic tests have two problems, firstly they are useless, secondly they are a virulent infection that can completely ruin your entire test suite.” A model that writes the first draft of a flaky test faster has produced the infection faster. It has not cured it.

We built Pie against that specific failure. Instead of generating a script bound to selectors and handing it back, Pie’s agents drive the app the way a person does and re-identify an element after a redesign by what it does and where it sits, so the author never writes or maintains a selector in the first place.

Why Level 1 Feels Like Progress for One Sprint

Level 1 tools feel like progress because the first week is genuinely great. You prompt, a suite appears, coverage numbers climb, and it looks like you bought your way out of the QA backlog. Then the product ships, the tests start failing on changes that aren’t bugs, and you are back to triaging red builds by hand. The AI helped you sprint to the starting line of the actual problem.

The survey data has already caught up with that experience. The Stack Overflow 2025 Developer Survey found 84% of developers using or planning to use AI tools, up from 76% a year earlier. Over the same year, trust moved the opposite direction, with 46% now actively distrusting the accuracy of AI output against 33% who trust it. Adoption is running well ahead of confidence, which is a reasonable place for a market to be and a terrible place to buy on the strength of a label.

Faster autocomplete is a feature, and a good one. The mistake is buying a feature as if it were autonomy. If a person on your team is still on the hook for every broken test, you didn’t automate testing. You automated typing.

What Changes When the AI Owns the Loop

Everything changes when the model owns the loop instead of the keyboard. An agent explores your app on its own, maps the flows that exist, generates the coverage, runs it, and repairs the tests when the interface shifts, all without a person writing or fixing a script. Your team moves from authoring and babysitting tests to reviewing results and setting intent. One tool makes the same job faster. The other changes what the job is.

The payoff shows up as release speed, not test counts. When maintenance stops gating every deploy, teams stop being a sprint behind their own product. Fi, the pet safety company behind the GPS smart collar, had 12 or more engineers locked into manual release testing and a validation cycle that ran two to three days. After Fi handed the exploration-and-maintenance loop to Pie’s agents, validation runs in a few hours and one dedicated QA engineer owns it.

What Level 3 buys is a release cadence that stops waiting on a person clearing a queue of false failures.

None of this replaces QA judgment, and it should not. Deciding which risks matter, designing the coverage strategy, and chasing the failures that actually signal a bug all stay human. What leaves is the execution and upkeep grind, which is the entire job we built Pie, an autonomous QA platform, to take. The teams pulling ahead are the ones handing over the loop, not the ones autocompleting inside it.

Five Questions That Separate Real AI Testing From a Label

You don’t need a spec sheet to sort a tool into the right pile. As more of your code gets written by a model, the case for handing over the loop only gets stronger, so sorting the tools wrong gets more expensive every quarter. You need five questions, and you need to watch the tool run on your app instead of the vendor’s demo app.

1. When a button moves, who fixes the test, a person or the model? If the answer is a person, you are at Level 1. The AI wrote the test and you own its failures.
2. Does it discover flows you never scripted, or only replay what you gave it? Replay is authoring with extra steps. Discovery is the model deciding what to test.
3. What happens on a screen it has never seen before? A real agent navigates it. A script generator needs you to record it first.
4. What are you maintaining in month three? If the answer is “the same selectors, now with AI suggestions,” the maintenance problem survived the purchase.
5. Will it run on your actual app, live, right now? Autonomy is easy to claim on a polished demo and hard to fake on a messy real product. Make the tool prove it on yours.

Run those five past any tool selling “AI testing” and the label stops mattering. Either the model owns the loop, or you still do.

Ask Who Owns the Loop

The category will keep blurring “AI writes your tests” and “AI tests your app,” because one of those is easy to ship and the other is hard to build. Vendors are not going to sort it out for you.

So sort it yourself, on one question. When the app changes and a test goes red, whose calendar does it land on? If the answer is still someone on your team, you bought authoring help and called it autonomy.

Pie exists because we wanted the second answer to be true rather than marketed. Ask any vendor the question and watch what they do with it. The ones who own the loop will just show you.

Dhaval Shreyas

CEO & Co-founder at Pie

13 years building mobile infrastructure at Square, Facebook, and Instacart. Now building the QA platform he wished existed the whole time. LinkedIn →