Software Factory Testing: How to Prove Agent-Built Software Works Before It Merges
Most software factory designs check the code, and customers meet the running product. See the stations on the line and the evidence each autonomy level needs.
Agents now write the code, the tests and the pull request, and the build comes back green. So what stopped any of them from breaking the sign-up page? In most published software factory designs, the running product gets one line at most.
If your team sits somewhere between reviewing every diff and writing a spec and checking back later, here is the evidence to collect before you hand the agents the next step.
What you’ll learn
- Which stations on a software factory line check the code, and which one sees what users see
- Why checks the agent can edit can’t vouch for its own work
- The evidence to collect before each step up Dan Shapiro’s autonomy levels
- How to tell, month over month, whether the factory is working
What an AI Software Factory Is
An AI software factory is a production line where coding agents turn a written spec into working software and automated stations decide what gets through, so people stop reading every diff. Software factory testing is the set of those stations that check the output before it merges, at least one of which uses the running product the way a customer will.
StrongDM’s AI team described one in February 2026, with specs and scenarios driving agents that write code and converge without human review. The line replaced the reader because the reader stopped scaling.
Software Factories Are Older Than the Agents
Cusumano’s history of Japan’s software factories dates the first facility run as a factory to Hitachi in 1969. By 2020 the US Department of Defense defined a software factory as CI/CD with mandated testing and security gates, so testing sat in the definition before any agent arrived.
The Los Angeles program office that stood up a military software factory for space systems is named Kobayashi Maru, after the unwinnable Starfleet Academy training scenario. Housing a software factory under the name of a test nobody can pass is either a warning or a dare.
One Station Sees What Users See
Draw your pipeline as stations and ask one question of each. Does it check the code, the behavior, or the product a customer will use?
The published designs spend their weight on the code side. Factory’s definition of a software factory lists “tested” as one verb in a chain of six. Katalon stacks lint, unit, integration and end-to-end checks so cheap failures stop the line early, and runs the end-to-end layer last. Addy Osmani’s examples of checks cheap enough to let a loop run dark are pass-or-fail oracles, type gates, property tests and a review agent with a rubric, and none of them opens the running app.
| Station | Who does the work | Checks | What it catches | What it misses |
|---|---|---|---|---|
| Spec | A person, often with an agent | Nothing yet, it sets the target | A wrong goal, early | Whatever nobody thought to write |
| Build | CI | The code | Compile, lint and type errors | Whether the feature works |
| Agent’s own tests | The coding agent | The code | Regressions against its own assumptions | The assumptions themselves |
| Held-out scenarios | A separate harness or judge | Behavior | Behavior the agent never saw tested | Flows nobody wrote a scenario for |
| Code review | A person or a review agent | The code | Logic slips and risky diffs | Bugs that only show when the app runs |
| Running-app check | A test agent on a preview or build | What users see | Broken flows a user would hit | Anything outside the flows it reaches |
| Merge gate | A person, behind rules the agent can’t edit | The decision | Weakened CI, unapproved merges | Whatever the stations before it missed |
| Release | Deploy tooling | Rollout | A failed deploy or health check | Bugs that pass a health check |
| Escape loop | Bug reports fed back to the line | The product, after release | What got through | Nothing, until someone turns it into a new scenario |
Read the Checks column top to bottom. Before the merge, the running-app check is the only station that uses the product the way your customer will.
Why the Builder Can’t Be the Inspector
Whoever writes the check decides what passing means, and the problem starts when the writer is the agent that wrote the code. A green build from those checks only says the agent agrees with itself.
A test stored in the codebase can be rewritten to match the code, and the agent is the one holding the pen. Simon Willison framed the resulting question as how you prove software works when agents write both the implementation and the tests.
Developers already feel it. In Sonar’s January 2026 survey of over 1,100 developers, 96% said they do not fully trust that AI-generated code is functionally correct, and only 48% said they always check it before committing.
We handed three models a function with a boundary bug in Can You Trust AI-Generated Tests?, and every suite they wrote passed on the buggy code. The models took their expected values from the code they were checking. The fix is a check the builder doesn’t control, and we go through scenarios in Scenarios vs Tests.
Where Held-Out Scenarios Stop
A held-out scenario tells you how the build behaves, and it runs in a world someone built for it, often with stand-ins for the services the software calls. Your customer opens the app on a phone or in a browser, and that gap is how tests pass while the app is broken.
Factory said the rest plainly when it launched Automated QA in April 2026. “The tests passed and the linters were green, yet the login form still broke in production because nobody actually clicked through it before shipping.” Factory built its fix inside its own coding agent, so the question that follows is who owns the check. A product check works best as its own station, run by something other than the agent that wrote the change.
Brain, writing on dark software factories, adds a second limit. A check can only test what someone thought to write down. A scenario list can’t cover the flow nobody imagined, and finding that flow is the job of autonomous QA. Pie explores your app and generates tests. Add one station that opens the product, and hand it to something other than the builder.
See What Your Users See
Point Pie at a preview of your next PR. See what a user would hit.
The Evidence Each Autonomy Level Needs
Dan Shapiro’s five levels describe what the human does as agents take over. They don’t say what has to be true before you climb. The evidence column below is ours.
| Shapiro level | What the human does | Evidence you need before you move up |
|---|---|---|
| 0 to 1 | Writes the code and hands the AI small tasks | CI and unit tests, and the human reads every line |
| 2 | Hands off “all your boring stuff” | A gate that blocks any PR that weakens CI, such as lowered coverage thresholds or removed tests |
| 3 | Reviews code all day as “the human in the loop” | A running-app check on each PR that has a preview or a build, so the reviewer reads evidence about the product and not only the diff |
| 4 | Writes a spec, leaves for 12 hours, then checks whether the tests pass | Checks the agent didn’t write and can’t edit, such as held-out scenarios, product tests kept outside the repo and a merge gate the agent can’t change |
| 5 | Nothing on the line, “a black box that turns specs into software” | All of the above running unattended and stopping the line on a failure, with every escaped bug turned into a new scenario |
On our own team, people review the high-level design and the way a change solves the problem, and a bot reviews the code. If the intent and the design hold up, the change goes ahead. The reading moved to a bot, and the judgment stayed with people.
The Level 2 gate is the cheapest to build and the easiest to skip. GitHub’s guidance on reviewing agent pull requests warns that agents may “remove the tests, skip the lint step, add || true to test commands”, and we lay out the full gate in How to Gate Agent-Written Pull Requests.
Mobile adds its own constraints at Level 3. A PR needs its own build that runs on a simulator or emulator before a running-app check can start, and How to Verify Agent-Built Mobile Apps takes that on.
What a Dark Software Factory Takes at Level 5
Shapiro names the top rung the Dark Factory, a line where no person is needed. Almost nobody is there. He knows a handful of people working this way, in teams of fewer than five.
Osmani’s rule for climbing applies here. Autonomy can’t grow past what you can check cheaply and reliably, and at Level 5 the checks are all that is left on the line.
Machines that run unattended never relied on blueprint reviews. Toyota calls the idea jidoka, a machine that stops itself when it detects something abnormal. On an agent line, that means the line stops when the product breaks, not only when the code looks wrong. Climb one rung at a time, and move up once the evidence on the rung below exists.
Where Pie Fits on the Line
A product station needs two things from your line, a place where it gets called in and something to open. This October a customer’s engineer walked us through their setup, with support tickets flowing into Claude through MCP for triage and the pipeline from ticket to fix being built that month. I told them the line needs a place where they can tag us in to test what was built.
The second need is a preview URL or a build upload for each pull request, so the first job on any repo is giving it one. Our guide to preview environments for PR testing covers that setup.
Pie, an agentic testing platform, is that station, and it can test your pull requests. Here is its bot’s verdict on one of our own, Pie testing Pie, with the four changes put in plain words. The PR changes how starter prompts behave in a chat composer, and the comment grades each change.
Three changes came back verified on the running preview. The fourth, a suggestion list above the composer, needed an integration that preview didn’t have, so the comment marks it not testable there and ends with “Check before merging.” The reviewer knows which three changes held up in the running app and which one still needs a person. Past the comment, the station plugs into the rest of the line.
- Issues: Pie groups related findings into issues.
- Fixes: When you use Auto-Fix, Pie opens proposed fixes as pull requests in repositories you connect, and merging them is always your team’s call.
- Agents: Your own agents can reach Pie through API and MCP. Keep the agent that writes a change away from the tests that judge it.
How to Tell the Factory Is Working
Start with what no station on this line checks, which is whether the code will still be maintainable in a year. Dex Horthy, who runs HumanLayer, points out that a line rewarded for passing tests pays no penalty for eroding maintainability. A product check doesn’t cover that either, so keep people reviewing the design, as our own team does, and track four numbers month over month.
- Escaped defects: Count the bugs customers found per agent-written PR.
- Product-station catches: Count the failures the running-app check caught after the code and behavior checks had passed.
- Human interventions: Count how often a person had to step in per merge.
- Satisfaction: Score the share of scenario runs that likely satisfied the user, a measure we take apart in Scenarios vs Tests.
If the third number isn’t falling, the line isn’t earning more autonomy yet. A coverage percentage won’t tell you otherwise.
Give the Product Its Own Station
Four of the five checks before the merge look at the code or its behavior, and a green run from them tells you the build agrees with itself. The login form in Factory’s story broke anyway, because nobody clicked through the product.
Pie is the station that uses the product the way your users do, and it can test your pull requests when it has a preview or a build to open. On one of our own, that meant three changes verified and one flagged for a person.
Agents can build the line. Somebody has to check what comes off it.
Add the Product Station
Hand Pie a preview URL from your next PR. See what it finds.
Frequently Asked Questions
A dark software factory is the top rung of Dan Shapiro's autonomy scale, Level 5, which he calls the Dark Factory. Specs go in and software comes out, with nobody needed on the line.
Shapiro knows only a handful of people running this way, in small teams.
Mostly. Lights-out manufacturing is the factory-floor version, a plant that runs unattended with the lights off. A dark software factory borrows the picture for software, where agents build without a person in the room.
With nobody watching, the automated checks carry the whole load.
CI/CD is the spine of it. The US Department of Defense's 2020 definition pairs CI/CD with mandated testing and security gates, and AI factories keep that shape.
What changes is who writes the code and the tests. Agents now do both, which makes the gates matter more.
Something the agent did not write and cannot edit, such as scenarios held outside the repo or a check that uses the running product the way a customer does.
A green result from tests the builder wrote only shows the builder agrees with itself.
Probably not at first. Digital twins are behavioral clones of the third-party services your software depends on. They let scenarios run at high volume without rate limits or API costs.
Start with a preview environment and test accounts, and add twins when limits or costs start to bite.
People write the spec and decide what merges. At Level 3 they review code all day, and at Level 4 they write a spec, leave, and check back after the tests run.
Only at Level 5 do they step off the line entirely.
Pie is the station that uses your product the way a user does. It explores your app and generates tests, and it can test your pull requests when it has a preview URL or a build upload.
When you use Auto-Fix, merging its proposed fixes is always your team's call.
13 years building mobile infrastructure at Square, Facebook, and Instacart. Now building the QA platform he wished existed the whole time. LinkedIn →