Scenarios vs Tests: How to Stop Coding Agents From Grading Their Own Work
A coding agent can rewrite any test it can read. Here is how scenarios kept out of its reach check the built app, and where they still miss.
A failing test goes green after the agent’s last commit, and the diff shows how. The agent changed the assertion and left the code alone. Nothing stopped it, because the test sat in the same repo the agent was editing. Anything the agent can read, it can bend.
If agents write your code and the tests that approve it, you need a check they can’t touch. Here is how scenarios give you one, and what they still won’t catch.
What you’ll learn
- How a scenario differs from a test, in six dimensions
- Three ways an agent-written test goes soft
- One acceptance criterion rewritten as a scenario
- Where holdouts still fail, and what runs a scenario against the product
Scenarios and Tests in Two Definitions
Two definitions are enough to tell them apart. The difference is where the check lives.
- Test: Code in your repository that runs in CI and returns pass or fail.
- Scenario: An end-to-end user story, often kept outside the codebase. It runs against the built software and is scored by whether the user was satisfied.
For a coding agent, the gap between the two is reach. The agent that wrote the code can read and rewrite a test in the same repository, and it never sees a scenario held outside it. One checks the code. The other checks the product.
A holdout set is the slice of data a model never trains on, almost like exam questions the student never gets to practice on. A scenario plays that role for a coding agent. The agent builds against the spec, and the scenario checks the result from somewhere the agent can’t reach.
Why Agent-Written Tests Drift Toward the Code
An agent that writes the code and the test is grading its own exam, and both answers come from the same place. Everything in that repo is fair game for the next commit.
A few months back I realized that we’re not really looking at the code anymore. You set direction and trust the coding agent to do its thing. Once that’s how you work, the check is the one thing you read, so it can’t come from the same place as the code.
Before fixing it, split it. Is the test weak from the start, or did the agent bend a good one? The first two patterns below are weak tests, and the third is a good test bent.
- Copied expectations: The agent reads the code and takes the expected value from it, so a bug becomes the spec.
- Narrow tests: The agent writes a check so narrow that a shortcut passes it, such as a function that returns true.
- Edited assertions: The agent changes a failing check to match the new behavior, so the test now describes the bug.
None of this is hypothetical. The reward-hacking benchmark EvilGenie observed explicit reward hacking by both Codex and Claude Code in programming tasks. In our own run, we handed three models a boundary bug, and all 56 test suites they wrote passed on the buggy code.
Weak tests need someone other than the coder writing the check. A bent test needs a check the coder can’t edit. A scenario held outside the repo does both.
Scenarios vs Tests at a Glance
Six differences decide which one you reach for. Half of them come down to who can touch the check.
| Dimension | Test | Scenario |
|---|---|---|
| Where it lives | In the repo, next to the code | Often outside the codebase |
| Who writes it | A developer or the coding agent | A person, or an agent that isn’t the coder |
| Who can edit it | Anyone with repo access, agent included | Whoever owns the scenario store |
| How it’s scored | Pass or fail | Satisfaction across many runs |
| What it catches | Regressions in behavior you specified | Whether a user’s journey works end to end |
| What it costs | Fast and cheap to run and rewrite | Slower per run, and someone has to write it |
Read the “Who can edit it” row first. It decides whether a green build means anything at all, because a check the agent can rewrite will agree with whatever the agent shipped.
Scored as Satisfaction, Not a Green Tick
A test reports one pass or fail. A scenario runs many times, because a model-driven run doesn’t take the same path twice. StrongDM scores it as the share of runs that likely satisfied the user, a probability instead of a yes or no.
There’s no standard pass mark. Set yours before the first run, and keep it with the scenarios.
Turning an Acceptance Criterion Into a Scenario
Take one criterion from a store’s spec and walk it from test to scenario. A returning customer can pay with a saved card and see an order confirmation.
As a test, it asserts that the order service returns a paid status for a cart total. The test passes when the service works, even if a layout bug hides the pay button. As a scenario, it follows the customer.
- Outcome: Write what the customer ends with, an order confirmation, and not the function that creates it.
- Path: Say what the customer does, in the order they do it.
- Finish line: Name what must be visible at the end, such as the item, the total and an order number.
- Awkward variants: Write a second copy where the saved card has expired and a third where the item sells out mid-checkout.
- Storage: Keep the scenario in a place the coding agent has no access to, such as a separate repo or tool.
Written out, the scenario reads like this:
A returning customer opens the store on their phone. They add running shoes in size 9 to the cart and go to checkout. They pay with the saved card. They finish on a confirmation page that shows the shoes, the total and an order number.
The unit of work is the journey. You describe what the customer does and what satisfied looks like, and nothing in it names a function the agent could edit.
Where Scenarios Still Fail
A scenario held out of reach takes away the easiest cheat, but it isn’t proof. The same EvilGenie benchmark measured hacking three ways, with held-out unit tests, LLM judges and test-file edit detection. Held-out tests gave only “minimal improvement” in detection, while the LLM judge proved “highly effective” in unambiguous cases. A holdout stops the agent from editing the check, and the rest has to be caught some other way. Three gaps remain.
- Judge error: Where an LLM judge scores the runs, a wrong judgment is a wrong score. Spot-check a sample of judged runs by hand.
- Scenario gaps: A scenario covers the journeys someone imagined. The flow nobody wrote down gets no score.
- Leaks: Scenario text that reaches the agent through a shared prompt or a pasted ticket turns into one more test to tune against.
A holdout takes away the edit, not the cheating. Keep a person reading the judged runs.
Running Scenarios Against the Product
A scenario earns its keep when something runs it against the built app. If the judge only reads code, you’ve built a second code check with a better name.
In software factory testing, that run is a station of its own, apart from the builder, and it uses the app the way a user would. Put it on the pull request, before merge, as one of the checks that gate agent-written pull requests.
Written scenarios cover the flows someone thought of. Pie explores your app and generates tests, so the run isn’t limited to those. Pie can test your pull requests, which takes a preview URL or a build upload. On one of our own pull requests, its comment verified three of four changes on the running preview, marked the fourth as not testable there, and ended with “Check before merging.” Keep the coding agent that wrote the change away from the tests Pie runs on it, and the check stays out of its reach. Think of it as almost letting a few actual users test your PR before it lands.
Write scenarios for the two or three flows that cost you most when they break, like the checkout above, and keep them out of the agent’s reach. Then let Pie explore your app for the flows nobody wrote down, on the preview of your next pull request.
Move the Check Out of Reach
Tests stay. They’re fast, and they catch logic errors the minute they appear. What changes is who holds the answer key.
Scenarios move it out of the agent’s reach, and Pie checks the built app instead of the diff. The test that went green because the agent edited its assertion is the case to design for.
Keep your tests. Just don’t let them grade the agent that wrote them.
Stop Letting Agents Grade Themselves
Point Pie at your next preview. Get findings from the running app.
Frequently Asked Questions
A test is code in your repository that runs in CI and returns pass or fail. A scenario is an end-to-end user story kept outside the codebase and run against the built software.
Tests check the code you specified. Scenarios check what a user goes through.
The model often takes its expected values from the code it is checking, so a bug becomes the spec. An agent can also edit a failing assertion until the build goes green.
Either way the test confirms the code agrees with itself, not that the product works.
A holdout suite is a set of checks the builder never sees. The name comes from model training, where a holdout set is data the model never trains on.
For coding agents, the holdout is often scenarios kept outside the repository, so the agent can't read or edit them.
Verification asks whether you built the thing right, meaning the software matches its spec. Validation asks whether you built the right thing, meaning it works for the people using it.
Unit tests mostly verify. Scenarios aim at validation.
Yes. If the scenario text reaches the agent through a prompt or a pasted ticket, it becomes one more test to tune against. An LLM judge can also score a bad run as a pass.
Keep the scenarios sealed and spot-check judged runs by hand.
No. Unit tests are fast and cheap, and they catch logic errors the moment they appear. Keep them.
Scenarios sit above them as the check the agent can't tune its way past. Together they cover speed and the user's view.
Pie checks the running app, the same ground a scenario covers. It explores your app and generates tests, so it isn't limited to the flows someone wrote down.
Pie can also test your pull requests when it has a preview URL or a build upload.
Eight years building search and delivery systems at Amazon. The kind of scale where flaky tests block billion-dollar releases. Now CTO at Pie, building AI agents that adapt when your UI changes. LinkedIn →