Blog / Test Automation Strategy: What to Automate First, in Order
Guide

Test Automation Strategy: What to Automate First, in Order

Every check looks worth automating, and no team can keep them all green. This test automation strategy ranks what to automate first and where to stop.

Your regression sheet runs to a couple of hundred checks, and every row is marked high priority. The quarter has room to script a few dozen. Choose badly and the suite spends the next sprint breaking on screens that moved.

Ranking is the core of a test automation strategy. Here is how to rank your own sheet so the checks you script pay back, and how to tell which ones belong with your testers.

What you’ll learn

  • The order to automate in, from the cheapest check to the costliest
  • Four questions that place any check on that order
  • Where to stop on each rung, and which checks stay with your testers
  • The two events that re-rank the order

Six Rungs, Ordered by Payoff Against Upkeep

A test automation strategy is a ranked order of what to script, with a stop line on every step. Automate in order of payoff against upkeep. A check pays off by running often and protecting a lot, and its upkeep depends on how often its screen changes. A rung opens only when the one beneath it runs green without repair.

RungWhat to automateUsual ownerStop line
1Unit tests on logic that has already brokenDevelopersEvery fixed bug leaves a test behind
2A smoke check on every buildDevelopers or QAIt signs in and reaches the core screen, nothing more
3API and service checksDevelopers or QAEach service contract has one check
4End-to-end flows that earn or lose moneyQAEvery flow whose break would page someone has a test
5Regression on steady screensQAThe repeat pass is gone for those screens
6Data combinations and device matricesQAA check costs more to keep than a miss costs

Rungs one to three usually belong to developers, so a QA lead’s first move is to check that they exist. The part of the strategy a QA team owns starts at rung four.

Four Questions That Place Any Check

Every row on a regression sheet can be placed by four questions asked in order. They extend the placement rule from our comparison of manual vs automated testing, with the cost of a miss added as the sort key.

  1. Frequency: Does the check run every release? One that runs twice a year never repays its script, so it stays manual.
  2. Clear result: Is there one expected result a script can read? A cart total that matches its line items is a yes. A page that feels right is a no, and a person answers it.
  3. Cost of a miss: What does it cost when this breaks? Lost sales, lost trust and safety rank highest. Sort the checks that survived the first two questions by this, highest first.
  4. Screen churn: Is the screen it touches steady? A steady screen takes a script. A screen that changes every sprint waits, or goes to a tool that does not depend on selectors.

Take a shopping app. Six checks from its regression sheet land in four places, and the cart total drops to the base of the ladder because it never needed a screen.

CheckQuestion it failsLands on
Cart total matches its line itemsNothing, and no screen is involvedRung 1
Pay with a saved cardNothing, and a miss loses salesRung 4
Edit a saved addressNothing, and the screen is steadyRung 5
Holiday banner copy is correctFrequency, it runs twice a yearManual
New filter is easy to findClear result, a person must judge itManual
Crop a profile photoScreen churn, redesigned last monthManual for now

What Each Rung Buys and Where to Stop

Each rung has a reason to sit where it does and a point past which more checks cost more than they return. The bottom of the ladder follows a rule Ham Vocke gave in The Practical Test Pyramid:

“Push your tests as far down the test pyramid as you can”
Ham Vocke, martinfowler.com, 2018

The pyramid settles where a check runs fastest. It does not settle how many end-to-end flows to keep. Here the cost of a miss and how often a screen changes decide that.

Pyramid: unit tests at the base, API checks in the middle, screen-bound checks on top
The pyramid sets the tier. The ladder decides which screen-bound checks stay.
  1. Past-bug unit tests: No screen is involved, so almost nothing breaks them. When a higher-level test catches a bug and no unit test fails beneath it, Vocke’s advice is to write the lower-level test. Jest, pytest and JUnit cover the common stacks. Make it a standing rule that every fixed bug leaves a test behind.
  2. Build smoke check: A short smoke test that signs in and reaches the core screen tells the team within minutes that a build is dead. Stop at the first screens a user meets, because a smoke suite that keeps growing turns into a slow regression suite.
  3. Service checks: Calls to your API run beneath the screens, so a screen redesign leaves them alone. One check per contract between services is enough, written in the same runner the developers already use.
  4. Money flows: Sign-up, sign-in, checkout and the one action customers pay for. There are only a few, and each one guards a lot. They are the best use of a slow, screen-bound test. Rank them by cost of a miss and stop when each flow that would page someone has one test.
  5. Stable-screen regression: The repeat pass your testers click through every release is where scripting returns the most tester time, and where upkeep bites hardest. As I see it, the real cost of an end-to-end suite is engineering time. You spend it once to write each test and again to keep it current as the code base grows. Begin with the screens that have stayed put longest, and read how to automate regression testing for the steps.
  6. Wide coverage: Data combinations, device matrices and rare edge paths multiply checks fast. Each one is another test to keep current and another result someone has to read. My advice is to grow this rung only as fast as your team can review what it reports. Ramp up faster and nobody has the bandwidth to read the results. Close the rung when a check costs more to keep than a miss would cost.

Name the Flows That Matter

Pick the flows that would page someone. Pie writes and runs their tests.

What Stays With Your Testers

Ranking by payoff leaves a list of checks that never earn a script, and that list is your manual team’s brief. Every hour that automation takes off the repeat pass goes to these checks.

  • Exploratory testing: A tester follows a hunch into a corner no written path reaches. Exploratory testing finds what a suite never thought to ask.
  • Usability and feel: Whether a button sits where people look is a judgment, and a script can only confirm that the button exists.
  • Features mid-build: When expected behavior changes every few days, a script written today is rewritten next week.
  • First pass on new work: A person runs a new feature once before anyone scripts it, and that run shows which checks deserve a script.
  • Device-in-hand checks: Pairing a physical product or pointing a camera at something stays with a person holding the device.
  • The final call: Whether a release does what the business asked is a human decision.

Two Triggers That Re-Rank the Ladder

The order holds until one of two things happens, and both show up in the suite itself.

  1. Screen churn: A screen starts changing every sprint. Its checks leave the scripted rungs, because a script finds each element by something written into the page and a redesign breaks it. Keep those checks manual until the screen settles.
  2. Noisy failures: Red runs stop meaning bugs. Martin Fowler’s test pyramid note warns that end-to-end tests are more prone to non-determinism, which can undermine trust in them. Once a rung’s failures are noise, freeze every rung above it until a red run means a bug again.

Write the Strategy on One Page

A test automation strategy fits on one page, with four lines on it:

  1. Rungs funded: Which rungs get checks this quarter.
  2. Owners: One named owner per rung.
  3. Stop lines: Where each rung stops growing.
  4. Triggers: The two events that send the page back for review.

Tool choice and CI wiring belong in the plan as well. They follow from the order. Once rung four is next, you know which flows your test automation tools have to drive.

Hand the Screen-Bound Rungs to Pie

Rungs four and five sit on screens, and they are the part of the ladder a QA team owns. Our agentic testing platform writes and runs the tests for those rungs on web, iOS and Android, while your developers keep rungs one to three.

  • Upkeep handled for you: Pie explores your app and generates the tests, and they self-heal. A changed selector does not break them, so your team is not repairing scripts after every redesign.
  • Checks from a sentence: Describe a manual check in Pie Chat and review the draft before it runs.
  • A map to rank against: Pie groups the tests it writes under your app’s key features, so the money flows from rung four are easy to find.
  • Testers stay in charge: Pie works alongside a manual QA team and your unit tests. It adds to both and replaces neither.
Pie Chat draft of a login page test, with objective, steps and checks
A drafted test in Pie Chat, reviewed before it runs.

Climb in Order and Stop on Purpose

The order is the strategy. Four questions place each check on a rung, and every rung has a line where it stops growing. A small ladder that stays green beats a large suite nobody trusts.

Open your sheet and find the row whose failure would page someone tonight. Does it have a test yet?

Script what pays. Leave the rest to people who can think about it.

Spend Testers on Judgment

Hand Pie the screen-bound rungs. Keep your testers on the hard calls.

Frequently Asked Questions

A strategy fits on one page with four lines. Name the rungs you fund this quarter and an owner for each. Set a stop line for every rung. List the two triggers that re-rank the order. Tool choice and CI wiring follow from that order.
Automate checks that run every release and have one clear result. Unit tests for logic that has broken come first, then a smoke check on every build and API checks. End-to-end flows that earn or lose money follow, once the screens they touch hold steady.
Leave exploratory testing and usability judgments with your testers, along with features still mid-build. Checks that need a physical device in hand stay manual too. A script cannot judge whether something feels right, and a check that changes weekly costs more to repair than to run by hand.
No percentage fits every team. Automate until the next check on the list fails the four questions, then stop. A coverage target pulls in checks that run rarely or sit on screens that keep changing, and those cost more to keep than they save.
The ladder takes its base from it, with fast low-level checks first. The pyramid does not say how many end-to-end flows to keep, so the ladder ranks those flows by what a miss costs and how steady their screens are.
Watch the suite itself. Every red run should point to a real bug. Each funded rung should run green without repair, and the repeat pass your testers click through should get shorter each release. When red runs stop meaning bugs, re-rank before you add another check.
No. Automation takes over repeat checks, and the work that needs judgment stays with people. Exploratory testing and the final call on a release remain with your testers. They get more hours for that work once the repeat pass runs on its own.
Pie writes and runs the tests for rungs four and five, the screen-bound flows, on web, iOS and Android. Developers keep the unit and API checks, and testers keep exploratory work and the release call. Pie adds to your team and does not replace it.
Jinoo Jain
Jinoo Jain
CPO & Co-founder at Pie

Spent a decade in B2B SaaS sales before building Pie. Now obsessed with helping engineering teams ship without the fear of breaking things. LinkedIn →