Guide

How to Build a Regression Test Suite (Step-by-Step)

A 9-step playbook for building a regression test suite that actually runs. Includes real auth flow and checkout flow test cases, code examples, and how autonomous discovery cuts first-suite time to 30 minutes.

Adithya Aggarwal

CTO & Co-founder at Pie

14 min read

Steps 1–3: Define Scope, Select Cases, and Structure Your Suite

Step 1: Define Your “Blast Radius”

Before writing a single test, answer one question: What would have to break for your users to notice (and panic) immediately?

For most products, that list is much shorter than engineering teams assume:

• Core Actions: Payment processing, authentication, core navigation.
• Account Mutations: High-stakes settings changes (like billing or permissions).
• Mobile Specifics: Crash-free app launch, deep link handling, push notification execution.

Write this list down in a short, plain-language document your entire team agrees on. This is your regression scope. Every test you write must trace back to this list. If it doesn’t, it belongs in a separate smoke suite or an integration test run—not your core regression suite.

Why this matters: When your suite inevitably grows to 400+ tests and CI pipelines start timing out, this scope statement tells you exactly which 80 tests to run on every single PR, and which 320 can wait for the nightly build.

Step 2: Select and Prioritize Test Cases by Risk

High-performing engineering teams don’t try to cover everything; they cover the right things. Instead of exhausting every possible edge case, prioritize your tests based on three real-world risk factors:

1. Revenue Impact: If it breaks, do you lose money or lock users out? (e.g., Checkout, Login). These run on every PR.
2. Code Volatility: Which areas of the codebase are your developers touching multiple times a week? Check your last 90 days of diffs. High-traffic code needs nightly validation.
3. Scar Tissue: Look at your bug reports from the last six months. Every production failure is a regression test waiting to be written.

Instead of creating massive matrices, bucket your priorities ruthlessly. Keep your P0s restricted to critical paths (Auth, Checkout happy paths) that run on every PR. Push environmental edge cases (expired cards, empty states) to P2s for pre-release windows.

Step 3: Architect Around User Flows, Not Code Features

Most regression suites fail structurally because they mimic the codebase architecture. They have a “login” folder, a “checkout” folder, and a “profile” folder. But when a bug spans across login and checkout, your tests break across multiple directories, and nobody can pinpoint the root cause.

Organize by user flow instead. A flow is a sequence of actions a real human takes to accomplish a goal.

• “New user signs up, adds a product, and completes a purchase” is a flow.
• “Login tests” is just a component check.

Flow-organized suites change the game in three ways:

• They catch integration failures that single-feature tests entirely miss.
• They produce failure messages that tell you exactly which user journey is broken, rather than just spitting out a broken page selector.
• They match the team’s mental model of how the product actually functions, making hand-offs seamless.

For 90% of applications, five to eight core flows will capture 80% of what actually matters. This follows the Pareto principle (also known as the 80/20 rule)—a small percentage of your flows drive the majority of your test coverage needs. Map out those 5 to 8 core flows, name them in plain language, and you have a bulletproof architectural blueprint.

Steps 4–6: Write Test Cases, Automate Execution, and Connect to CI

Scope without execution is documentation. Steps 4 through 6 are where your regression suite becomes a real engineering artifact.

Step 4: Write Test Cases with Concrete Examples

Every guide on this topic says “write test cases for your critical flows.” None of them show you what that looks like. Here are two concrete examples.

Auth flow test case structure (8 steps):

Before you write auth tests, decide what “authentication working” means end-to-end: not just that the login button submits a form, but that a valid user reaches their authenticated home state, an invalid user sees the right error, and a session-expired user is redirected correctly. Each of those is a separate test case, not a single “login test.” Here is what a concrete auth flow structure looks like:

Test Suite: Authentication Flow
Test Case: TC-AUTH-001 — Valid user login (email + password)

Preconditions:
  - Test account credentials available (email: [email protected])
  - User is logged out (session cleared)

Steps:
  1. Launch app / navigate to login screen
  2. Enter valid email address
  3. Enter valid password
  4. Tap "Sign In" button
  5. Wait for home screen to render

Expected Results:
  - Home screen renders within 3 seconds
  - User display name visible in header
  - Session token present in storage
  - No error toast or modal displayed

Test Case: TC-AUTH-002 — Invalid credentials show correct error

Steps:
  1. Launch app / navigate to login screen
  2. Enter valid email, enter wrong password
  3. Tap "Sign In"

Expected Results:
  - Error message displayed: "Incorrect email or password"
  - User remains on login screen
  - Password field cleared, email field retained
  - No session token created

Test Case: TC-AUTH-003 — Session expiry redirects to login

Preconditions:
  - User is logged in with a session token
  - Token expiry manually triggered (test environment only)

Steps:
  1. Trigger session expiry
  2. Attempt any authenticated action (e.g., load profile)

Expected Results:
  - Redirect to login screen
  - Session-expired message shown (if applicable)
  - Previous destination preserved for post-login redirect

Checkout flow test case structure (12 steps):

Checkout is where most revenue-critical regressions live. Most bugs here are not “checkout broken” bugs. They are “checkout broken under specific conditions” bugs: a specific payment method, a specific promo code format, a specific device size. Structure your checkout tests to catch the conditions, not just the happy path.

Test Suite: Checkout Flow
Test Case: TC-CHK-001 — Happy path: single item checkout with saved card

Preconditions:
  - User logged in
  - One item in cart (in-stock SKU)
  - Valid saved payment method on account

Steps:
  1. Navigate to cart
  2. Verify item shown with correct price
  3. Tap "Proceed to Checkout"
  4. Verify shipping address pre-populated
  5. Verify saved card shown as default
  6. Tap "Place Order"
  7. Wait for order confirmation screen

Expected Results:
  - Order confirmation displayed with order ID
  - Confirmation email triggered (check test inbox)
  - Cart cleared
  - Inventory decremented in system (API check)

Test Case: TC-CHK-002 — Coupon code applied before checkout

Steps:
  1. Navigate to cart with item
  2. Enter valid promo code in coupon field
  3. Tap "Apply"
  4. Proceed to checkout

Expected Results:
  - Discount applied to cart total
  - Discounted price visible on checkout summary
  - Order total reflects discount in order confirmation

Test Case: TC-CHK-003 — Expired card shows payment error

Steps:
  1. Navigate to checkout
  2. Select expired card as payment method
  3. Attempt to place order

Expected Results:
  - Payment declined error displayed
  - User prompted to update payment method
  - Order NOT created in system
  - Cart items preserved

Step 5: Automate Test Execution

Manual test cases written above are the spec. Automation is how you run them at scale without human involvement. For most mobile and web teams, there are two practical approaches:

• Selector-based automation (Playwright, Espresso, XCUITest) — Write code that locates elements by ID, accessibility label, or XPath, then interacts with them. Fast to write initially. Expensive to maintain when the UI changes, because every locator is a hardcoded reference to implementation details.

• Behavior-based automation (vision-based or model-driven) — Tests locate elements by what they look like and what they are, not by their selector. More resilient to UI changes. Self-healing tests extend this further: when an element moves or gets renamed, the test adapts automatically rather than failing.

Here is what selector-based automation looks like for TC-CHK-001 in a mobile context (pseudocode, framework-agnostic):

# TC-CHK-001: Happy path checkout with saved card
# This test validates the complete purchase flow from cart to confirmation.
# Run this on every PR that touches cart, checkout, or payment code.

def test_checkout_happy_path_saved_card(driver, test_user):
    # Setup: ensure cart has exactly one in-stock item
    add_item_to_cart(driver, sku="ITEM-001", quantity=1)

    # Navigate to cart and verify state
    navigate_to_cart(driver)
    assert get_cart_item_count(driver) == 1
    assert get_cart_item_price(driver, "ITEM-001") == EXPECTED_PRICE

    # Proceed through checkout
    tap_element(driver, label="Proceed to Checkout")
    assert shipping_address_is_prefilled(driver, test_user.address)
    assert default_payment_method_is_visible(driver, test_user.saved_card)

    # Place order
    tap_element(driver, label="Place Order")
    wait_for_screen(driver, screen_id="order_confirmation", timeout=5)

    # Verify confirmation state
    order_id = get_order_id_from_screen(driver)
    assert order_id is not None
    assert cart_is_empty(driver)
    assert order_exists_in_system(order_id)  # API call to backend

The key discipline here: each automated test verifies state at multiple points, not just at the end. If the confirmation screen loads but the cart was not cleared, that is a bug. If the order ID is displayed but the backend record does not exist, that is a bug. Tests that only check the final screen miss half the failure modes.

Step 6: Integrate with Your CI/CD Pipeline

Tests that do not run automatically are documentation, not regression protection.

Every modern CI system (GitHub Actions, GitLab CI, CircleCI, Bitrise for mobile) supports a test stage before merge. Configure yours to run at minimum: smoke tests on every PR, full regression suite on merge to main, and a pre-release gate before any production deployment.

For end-to-end testing in CI/CD, the rule that matters most is test segmentation: not every test needs to run on every trigger. Use labels or tags to define three tiers:

• @smoke — runs on every PR (under 10 minutes total)
• @regression — runs nightly and on merge to main (can be up to 45 minutes)
• @release — full suite plus edge cases, runs before production deploy

Keep your smoke tier ruthlessly small. When smoke tests start taking 25 minutes, developers stop waiting for them. When developers stop waiting, CI loses its value as a quality gate.

Steps 7–9: Add Mobile Coverage, Eliminate Flakiness, and Build for Maintenance

The first six steps build a suite. Steps 7 through 9 determine whether it survives contact with production engineering.

Step 7: Build Mobile-First Coverage into Your Suite

Most regression guides treat mobile as an afterthought. For teams building native apps, that framing is backwards. Your mobile app testing regression suite needs to cover scenarios that have no equivalent on web:

• Gesture coverage: Tap, swipe, pinch, scroll-to-load are not edge cases on mobile. A swipe-to-delete gesture on a cart item that silently fails is a checkout regression. Add gesture interactions for every core flow that uses them.

• Deep link handling: If your app supports deep links (and most do), verify that authenticated deep links work, unauthenticated deep links redirect to login and return the user afterward, and malformed deep links do not crash the app.

• OS version and screen size: Regression does not mean “works on my device.” Test across the OS versions your users are actually running. Apple and Google both provide device distribution data through their respective developer portals. Run your P0 tests across at minimum: latest OS, one previous major version, and your lowest-supported version.

• Network condition testing: Checkout flows that depend on real-time payment authorization need to be tested under degraded network conditions. A flow that succeeds on WiFi but fails on 3G is a real regression waiting to ship.

• Background and foreground transitions: On mobile, users switch apps mid-flow. A user who puts your app in the background during checkout and returns 90 seconds later should find their session intact and their cart unchanged. If they do not, that is a regression.

Step 8: Address Flakiness Before It Poisons Your Suite

Here is the trap most teams fall into: they build a 300-test suite, it works well for four months, then one by one tests start failing randomly. The team disables the failing tests to unblock merges. Within six months, 30% of the suite is quarantined and everyone has stopped trusting it.

Flakiness comes from deeper causes than most engineers assume. The failure modes split into three categories:

• Timing dependencies — Tests assume a loading state ends in exactly 2 seconds, rather than waiting for a specific element to appear or a ready state to signal. The fix is immediate: replace hardcoded sleeps with explicit wait conditions tied to actual page readiness signals. Wait for the element to render, not for a duration.

• Shared test state — Tests pass or fail depending on execution order because they read or write shared data (cache, database, login session). If Test A runs before Test B and leaves stale data behind, Test B fails. The fix: each test should set up and tear down its own data. No test should assume a previous test has run or left artifacts behind.

• Element location brittleness — Tests break when a button moves one pixel or gets a new CSS class name, because selectors are hardcoded references to implementation details, not to purpose. The fix: self-healing test infrastructure handles this automatically by locating elements by visual fingerprint rather than selector. When the button moves, the test adapts.

Quarantining flaky tests is a reasonable short-term tool. It is not a strategy. Every quarantined test is a regression that can now ship without detection. Track your quarantine list, review it weekly, and treat it as technical debt with a deadline.

Step 9: Build Maintenance into the Architecture, Not the Backlog

Maintenance is the number one killer of test suites. It’s rarely flaky logic that dooms a QA pipeline; it’s the sheer volume of upkeep.

When a UI redesign breaks 80 locators overnight, someone has to fix them. If that requires an engineer to manually hunt down and update selectors, your testing pipeline will immediately fall behind your deployment cycle. Feature work will always win out over test maintenance, meaning your suite is on a fast track to being abandoned.

To break this cycle, you need to make two core architectural shifts:

1. Prioritize Coverage Density Over Suite Size

More tests do not equal better testing. In fact, overlapping coverage just doubles your maintenance tax.

Audit your suite periodically by asking a simple question: Which single test should catch this bug first? For every critical failure, there should be one primary validator, not six. By pruning redundant tests that cover identical code paths, you can easily cut your suite size in half—drastically reducing execution times and maintenance overhead without dropping your guard.

2. Let Autonomous Discovery Handle the Drudge Work

When your application evolves, your tests need to evolve with it automatically. Instead of forcing a human engineer to manually audit and update the suite after every major release, use autonomous discovery to continuously map your application as it exists in production today.

Autonomous discovery crawls your current UI, identifies gaps where new features lack coverage, and updates flows dynamically.

The Fast Track: If you’re building a regression suite from scratch, Pie’s autonomous discovery allows you to skip weeks of manual scripting and hit 60–80% baseline coverage in 30 minutes. If you already have a suite, it’s how you stop the coverage gap from widening every time your developers ship code.

Build Once, Run Forever

Building a regression suite is easy. Keeping one alive is what breaks teams. You’ll hit the maintenance wall between month four and month eight—the same wall every team hits. When you do, you’ll have a choice: maintain the suite or ship features. Those two things can’t coexist with traditional automation.

Pie was designed for teams that need both. Our autonomous testing platform generates your baseline suite in 30 minutes, then automatically maintains and expands it as your product grows. No more choosing between shipping and testing.

Adithya Aggarwal

CTO & Co-founder at Pie

Eight years building search and delivery systems at Amazon. The kind of scale where flaky tests block billion-dollar releases. Now CTO at Pie, building AI agents that adapt when your UI changes. LinkedIn →