Regression Test Selection: How to Run Fewer Tests Without Missing Failures
Regression test selection always has a change it cannot see. Pick tests per change and audit your selector against a full run so it misses no failures.
A test selector has one job: run only the tests a change can break. We wrote one for a small shop app, 32 lines on top of coverage.py, and changed one line in a config file. It picked 1 test out of 12, and that test passed. A full run of the same commit failed 4. The selector had no test mapped to that file.
Every way of picking regression tests has a change it cannot see. Here is how to pick tests per change and prove your selector is not hiding the failures that matter.
What you’ll learn
- The five signals that decide which regression tests run for a change
- A rule for which kinds of change need every test
- Three real misses from a selector script we wrote and ran
- A 17-line audit that catches a selector skipping failures
What Is Regression Test Selection?
Regression test selection is choosing which tests from an existing suite to run for one specific change, based on what that change can affect. It cuts run time, and it adds one risk. A test that does not run cannot fail, so if the selector leaves out a test that would have failed, the bug gets through.
Two other practices sound similar. Prioritization runs every test and puts the likely failures first. Minimization deletes redundant tests from the suite for good. Selection keeps the whole regression suite and, for each change, runs only the slice that change can affect. Because the full suite is still there, you can audit a selector against a full run.
The pressure to select shows up once the suite is the slowest step in the pipeline, the pattern behind append-only suites. Selection starts the moment a change lands and someone has to decide which regression tests run.
Run the union of five signals, force a full run for any change the signals cannot see, and compare against a full run to catch every miss.
Five Signals for Picking Tests for a Change
Five signals decide what a selector picks. Each answers a different question about the change, and each has a blind spot.
| Signal | The question it answers | Where it skips a real failure |
|---|---|---|
| Changed files | Which tests live next to the files in the diff? | Shared code, and any behavior a test reaches through another module |
| Coverage or dependency map | Which tests execute the code that changed? | Config, data files, dependency upgrades, and anything built after the map was |
| Risk | Which flows cost the most if this change broke them? | A judgment call that goes stale and ignores what the diff touched |
| Failure history | Which tests failed lately and could fail again? | New code with no history, and flaky tests that look like signal |
| Always-run smoke | What must work after every change? | Depth: a passing smoke test says nothing about the flows below it |
Changed Files
The cheapest signal is the diff itself. A rule that maps src/checkout/** to tests/checkout takes minutes to write and breaks the first time a change lands in shared code. Playwright’s --only-changed flag goes one step further. Its release notes say it runs the test files changed since a git ref, plus every test file that imports a changed file. It follows imports, not what a test executes, so a change that reaches a test any other way stays invisible to it.
Coverage and Dependency Maps
A map records which tests execute which files. It works almost like a book’s index, where you look up a file and find every test that ran it. Datadog’s Test Impact Analysis builds it from per-test coverage and cross-references it with the files in a new commit. Google’s CI chapter describes the same idea at a larger scale, where the main mechanism is analysis of the downstream dependency graph.
A map only knows what tests executed. Datadog’s docs list what its default setup cannot detect: library dependencies, compiler options, external services, and data files used by data-driven tests.
Risk
Risk asks what it would cost if this area broke. Authentication, checkout, and payments earn a place in every selection, whatever the diff says. A risk list is one person’s judgment, so give it an owner and a review date, or it describes last year’s product.
Failure History
Tests that failed recently are good bets to fail again, so they ride along with the next change. Azure’s Test Impact Analysis includes previously failing tests next to the impacted and newly added ones. Our selector reads a plain list of recent failures and adds them to the pick. Quarantine flaky tests first, because history carries them along too.
Always-Run Smoke
A small tagged tier runs on every change, whatever the diff says. Treat it as the floor under the other signals, the same job a smoke test does anywhere. Its limit is depth. In our run the smoke test passed while four other tests failed.
A Rule for Each Kind of Change
Pick the union of the signals, then ask whether the map can see this change. When it cannot, run everything.
| The change | What runs | Why |
|---|---|---|
| A source file the map covers | Tests that cover it, the smoke tier, recent failures | The map has seen this file run |
| A new or edited test | That test, plus the pick above | A new test has no coverage or history yet |
| Config, data files, flags, CI settings | Everything | Coverage never measured those files |
| A dependency or lockfile upgrade | Everything | Datadog lists library dependencies as invisible by default |
| A new module, or code moved between files | Everything, then rebuild the map | The map has never seen the new path |
| Markup or styles only | Everything | Coverage never ran markup or styles, and Azure’s analysis falls back to all tests on HTML and CSS changes |
What a Selector Misses: A Small Project We Ran
We built a small shop app with 12 pytest tests and a tax rate in a JSON file, then wrote a 32-line selector on top of coverage.py’s per-test contexts. The script takes the files in git diff, looks up the tests that ran lines of them, and adds the smoke tier and a list of recent failures. A change to a tracked file runs every test.
"""Pick tests for a change: smoke, tests that cover the changed files, recent failures."""
import fnmatch
import subprocess
import sys
from pathlib import Path
from coverage import CoverageData
base, tracked = sys.argv[1], sys.argv[2:]
changed = subprocess.run(["git", "diff", "--name-only", base],
capture_output=True, text=True, check=True).stdout.split()
# Files the map cannot see (config, lockfiles): any change to them runs everything.
if any(fnmatch.fnmatch(f, g) for f in changed for g in tracked):
print("tests") # the whole suite
sys.exit()
data = CoverageData(".coverage")
data.read()
covers = {} # file -> tests that executed a line of it
for path in data.measured_files():
rel = Path(path).resolve().relative_to(Path.cwd()).as_posix()
for contexts in data.contexts_by_lineno(path).values():
covers.setdefault(rel, set()).update(c.split("|")[0] for c in contexts if c)
smoke = subprocess.run([sys.executable, "-m", "pytest", "--collect-only", "-q", "-m", "smoke"],
capture_output=True, text=True).stdout.split()
history = Path("tools/recent-failures.txt")
recent = history.read_text().split() if history.exists() else []
picked = {t for f in changed for t in covers.get(f, ())} | {t for t in smoke if "::" in t} | set(recent)
print("\n".join(sorted(picked)))
Build the map once, then pick for each change and run what it picked:
COVERAGE_CORE=ctrace python -m pytest --cov=shop --cov-context=test
python tools/pick_tests.py HEAD "config/*" > picked.txt
python -m pytest @picked.txt --junitxml=selected.xml
python -m pytest --junitxml=full.xml
We ran it on Python 3.14.0 with pytest 9.2.0, pytest-cov 7.1.0, and coverage.py 7.17.2. Each row below is a change we made, the pick, and what the full run of the same commit said.
| The change | Tests picked | Failed in full run | Failed in selected run |
|---|---|---|---|
Coupon rate in discounts.py from 10 to 15 | 4 of 12 | 1 | 1 |
US tax rate in config/tax.json, no tracked files | 1 of 12 | 4 | 0 |
Same change, config/* tracked | 12 of 12 | 4 | 4 |
rounding.py switched from rounding to flooring, map built before the refactor | 1 of 12 | 1 | 0 |
| Same change, map rebuilt after the refactor | 4 of 12 | 1 | 1 |
A Config File Nobody Mapped
The tax rate lives in config/tax.json. Coverage measures Python files, so the map has no entry for it, and the diff matched nothing. The selector fell back to the smoke test, which passed. The full run failed four tests, from the pricing unit test up to the checkout totals. Listing config/* as tracked makes the same change run all 12, which is the fix Datadog’s docs call tracked files.
A Map Built Before the Code Moved
We refactored rounding out of checkout into a new shop/rounding.py and did not rebuild the map. A later change to that file matched nothing, because the old map had never seen it. Only the smoke test ran, and the full run failed the tax-rounding test. After we rebuilt the map on the refactored commit, the same change picked 4 tests and the rounding test failed where it should.
A Map That Was Quietly Incomplete
Our first map build on Python 3.14 used coverage.py’s default sys.monitoring core. It raised a single warning in the pytest summary, and the pick on the coupon change had 2 tests where the right answer was 4, so the audit flagged the coupon test as missed. coverage.py’s own message reference says dynamic contexts are not supported with that core and the context data may be incomplete. Setting COVERAGE_CORE=ctrace fixed it. The build exited cleanly, so read the map builder’s warnings as part of the selector.
Audit a Suite You Never Scripted
Point Pie at your app. Get end-to-end tests worth auditing.
How to Check a Selector Is Not Skipping Failures
All three misses surfaced the same way, as full-run failures in tests the selected run never ran. Comparing the two runs is the entire audit. Microsoft’s Azure Pipelines docs describe the same check for impact analysis: run the selected tests, run all tests, and confirm the full run reports the same failures.
"""Compare the failures of a selected run with a full run of the same commit."""
import sys
import xml.etree.ElementTree as ET
def failed(path):
return {f"{c.get('classname')}.{c.get('name')}"
for c in ET.parse(path).getroot().iter("testcase")
if c.find("failure") is not None or c.find("error") is not None}
full, picked = failed(sys.argv[1]), failed(sys.argv[2])
missed = sorted(full - picked)
print(f"full run failed {len(full)}, selected run failed {len(picked)}, missed {len(missed)}")
for name in missed:
print(" MISSED", name)
sys.exit(1 if missed else 0)
On the config change with no tracked files, python tools/audit_selection.py full.xml selected.xml printed this and exited with code 1:
full run failed 4, selected run failed 0, missed 4
MISSED tests.test_checkout.test_checkout_rounds_tax_to_nearest_cent
MISSED tests.test_checkout.test_checkout_us_total
MISSED tests.test_checkout.test_checkout_with_coupon
MISSED tests.test_pricing.test_tax_rate_known_region
Turn it into a routine in five steps:
- Shadow the selected run with a full run. Run every test on the same commit at merge or nightly, whichever you can afford. Azure’s docs recommend a full run at a set interval, and Datadog leaves selection off the default branch so every test runs before production.
- Diff the failures. A miss is any test that failed in the full run and did not fail in the selected run.
- Name the cause of every miss. A file type the map cannot see goes on the tracked list. A stale map gets rebuilt. An incomplete map sends you back to the builder’s warnings.
- Rebuild the map from the full run. The nightly run with coverage contexts feeds the audit and the map, so the second job rides on a run you already do.
- Count misses per week. The target is zero. A clean week with few changes proves little, because the audit only sees failures that changes caused, so judge the selector over many changes.
Pie Runs the End-to-End Side of the Check
Pie adds an end-to-end layer that works alongside your QA team and your unit tests. Our agentic testing platform explores your app and writes tests for web, iOS, and Android, so the suite you audit does not wait on someone to script it.
- Test the change itself: Comment
@pie-pr-bot teston a GitHub pull request or/pieon GitLab, and Pie tests the change. It needs something to run against: a preview URL for a web app, or an uploaded build for a mobile app. - Run the end-to-end suite on a schedule: A Loop runs the suites you pick at the time you set, such as every night. That nightly run is the full-suite baseline for your end-to-end layer.
A Selector Is a Bet You Can Audit
Picking tests per change pays for itself, and every method is a bet on what a change can touch. Coverage maps lose to config files, path rules lose to shared code, and risk lists lose to last quarter’s product.
Shadowing each pick with a full run, and counting what the selector missed, keeps the bet honest. Ours missed tests three ways, and each miss named its own fix.
A skipped test is a promise that it would have passed. Check the promise.
Run the End-to-End Suite Nightly
Put the end-to-end run on a Loop. Keep your selector for unit tests.