How-To

How to Test Webhooks End-to-End (Without Flaky Tests)

Most webhook guides prove the callback arrived. They never prove it did its job. Here is the architecture that tests the outcome, deterministically, in CI.

Adithya Aggarwal

CTO & Co-founder at Pie

11 min read Posted Sep 22, 2026

A webhook is not a request your server answers. It is a callback that arrives later, on the provider’s schedule, to tell you something already happened somewhere else. Search for how to test one and the first page of results teaches the same drill. Expose localhost with a tunnel, watch the payload land, confirm your endpoint returned 200.

The drill tests delivery. It never tests the promise the webhook was making. The payment succeeded, so the subscription goes active. The repo got a push, so the build kicks off.

Asserting the 200 is almost like signing for a package without opening it. You have proof it arrived and no idea whether what is inside is what you ordered.

The test that matters proves the outcome, and it has to prove it deterministically, in CI, with no public tunnel and no fixed sleep. Here is how we build that test, and where it still bites.

Why Webhook Tests Flake (and Why 200 OK Is Not a Test)

Webhook tests flake because the thing you want to assert happens after the request your test can see, and somewhere else. Your test triggers an event on a third party. The third party sends a callback on its own schedule.

Your handler then has to acknowledge that callback quickly or the provider counts the delivery as failed and retries it, so the real work almost always goes on a queue. The subscription flipping to active lands last, and a test that checks for a 200 is measuring the first domino and calling the game.

Four failure modes hide in that gap, and every one of them is an architecture gap in the test rather than a bug in your app:

• Asynchronous callback: The event does not arrive on a fixed timeline. A test that sleeps and then asserts is betting the callback beats the clock. Under CI load it loses, and the test goes red on state that simply had not settled yet.
• Tunnel dependency: The standard local setup exposes your machine with ngrok or a similar tunnel. It works at a desk. In CI it means a public URL that changes every run, has to be registered with the provider, and collides the moment two workers run in parallel.
• Asserting on delivery, not outcome: Returning 200 acknowledges receipt. It says nothing about whether the handler parsed the payload, verified the signature, and committed the right change. Teams ship a broken handler behind a green suite for exactly this reason.
• Duplicate and out-of-order delivery: Providers retry. A test that assumes exactly-once, in-order delivery passes on a clean run and breaks the day production sends the same event twice.

Close all four and webhook tests stop being the ones you re-run until they pass. They become what a flaky test in CI always was, an architecture problem rather than a retry problem.

Assert the Outcome, Not the Delivery

The single change that fixes webhook testing is moving the assertion from the callback to its consequence. Do not assert that /webhooks/stripe returned 200. Assert that the user who just paid can now use the feature they paid for. The webhook is plumbing between those two facts, and plumbing is worth testing only by what comes out the other end.

Moving the assertion downstream reframes the whole test. A delivery test needs to see the raw HTTP request, which is why every guide reaches for a tunnel or a request inspector.

An outcome test does not care about the request at all. It triggers the source event, then watches your application’s own state, the database row, the API response, the screen, until the expected change appears. The callback becomes an implementation detail your test never has to intercept.

Outcome-first is also the honest definition of end-to-end. Testing that a POST reached your server is an integration test wearing an E2E costume. Testing that a real event produces a real user-visible result is the actual journey, and it is the one that catches the wrong-environment endpoint, the missed signature check, and the handler that quietly swallows an error.

Four Pieces That Make Webhook Tests Deterministic

An outcome-first webhook test has four moving parts. Get these four right and the flake has nowhere to live.

1. A Way to Trigger the Real Event on Demand

You need to fire the source event from your test rather than wait for one to happen. Most providers ship a first-party way to do it. The Stripe CLI has stripe trigger, GitHub lets you redeliver a past delivery from the web interface or the REST API, and many platforms expose a send-test-event button. First-party triggers matter because they emit the provider’s exact payload, headers and all.

The limit worth naming up front is that not every provider ships one. If yours does not, you fall back to a hosted gateway or a hand-built request, and you give up either the CI story or the real payload shape. Check for a CLI before you design the rest of the test around having one.

2. A Capture Point Your App Actually Runs Behind

The callback has to reach a running instance of your app, not a bin that only displays it. In CI that means the provider CLI forwarding to your local server, or a hosted relay routing to it. The capture point’s only job is to get a genuine, signed event into your real handler.

3. An Assertion on the Side Effect

Query your own system for the change the event should have caused. A subscription status, an order marked paid, a verified flag. Only that assertion means anything, and it belongs on state you own and can read directly.

4. Control Over Signature and Time

Signature verification and token or event expiry are security features, so you test them by controlling both. Sign a payload yourself to test rejection, and age an event through a test-only hook to test staleness, rather than waiting on real minutes.

Testing a Webhook End-to-End in Five Steps

Put the four pieces together on the flow every subscription business lives or dies on, where a customer pays and their account has to unlock. Using Stripe as the concrete example, the test is five steps.

1. Point the callback at your running app: Start your server, then forward events to it with the provider CLI. No tunnel to register, no public URL to rotate.

# Forward Stripe events to your local endpoint, signed with the secret the CLI prints.
stripe listen --forward-to localhost:3000/webhooks/stripe

2. Drive the flow that should cause the event: Complete the checkout in the UI with a test card, or create the subscription through the API. Skipping this user action is exactly what makes a delivery-only test hollow.
3. Let the real event fire, or bind the triggered one to your user: The checkout in step 2 emits payment_intent.succeeded on its own, and that delivery is the one this test should ride. If you fire the event on demand instead, bind it to the user you are about to assert on with stripe trigger payment_intent.succeeded --add payment_intent:customer=cus_xxx. A bare trigger builds its own fixtures, so the event arrives for a customer your app has never seen, the handler runs, and the plan never flips. Trigger commands also emit companion events on the way to the one you asked for, so match on the event type rather than assuming a single delivery.
4. Poll for the side effect: Query your own API or database until the subscription reads active, up to a ceiling. Not a fixed wait. The exact pattern is in the polling section below.
5. Assert the user-visible outcome: Confirm the account now has access, so the pro-only route loads, the entitlement flag is set, and the paywall is gone.

Step five is the point of the whole exercise. Subscription apps live and die on the paywall flow, and the webhook is the seam where “paid” becomes “provisioned.” A broken handler there is silent revenue loss, invisible to any test that stopped at the 200. It belongs in the same suite as testing 2FA and OTP flows, one more leg of the end-to-end testing you already run, and once it asserts on the outcome it should be just as boring and green as the rest.

Five Ways to Capture the Inbound Webhook, Compared

Step one hinged on getting a real event into your handler. There are five common ways to do it, and they differ most on what CI actually cares about. Does it run cleanly under parallelism, does it deliver the provider’s own payload, and can your test trigger an event on demand.

The two rows most guides lead with, the tunnel and the request inspector, are development tools rather than test infrastructure. A tunnel gives you a public URL to watch events by hand and does not survive CI parallelism. A request inspector shows you the payload but never runs your handler, so it cannot test an outcome at all.

For automated end-to-end tests the provider CLI is the workhorse, because it forwards genuinely signed events with no public URL and triggers them on command. A hosted gateway like Hookdeck or Svix earns its keep once you are ingesting from many providers at volume. Keep the hand-built POST for one job, the fast offline signature and negative-path tests where you want to fabricate a bad event on purpose. TLDR, provider CLI for the flow, hand-built POST for the security cases, and nothing else in the pipeline.

Four Edge Cases That Break in Production

The happy path is the easy part. Webhook incidents come from the cases teams never wrote a test for, and there are four worth making explicit.

Signature verification. An endpoint that trusts any POST is an open door for a forged event that grants access or credits an account. Stripe’s own reference handler answers a signature verification error with a 400, so test both sides against that:

// Forged or missing signature must be rejected.
const bad = await postWebhook(payload, { signature: 't=1700000000,v1=deadbeef' });
expect(bad.status).toBe(400);

// Only a correctly signed event is accepted and processed.
const good = await postWebhook(payload, { signature: sign(payload, secret) });
expect(good.status).toBe(200);

Duplicate delivery and idempotency. Stripe warns that an endpoint might occasionally receive the same event more than once, and Amazon SNS says the same about its own distributed delivery. Replay one event twice and assert the side effect happens exactly once. A non-idempotent handler double-charges the first time a retry lands.

Retries and failure. When your endpoint errors or times out, the provider retries. Stripe keeps attempting delivery for up to three days with exponential backoff in live mode, while a sandbox gives up after a handful of attempts over a few hours, which is a gap worth knowing before you tune a CI timeout to the wrong number. Return a 500 on purpose, confirm the retry lands, and confirm the eventual success is still processed once.

Out-of-order events. Stripe does not guarantee that events arrive in the order they were generated. Deliver subscription.updated before subscription.created and assert your final state is still correct. It is the same class of non-deterministic test that makes any concurrent system hard to reason about, and the fix is the same. Assert on the settled end state rather than the sequence.

The Polling Pattern That Kills the Flake

Whichever capture method you chose, one line still decides whether the test is reliable, and it is how you wait for the side effect. The fixed sleep is what makes webhook tests flaky. It encodes a guess about how long the callback plus the async handler will take, and when the guess is wrong in either direction the test is slow or red. Polling waits for the actual condition and returns the instant it is true.

Test frameworks solved this inside the browser years ago. Playwright’s auto-retrying assertions re-fetch the element and check it over and over, in the docs’ own words, “until the condition is met or until the timeout is reached,” and the same page warns that non-retrying assertions on asynchronous pages lead to flaky tests. Apply that rule one layer out, to your own application state:

// Anti-pattern: guess how long the webhook takes, then hope.
await page.waitForTimeout(5000);
const sub = await api.getSubscription(userId);
expect(sub.status).toBe('active');     // races the callback under CI load

// Deterministic: poll for the real side effect, with a ceiling.
async function waitForState(check, { timeout = 20000, interval = 500 } = {}) {
  const deadline = Date.now() + timeout;
  while (Date.now() < deadline) {
    const result = await check();
    if (result) return result;         // returns the instant the effect lands
    await new Promise((r) => setTimeout(r, interval));
  }
  throw new Error(`Side effect never settled within ${timeout}ms`);
}

// Trigger the real event, then wait for what it was supposed to do.
await completeCheckout(userId);
const active = await waitForState(async () => {
  const sub = await api.getSubscription(userId);
  return sub.status === 'active' ? sub : null;
});
expect(active.plan).toBe('pro');

The difference is not stylistic. The sleep version couples your pass rate to the provider’s worst-case latency on a loaded runner. The polling version is indifferent to it, because it returns as soon as the state you actually care about is real.

The failure message earns its keep too. A timed-out poll has already split the problem in half for you. Either the event never reached your handler, which is a capture problem, or it reached it and the handler never committed, which is a business-logic problem.

Two different owners, two different fixes, and you know which one before you open a log. A blown sleep tells you none of that, which is the whole reason fixing flaky tests beats retrying around them.

How Pie Tests Webhook-Driven Flows

Everything above is the pattern, and most of it is wiring you build once and then keep alive. The two pieces that rot fastest are the selectors on the pages either side of the callback and the wait in the middle.

Pie is an autonomous QA platform that drives your web, iOS, and Android app the way a person would, which is why how Pie works on a webhook-driven feature looks nothing like intercepting a request. The test starts where the user starts, at the trigger action, and ends where the user notices, at the visible result.

Three phases:

1. Drive the trigger flow: The agent completes the checkout or performs the action that emits the event, confirming each screen as it goes. You never write or maintain a selector, so a redesign of the pricing page does not turn into a red test the next morning.
2. Wait on the observed outcome: Instead of sleeping for a guessed duration, the agent re-checks the running app after each step and moves on when the state it is waiting for is genuinely there.
3. Verify the user-visible result: The agent confirms the feature the webhook was supposed to enable is now usable, which is the assertion that actually protects revenue.

Worth naming the boundary, because it decides whether this fits your case. Pie asserts on what a user can see. If your webhook’s only consequence is a row in a table with no interface anywhere, that assertion belongs in the polling helper above rather than in a UI test. Where the webhook does surface as a plan badge, an unlocked route, or an order status, you are testing the exact thing a customer would file a ticket about.

A test case reads about like this. The checkout with a test card is the trigger, so the event arrives for the exact customer the agent just created and there is no separate stripe trigger step to keep in sync with it:

[open app on the pricing page]
[start the Pro subscription checkout with a test card]
[verify the "processing payment" state appears]
[wait until the account shows the Pro plan as active]
[verify a Pro-only feature is now unlocked]

The same end-to-end run that covers the rest of your app covers the webhook leg, including the native case where provisioning has to show up in an iOS or Android build. You describe the flow once and the visual layer absorbs the UI churn that breaks selector-based tests on every redesign.

Test the Outcome

Webhooks are the one integration where the interesting behavior happens after the request you can see, which is why so many suites end up testing the wrong half. But the 200 never lost anyone a customer. A subscription that failed to provision after a successful charge did.

Give the webhook the same treatment you give every other dependency. Trigger the real event, capture it behind your real handler, and assert on the side effect by polling for it instead of sleeping.

We built Pie to hand you that leg by default. Drive the trigger, wait on the real outcome, no tunnel and no fixed sleep to flake on. Describe the flow once and stop babysitting it.

Adithya Aggarwal

CTO & Co-founder at Pie

Eight years building search and delivery systems at Amazon. The kind of scale where flaky tests block billion-dollar releases. Now CTO at Pie, building AI agents that adapt when your UI changes. LinkedIn →