How-To

How to Test Real-Time and WebSocket Features End-to-End (Without Flaky Tests)

Real-time tests flake because teams treat a pushed message like a request they can wait out. Here's the architecture that makes WebSocket flows deterministic in CI.

Adithya Aggarwal

CTO & Co-founder at Pie

11 min read Posted Sep 22, 2026

A WebSocket stays open so the server can send you something you never asked for. RFC 6455 puts it plainly: each side “can, independently from the other, send data at will.” That is the entire point of the transport, and it is also why the chat test is the one nobody on your team trusts.

Your test code was built for the opposite arrangement. It calls something, then waits for the answer to that call. When Luo and colleagues classified flaky-test fixes across 51 open-source projects in their 2014 empirical study, “Async Wait” was the largest category by a wide margin, 74 of the 161 commits they were able to classify.

A chat message, a live notification, a presence dot going green, a collaborative cursor moving. Every one of those is a message the server pushes on its own schedule, and your test has to be ready for it without knowing when it lands. Treat that pushed message like a request you can wait out, and you have built a flaky test on purpose. The architecture below makes the flow deterministic instead.

Why Real-Time Tests Flake and Ordinary Waits Cannot Fix It

Ordinary test code is built for events the client starts. It calls an endpoint and waits for the response to that call, and the waiting is free because the request itself is the thing being awaited. A persistent socket removes that anchor. Once the connection is open the server can push a frame at any moment, and your test has no request to hold on to, so teams reach for the one thing that looks like waiting and hard-code a duration.

That duration is a guess about the socket’s latency on a machine you do not control. Guess low and the test is red on a busy CI runner. Guess high and every real-time test in the suite pays the tax on every run, which is how a chat spec quietly becomes the slowest file in the repo.

The second problem is worse, and it is the one that costs you an afternoon. When a real-time test goes red, split it before you touch anything. Did the frame never arrive, or did it arrive and the UI never rendered it?

A broken broadcast and a broken message handler fail identically under a fixed sleep, because all the sleep can tell you is that the screen was still empty when the timer expired. The instrument that hides the difference is the same instrument that caused the flake.

Three Pieces That Make Socket Tests Deterministic

Three pieces turn a real-time flow from a coin flip into a deterministic test. A real connection, more than one client, and a wait on observable state rather than on time. Each one closes a gap a fixed sleep leaves open.

1. A Real Connection, Not a Mock

Test end to end against an actual socket, not a stub that replays a canned frame. The parts of a real-time feature that break in production, reconnection after a dropped connection, message ordering under load, duplicate delivery, all live where the connection itself lives. A mock replays a perfect frame every time and never exercises any of them. Keep mocked sockets for unit tests of a single handler, and let the end-to-end test hold a live connection open.

2. Two Clients, Because Real-Time Is Multiplayer

Most real-time features exist to move state between users, so a single-client test proves almost nothing. Open two independent contexts, send from one, and assert the push arrives on the other. In Playwright that is two browser contexts in one test, which its own docs recommend for exactly this, and in a native mobile suite it is two sessions. The test now exercises the real broadcast path instead of a local echo, which is where presence, typing indicators and shared documents actually fail.

3. A Wait on State, Not on Time

Wait for the result the message causes, the new row, the incremented badge, the cursor that moved, and poll for it up to a ceiling. It is the single piece that removes the flake, and it gets the full treatment with code further down.

Six Steps to Test a Real-Time Message Flow

Put those three pieces together on a chat feature and a real-time message test falls out as six concrete steps. User A sends a message, and user B, connected on a separate session, should see it appear without a refresh. Run it against a live connection, with the assertion on user B’s screen.

1. Open two independent clients: Spin up two contexts or sessions, one for the sender and one for the receiver, each with its own socket. Never share a context, or you are testing a local echo instead of a broadcast.
2. Confirm both connections are open: Wait until each client shows a connected state, the presence dot, the “online” label or a ready flag, before acting. Sending before the receiver’s socket is open is a classic false failure, and every multi-user test stacked on top of it inherits the flake.
3. Act as the sender: As user A, type a message carrying a unique marker such as a UUID, and submit. The marker is what makes the later assertion exact instead of “some message appeared.”
4. Assert on the receiver: On user B’s screen, wait for the message bubble carrying that marker to appear. Use a retrying assertion with a timeout, never a sleep followed by a check.
5. Assert the sender’s own view: Confirm user A sees the message move from pending to sent, which catches the case where the broadcast works but the sender’s optimistic update never reconciles.
6. Assert ordering and count: Send a short burst and assert both clients show the same messages in the same order, which is where naive real-time implementations quietly drop or reorder frames.

The whole value sits in step 4 landing on the other client. A single-client chat test can pass while the broadcast is completely broken, because the sender’s optimistic UI shows the message locally whether or not it ever left the device. The same discipline governs the rest of your end-to-end auth and session tests, and once this flow waits on state it should be every bit as boring and green as they are.

Four Real-Time Failures the Happy Path Never Reaches

The happy path is the easy half. Real-time features fail in four specific places, and those are exactly the places a fixed-sleep test can never reach. None of them is exotic. They are what every user on a subway or a spotty office wifi triggers daily, so test each as an explicit case and force the condition rather than waiting for it.

• Reconnection: Cut the connection through a network-control hook or by closing the socket from a test seam, then assert the client reconnects and that messages sent during the outage are delivered or resynced afterward. This is the highest-value case, because reconnection is the code path mocks never cover and users hit constantly.
• Message ordering: Push several messages in quick succession and assert both clients render them in send order. Systems that fan out through multiple workers can deliver out of order under load.
• Duplicate delivery: Delivery guarantees vary by transport and most of them are configurable, so do not assume the one you are on rules out a replay. Assert that the client de-duplicates a repeated frame instead of showing the message twice.
• Stale state on reload: Reload user B mid-conversation and assert the history rehydrates correctly, which catches the gap between the socket stream and the initial fetch.

Forcing these conditions is the same discipline as isolating a test from shared state. Reach the state you need directly through a test seam, rather than driving the whole system in real time and hoping it lands. A reconnection you trigger on demand is a deterministic assertion. A reconnection you wait for is another flaky sleep.

Four Ways to Assert on a Pushed Event, Compared

Every step above hinged on one decision. How do you wait for a message you did not request? There are four common answers, and they differ on the three things CI actually cares about. Does the wait return as soon as the event lands, does it survive a redesign, and does it prove anything rendered.

The fixed-sleep row fails the only column that decides your CI time, so it is either slow or red and usually both. The interesting row is the third one.

Listening on the socket is fast, precise and completely redesign-proof, which is why it is the answer most engineers reach for second. It also proves that a frame arrived and nothing else. Break the client’s message handler so the bubble never renders, and the frame-level assertion stays green while every user sees a dead chat window.

A retrying assertion on the UI is the right default in framework code, and Playwright’s WebSocket class lets you hold both ends at once, asserting that the frame was sent and that the interface changed because of it. The last row, waiting on visible state, is how a person verifies real-time and how Pie does it. The assertion is what the user actually sees, so it survives the selector churn that eventually breaks row two.

Replace Every Fixed Sleep With a Condition

Whichever surface you assert on, one function decides the outcome, and it is the one that encodes your guess about latency. A poll waits for the actual condition, that the message exists, up to a ceiling. It returns the instant the frame lands and fails only if the frame genuinely never arrives.

Every serious framework already treats fixed waits as an anti-pattern. Playwright’s web-first assertions re-check the condition until it is met or the timeout is reached, rather than pausing. Apply the same rule to a pushed event, whether you observe it on the socket or in the DOM.

// Anti-pattern: a fixed guess. Slow when the socket is fast, red when it is slow.
await page.waitForTimeout(3000);
expect(await getMessages(page)).toContain(marker);

// Deterministic: wait for the specific frame, with a ceiling.
async function waitForMessage(socket, predicate, { timeout = 15000 } = {}) {
  return new Promise((resolve, reject) => {
    const onMessage = (raw) => {
      const msg = JSON.parse(raw);
      if (predicate(msg)) {            // resolves the instant it arrives
        cleanup();
        resolve(msg);
      }
    };
    const timer = setTimeout(() => {
      cleanup();
      reject(new Error(`No matching message within ${timeout}ms`));
    }, timeout);
    // Detach on both exits, or every call leaves a listener behind and they pile up.
    const cleanup = () => {
      clearTimeout(timer);
      socket.off('message', onMessage);
    };
    socket.on('message', onMessage);
  });
}

// Or assert on the rendered result, which auto-retries until the bubble appears.
await expect(receiver.getByText(marker)).toBeVisible({ timeout: 15000 });

The difference is not stylistic. The sleep version couples your pass rate to the socket’s worst-case latency on a loaded CI runner, and the condition version is indifferent to it. One change moves a real-time test out of the retry-until-green pile, which is the whole point of how to fix flaky tests instead of retrying around them.

How Pie Runs a Real-Time Test, and Where It Stops

Everything above is the pattern, and none of it belongs to any one tool. Pie approaches it from a different direction, and the difference shows up at exactly the step this post is about.

The agent does not run a script against your app. It takes a screenshot at each step, and a model decides what is on the screen and what to do next, almost like a new hire working the app for the first time and checking whether the thing they expected actually showed up.

So the wait for a pushed message is never a duration. It is a question asked of the screen, over and over, until the answer is yes. Three things follow from that design.

1. The wait is on visible state: Instead of a hard-coded pause, the model looks at each screenshot and decides ready or not ready yet, so a step advances the moment the message bubble renders and not a second before.
2. You still set the ceiling: A step can carry its own timeout assertion, so a screen that has not updated in fifteen seconds fails the test instead of hanging on it.
3. Nothing is anchored to a selector: There is no locator for you to write or maintain, so the same autonomous test survives the redesign that would break the retrying-assertion row in the table above.

For the setup around the flow, execution scripts make API or shell calls before, during or after a run. That is the seam you would use to put an account in the right state, or to have your own backend close a connection at the moment you want the reconnect path exercised. The end-to-end run flow in the docs walks through what happens at each step, and how Pie works covers the layer above it.

Two limits, stated plainly, because a real-time feature will find both of them. Two agents cannot coordinate inside one session yet.

A user A to user B handoff has to be authored as a sequential test, logging in as A, acting, then logging in as B to verify what landed, and autonomous discovery will not find that flow on its own. Anything that needs both users live in the same instant, a presence dot, a typing indicator, a shared cursor, is out of scope for now. Everything single-session runs fine.

The second limit is smaller and easier to design around. A toast that flashes for a moment can fall between screenshots, so assert on something that persists, the message sitting in the thread rather than the banner that announced it.

A test case is written in plain English rather than code, and it reads close to the six steps above. The #{...} call runs a registered execution script and @{...} injects a stored credential. Everything between the square brackets is a step the agent performs and verifies on screen.

#{reset_thread}
[log in as @{user_a}]
[send message "hello-{uuid}" to the shared thread]
[verify the message shows as sent, not pending]
[log out]
[log in as @{user_b}]
[verify message "hello-{uuid}" is visible in the thread]
[verify the message appears once, not twice]

TLDR on the section. Pie takes the timer and the selector out of a real-time test. It does not yet take out the second live user.

Synchronize on State, Not the Clock

You already refuse to guess at timing everywhere else in your suite. You await a response instead of sleeping through a request, and you poll an inbox instead of betting on delivery. Real-time is the one place teams still reach for a fixed sleep, which is exactly why the chat test is the one nobody trusts.

Give the socket a real connection, a second client, and a wait on the state the message produces instead of on the clock. Do that and the flakiest feature in your app stops being the test you rerun until it goes green.

Pie takes the timer and the selector out of that test, because the agent waits on what is actually on the screen. After that, the socket can be as slow as it likes.

Adithya Aggarwal

CTO & Co-founder at Pie

Eight years building search and delivery systems at Amazon. The kind of scale where flaky tests block billion-dollar releases. Now CTO at Pie, building AI agents that adapt when your UI changes. LinkedIn →