Recover a dead host connection within seconds of returning to the foreground (#9086)

* fix(client): recover a dead connection within seconds of returning to the foreground

A socket that died while the OS had JS suspended stays OPEN and keeps accepting
sends, so nothing but inbound frames can prove it alive. Since #8651 the
foreground check rode the regular heartbeat: a 15 s window, dedupe onto a probe
that straddled the suspension, misses from before background carried over, and
a 1.5 s backoff after teardown. Measured at the transport: 40 s to tear down and
41.5 s to reconnect, with 15 s to 30 s variations depending on what was in
flight when the app was suspended.

Foreground now starts a fresh cycle inside the same heartbeat owner: two 5 s
windows back to back, then teardown and an immediate reconnect. An outstanding
heartbeat probe is dropped without a miss because it measured the suspension,
not the socket; a latency measurement keeps its own deadline and is shared.
Repeated foreground events join the running cycle instead of restarting it.
The rule that any validated inbound frame voids a window is unchanged, so a
slow multi-frame transfer still survives, and after any progress the heartbeat
returns to its 10 s / 15 s cadence. Ordinary heartbeat teardowns keep their
backoff; only the foreground cycle reconnects at once, and a failed immediate
attempt backs off as before.

A single frame that takes longer than the two windows to arrive after
foreground is still torn down. That needs smaller daemon-to-client frames and
is documented as a known limit.

* fix(client): count only fresh evidence in the foreground liveness cycle

Two gaps in the foreground cycle let stale evidence decide a fresh check.

A latency measurement outstanding at foreground is shared by the check. If a
frame had arrived during that measurement before the app was suspended, the
check read that old activity as "alive" and went back to the normal cadence
without running its windows. The check now clears the shared probe's activity
flag when it begins, so only frames that arrive from then on count.

If the app was suspended again inside the second window, the next resume
either found the queued expiry already fired (teardown before AppState) or
deduped onto the old probe and let the expiry fire right after (teardown with
no fresh window). A foreground window whose timer fires well past its deadline
on the wall clock spanned a suspension; it is now discarded together with any
activity it saw and the misses before it, and the cycle starts over. The wall
clock is used because the monotonic clock may not advance while the OS has the
process suspended.

* fix(client): keep latency measurements alive when foreground replaces a heartbeat

A latency measurement that found a heartbeat ping outstanding joins that
probe. The foreground check used to fail the probe outright to start its
fresh window, which rejected the measurement before its own deadline on a
healthy socket; the diagnostics collector treats that as a failed
collection. The superseded probe is now detached without settling and
takes the outcome of the fresh window, so the heartbeat tick and any
measurement waiting on it see the next pong or the window's timeout
instead of a spurious error.

* fix(client): let latency measurements wait for the next pong instead of a probe

A measurement that joined an outstanding heartbeat probe took that
probe's fate. The foreground check replaces such a probe, so the
measurement failed early: first with the supersede error, then, after
the hand-off to the fresh window, with that window's five-second timeout
even when the caller had asked for a longer deadline and the pong was
about to arrive.

Measurements now register as pong waiters. A waiter settles on the next
pong, on a connection failure (close, transport dispose, ping send
failure) or on its own deadline, and no longer cares which probe put the
ping on the wire. The foreground check goes back to dropping the
straddling heartbeat probe outright, since the only thing waiting on a
probe's promise is the heartbeat tick.

* fix(client): report the joined ping's round trip from a latency measurement

A measurement that joins an in-flight heartbeat ping used to start its
clock at the join, so it reported only the remaining wait for the pong.
The clock now starts at the send of the ping the measurement waits on,
which matches what the heartbeat records for the same pong.
This commit is contained in:
Mohamed Boudra
2026-11-08 14:01:51 +07:00
committed by GitHub
parent ac08077f7a
commit f539ccf1c0
3 changed files with 581 additions and 69 deletions
+439 -21
View File
@@ -728,6 +728,10 @@ class FakeDaemon {
this.onClose({ reason });
}
daemonSendsPong(): void {
this.onMessage(JSON.stringify({ type: "pong" }));
}
triggerError(event?: unknown): void {
this.onError(event);
}
@@ -2605,7 +2609,7 @@ test("keeps default connect timeout shorter than session RPC waiters", async ()
}
});
test("foreground checks a silently broken socket through two normal heartbeat misses", async () => {
test("foreground tears down a silently dead socket after two fresh five-second windows and reconnects at once", async () => {
useHeartbeatClock();
const first = new FakeDaemon();
const second = new FakeDaemon();
@@ -2623,14 +2627,23 @@ test("foreground checks a silently broken socket through two normal heartbeat mi
await connection;
client.ensureConnected({ verify: true });
await vi.advanceTimersByTimeAsync(15_000);
expect(first.pingTimestamps()).toEqual(["0s"]);
await vi.advanceTimersByTimeAsync(5_000);
// The first silent window is followed by a second check at once, not ten seconds later.
expect(first.closesFromClient()).toEqual([]);
await vi.advanceTimersByTimeAsync(25_000);
expect(first.pingTimestamps()).toEqual(["0s", "5s"]);
await vi.advanceTimersByTimeAsync(4_999);
expect(first.closesFromClient()).toEqual([]);
await vi.advanceTimersByTimeAsync(1);
expect(first.closesFromClient()).toEqual([{ code: 1001, reason: "Liveness check timed out" }]);
client.ensureConnected();
// The reconnect is immediate: no 1.5s backoff after a foreground liveness teardown.
expect(attempts).toBe(2);
second.openConnection();
expect(client.getConnectionState()).toEqual({ status: "connected" });
// The replacement runs the normal heartbeat cadence.
await vi.advanceTimersByTimeAsync(10_000);
expect(second.pingTimestamps()).toEqual(["20s"]);
expect(second.closesFromClient()).toEqual([]);
});
test("foreground verification preserves a healthy socket and deduplicates simultaneous checks", async () => {
@@ -2675,22 +2688,26 @@ test("chunks prevent heartbeat misses until silence causes two misses", async ()
void client.measureLatency({ timeoutMs: 5_000 }).catch((error: unknown) => {
measurementError = error;
});
await vi.advanceTimersByTimeAsync(5_000);
// Chunks complete every four seconds for the first 36 seconds; no pong ever arrives.
for (let index = 0; index < 9; index++) {
await vi.advanceTimersByTimeAsync(4_000);
daemon.receiveChunk(index === 0);
expect(daemon.closesFromClient()).toEqual([]);
}
expect(measurementError).toEqual(
expect.objectContaining({
message: "Latency measurement timed out (5000ms)",
}),
);
for (let index = 0; index < 4; index++) {
await vi.advanceTimersByTimeAsync(index === 0 ? 5_000 : 10_000);
daemon.receiveChunk(index === 0);
expect(daemon.closesFromClient()).toEqual([]);
}
// No complete message or pong has arrived, but progress prevents misses.
expect(daemon.pingTimestamps()).toEqual(["0s", "25s"]);
await vi.advanceTimersByTimeAsync(25_000);
// Progress inside the foreground window returned the heartbeat to its normal cadence.
expect(daemon.pingTimestamps()).toEqual(["0s", "15s"]);
await vi.advanceTimersByTimeAsync(19_000);
expect(daemon.closesFromClient()).toEqual([]);
await vi.advanceTimersByTimeAsync(25_000);
expect(daemon.pingTimestamps()).toEqual(["0s", "15s", "40s"]);
await vi.advanceTimersByTimeAsync(24_999);
expect(daemon.closesFromClient()).toEqual([]);
await vi.advanceTimersByTimeAsync(1);
expect(daemon.pingTimestamps()).toEqual(["0s", "15s", "40s", "65s"]);
expect(daemon.closesFromClient()).toEqual([{ code: 1001, reason: "Liveness check timed out" }]);
});
@@ -2712,13 +2729,14 @@ test("foreground preserves a multi-MB transfer with its pong queued behind chunk
client.ensureConnected({ verify: true });
await vi.advanceTimersByTimeAsync(3_001);
expect(daemon.closesFromClient()).toEqual([]);
for (let index = 0; index < 4; index++) {
await vi.advanceTimersByTimeAsync(10_000);
for (let index = 0; index < 5; index++) {
await vi.advanceTimersByTimeAsync(4_000);
daemon.receiveChunk(index === 0);
expect(daemon.closesFromClient()).toEqual([]);
}
// No complete message or pong has arrived, but progress prevents misses.
expect(daemon.pingTimestamps()).toEqual(["0s", "25s"]);
// The first chunk landed seven seconds after foreground: the first window was silent, the
// second saw progress, so the cycle ended without a teardown and the cadence is normal again.
expect(daemon.pingTimestamps()).toEqual(["0s", "5s", "20s"]);
daemon.finishTransfer();
await vi.advanceTimersByTimeAsync(30_000);
expect(client.getConnectionState()).toEqual({ status: "connected" });
@@ -2742,13 +2760,14 @@ test("chunk progress beyond the daemon lease keeps sending heartbeat pings", asy
await connection;
client.ensureConnected({ verify: true });
for (let index = 0; index < 12; index++) {
await vi.advanceTimersByTimeAsync(5_000);
await vi.advanceTimersByTimeAsync(4_000);
daemon.receiveChunk(index === 0);
// Each completed window starts the next ping ten seconds later,
// even though the response and all queued pongs are still in flight.
const now = (index + 1) * 4;
const expected = ["0s"];
if (index >= 4) expected.push("25s");
if (index >= 9) expected.push("50s");
if (now >= 15) expected.push("15s");
if (now >= 40) expected.push("40s");
expect(daemon.pingTimestamps()).toEqual(expected);
expect(daemon.closesFromClient()).toEqual([]);
}
@@ -2758,6 +2777,393 @@ test("chunk progress beyond the daemon lease keeps sending heartbeat pings", asy
expect(daemon.closesFromClient()).toEqual([]);
});
function connectThroughReplacements(
clientId: string,
daemons: FakeDaemon[],
options?: { reconnect?: { enabled: boolean } },
): { client: DaemonClient; attempts: () => number } {
let attempts = 0;
const client = new DaemonClient({
url: "ws://test",
clientId,
logger: noopLogger,
...(options?.reconnect ? { reconnect: options.reconnect } : {}),
transportFactory: () => {
const daemon = daemons[Math.min(attempts, daemons.length - 1)]!;
attempts += 1;
return daemon.transport;
},
});
clients.push(client);
return { client, attempts: () => attempts };
}
test("foreground supersedes a heartbeat that straddled the suspension without counting a miss", async () => {
useHeartbeatClock();
const first = new FakeDaemon();
const second = new FakeDaemon();
const { client, attempts } = connectThroughReplacements("resume-before-expiry", [first, second]);
const connection = client.connect();
first.openConnection();
await connection;
first.daemonGoesSilent();
await vi.advanceTimersByTimeAsync(10_000); // heartbeat ping outstanding when the OS suspends JS
await vi.advanceTimersByTimeAsync(14_990); // resume: AppState lands before the stale timer
client.ensureConnected({ verify: true });
expect(first.pingTimestamps()).toEqual(["10s", "24.99s"]);
await vi.advanceTimersByTimeAsync(10); // the stale deadline passes without a miss
await vi.advanceTimersByTimeAsync(4_990);
expect(first.closesFromClient()).toEqual([]);
expect(first.pingTimestamps()).toEqual(["10s", "24.99s", "29.99s"]);
await vi.advanceTimersByTimeAsync(5_000);
expect(first.closesFromClient()).toEqual([{ code: 1001, reason: "Liveness check timed out" }]);
expect(attempts()).toBe(2);
});
test("foreground does not inherit a miss recorded before the app was suspended", async () => {
useHeartbeatClock();
const first = new FakeDaemon();
const second = new FakeDaemon();
const { client, attempts } = connectThroughReplacements("resume-after-expiry", [first, second]);
const connection = client.connect();
first.openConnection();
await connection;
first.daemonGoesSilent();
await vi.advanceTimersByTimeAsync(25_000); // ping at 10s missed at 25s, before or during suspension
client.ensureConnected({ verify: true });
expect(first.pingTimestamps()).toEqual(["10s", "25s"]);
await vi.advanceTimersByTimeAsync(5_000);
// Two fresh windows: the old miss would have torn down here.
expect(first.closesFromClient()).toEqual([]);
expect(first.pingTimestamps()).toEqual(["10s", "25s", "30s"]);
await vi.advanceTimersByTimeAsync(5_000);
expect(first.closesFromClient()).toEqual([{ code: 1001, reason: "Liveness check timed out" }]);
expect(attempts()).toBe(2);
});
test("a slow but healthy socket survives foreground after a miss before the suspension", async () => {
useHeartbeatClock();
const daemon = new FakeDaemon();
const { client, attempts } = connectThroughReplacements("resume-slow-pong", [daemon]);
const connection = client.connect();
daemon.openConnection();
await connection;
daemon.daemonGoesSilent();
await vi.advanceTimersByTimeAsync(25_000); // one miss
daemon.daemonAnswersPingsAfter("5.5s");
client.ensureConnected({ verify: true });
await vi.advanceTimersByTimeAsync(60_000);
expect(daemon.closesFromClient()).toEqual([]);
expect(client.getConnectionState()).toEqual({ status: "connected" });
expect(attempts()).toBe(1);
});
test("a pong that arrives after the first foreground window keeps the connection and restores the cadence", async () => {
useHeartbeatClock();
const daemon = new FakeDaemon();
daemon.daemonAnswersPingsAfter("5.5s");
const { client, attempts } = connectThroughReplacements("slow-pong", [daemon]);
const connection = client.connect();
daemon.openConnection();
await connection;
client.ensureConnected({ verify: true });
await vi.advanceTimersByTimeAsync(5_000);
expect(daemon.pingTimestamps()).toEqual(["0s", "5s"]);
await vi.advanceTimersByTimeAsync(55_000);
expect(client.getConnectionState()).toEqual({ status: "connected" });
expect(daemon.closesFromClient()).toEqual([]);
expect(attempts()).toBe(1);
// After the late pong the heartbeat runs ten seconds after each settled probe again.
expect(daemon.pingTimestamps()).toEqual(["0s", "5s", "15.5s", "31s", "46.5s"]);
});
test("a latency measurement outstanding at foreground keeps its own deadline, then the cycle runs", async () => {
useHeartbeatClock();
const first = new FakeDaemon();
const second = new FakeDaemon();
first.daemonGoesSilent();
const { client, attempts } = connectThroughReplacements("latency-then-foreground", [
first,
second,
]);
const connection = client.connect();
first.openConnection();
await connection;
const measurement = client.measureLatency({ timeoutMs: 1_000 });
const measurementError = measurement.then(
() => null,
(error: unknown) => error,
);
client.ensureConnected({ verify: true });
expect(first.pingTimestamps()).toEqual(["0s"]);
await vi.advanceTimersByTimeAsync(1_000);
await expect(measurementError).resolves.toEqual(
expect.objectContaining({ message: "Latency measurement timed out (1000ms)" }),
);
expect(first.closesFromClient()).toEqual([]);
await vi.advanceTimersByTimeAsync(5_000);
expect(first.pingTimestamps()).toEqual(["0s", "1s", "6s"]);
expect(first.closesFromClient()).toEqual([]);
await vi.advanceTimersByTimeAsync(5_000);
expect(first.closesFromClient()).toEqual([{ code: 1001, reason: "Liveness check timed out" }]);
expect(attempts()).toBe(2);
});
test("a shared latency probe's activity from before foreground does not satisfy the fresh check", async () => {
useHeartbeatClock();
const first = new FakeDaemon();
const second = new FakeDaemon();
first.daemonGoesSilent();
const { client, attempts } = connectThroughReplacements("stale-activity", [first, second]);
const connection = client.connect();
first.openConnection();
await connection;
void client.measureLatency({ timeoutMs: 5_000 }).catch(() => {});
first.receiveChunk(true); // a frame arrived while the measurement was outstanding, then the socket died
client.ensureConnected({ verify: true });
expect(first.pingTimestamps()).toEqual(["0s"]);
await vi.advanceTimersByTimeAsync(5_000);
// The measurement's own deadline passed; the foreground cycle starts its first fresh window now.
expect(first.closesFromClient()).toEqual([]);
expect(first.pingTimestamps()).toEqual(["0s", "5s"]);
await vi.advanceTimersByTimeAsync(5_000);
expect(first.pingTimestamps()).toEqual(["0s", "5s", "10s"]);
await vi.advanceTimersByTimeAsync(5_000);
expect(first.closesFromClient()).toEqual([{ code: 1001, reason: "Liveness check timed out" }]);
expect(attempts()).toBe(2);
});
test("a latency measurement that joined a heartbeat probe survives the foreground check", async () => {
useHeartbeatClock();
const daemon = new FakeDaemon();
daemon.daemonAnswersPingsAfter("2s");
const { client, attempts } = connectThroughReplacements("latency-joins-heartbeat", [daemon]);
const connection = client.connect();
daemon.openConnection();
await connection;
await vi.advanceTimersByTimeAsync(10_000); // the first heartbeat ping is outstanding
const measurement = client.measureLatency({ timeoutMs: 5_000 });
await vi.advanceTimersByTimeAsync(1_000);
client.ensureConnected({ verify: true });
expect(daemon.pingTimestamps()).toEqual(["10s", "11s"]);
// The heartbeat's pong lands at 12s and answers the measurement that joined it.
await vi.advanceTimersByTimeAsync(1_000);
await expect(measurement).resolves.toBe(2000);
await vi.advanceTimersByTimeAsync(19_000);
expect(client.getConnectionState()).toEqual({ status: "connected" });
expect(daemon.closesFromClient()).toEqual([]);
expect(attempts()).toBe(1);
expect(daemon.pingTimestamps()).toEqual(["10s", "11s", "22s"]);
});
test("a joined latency measurement outlives a silent foreground window until its own deadline", async () => {
useHeartbeatClock();
const daemon = new FakeDaemon();
daemon.daemonAnswersPingsAfter("7s");
const { client, attempts } = connectThroughReplacements("latency-outlives-window", [daemon]);
const connection = client.connect();
daemon.openConnection();
await connection;
await vi.advanceTimersByTimeAsync(10_000); // the first heartbeat ping is outstanding
const measurement = client.measureLatency({ timeoutMs: 10_000 });
await vi.advanceTimersByTimeAsync(1_000);
client.ensureConnected({ verify: true });
await vi.advanceTimersByTimeAsync(5_000); // the first window closes silent at 16s
expect(daemon.pingTimestamps()).toEqual(["10s", "11s", "16s"]);
// The heartbeat's pong lands at 17s, inside the second window and before the 20s deadline.
await vi.advanceTimersByTimeAsync(1_000);
await expect(measurement).resolves.toBe(7000);
await vi.advanceTimersByTimeAsync(10_000);
expect(client.getConnectionState()).toEqual({ status: "connected" });
expect(daemon.closesFromClient()).toEqual([]);
expect(attempts()).toBe(1);
expect(daemon.pingTimestamps()).toEqual(["10s", "11s", "16s", "27s"]);
});
// Fake timers cannot suspend: a suspension is modelled as the wall clock jumping forward while
// the pending window timer stays queued, then firing late on resume.
async function suspendFor(ms: number): Promise<void> {
vi.setSystemTime(Date.now() + ms);
}
test("a foreground window that spanned a second suspension is discarded when the timer fires first", async () => {
useHeartbeatClock();
const daemon = new FakeDaemon();
daemon.daemonGoesSilent();
const { client, attempts } = connectThroughReplacements("suspended-mid-cycle-timer", [daemon]);
const connection = client.connect();
daemon.openConnection();
await connection;
client.ensureConnected({ verify: true });
await vi.advanceTimersByTimeAsync(5_000); // window 1 silent, window 2 outstanding
expect(daemon.pingTimestamps()).toEqual(["0s", "5s"]);
await suspendFor(60_000);
daemon.daemonAnswersPingsAfter("fast"); // the socket is healthy on resume
await vi.advanceTimersByTimeAsync(5_000); // window 2's timer fires late
client.ensureConnected({ verify: true }); // then AppState lands
await vi.advanceTimersByTimeAsync(1_000);
expect(daemon.closesFromClient()).toEqual([]);
expect(client.getConnectionState()).toEqual({ status: "connected" });
expect(attempts()).toBe(1);
});
test("a foreground window that spanned a second suspension is discarded when AppState fires first", async () => {
useHeartbeatClock();
const daemon = new FakeDaemon();
daemon.daemonGoesSilent();
const { client, attempts } = connectThroughReplacements("suspended-mid-cycle-appstate", [daemon]);
const connection = client.connect();
daemon.openConnection();
await connection;
client.ensureConnected({ verify: true });
await vi.advanceTimersByTimeAsync(5_000);
expect(daemon.pingTimestamps()).toEqual(["0s", "5s"]);
await suspendFor(60_000);
daemon.daemonAnswersPingsAfter("fast");
client.ensureConnected({ verify: true }); // AppState lands before the queued timer
await vi.advanceTimersByTimeAsync(5_000); // window 2's timer fires late
await vi.advanceTimersByTimeAsync(1_000);
expect(daemon.closesFromClient()).toEqual([]);
expect(client.getConnectionState()).toEqual({ status: "connected" });
expect(attempts()).toBe(1);
});
test("a dead socket after a second suspension mid-cycle still gets two fresh windows", async () => {
useHeartbeatClock();
const first = new FakeDaemon();
const second = new FakeDaemon();
first.daemonGoesSilent();
const { client, attempts } = connectThroughReplacements("suspended-mid-cycle-dead", [
first,
second,
]);
const connection = client.connect();
first.openConnection();
await connection;
client.ensureConnected({ verify: true });
await vi.advanceTimersByTimeAsync(5_000);
first.receiveChunk(true); // a frame before the second suspension is not evidence after it
await suspendFor(60_000);
client.ensureConnected({ verify: true });
await vi.advanceTimersByTimeAsync(5_000); // the late timer discards window 2 and starts over
expect(first.pingTimestamps()).toEqual(["0s", "5s", "10s"]);
await vi.advanceTimersByTimeAsync(5_000);
expect(first.closesFromClient()).toEqual([]);
expect(first.pingTimestamps()).toEqual(["0s", "5s", "10s", "15s"]);
await vi.advanceTimersByTimeAsync(5_000);
expect(first.closesFromClient()).toEqual([{ code: 1001, reason: "Liveness check timed out" }]);
expect(attempts()).toBe(2);
});
test("repeated foreground events share one cycle and do not restart its windows", async () => {
useHeartbeatClock();
const first = new FakeDaemon();
const second = new FakeDaemon();
first.daemonGoesSilent();
const { client, attempts } = connectThroughReplacements("repeated-foreground", [first, second]);
const connection = client.connect();
first.openConnection();
await connection;
client.ensureConnected({ verify: true });
for (let elapsedSeconds = 1; elapsedSeconds <= 9; elapsedSeconds++) {
await vi.advanceTimersByTimeAsync(1_000);
client.ensureConnected({ verify: true });
expect(first.closesFromClient()).toEqual([]);
}
expect(first.pingTimestamps()).toEqual(["0s", "5s"]);
await vi.advanceTimersByTimeAsync(1_000);
expect(first.closesFromClient()).toEqual([{ code: 1001, reason: "Liveness check timed out" }]);
expect(attempts()).toBe(2);
second.openConnection();
expect(client.getConnectionState()).toEqual({ status: "connected" });
});
test("a late pong from the replaced socket does not count for the replacement", async () => {
useHeartbeatClock();
const first = new FakeDaemon();
const second = new FakeDaemon();
first.daemonGoesSilent();
second.daemonGoesSilent();
const { client, attempts } = connectThroughReplacements("late-pong", [first, second]);
const connection = client.connect();
first.openConnection();
await connection;
client.ensureConnected({ verify: true });
await vi.advanceTimersByTimeAsync(10_000);
expect(attempts()).toBe(2);
second.openConnection();
client.ensureConnected({ verify: true });
expect(second.pingTimestamps()).toEqual(["10s"]);
await vi.advanceTimersByTimeAsync(2_000);
first.daemonSendsPong();
await vi.advanceTimersByTimeAsync(8_000);
expect(second.pingTimestamps()).toEqual(["10s", "15s"]);
expect(second.closesFromClient()).toEqual([{ code: 1001, reason: "Liveness check timed out" }]);
expect(first.closesFromClient()).toEqual([{ code: 1001, reason: "Liveness check timed out" }]);
});
test("closing the client in the middle of a foreground cycle stops it", async () => {
useHeartbeatClock();
const first = new FakeDaemon();
const second = new FakeDaemon();
first.daemonGoesSilent();
const { client, attempts } = connectThroughReplacements("close-mid-cycle", [first, second]);
const connection = client.connect();
first.openConnection();
await connection;
client.ensureConnected({ verify: true });
await vi.advanceTimersByTimeAsync(5_000);
expect(first.pingTimestamps()).toEqual(["0s", "5s"]);
await client.close();
await vi.advanceTimersByTimeAsync(60_000);
expect(first.pingTimestamps()).toEqual(["0s", "5s"]);
expect(first.closesFromClient()).toEqual([{ code: 1000, reason: "Client closed" }]);
expect(attempts()).toBe(1);
expect(client.getConnectionState()).toEqual({ status: "disposed" });
});
test("a foreground teardown with reconnect disabled reports the timeout and stays down", async () => {
useHeartbeatClock();
const daemon = new FakeDaemon();
daemon.daemonGoesSilent();
const { client, attempts } = connectThroughReplacements("no-reconnect", [daemon], {
reconnect: { enabled: false },
});
const connection = client.connect();
daemon.openConnection();
await connection;
client.ensureConnected({ verify: true });
await vi.advanceTimersByTimeAsync(10_000);
expect(client.getConnectionState()).toEqual({
status: "disconnected",
reason: "Liveness check timed out (5000ms)",
});
await vi.advanceTimersByTimeAsync(10_000);
expect(attempts()).toBe(1);
});
test("the immediate reconnect after a foreground teardown backs off normally if it fails", async () => {
useHeartbeatClock();
const first = new FakeDaemon();
const second = new FakeDaemon();
const third = new FakeDaemon();
first.daemonGoesSilent();
const { client, attempts } = connectThroughReplacements("no-storm", [first, second, third]);
const connection = client.connect();
first.openConnection();
await connection;
client.ensureConnected({ verify: true });
await vi.advanceTimersByTimeAsync(10_000);
expect(attempts()).toBe(2);
second.daemonClosesWith("connection refused");
await vi.advanceTimersByTimeAsync(1_499);
expect(attempts()).toBe(2);
await vi.advanceTimersByTimeAsync(1);
expect(attempts()).toBe(3);
third.openConnection();
expect(client.getConnectionState()).toEqual({ status: "connected" });
});
test("an obsolete heartbeat cannot close a replacement connection", async () => {
useHeartbeatClock();
const first = new FakeDaemon();
@@ -2887,6 +3293,18 @@ test("a heartbeat sharing a latency measurement still saves the RTT", async () =
expect(session.lastLivenessRttMs()).toBe(2000);
});
test("a measurement that joins an in-flight ping reports that ping's round trip", async () => {
useHeartbeatClock();
const session = new DaemonClientSession();
session.daemonAnswersPingsAfter("2s");
await session.connect();
await session.advance(11_000); // the heartbeat ping sent at 10s is still outstanding
const measurement = session.measureLatency({ timeoutMs: 5_000 });
await session.advance(1_000);
await expect(measurement).resolves.toBe(2000);
expect(session.lastLivenessRttMs()).toBe(2000);
});
test("reports the round-trip time of the last successful heartbeat", async () => {
useHeartbeatClock();
const session = new DaemonClientSession();
+141 -47
View File
@@ -1143,6 +1143,12 @@ const DEFAULT_CONNECT_TIMEOUT_MS = 15_000;
const DEFAULT_LIVENESS_TIMEOUT_MS = 5000;
const LIVENESS_HEARTBEAT_INTERVAL_MS = 10_000;
const LIVENESS_HEARTBEAT_TIMEOUT_MS = 15_000;
// Window of each check after the app returns to the foreground. A socket that silently died
// while the app was suspended stays OPEN and accepts sends, so only inbound frames prove it.
const FOREGROUND_LIVENESS_TIMEOUT_MS = 5_000;
// A foreground window whose timer fires this much later than scheduled (wall clock) spanned
// another suspension or a blocked JS thread, so its silence is no evidence either way.
const SUSPENDED_WINDOW_SLACK_MS = 2_000;
const LIVENESS_FAILURE_RECONNECT_THRESHOLD = 2;
/** Default timeout for waiting for connection before sending queued messages */
@@ -1305,13 +1311,27 @@ interface PingProbe {
reject: (error: Error) => void;
timeoutHandle: ReturnType<typeof setTimeout>;
startedAt: number;
// Wall clock, which keeps running while the OS has JS suspended; perfNow may not.
startedAtWallMs: number;
receivedActivity: boolean;
// Whether a timeout on this ping should be recorded as a liveness failure. Only the
// heartbeat sets this; a latency measurement never drives teardown, even when a
// heartbeat tick shares (dedupes onto) an in-flight measurement ping.
drivesLivenessFailure: boolean;
role: PingProbeRole;
}
// A latency measurement waits for the next pong, not for a particular probe: the liveness
// cycle may replace the probe the measurement joined, and a pong that answers an expired window
// still proves the socket. It settles on a pong, a connection failure, or its own deadline.
interface PongWaiter {
startedAt: number;
resolve: (rttMs: number) => void;
reject: (error: Error) => void;
}
// Who started the probe. A "latency" probe never drives teardown, even when a heartbeat
// tick shares (dedupes onto) it. A "heartbeat" probe runs the regular cadence. A
// "foreground" probe belongs to the verification cycle that ensureConnected({ verify })
// runs after the app resumes: short window, immediate second check, reconnect at once.
type PingProbeRole = "latency" | "heartbeat" | "foreground";
// COMPAT(providerUsageList): added in v0.1.98, remove after 2027-03-26.
export function supportsUsageReports(features: ServerInfoStatusPayload["features"]): boolean {
return features?.usageSources === true || features?.providerUsageList === true;
@@ -1463,6 +1483,7 @@ export class DaemonClient {
private runtimeMetricsInterval: ReturnType<typeof setInterval> | null = null;
private runtimeMetrics: DaemonClientRuntimeMetrics | null = null;
private pingProbe: PingProbe | null = null;
private readonly pongWaiters = new Set<PongWaiter>();
private livenessHeartbeatTimer: ReturnType<typeof setTimeout> | null = null;
private lastLivenessRttMs: number | null = null;
private consecutiveLivenessFailures = 0;
@@ -1788,6 +1809,7 @@ export class DaemonClient {
this.creations.close();
this.rejectPendingSendQueue(new Error("Daemon client closed"));
this.rejectPingProbe(new Error("Daemon client closed"));
this.rejectPongWaiters(new Error("Daemon client closed"));
this.terminalStreams.clearSlots();
this.lastServerInfoMessage = null;
if (this.runtimeMetricsInterval) {
@@ -1810,7 +1832,7 @@ export class DaemonClient {
this.shouldReconnect = true;
}
if (this.connectionState.status === "connected") {
if (options?.verify) this.tickLivenessHeartbeat();
if (options?.verify) this.verifyConnectionOnForeground();
return;
}
if (this.connectionState.status === "connecting") return;
@@ -2561,43 +2583,41 @@ export class DaemonClient {
measureLatency(params?: { timeoutMs?: number }): Promise<number> {
const timeoutMs = Math.max(1, params?.timeoutMs ?? DEFAULT_LIVENESS_TIMEOUT_MS);
let deadline: ReturnType<typeof setTimeout>;
const timeout = new Promise<never>((_resolve, reject) => {
deadline = setTimeout(() => reject(new PingTimeoutError(timeoutMs)), timeoutMs);
});
return Promise.race([
this.sendPingAwaitRtt({ timeoutMs, drivesLivenessFailure: false }),
timeout,
])
.catch((error) => {
throw toTimeoutError(error, "Latency measurement", timeoutMs);
})
.finally(() => clearTimeout(deadline));
}
private async livenessPing(params?: { timeoutMs?: number }): Promise<number> {
const timeoutMs = Math.max(1, params?.timeoutMs ?? DEFAULT_LIVENESS_TIMEOUT_MS);
try {
const rttMs = await this.sendPingAwaitRtt({ timeoutMs, drivesLivenessFailure: true });
this.lastLivenessRttMs = rttMs;
return rttMs;
} catch (error) {
throw toTimeoutError(error, "Liveness check", timeoutMs);
}
}
private sendPingAwaitRtt(params: {
timeoutMs: number;
drivesLivenessFailure: boolean;
}): Promise<number> {
if (this.connectionState.status !== "connected" || !this.transport) {
return Promise.reject(
new Error(`Transport not connected (status: ${this.connectionState.status})`),
);
}
const waiter: PongWaiter = { startedAt: perfNow(), resolve: () => {}, reject: () => {} };
const pong = new Promise<number>((resolve, reject) => {
waiter.resolve = resolve;
waiter.reject = reject;
});
let deadline: ReturnType<typeof setTimeout>;
const timeout = new Promise<never>((_resolve, reject) => {
deadline = setTimeout(() => reject(new PingTimeoutError(timeoutMs)), timeoutMs);
});
this.pongWaiters.add(waiter);
// The probe only puts a ping on the wire; its own outcome is the heartbeat's concern. A
// measurement that joins an in-flight ping reports that ping's round trip, so the clock
// starts at the send, not at the join.
const probe = this.ensurePingProbe({ timeoutMs, role: "latency" });
void probe.promise.catch(() => {});
waiter.startedAt = probe.startedAt;
return Promise.race([pong, timeout])
.catch((error) => {
throw toTimeoutError(error, "Latency measurement", timeoutMs);
})
.finally(() => {
clearTimeout(deadline);
this.pongWaiters.delete(waiter);
});
}
// One ping is outstanding at a time; a caller that finds one shares it.
private ensurePingProbe(params: { timeoutMs: number; role: PingProbeRole }): PingProbe {
if (this.pingProbe) {
return this.pingProbe.promise;
return this.pingProbe;
}
const startedAt = perfNow();
@@ -2613,10 +2633,20 @@ export class DaemonClient {
this.pingProbe = null;
const error = new PingTimeoutError(timeoutMs);
probe.reject(error);
if (
probe.role === "foreground" &&
Date.now() - probe.startedAtWallMs > timeoutMs + SUSPENDED_WINDOW_SLACK_MS
) {
// The app was suspended again inside this window. Discard it, any frame it saw and
// the misses before it; the heartbeat tick that owns this probe starts the cycle over.
probe.receivedActivity = false;
this.consecutiveLivenessFailures = 0;
return;
}
// Keep the deadline fixed so subsequent pings renew the daemon socket lease
// during a long download. Progress proves life even when its pong is queued.
if (probe.drivesLivenessFailure && !probe.receivedActivity) {
this.recordLivenessFailure(toTimeoutError(error, "Liveness check", timeoutMs));
if (probe.role !== "latency" && !probe.receivedActivity) {
this.recordLivenessFailure(toTimeoutError(error, "Liveness check", timeoutMs), probe.role);
}
};
const probe: PingProbe = {
@@ -2626,7 +2656,8 @@ export class DaemonClient {
timeoutHandle: setTimeout(expire, timeoutMs),
receivedActivity: false,
startedAt,
drivesLivenessFailure: params.drivesLivenessFailure,
startedAtWallMs: Date.now(),
role: params.role,
};
this.pingProbe = probe;
@@ -2634,7 +2665,8 @@ export class DaemonClient {
if (this.pingProbe !== probe) return;
const sendError = error instanceof Error ? error : new Error(String(error));
this.rejectPingProbe(sendError);
if (probe.drivesLivenessFailure) this.recordLivenessFailure(sendError);
this.rejectPongWaiters(sendError);
if (probe.role !== "latency") this.recordLivenessFailure(sendError, probe.role);
};
try {
void this.sendJsonMessage("ping", "ping", { type: "ping" }).catch(rejectSend);
@@ -2642,7 +2674,7 @@ export class DaemonClient {
rejectSend(error);
}
return promise;
return probe;
}
private startLivenessHeartbeat(): void {
@@ -2669,14 +2701,55 @@ export class DaemonClient {
}, LIVENESS_HEARTBEAT_INTERVAL_MS);
}
private tickLivenessHeartbeat(): void {
/**
* The app came back to the foreground. Start a fresh verification cycle: two
* FOREGROUND_LIVENESS_TIMEOUT_MS windows back to back, then teardown and an immediate
* reconnect. Misses recorded before the app was suspended do not count, and a heartbeat
* probe that straddled the suspension measured the suspension, not the socket, so it is
* dropped without a miss. Latency measurements wait for the next pong, so replacing the
* probe they joined does not fail them.
*/
private verifyConnectionOnForeground(): void {
// A cycle already running keeps its deadline and its window count.
if (this.pingProbe?.role === "foreground") return;
if (this.pingProbe?.role === "heartbeat") {
this.rejectPingProbe(new Error("Heartbeat superseded by foreground check"));
}
// A shared latency probe keeps its deadline, but only frames that arrive from now on count
// for this check; a latency probe never reads this flag for itself.
if (this.pingProbe) this.pingProbe.receivedActivity = false;
this.consecutiveLivenessFailures = 0;
this.tickLivenessHeartbeat("foreground");
}
private tickLivenessHeartbeat(check: "heartbeat" | "foreground" = "heartbeat"): void {
this.stopLivenessHeartbeat();
const transport = this.transport;
const tick = this.livenessPing({ timeoutMs: LIVENESS_HEARTBEAT_TIMEOUT_MS });
void tick
.catch(() => {})
.finally(() => {
if (this.transport === transport) this.scheduleNextLivenessHeartbeat();
if (this.connectionState.status !== "connected" || !transport) return;
const probe = this.ensurePingProbe({
timeoutMs:
check === "foreground" ? FOREGROUND_LIVENESS_TIMEOUT_MS : LIVENESS_HEARTBEAT_TIMEOUT_MS,
role: check,
});
void probe.promise
.then(
(rttMs) => {
this.lastLivenessRttMs = rttMs;
return true;
},
() => probe.receivedActivity,
)
.then((alive) => {
// A replaced transport, or a probe that superseded this one, owns the next step.
if (this.transport !== transport || this.pingProbe) return;
if (!alive && check === "foreground") {
// Zero inbound frames in a foreground window: check again now. The second silent
// window reaches the threshold inside recordLivenessFailure.
this.tickLivenessHeartbeat("foreground");
return;
}
this.scheduleNextLivenessHeartbeat();
return;
});
}
@@ -7197,6 +7270,7 @@ export class DaemonClient {
messageType: "pong",
});
this.resolvePingProbe();
this.resolvePongWaiters();
this.runtimeMetrics?.recordMessage("pong", bytes, perfNow() - startMs);
return;
}
@@ -7446,6 +7520,8 @@ export class DaemonClient {
reason?: string;
event?: string;
reasonCode?: string;
// Attempt now instead of after the backoff delay. Failures of that attempt back off as usual.
immediate?: boolean;
}): void {
if (this.reconnectTimeout) {
clearTimeout(this.reconnectTimeout);
@@ -7476,6 +7552,7 @@ export class DaemonClient {
this.clearWaiters(connectionError, true);
this.rejectPendingSendQueue(connectionError);
this.rejectPingProbe(connectionError);
this.rejectPongWaiters(connectionError);
this.terminalStreams.clearSlots();
this.lastServerInfoMessage = null;
@@ -7495,6 +7572,10 @@ export class DaemonClient {
return;
}
if (input?.immediate) {
void this.attemptConnect();
return;
}
this.armReconnectTimer();
}
@@ -7569,7 +7650,18 @@ export class DaemonClient {
probe.reject(error);
}
private recordLivenessFailure(error: Error): void {
private resolvePongWaiters(): void {
const now = perfNow();
for (const waiter of this.pongWaiters) waiter.resolve(now - waiter.startedAt);
this.pongWaiters.clear();
}
private rejectPongWaiters(error: Error): void {
for (const waiter of this.pongWaiters) waiter.reject(error);
this.pongWaiters.clear();
}
private recordLivenessFailure(error: Error, role: PingProbeRole): void {
this.consecutiveLivenessFailures += 1;
if (this.consecutiveLivenessFailures < LIVENESS_FAILURE_RECONNECT_THRESHOLD) {
return;
@@ -7581,6 +7673,8 @@ export class DaemonClient {
reason: error.message,
event: "LIVENESS_TIMEOUT",
reasonCode: "liveness_timeout",
// The foreground cycle already waited two windows; the user is looking at the app.
immediate: role === "foreground",
});
}