Recover a dead host connection within seconds of returning to the foreground (#9086)

* fix(client): recover a dead connection within seconds of returning to the foreground

A socket that died while the OS had JS suspended stays OPEN and keeps accepting
sends, so nothing but inbound frames can prove it alive. Since #8651 the
foreground check rode the regular heartbeat: a 15 s window, dedupe onto a probe
that straddled the suspension, misses from before background carried over, and
a 1.5 s backoff after teardown. Measured at the transport: 40 s to tear down and
41.5 s to reconnect, with 15 s to 30 s variations depending on what was in
flight when the app was suspended.

Foreground now starts a fresh cycle inside the same heartbeat owner: two 5 s
windows back to back, then teardown and an immediate reconnect. An outstanding
heartbeat probe is dropped without a miss because it measured the suspension,
not the socket; a latency measurement keeps its own deadline and is shared.
Repeated foreground events join the running cycle instead of restarting it.
The rule that any validated inbound frame voids a window is unchanged, so a
slow multi-frame transfer still survives, and after any progress the heartbeat
returns to its 10 s / 15 s cadence. Ordinary heartbeat teardowns keep their
backoff; only the foreground cycle reconnects at once, and a failed immediate
attempt backs off as before.

A single frame that takes longer than the two windows to arrive after
foreground is still torn down. That needs smaller daemon-to-client frames and
is documented as a known limit.

* fix(client): count only fresh evidence in the foreground liveness cycle

Two gaps in the foreground cycle let stale evidence decide a fresh check.

A latency measurement outstanding at foreground is shared by the check. If a
frame had arrived during that measurement before the app was suspended, the
check read that old activity as "alive" and went back to the normal cadence
without running its windows. The check now clears the shared probe's activity
flag when it begins, so only frames that arrive from then on count.

If the app was suspended again inside the second window, the next resume
either found the queued expiry already fired (teardown before AppState) or
deduped onto the old probe and let the expiry fire right after (teardown with
no fresh window). A foreground window whose timer fires well past its deadline
on the wall clock spanned a suspension; it is now discarded together with any
activity it saw and the misses before it, and the cycle starts over. The wall
clock is used because the monotonic clock may not advance while the OS has the
process suspended.

* fix(client): keep latency measurements alive when foreground replaces a heartbeat

A latency measurement that found a heartbeat ping outstanding joins that
probe. The foreground check used to fail the probe outright to start its
fresh window, which rejected the measurement before its own deadline on a
healthy socket; the diagnostics collector treats that as a failed
collection. The superseded probe is now detached without settling and
takes the outcome of the fresh window, so the heartbeat tick and any
measurement waiting on it see the next pong or the window's timeout
instead of a spurious error.

* fix(client): let latency measurements wait for the next pong instead of a probe

A measurement that joined an outstanding heartbeat probe took that
probe's fate. The foreground check replaces such a probe, so the
measurement failed early: first with the supersede error, then, after
the hand-off to the fresh window, with that window's five-second timeout
even when the caller had asked for a longer deadline and the pong was
about to arrive.

Measurements now register as pong waiters. A waiter settles on the next
pong, on a connection failure (close, transport dispose, ping send
failure) or on its own deadline, and no longer cares which probe put the
ping on the wire. The foreground check goes back to dropping the
straddling heartbeat probe outright, since the only thing waiting on a
probe's promise is the heartbeat tick.

* fix(client): report the joined ping's round trip from a latency measurement

A measurement that joins an in-flight heartbeat ping used to start its
clock at the join, so it reported only the remaining wait for the pong.
The clock now starts at the send of the ping the measurement waits on,
which matches what the heartbeat records for the same pong.
This commit is contained in:
Mohamed Boudra
2026-11-08 14:01:51 +07:00
committed by GitHub
parent ac08077f7a
commit f539ccf1c0
3 changed files with 581 additions and 69 deletions
+1 -1
View File
@@ -276,7 +276,7 @@ its teardown; reconnect creates new server IDs for surviving client handles. See
**Top-level WS envelopes** are `hello`, `recording_state`, `ping`/`pong`, and `session` (which wraps the rich union of session messages).
Client liveness checks use the top-level JSON `ping`/`pong` envelope, not a session RPC or RFC6455 control ping. Current clients ping every 10 seconds, beginning one interval after connecting. The first ping claims an application-ownership lease for that physical socket, all later inbound activity renews it, and the daemon forcibly terminates the socket if the lease expires. A legacy or raw socket that never sends an application ping never enters this lease and is not closed for omitting one. Session RPC timeouts are operation failures, not proof that the socket is dead. A subscription bootstrap timeout closes its source to clear any unknown server-owned registration; the failed handle is released before other surviving handles reconnect.
Client liveness checks use the top-level JSON `ping`/`pong` envelope, not a session RPC or RFC6455 control ping. Current clients ping every 10 seconds, beginning one interval after connecting; a window with no validated inbound frame is a miss, and two misses in a row tear the socket down. When the app returns to the foreground the client runs a fresh cycle of two 5-second windows back to back, then reconnects without backoff, because a socket that died while the OS had JS suspended stays `OPEN` and accepts sends: only inbound frames prove it. Misses and probes from before the suspension do not count toward that cycle, and a window whose timer fires long after its deadline spanned another suspension, so it is discarded and the cycle starts over. Any complete frame inside a window keeps the connection, so a slow multi-frame transfer survives; a single frame that takes longer than the two windows to arrive after foreground does not, and the fix for that is smaller daemon-to-client frames, not a longer window. The first ping claims an application-ownership lease for that physical socket, all later inbound activity renews it, and the daemon forcibly terminates the socket if the lease expires. A legacy or raw socket that never sends an application ping never enters this lease and is not closed for omitting one. Session RPC timeouts are operation failures, not proof that the socket is dead. A subscription bootstrap timeout closes its source to clear any unknown server-owned registration; the failed handle is released before other surviving handles reconnect.
Every physical send path enforces an 8 MiB outbound high-water mark, including JSON broadcasts, binary terminal frames, and the encrypted relay adapter's asynchronous queue. This sits above the terminal stream's 4 MiB soft backpressure threshold, leaving room for snapshot catch-up before the hard cutoff. JSON is serialized once per broadcast after sockets already at the limit are removed, then its exact byte length is checked for every remaining socket. A frame that would cross the limit is not sent; that physical socket is forcibly terminated without disturbing other sockets attached to the same logical session. Multiple tabs and simultaneous direct and relay paths may legitimately share a client id.