common/hermes
2
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

fix(gateway): use transcript timestamp for auto-continue freshness

Browse Source 
Follow-up to PR #16802 (BeliefanX). The original fix read
`agent_history[-1].get("timestamp")` for the tool-tail freshness gate,
but `gateway/run.py` strips the `timestamp` field off all tool/tool_call
rows when building `agent_history` from the raw transcript (see
`clean_msg = {k: v for k, v in msg.items() if k != "timestamp"}`).  At
runtime the tool-tail branch always saw `None` and silently took the
legacy-fresh path — the stale-guard never fired for the tool-tail case
it was supposed to cover.

Changes:
- Read the freshness signal from the RAW `history` list (via new
  `_last_transcript_timestamp()` helper) BEFORE the strip.  Both the
  resume_pending branch and the tool-tail branch use this single signal,
  replacing the two divergent ones.
- Default window bumped 15 min → 1 hour via new
  `_AUTO_CONTINUE_FRESHNESS_SECS_DEFAULT`.  The 15-minute default was
  shorter than the default `gateway_timeout` of 30 min, so a legitimate
  long-running turn interrupted near its timeout boundary and resumed
  shortly after would have been misclassified as stale.
- Configurable via `config.yaml` `agent.gateway_auto_continue_freshness`
  (bridged to `HERMES_AUTO_CONTINUE_FRESHNESS` at gateway startup — same
  pattern as `gateway_timeout`).  Set to 0 to disable the gate.
- `_coerce_gateway_timestamp` now explicitly rejects bool (which is a
  subclass of int and would otherwise coerce to 0.0/1.0).
- Tests rewritten to exercise the real production data shape: raw
  `history` → `_build_agent_history` strip → freshness decision.  A
  regression guard (`test_stale_tool_tail_with_production_data_shape`)
  asserts `agent_history` tool rows carry NO timestamp, protecting
  against someone "fixing" the original bug by re-adding the stripped
  field (which would break the OpenAI tool-result message contract).

Add BeliefanX to scripts/release.py AUTHOR_MAP.

E2E verified: config.yaml → env var bridge → helper returns configured
value; default 1h window; malformed/empty env var falls back to default;
ISO-Z timestamps parse; ms-epoch coerced; bool rejected.
This commit is contained in:
teknium1
2026-04-28 05:00:26 -07:00
committed by Teknium
parent 93feffbcfa
commit 7444e49d4e
4 changed files with 407 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files

121
gateway/run.py
View File

@@ -43,7 +43,22 @@ _AGENT_CACHE_IDLE_TTL_SECS = 3600.0 # evict agents idle for >1h
# Only auto-continue interrupted gateway turns while the interruption is fresh.
# Stale tool-tail/resume markers can otherwise revive an unrelated old task
# after a gateway restart when the user's next message starts new work.
_AUTO_CONTINUE_FRESHNESS_SECS = 15 * 60
#
# The freshness signal is the timestamp of the last transcript row, which
# ``hermes_state.get_messages`` carries on every persisted message. This
# handles the two auto-continue cases uniformly:
# * resume_pending (gateway restart/shutdown watchdog marked the session)
# * tool-tail (last persisted message is a tool result the agent
# never got to reply to)
# In both cases "when did we last do anything on this transcript" is the
# correct freshness question, so one signal replaces two divergent ones.
#
# Default window: 1 hour. This comfortably covers ``agent.gateway_timeout``
# (30 min default) plus runtime slack — a legitimate long-running turn that
# gets interrupted near its timeout boundary and is resumed shortly after
# is still classified fresh. Override via
# ``config.yaml`` ``agent.gateway_auto_continue_freshness``.
_AUTO_CONTINUE_FRESHNESS_SECS_DEFAULT = 60 * 60
def _coerce_gateway_timestamp(value: Any) -> Optional[float]:
@@ -51,11 +66,16 @@ def _coerce_gateway_timestamp(value: Any) -> Optional[float]:
Missing/unparseable timestamps return None so legacy transcripts keep the
historical auto-continue behaviour instead of being silently dropped.
Accepts: datetime, epoch seconds (int/float), epoch milliseconds (when
the magnitude exceeds year-2286), ISO-8601 strings (with or without a
trailing ``Z``), and numeric strings.
"""
if value is None:
return None
if isinstance(value, datetime):
return value.timestamp()
if isinstance(value, bool): # bool is a subclass of int — skip it
return None
if isinstance(value, (int, float)):
# Some platform events use milliseconds; Hermes state rows use seconds.
return float(value) / 1000.0 if float(value) > 10_000_000_000 else float(value)
@@ -75,17 +95,77 @@ def _coerce_gateway_timestamp(value: Any) -> Optional[float]:
return None
def _is_fresh_gateway_interruption(value: Any, *, now: Optional[float] = None) -> bool:
def _auto_continue_freshness_window() -> float:
"""Return the configured auto-continue freshness window in seconds.
Reads ``HERMES_AUTO_CONTINUE_FRESHNESS`` (bridged from
``config.yaml`` ``agent.gateway_auto_continue_freshness`` at gateway
startup, same pattern as ``HERMES_AGENT_TIMEOUT``). Falls back to the
module default when unset or malformed. Non-positive values disable
the freshness gate (restores the pre-fix "always fresh" behaviour for
users who want to opt out).
"""
raw = os.environ.get("HERMES_AUTO_CONTINUE_FRESHNESS")
if raw is None or raw == "":
return float(_AUTO_CONTINUE_FRESHNESS_SECS_DEFAULT)
try:
return float(raw)
except (TypeError, ValueError):
return float(_AUTO_CONTINUE_FRESHNESS_SECS_DEFAULT)
def _is_fresh_gateway_interruption(
value: Any,
*,
now: Optional[float] = None,
window_secs: Optional[float] = None,
) -> bool:
"""Return True when an interruption marker is fresh enough to auto-continue.
Unknown timestamps are treated as fresh for backward compatibility with
legacy transcripts/tests that predate timestamp persistence.
legacy transcripts (pre-dating timestamp persistence) and with in-memory
test scaffolding that constructs history entries without timestamps.
A non-positive ``window_secs`` disables the gate (always fresh), which
restores the pre-fix behaviour for users who opt out via config.
"""
window = (
float(window_secs)
if window_secs is not None
else float(_AUTO_CONTINUE_FRESHNESS_SECS_DEFAULT)
)
if window <= 0:
return True
timestamp = _coerce_gateway_timestamp(value)
if timestamp is None:
return True
current = time.time() if now is None else now
return current - timestamp <= _AUTO_CONTINUE_FRESHNESS_SECS
return current - timestamp <= window
def _last_transcript_timestamp(history: Optional[List[Dict[str, Any]]]) -> Any:
"""Return the ``timestamp`` of the last usable transcript row, if any.
Skips metadata-only rows (``session_meta``, system injections) that are
dropped before being handed to the agent. Returns ``None`` when no
usable row carries a timestamp — callers should treat that as "fresh"
for backward compatibility.
"""
if not history:
return None
for msg in reversed(history):
if not isinstance(msg, dict):
continue
role = msg.get("role")
if not role or role in ("session_meta", "system"):
continue
ts = msg.get("timestamp")
if ts is not None:
return ts
# First non-meta row without a timestamp — legacy transcript row.
# Returning None lets the caller fall through to the legacy-fresh path.
return None
return None
# ---------------------------------------------------------------------------
@@ -260,6 +340,13 @@ if _config_path.exists():
os.environ["HERMES_AGENT_NOTIFY_INTERVAL"] = str(_agent_cfg["gateway_notify_interval"])
if "restart_drain_timeout" in _agent_cfg and "HERMES_RESTART_DRAIN_TIMEOUT" not in os.environ:
os.environ["HERMES_RESTART_DRAIN_TIMEOUT"] = str(_agent_cfg["restart_drain_timeout"])
if (
"gateway_auto_continue_freshness" in _agent_cfg
and "HERMES_AUTO_CONTINUE_FRESHNESS" not in os.environ
):
os.environ["HERMES_AUTO_CONTINUE_FRESHNESS"] = str(
_agent_cfg["gateway_auto_continue_freshness"]
)
_display_cfg = _cfg.get("display", {})
if _display_cfg and isinstance(_display_cfg, dict):
if "busy_input_mode" in _display_cfg and "HERMES_GATEWAY_BUSY_INPUT_MODE" not in os.environ:
@@ -10554,26 +10641,38 @@ class GatewayRunner:
# anything (tool, assistant with unfinished work, etc.), so we
# give a stronger, reason-aware instruction that subsumes the
# tool-tail case.
#
# Freshness gate (#16802): both branches are gated on the age
# of the last persisted transcript row. That is the correct
# "when did we last do anything here" signal for both the
# resume_pending path (restart watchdog) and the tool-tail
# path (in-flight tool loop killed). We read ``history[-1]``
# here because ``agent_history`` has already stripped the
# ``timestamp`` field off tool/tool_call rows for API purity
# (see the `k != "timestamp"` filter above). Rows without a
# timestamp (legacy transcripts) are treated as fresh so the
# historical auto-continue behaviour is preserved.
_freshness_window = _auto_continue_freshness_window()
_interruption_is_fresh = _is_fresh_gateway_interruption(
_last_transcript_timestamp(history),
window_secs=_freshness_window,
)
_resume_entry = None
if session_key:
try:
_resume_entry = self.session_store._entries.get(session_key)
except Exception:
_resume_entry = None
_resume_marked_at = (
getattr(_resume_entry, "last_resume_marked_at", None)
if _resume_entry is not None
else None
)
_is_resume_pending = bool(
_resume_entry is not None
and getattr(_resume_entry, "resume_pending", False)
and _is_fresh_gateway_interruption(_resume_marked_at)
and _interruption_is_fresh
)
_has_fresh_tool_tail = bool(
agent_history
and agent_history[-1].get("role") == "tool"
and _is_fresh_gateway_interruption(agent_history[-1].get("timestamp"))
and _interruption_is_fresh
)
if _is_resume_pending: