fix(nous): don't trip cross-session rate breaker on upstream-capacity 429s (#15898)
Nous Portal multiplexes multiple upstream providers (DeepSeek, Kimi, MiMo, Hermes) behind one endpoint. Before this fix, any 429 on any of those models recorded a cross-session file breaker that blocked EVERY model on Nous for the cooldown window -- even though the caller's own RPM/RPH/TPM/TPH buckets were healthy. Users hit a DeepSeek V4 Pro capacity error, restarted, switched to Kimi 2.6, and still got 'Nous Portal rate limit active -- resets in 46m 53s'. Nous already emits the full x-ratelimit-* header suite on every response (captured by rate_limit_tracker into agent._rate_limit_state). We now gate the breaker on that data: trip it only when either the 429's own headers or the last-known-good state show a bucket with remaining == 0 AND a reset window >= 60s. Upstream-capacity 429s (healthy buckets everywhere, but upstream out of capacity) fall through to normal retry/fallback and the breaker is never written. Note: the in-memory 'restart TUI/gateway to clear' workaround circulated in Discord does NOT work -- the breaker is file-backed at ~/.hermes/rate_limits/nous.json. The workaround for users still affected by a bad state file is to delete it. Reported in Discord by CrazyDok1 and KYSIV (Apr 2026).
This commit is contained in:
61
run_agent.py
61
run_agent.py
@@ -11007,36 +11007,69 @@ class AIAgent:
|
||||
continue
|
||||
|
||||
# ── Nous Portal: record rate limit & skip retries ─────
|
||||
# When Nous returns a 429, record the reset time to a
|
||||
# shared file so ALL sessions (cron, gateway, auxiliary)
|
||||
# know not to pile on. Then skip further retries —
|
||||
# each one burns another RPH request and deepens the
|
||||
# rate limit hole. The retry loop's top-of-iteration
|
||||
# guard will catch this on the next pass and try
|
||||
# fallback or bail with a clear message.
|
||||
# When Nous returns a 429 that is a genuine account-
|
||||
# level rate limit, record the reset time to a shared
|
||||
# file so ALL sessions (cron, gateway, auxiliary) know
|
||||
# not to pile on, then skip further retries -- each
|
||||
# one burns another RPH request and deepens the hole.
|
||||
# The retry loop's top-of-iteration guard will catch
|
||||
# this on the next pass and try fallback or bail.
|
||||
#
|
||||
# IMPORTANT: Nous Portal multiplexes multiple upstream
|
||||
# providers (DeepSeek, Kimi, MiMo, Hermes). A 429 can
|
||||
# also mean an UPSTREAM provider is out of capacity
|
||||
# for one specific model -- transient, clears in
|
||||
# seconds, nothing to do with the caller's quota.
|
||||
# Tripping the cross-session breaker on that would
|
||||
# block every Nous model for minutes. We use
|
||||
# ``is_genuine_nous_rate_limit`` to tell the two
|
||||
# apart via the 429's own x-ratelimit-* headers and
|
||||
# the last-known-good state captured on the previous
|
||||
# successful response.
|
||||
if (
|
||||
is_rate_limited
|
||||
and self.provider == "nous"
|
||||
and classified.reason == FailoverReason.rate_limit
|
||||
and not recovered_with_pool
|
||||
):
|
||||
_genuine_nous_rate_limit = False
|
||||
try:
|
||||
from agent.nous_rate_guard import record_nous_rate_limit
|
||||
from agent.nous_rate_guard import (
|
||||
is_genuine_nous_rate_limit,
|
||||
record_nous_rate_limit,
|
||||
)
|
||||
_err_resp = getattr(api_error, "response", None)
|
||||
_err_hdrs = (
|
||||
getattr(_err_resp, "headers", None)
|
||||
if _err_resp else None
|
||||
)
|
||||
record_nous_rate_limit(
|
||||
_genuine_nous_rate_limit = is_genuine_nous_rate_limit(
|
||||
headers=_err_hdrs,
|
||||
error_context=error_context,
|
||||
last_known_state=self._rate_limit_state,
|
||||
)
|
||||
if _genuine_nous_rate_limit:
|
||||
record_nous_rate_limit(
|
||||
headers=_err_hdrs,
|
||||
error_context=error_context,
|
||||
)
|
||||
else:
|
||||
logging.info(
|
||||
"Nous 429 looks like upstream capacity "
|
||||
"(no exhausted bucket in headers or "
|
||||
"last-known state) -- not tripping "
|
||||
"cross-session breaker."
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
# Skip straight to max_retries — the top-of-loop
|
||||
# guard will handle fallback or bail cleanly.
|
||||
retry_count = max_retries
|
||||
continue
|
||||
if _genuine_nous_rate_limit:
|
||||
# Skip straight to max_retries -- the
|
||||
# top-of-loop guard will handle fallback or
|
||||
# bail cleanly.
|
||||
retry_count = max_retries
|
||||
continue
|
||||
# Upstream capacity 429: fall through to normal
|
||||
# retry logic. A different model (or the same
|
||||
# model a moment later) will typically succeed.
|
||||
|
||||
is_payload_too_large = (
|
||||
classified.reason == FailoverReason.payload_too_large
|
||||
|
||||
Reference in New Issue
Block a user