fix(auxiliary): detect quota exhaustion as payment error; allow capacity-error fallback for explicit providers

Closes #26803

Root causes:
1. _is_payment_error() checked for billing keywords (credits, insufficient
   funds, billing, payment required) but missed daily token quota exhaustion
   phrases used by Bedrock, Vertex AI, and LiteLLM proxies — e.g.
   'Too many tokens per day', 'quota exceeded', 'resource exhausted',
   'daily limit'. These are functionally identical to credit exhaustion
   (provider cannot serve the request) but don't trigger fallback.

2. The call_llm() fallback chain was gated on resolved_provider == 'auto'.
   When a task resolves to a specific provider (e.g. 'custom' for a LiteLLM
   proxy, or 'openrouter'), capacity failures (payment/quota/connection)
   silently raise instead of trying alternatives. This is overly conservative:
   capacity errors mean the provider *cannot* serve the request regardless of
   user intent, so alternatives should always be tried.

Fixes:
- Add quota-related keywords to _is_payment_error(): quota_exceeded,
  too many tokens per day, daily limit, tokens per day, daily quota,
  resource exhausted (Vertex AI gRPC code).
- Allow fallback for capacity errors (payment + connection) even when
  resolved_provider is not 'auto'. Rate-limit fallback stays gated on
  is_auto to honour explicit provider constraints for transient limits.
- Apply both fixes to sync call_llm() and async acall_llm() paths.
- Add 6 targeted tests for the new quota-error detection cases.
This commit is contained in:
Bartok9
2026-05-16 03:36:36 -04:00
committed by Teknium
parent 569bc94b59
commit 24c209f112
2 changed files with 72 additions and 9 deletions

View File

@@ -925,6 +925,44 @@ class TestIsPaymentError:
exc = Exception("connection reset")
assert _is_payment_error(exc) is False
# ── Daily / monthly quota exhaustion (#26803) ────────────────────────────
def test_429_quota_exceeded(self):
"""Cloud provider quota exhaustion (e.g. Vertex AI) is a payment error."""
exc = Exception("RESOURCE_EXHAUSTED: quota exceeded for project")
exc.status_code = 429
assert _is_payment_error(exc) is True
def test_429_too_many_tokens_per_day(self):
"""Bedrock / LiteLLM daily token limit is a payment error."""
exc = Exception("Too many tokens per day: 1000000 used, 1000000 limit")
exc.status_code = 429
assert _is_payment_error(exc) is True
def test_429_daily_limit_phrase(self):
"""Generic 'daily limit' phrasing is a payment error."""
exc = Exception("You have exceeded your daily limit.")
exc.status_code = 429
assert _is_payment_error(exc) is True
def test_429_resource_exhausted_grpc(self):
"""Vertex AI gRPC RESOURCE_EXHAUSTED maps to payment error."""
exc = Exception("resource exhausted")
exc.status_code = 429
assert _is_payment_error(exc) is True
def test_429_daily_quota_phrase(self):
"""'daily quota' phrasing is a payment error."""
exc = Exception("Daily quota of 500 requests reached.")
exc.status_code = 429
assert _is_payment_error(exc) is True
def test_429_transient_rate_limit_not_quota(self):
"""Transient 429 rate limit without quota keywords is NOT a payment error."""
exc = Exception("Rate limit exceeded. Retry after 10s.")
exc.status_code = 429
assert _is_payment_error(exc) is False
class TestIsRateLimitError:
"""_is_rate_limit_error detects 429 rate-limit errors warranting fallback."""