fix(auxiliary): detect quota exhaustion as payment error; allow capacity-error fallback for explicit providers
Closes #26803 Root causes: 1. _is_payment_error() checked for billing keywords (credits, insufficient funds, billing, payment required) but missed daily token quota exhaustion phrases used by Bedrock, Vertex AI, and LiteLLM proxies — e.g. 'Too many tokens per day', 'quota exceeded', 'resource exhausted', 'daily limit'. These are functionally identical to credit exhaustion (provider cannot serve the request) but don't trigger fallback. 2. The call_llm() fallback chain was gated on resolved_provider == 'auto'. When a task resolves to a specific provider (e.g. 'custom' for a LiteLLM proxy, or 'openrouter'), capacity failures (payment/quota/connection) silently raise instead of trying alternatives. This is overly conservative: capacity errors mean the provider *cannot* serve the request regardless of user intent, so alternatives should always be tried. Fixes: - Add quota-related keywords to _is_payment_error(): quota_exceeded, too many tokens per day, daily limit, tokens per day, daily quota, resource exhausted (Vertex AI gRPC code). - Allow fallback for capacity errors (payment + connection) even when resolved_provider is not 'auto'. Rate-limit fallback stays gated on is_auto to honour explicit provider constraints for transient limits. - Apply both fixes to sync call_llm() and async acall_llm() paths. - Add 6 targeted tests for the new quota-error detection cases.
This commit is contained in:
@@ -925,6 +925,44 @@ class TestIsPaymentError:
|
||||
exc = Exception("connection reset")
|
||||
assert _is_payment_error(exc) is False
|
||||
|
||||
# ── Daily / monthly quota exhaustion (#26803) ────────────────────────────
|
||||
|
||||
def test_429_quota_exceeded(self):
|
||||
"""Cloud provider quota exhaustion (e.g. Vertex AI) is a payment error."""
|
||||
exc = Exception("RESOURCE_EXHAUSTED: quota exceeded for project")
|
||||
exc.status_code = 429
|
||||
assert _is_payment_error(exc) is True
|
||||
|
||||
def test_429_too_many_tokens_per_day(self):
|
||||
"""Bedrock / LiteLLM daily token limit is a payment error."""
|
||||
exc = Exception("Too many tokens per day: 1000000 used, 1000000 limit")
|
||||
exc.status_code = 429
|
||||
assert _is_payment_error(exc) is True
|
||||
|
||||
def test_429_daily_limit_phrase(self):
|
||||
"""Generic 'daily limit' phrasing is a payment error."""
|
||||
exc = Exception("You have exceeded your daily limit.")
|
||||
exc.status_code = 429
|
||||
assert _is_payment_error(exc) is True
|
||||
|
||||
def test_429_resource_exhausted_grpc(self):
|
||||
"""Vertex AI gRPC RESOURCE_EXHAUSTED maps to payment error."""
|
||||
exc = Exception("resource exhausted")
|
||||
exc.status_code = 429
|
||||
assert _is_payment_error(exc) is True
|
||||
|
||||
def test_429_daily_quota_phrase(self):
|
||||
"""'daily quota' phrasing is a payment error."""
|
||||
exc = Exception("Daily quota of 500 requests reached.")
|
||||
exc.status_code = 429
|
||||
assert _is_payment_error(exc) is True
|
||||
|
||||
def test_429_transient_rate_limit_not_quota(self):
|
||||
"""Transient 429 rate limit without quota keywords is NOT a payment error."""
|
||||
exc = Exception("Rate limit exceeded. Retry after 10s.")
|
||||
exc.status_code = 429
|
||||
assert _is_payment_error(exc) is False
|
||||
|
||||
|
||||
class TestIsRateLimitError:
|
||||
"""_is_rate_limit_error detects 429 rate-limit errors warranting fallback."""
|
||||
|
||||
Reference in New Issue
Block a user