fix: increase stream read timeout default to 120s, auto-raise for local LLMs (#6967)
Raise the default httpx stream read timeout from 60s to 120s for all providers. Additionally, auto-detect local LLM endpoints (Ollama, llama.cpp, vLLM) and raise the read timeout to HERMES_API_TIMEOUT (1800s) since local models can take minutes for prefill on large contexts before producing the first token. The stale stream timeout already had this local auto-detection pattern; the httpx read timeout was missing it — causing a hard 60s wall that users couldn't find (HERMES_STREAM_READ_TIMEOUT was undocumented). Changes: - Default HERMES_STREAM_READ_TIMEOUT: 60s -> 120s - Auto-detect local endpoints -> raise to 1800s (user override respected) - Document HERMES_STREAM_READ_TIMEOUT and HERMES_STREAM_STALE_TIMEOUT - Add 10 parametrized tests Reported-by: Pavan Srinivas (@pavanandums)
This commit is contained in:
12
run_agent.py
12
run_agent.py
@@ -4433,7 +4433,17 @@ class AIAgent:
|
||||
"""Stream a chat completions response."""
|
||||
import httpx as _httpx
|
||||
_base_timeout = float(os.getenv("HERMES_API_TIMEOUT", 1800.0))
|
||||
_stream_read_timeout = float(os.getenv("HERMES_STREAM_READ_TIMEOUT", 60.0))
|
||||
_stream_read_timeout = float(os.getenv("HERMES_STREAM_READ_TIMEOUT", 120.0))
|
||||
# Local providers (Ollama, llama.cpp, vLLM) can take minutes for
|
||||
# prefill on large contexts before producing the first token.
|
||||
# Auto-increase the httpx read timeout unless the user explicitly
|
||||
# overrode HERMES_STREAM_READ_TIMEOUT.
|
||||
if _stream_read_timeout == 120.0 and self.base_url and is_local_endpoint(self.base_url):
|
||||
_stream_read_timeout = _base_timeout
|
||||
logger.debug(
|
||||
"Local provider detected (%s) — stream read timeout raised to %.0fs",
|
||||
self.base_url, _stream_read_timeout,
|
||||
)
|
||||
stream_kwargs = {
|
||||
**api_kwargs,
|
||||
"stream": True,
|
||||
|
||||
Reference in New Issue
Block a user