fix: use ceiling division for token estimation, deduplicate inline formula

Switch estimate_tokens_rough(), estimate_messages_tokens_rough(), and
estimate_request_tokens_rough() from floor division (len // 4) to
ceiling division ((len + 3) // 4). Short texts (1-3 chars) previously
estimated as 0 tokens, causing the compressor and pre-flight checks to
systematically undercount when many short tool results are present.

Also replaced the inline duplicate formula in run_conversation()
(total_chars // 4) with a call to the shared
estimate_messages_tokens_rough() function.

Updated 4 tests that hardcoded floor-division expected values.

Related: issue #6217, PR #6629
This commit is contained in:
Teknium
2026-04-11 16:33:35 -07:00
parent 6d272ba477
commit 5c2ecdec49
3 changed files with 19 additions and 11 deletions

View File

@@ -8049,7 +8049,7 @@ class AIAgent:
# Calculate approximate request size for logging
total_chars = sum(len(str(msg)) for msg in api_messages)
approx_tokens = total_chars // 4 # Rough estimate: 4 chars per token
approx_tokens = estimate_messages_tokens_rough(api_messages)
# Thinking spinner for quiet mode (animated during API call)
thinking_spinner = None