fix: sanitize surrogate characters from clipboard paste to prevent UnicodeEncodeError (#3624)
Pasting text from rich-text editors (Google Docs, Word, etc.) can inject lone surrogate characters (U+D800..U+DFFF) that are invalid UTF-8. The OpenAI SDK serializes messages with ensure_ascii=False, then encodes to UTF-8 for the HTTP body — surrogates crash this with: UnicodeEncodeError: 'utf-8' codec can't encode character '\udce2' Three-layer fix: 1. Primary: sanitize user_message at the top of run_conversation() 2. CLI: sanitize in chat() before appending to conversation_history 3. Safety net: catch UnicodeEncodeError in the API error handler, sanitize the entire messages list in-place, and retry once. Also exclude UnicodeEncodeError from is_local_validation_error so it doesn't get classified as non-retryable. Includes 14 new tests covering the sanitization helpers and the integration with run_conversation().
This commit is contained in:
7
cli.py
7
cli.py
@@ -5534,6 +5534,13 @@ class HermesCLI:
|
||||
except Exception as e:
|
||||
logging.debug("@ context reference expansion failed: %s", e)
|
||||
|
||||
# Sanitize surrogate characters that can arrive via clipboard paste from
|
||||
# rich-text editors (Google Docs, Word, etc.). Lone surrogates are invalid
|
||||
# UTF-8 and crash JSON serialization in the OpenAI SDK.
|
||||
if isinstance(message, str):
|
||||
from run_agent import _sanitize_surrogates
|
||||
message = _sanitize_surrogates(message)
|
||||
|
||||
# Add user message to history
|
||||
self.conversation_history.append({"role": "user", "content": message})
|
||||
|
||||
|
||||
Reference in New Issue
Block a user