common/hermes
2
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

feat(grok): apply OpenAI execution guidance to xAI Grok / xai-oauth models (#27797)

Browse Source 
Grok models hit the same failure modes that OPENAI_MODEL_EXECUTION_GUIDANCE
addresses for GPT/Codex: claiming completion without tool calls
('to be honest, I didn't create the file yet'), suggesting workarounds
instead of using existing tools (proposing a folder-based memory system
when the memory tool exists), replying with plans instead of executing.

TOOL_USE_ENFORCEMENT_GUIDANCE was already injected for any model whose
name contains 'grok' (TOOL_USE_ENFORCEMENT_MODELS). This extends the
follow-on family-specific block — OPENAI_MODEL_EXECUTION_GUIDANCE
(tool_persistence / mandatory_tool_use / act_dont_ask / prerequisite_checks
/ verification / missing_context) — to grok-named models too.

The OPENAI_ prefix is retained for backwards compat with imports/tests;
docstring + inline comment now note that the body is family-agnostic and
the prefix reflects origin, not exclusivity.

Tests cover the OpenRouter slug (x-ai/grok-4.3) and the xai-oauth bare
name (grok-4.3), plus a negative control on claude.

E2E verified against a real AIAgent build of the system prompt for both
xai-oauth and openrouter grok models.
This commit is contained in:
Teknium
2026-05-17 23:00:37 -07:00
committed by GitHub
parent 43e566f77e
commit 9b91377bec
3 changed files with 42 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

34
tests/run_agent/test_run_agent.py
View File

@@ -1074,6 +1074,40 @@ class TestToolUseEnforcementConfig:
prompt = agent._build_system_prompt()
assert TOOL_USE_ENFORCEMENT_GUIDANCE not in prompt
def test_auto_injects_for_grok(self):
"""xAI Grok / xai-oauth models hit the same enforcement path as GPT."""
from agent.prompt_builder import TOOL_USE_ENFORCEMENT_GUIDANCE
agent = self._make_agent(model="x-ai/grok-4.3", tool_use_enforcement="auto")
prompt = agent._build_system_prompt()
assert TOOL_USE_ENFORCEMENT_GUIDANCE in prompt
def test_auto_injects_execution_guidance_for_grok(self):
"""Grok also gets OPENAI_MODEL_EXECUTION_GUIDANCE (verification,
mandatory_tool_use, act_dont_ask). Same failure modes as GPT in
practice — claims completion without tool calls, suggests workarounds
instead of using existing tools.
"""
from agent.prompt_builder import OPENAI_MODEL_EXECUTION_GUIDANCE
agent = self._make_agent(model="x-ai/grok-4.3", tool_use_enforcement="auto")
prompt = agent._build_system_prompt()
assert OPENAI_MODEL_EXECUTION_GUIDANCE in prompt
def test_auto_injects_execution_guidance_for_xai_oauth_model(self):
"""xai-oauth bare model names (no slash) also match the grok pattern."""
from agent.prompt_builder import OPENAI_MODEL_EXECUTION_GUIDANCE
agent = self._make_agent(model="grok-4.3", tool_use_enforcement="auto")
prompt = agent._build_system_prompt()
assert OPENAI_MODEL_EXECUTION_GUIDANCE in prompt
def test_auto_does_not_inject_execution_guidance_for_claude(self):
"""Sanity: execution guidance stays off for non-targeted families."""
from agent.prompt_builder import OPENAI_MODEL_EXECUTION_GUIDANCE
agent = self._make_agent(
model="anthropic/claude-sonnet-4", tool_use_enforcement="auto"
)
prompt = agent._build_system_prompt()
assert OPENAI_MODEL_EXECUTION_GUIDANCE not in prompt
def test_true_forces_for_all_models(self):
from agent.prompt_builder import TOOL_USE_ENFORCEMENT_GUIDANCE
agent = self._make_agent(model="anthropic/claude-sonnet-4", tool_use_enforcement=True)