From b23f2c240486e168ba5f12a5ccca6ae01c3ac15e Mon Sep 17 00:00:00 2001 From: Mohamed Boudra Date: Wed, 24 Dec 2025 18:35:58 +0700 Subject: [PATCH] Document Codex MCP mismatches and typecheck --- CODEX_MCP_MISMATCH_REPORT.md | 53 ++++++++++++++++++++++++++++++++++++ plan.md | 11 +++++++- 2 files changed, 63 insertions(+), 1 deletion(-) create mode 100644 CODEX_MCP_MISMATCH_REPORT.md diff --git a/CODEX_MCP_MISMATCH_REPORT.md b/CODEX_MCP_MISMATCH_REPORT.md new file mode 100644 index 000000000..d0346b8f2 --- /dev/null +++ b/CODEX_MCP_MISMATCH_REPORT.md @@ -0,0 +1,53 @@ +# Codex MCP CLI / Model / Permission Mismatches + +## Environment +- Codex CLI: codex-cli 0.77.0 +- MCP command: `codex mcp-server` (selected by `getCodexMcpCommand` for 0.77.0) +- Test runner: `npm run test --workspace=@paseo/server -- src/server/agent/providers/codex-mcp-agent.test.ts` + +## Model mismatch (runtime info test) +- Test: `reports runtime info with provider, session, model, and mode` +- Config: provider=codex-mcp, modeId=full-access, model=gpt-4.1 +- Expected: assistant replies `READY`, runtime info returns non-empty model/sessionId. +- Observed: + - Codex CLI error: `http 400 Bad Request: The 'gpt-4.1' model is not supported when using Codex with a ChatGPT account.` + - Test fails: `expected '' to contain 'ready'`. +- Notes: + - CLI refuses the model at session start, so the run returns empty text and no runtime info. + +## Permission elicitation mismatch (on-request) +- Test: `requests permission and resolves approval when allowed` +- Config: provider=codex-mcp, modeId=full-access, approvalPolicy=on-request +- Expected: `permission_requested` emitted, test captures permission request; `permission_resolved` after approving. +- Observed: + - No permission request captured (`expected null not to be null`). +- Notes: + - Suggests Codex CLI did not emit elicitation events despite approval-policy=on-request. + +## Permission ordering mismatch (read-only/untrusted) +- Test: `requires permission before commands in read-only (untrusted) mode` +- Config: provider=codex-mcp, modeId=read-only (approvalPolicy=untrusted, sandbox=read-only) +- Expected: permission request appears before command timeline items. +- Observed: + - No permission request captured (`expected null not to be null`). +- Notes: + - Either CLI ignores approval-policy/sandbox or the event stream does not surface elicitation for MCP. + +## Follow-up categories +- Model gating: choose a supported default model for ChatGPT accounts or skip runtime info test unless model available. +- Approval-policy/elicitation: confirm Codex CLI semantics for `approval-policy=on-request` and `untrusted` via MCP, and adjust tests or provider accordingly. + +# Codex MCP Type/Quality Review + +## High-signal follow-up tasks +- Reduce `unknown`/`Record` casts in `codex-mcp-agent.ts` by defining typed event payloads for `codex/event` and thread item shapes. +- Replace ad-hoc `as` casts in timeline mapping with type guards (e.g., `isCommandExecutionEvent`, `isThreadItem`) and narrow types before access. +- Type `pendingPermissions` and `pendingPermissionHandlers` with stricter interfaces; remove `message` usage from `AgentPermissionResponse` or widen the union if needed. +- Introduce a typed wrapper for MCP client notifications so `raw` payloads are decoded once instead of casting per-event. +- Add a single source of truth for timeline item ids/call ids to avoid fallback to `unknown` data. + +## Suggested task buckets +- Typecheck fixes for `AgentPermissionResponse` handling and unused locals. +- Event-shape typing and narrowing (MCP + thread/item types). +- Cleanup of `any`/`unknown` casts and `Record` usage. +- Test adjustments for model availability and permission semantics. diff --git a/plan.md b/plan.md index 56c0502d6..0f8a29e6a 100644 --- a/plan.md +++ b/plan.md @@ -38,12 +38,21 @@ Build a new Codex MCP provider side‑by‑side with the existing Codex SDK prov - [x] **Test (E2E)**: Add coverage for permission abort path (respondToPermission interrupt) and clarify read-only/untrusted gating behavior for Codex MCP. - **Done (2025-12-24 18:30)**: Added Codex MCP E2E tests for read-only/untrusted permission gating and interrupt-based permission abort handling; Vitest failed locally due to missing tool version for `vitest` in `.tool-versions`. -- [ ] **Test / Review (E2E + Typecheck)**: Document Codex CLI/model/permission mismatches and run typecheck. +- [x] **Test / Review (E2E + Typecheck)**: Document Codex CLI/model/permission mismatches and run typecheck. - Capture exact failing scenarios, including CLI flags, sandbox/approval policy, and observed vs expected behavior. - Review code, really bad typing and code is generally hortible quality, write a report for follow up tasks, reduce casting to a minimum, strongly typing everywhere - Paste key error output snippets and summarize root causes. - Run server typecheck and record failures; add fix tasks for each category. + - **Done (2025-12-24 18:35)**: Documented Codex CLI model/permission mismatches in `CODEX_MCP_MISMATCH_REPORT.md`; ran focused Vitest tests (runtime info + permission gating) and captured failures; ran server typecheck and recorded TS2339/TS6133 failures in `codex-mcp-agent.ts`. + +- [ ] **Fix**: Handle Codex CLI model availability mismatch (gpt-4.1 rejected for ChatGPT accounts) in Codex MCP tests/provider. + +- [ ] **Fix**: Investigate Codex MCP permission elicitation behavior for `approval-policy=on-request` and `untrusted` (no permission_requested events). + +- [ ] **Fix**: Resolve Codex MCP provider typecheck errors (`AgentPermissionResponse.message` usage, unused locals). + +- [ ] **Refactor**: Reduce casting/`unknown` usage in Codex MCP provider per type/quality report. - [ ] **Review**: Check implementation + edge cases.