feat(image-input): native multimodal routing based on model vision capability (#16506)

* feat(image-input): native multimodal routing based on model vision capability

Attach user-sent images as OpenAI-style content parts on the user turn when
the active model supports native vision, so vision-capable models see real
pixels instead of a lossy text description from vision_analyze.

Routing decision (agent/image_routing.py::decide_image_input_mode):

  agent.image_input_mode = auto | native | text  (default: auto)

In auto mode:
  - If auxiliary.vision.provider/model is explicitly configured, keep the
    text pipeline (user paid for a dedicated vision backend).
  - Else if models.dev reports supports_vision=True for the active
    provider/model, attach natively.
  - Else fall back to text (current behaviour).

Call sites updated: gateway/run.py (all messaging platforms), tui_gateway
(dashboard/Ink), cli.py (interactive /attach + drag-drop).

run_agent.py changes:
  - _prepare_anthropic_messages_for_api now passes image parts through
    unchanged when the model supports vision — the Anthropic adapter
    translates them to native image blocks. Previous behaviour
    (vision_analyze → text) only runs for non-vision Anthropic models.
  - New _prepare_messages_for_non_vision_model mirrors the same contract
    for chat.completions and codex_responses paths, so non-vision models
    on any provider get text-fallback instead of failing at the provider.
  - New _model_supports_vision() helper reads models.dev caps.

vision_analyze description rewritten: positions it as a tool for images
NOT already visible in the conversation (URLs, tool output, deeper
inspection). Prevents the model from redundantly calling it on images
already attached natively.

Config default: agent.image_input_mode = auto.

Tests: 35 new (test_image_routing.py + test_vision_aware_preprocessing.py),
all existing tests that reference _prepare_anthropic_messages_for_api
still pass (198 targeted + new tests green).

* feat(image-input): size-cap + resize oversized images, charge image tokens in compressor

Two follow-ups that make the native image routing safer for long / heavy
sessions:

1) Oversize handling in build_native_content_parts:
   - 20 MB ceiling per image (matches vision_tools._MAX_BASE64_BYTES,
     the most restrictive provider — Gemini inline data).
   - Delegates to vision_tools._resize_image_for_vision (Pillow-based,
     already battle-tested) to downscale to 5 MB first-try.
   - If Pillow is missing or resize still overshoots, the image is
     dropped and reported back in skipped[]; caller falls back to text
     enrichment for that image.

2) Image-token accounting in context_compressor:
   - New _IMAGE_TOKEN_ESTIMATE = 1600 (matches Claude Code's constant;
     within the realistic range for Anthropic/GPT-4o/Gemini billing).
   - _content_length_for_budget() helper: sums text-part lengths and
     charges _IMAGE_CHAR_EQUIVALENT (1600 * 4 chars) per image/image_url/
     input_image part.  Base64 payload inside image_url is NOT counted
     as chars — dimensions don't matter, only image-presence.
   - Both tail-cut sites (_prune_old_tool_results L527 and
     _find_tail_cut_by_tokens L1126) now call the helper so multi-image
     conversations don't slip past compression budget.

Tests: 9 new in test_image_routing.py (oversize triggers resize,
resize-fails-returns-None, oversize-skipped-reported), 11 new in
test_compressor_image_tokens.py (flat charge per image, multiple images,
Responses-API / Anthropic-native / OpenAI-chat shapes, no-inflation on
raw base64, bounds-check on the constant, integration test that an
image-heavy tail actually gets trimmed).

* fix(image-input): replace blanket 20MB ceiling with empirically-verified per-provider limits

The previous commit imposed a hardcoded 20 MB base64 ceiling on all
providers, triggering auto-resize on anything larger. This was wrong in
both directions:

  * Too loose for Anthropic — actual limit is 5 MB (returns HTTP 400
    'image exceeds 5 MB maximum' above that).
  * Too strict for OpenAI / Codex / OpenRouter — accept 49 MB+ without
    complaint (empirically verified April 2026 with progressive PNG
    sizes).

New behaviour:

  * _PROVIDER_BASE64_CEILING table: only anthropic and bedrock have a
    ceiling (5 MB, since bedrock-on-Claude shares Anthropic's decoder).
  * Providers NOT in the table get no ceiling — images attach at native
    size and we trust the provider to return its own error if it
    disagrees. A provider-specific 400 message is clearer than us
    guessing wrong and silently degrading image quality.
  * build_native_content_parts() gains a keyword-only provider arg;
    gateway/CLI/TUI pass the active provider so Anthropic users get
    auto-resize protection while OpenAI users don't pay it.
  * Resize target dropped from 5 MB to 4 MB to slide safely under
    Anthropic's boundary with header overhead.

Empirical measurements (direct API, no Hermes in the loop):

    image b64     anthropic   openrouter/gpt5.5   codex-oauth/gpt5.5
    0.19 MB       ✓           ✓                   ✓
    12.37 MB      ✗ 400 5MB   ✓                   ✓
    23.85 MB      ✗ 400 5MB   ✓                   ✓
    49.46 MB      ✗ 413       ✓                   ✓

Tests: rewrote TestOversizeHandling (5 tests): no-ceiling pass-through,
Anthropic resize fires, Anthropic skip on resize-fail, build_native_parts
routes ceiling by provider, unknown provider gets no ceiling. All 52
targeted tests pass.

* refactor(image-input): attempt native, shrink-and-retry on provider reject

Replace proactive per-provider size ceilings with a reactive shrink path
on the provider's actual rejection. All providers now attempt native
full-size attachment first; if the provider returns an image-too-large
error, the agent silently shrinks and retries once.

Why the previous design was wrong: hardcoding provider ceilings
(anthropic=5MB, others=unlimited) meant OpenAI users on a 10MB image
paid no tax, but Anthropic users lost quality on anything >5MB even
though the empirical behaviour at provider-reject time is the same
(shrink + retry). Baking the table into the routing layer also
requires updating Hermes every time a provider's limit changes.

Reactive design:
  - image_routing.py: _file_to_data_url encodes native size, no ceiling.
    build_native_content_parts drops its provider kwarg.
  - error_classifier.py: new FailoverReason.image_too_large + pattern
    match ("image exceeds", "image too large", etc.) checked BEFORE
    context_overflow so Anthropic's 5MB rejection lands in the right
    bucket.
  - run_agent.py: new _try_shrink_image_parts_in_messages walks api
    messages in-place, re-encodes oversized data: URL image parts
    through vision_tools._resize_image_for_vision to fit under 4MB,
    handles both chat.completions (dict image_url) and Responses
    (string image_url) shapes, ignores http URLs (provider-fetched).
    New image_shrink_retry_attempted flag in the retry loop fires the
    shrink exactly once per turn after credential-pool recovery but
    before auth retries.

E2E verified live against Anthropic claude-sonnet-4-6:
  - 17.9MB PNG (23.9MB b64) attached at native size
  - Anthropic returns 400 "image exceeds 5 MB maximum"
  - Agent logs '📐 Image(s) exceeded provider size limit — shrank and
    retrying...'
  - Retry succeeds, correct response delivered in 6.8s total.

Tests: 12 new (8 shrink-helper shapes + 4 classifier signals),
replaces 5 proactive-ceiling tests with 3 simpler 'native attach works'
tests. 181 targeted tests pass. test_enum_members_exist in
test_error_classifier.py updated for the new enum value.

tests/agent/test_compressor_image_tokens.py

@@ -0,0 +1,141 @@
+"""Tests for image-token accounting in the context compressor.
+
+Covers the native-image-routing PR's companion change: the compressor's
+multimodal message length counter now charges ~1600 tokens per attached
+image part instead of 0, so tail-cut / prune decisions are accurate for
+creative workflows that iterate on images across many turns.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+from agent.context_compressor import (
+    _CHARS_PER_TOKEN,
+    _IMAGE_CHAR_EQUIVALENT,
+    _IMAGE_TOKEN_ESTIMATE,
+    _content_length_for_budget,
+)
+
+
+class TestContentLengthForBudget:
+    def test_plain_string(self):
+        assert _content_length_for_budget("hello world") == 11
+
+    def test_empty_string(self):
+        assert _content_length_for_budget("") == 0
+
+    def test_none_coerces_to_zero(self):
+        assert _content_length_for_budget(None) == 0
+
+    def test_text_only_list(self):
+        content = [
+            {"type": "text", "text": "first"},
+            {"type": "text", "text": "second"},
+        ]
+        assert _content_length_for_budget(content) == 5 + 6
+
+    def test_single_image_part_charges_fixed_budget(self):
+        content = [
+            {"type": "text", "text": "look"},
+            {"type": "image_url", "image_url": {"url": "data:image/png;base64,XXXX"}},
+        ]
+        # 4 chars of text + 1 image at fixed char-equivalent
+        assert _content_length_for_budget(content) == 4 + _IMAGE_CHAR_EQUIVALENT
+
+    def test_image_url_raw_base64_is_not_counted_as_chars(self):
+        """A 1MB base64 blob inside an image_url must NOT inflate token count.
+
+        The flat image estimate is what the provider actually bills; the raw
+        base64 is transport payload, not context tokens.
+        """
+        huge_url = "data:image/png;base64," + ("A" * 1_000_000)
+        content = [
+            {"type": "image_url", "image_url": {"url": huge_url}},
+        ]
+        # Exactly one image's worth, not 1M + something.
+        assert _content_length_for_budget(content) == _IMAGE_CHAR_EQUIVALENT
+
+    def test_multiple_image_parts(self):
+        content = [
+            {"type": "text", "text": "compare"},
+            {"type": "image_url", "image_url": {"url": "data:image/png;base64,AAA"}},
+            {"type": "image_url", "image_url": {"url": "data:image/png;base64,BBB"}},
+            {"type": "image_url", "image_url": {"url": "data:image/png;base64,CCC"}},
+        ]
+        assert _content_length_for_budget(content) == 7 + 3 * _IMAGE_CHAR_EQUIVALENT
+
+    def test_openai_responses_input_image_shape(self):
+        """Responses API uses type=input_image with top-level image_url string."""
+        content = [
+            {"type": "input_text", "text": "hey"},
+            {"type": "input_image", "image_url": "data:image/png;base64,XX"},
+        ]
+        # input_text has .text "hey" (3 chars) + 1 image
+        assert _content_length_for_budget(content) == 3 + _IMAGE_CHAR_EQUIVALENT
+
+    def test_anthropic_native_image_shape(self):
+        """Anthropic native shape: {type: image, source: {...}}."""
+        content = [
+            {"type": "text", "text": "hi"},
+            {"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": "XX"}},
+        ]
+        assert _content_length_for_budget(content) == 2 + _IMAGE_CHAR_EQUIVALENT
+
+    def test_bare_string_part_in_list(self):
+        """Older code paths sometimes produce mixed list-of-strings content."""
+        content = ["hello", {"type": "text", "text": "world"}]
+        assert _content_length_for_budget(content) == 5 + 5
+
+    def test_image_estimate_constant_is_reasonable(self):
+        """Sanity-check the estimate aligns with real provider billing.
+
+        Anthropic ≈ width*height/750 → ~1600 for 1000×1200.
+        OpenAI GPT-4o high-detail 2048×2048 ≈ 1445.
+        Gemini 258/tile × 6 tiles for a 2048×2048 ≈ 1548.
+        Anything in the 800-2000 range is defensible. Enforce bounds so an
+        accidental edit doesn't drop it to e.g. 16.
+        """
+        assert 800 <= _IMAGE_TOKEN_ESTIMATE <= 2500
+        assert _IMAGE_CHAR_EQUIVALENT == _IMAGE_TOKEN_ESTIMATE * _CHARS_PER_TOKEN
+
+
+class TestTokenBudgetWithImages:
+    """Integration: the compressor's tail-cut decision now respects image cost."""
+
+    def test_image_heavy_turns_count_toward_budget(self):
+        """A tail with 5 image-bearing turns should blow past a 5K token budget."""
+        from agent.context_compressor import ContextCompressor
+
+        # Minimal compressor fixture — just enough to call _find_tail_cut_by_tokens
+        cc = object.__new__(ContextCompressor)
+        cc.tail_token_budget = 5000
+
+        # Build 10 messages: 5 with images, 5 with short text. Without the
+        # image-tokens fix, the compressor would think all 10 fit in 5K and
+        # protect them all. With the fix, images alone cost 5 × 1600 = 8K,
+        # so the tail should be trimmed.
+        messages = [{"role": "system", "content": "sys"}]
+        for i in range(5):
+            messages.append({
+                "role": "user",
+                "content": [
+                    {"type": "text", "text": f"turn {i}"},
+                    {"type": "image_url", "image_url": {"url": "data:image/png;base64,AAA"}},
+                ],
+            })
+            messages.append({
+                "role": "assistant",
+                "content": f"response {i}",
+            })
+
+        cut = cc._find_tail_cut_by_tokens(messages, head_end=0, token_budget=5000)
+
+        # Budget is 5K, soft ceiling 7.5K. 5 images alone = 8000 image-tokens.
+        # Walking backward, the compressor should stop before including all 5.
+        # Exact cut depends on text lengths and min_tail, but it MUST be > 1
+        # (at least some head-side messages should be compressible).
+        assert cut > 1, (
+            f"Expected image-heavy tail to be trimmed; compressor placed cut at "
+            f"{cut} out of {len(messages)} (image tokens were likely ignored)."
+        )

tests/agent/test_error_classifier.py

@@ -54,7 +54,7 @@ class TestFailoverReason:
         expected = {
             "auth", "auth_permanent", "billing", "rate_limit",
             "overloaded", "server_error", "timeout",
-            "context_overflow", "payload_too_large",
+            "context_overflow", "payload_too_large", "image_too_large",
             "model_not_found", "format_error",
             "provider_policy_blocked",
             "thinking_signature", "long_context_tier", "unknown",

tests/agent/test_image_routing.py

@@ -0,0 +1,213 @@
+"""Tests for agent/image_routing.py — the per-turn image input mode decision."""
+
+from __future__ import annotations
+
+import base64
+from pathlib import Path
+from unittest.mock import patch
+
+import pytest
+
+from agent.image_routing import (
+    _coerce_mode,
+    _explicit_aux_vision_override,
+    build_native_content_parts,
+    decide_image_input_mode,
+)
+
+
+# ─── _coerce_mode ────────────────────────────────────────────────────────────
+
+
+class TestCoerceMode:
+    def test_valid_modes_pass_through(self):
+        assert _coerce_mode("auto") == "auto"
+        assert _coerce_mode("native") == "native"
+        assert _coerce_mode("text") == "text"
+
+    def test_case_insensitive(self):
+        assert _coerce_mode("NATIVE") == "native"
+        assert _coerce_mode("Auto") == "auto"
+
+    def test_invalid_falls_back_to_auto(self):
+        assert _coerce_mode("nonsense") == "auto"
+        assert _coerce_mode("") == "auto"
+        assert _coerce_mode(None) == "auto"
+        assert _coerce_mode(42) == "auto"
+
+    def test_strips_whitespace(self):
+        assert _coerce_mode("  native  ") == "native"
+
+
+# ─── _explicit_aux_vision_override ───────────────────────────────────────────
+
+
+class TestExplicitAuxVisionOverride:
+    def test_none_config(self):
+        assert _explicit_aux_vision_override(None) is False
+
+    def test_empty_config(self):
+        assert _explicit_aux_vision_override({}) is False
+
+    def test_default_auto_is_not_explicit(self):
+        cfg = {"auxiliary": {"vision": {"provider": "auto", "model": "", "base_url": ""}}}
+        assert _explicit_aux_vision_override(cfg) is False
+
+    def test_provider_set_is_explicit(self):
+        cfg = {"auxiliary": {"vision": {"provider": "openrouter", "model": ""}}}
+        assert _explicit_aux_vision_override(cfg) is True
+
+    def test_model_set_is_explicit(self):
+        cfg = {"auxiliary": {"vision": {"provider": "auto", "model": "google/gemini-2.5-flash"}}}
+        assert _explicit_aux_vision_override(cfg) is True
+
+    def test_base_url_set_is_explicit(self):
+        cfg = {"auxiliary": {"vision": {"provider": "auto", "base_url": "http://localhost:11434"}}}
+        assert _explicit_aux_vision_override(cfg) is True
+
+
+# ─── decide_image_input_mode ─────────────────────────────────────────────────
+
+
+class TestDecideImageInputMode:
+    def test_explicit_native_overrides_everything(self):
+        cfg = {"agent": {"image_input_mode": "native"}}
+        # Non-vision model, aux-vision explicitly configured: native still wins.
+        cfg["auxiliary"] = {"vision": {"provider": "openrouter", "model": "foo"}}
+        with patch("agent.image_routing._lookup_supports_vision", return_value=False):
+            assert decide_image_input_mode("openrouter", "some-non-vision-model", cfg) == "native"
+
+    def test_explicit_text_overrides_everything(self):
+        cfg = {"agent": {"image_input_mode": "text"}}
+        with patch("agent.image_routing._lookup_supports_vision", return_value=True):
+            assert decide_image_input_mode("anthropic", "claude-sonnet-4", cfg) == "text"
+
+    def test_auto_with_vision_capable_model(self):
+        with patch("agent.image_routing._lookup_supports_vision", return_value=True):
+            assert decide_image_input_mode("anthropic", "claude-sonnet-4", {}) == "native"
+
+    def test_auto_with_non_vision_model(self):
+        with patch("agent.image_routing._lookup_supports_vision", return_value=False):
+            assert decide_image_input_mode("openrouter", "qwen/qwen3-235b", {}) == "text"
+
+    def test_auto_with_unknown_model(self):
+        with patch("agent.image_routing._lookup_supports_vision", return_value=None):
+            assert decide_image_input_mode("openrouter", "brand-new-slug", {}) == "text"
+
+    def test_auto_respects_aux_vision_override_even_for_vision_model(self):
+        """If the user configured a dedicated vision backend, don't bypass it."""
+        cfg = {"auxiliary": {"vision": {"provider": "openrouter", "model": "google/gemini-2.5-flash"}}}
+        with patch("agent.image_routing._lookup_supports_vision", return_value=True):
+            assert decide_image_input_mode("anthropic", "claude-sonnet-4", cfg) == "text"
+
+    def test_none_config_is_auto(self):
+        with patch("agent.image_routing._lookup_supports_vision", return_value=True):
+            assert decide_image_input_mode("anthropic", "claude-sonnet-4", None) == "native"
+
+    def test_invalid_mode_coerces_to_auto(self):
+        cfg = {"agent": {"image_input_mode": "weird-value"}}
+        with patch("agent.image_routing._lookup_supports_vision", return_value=True):
+            assert decide_image_input_mode("anthropic", "claude-sonnet-4", cfg) == "native"
+
+
+# ─── build_native_content_parts ──────────────────────────────────────────────
+
+
+def _png_bytes() -> bytes:
+    """Return a tiny valid 1x1 transparent PNG."""
+    return base64.b64decode(
+        "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR4nGNgYGBgAAAABQABpfZFQAAAAABJRU5ErkJggg=="
+    )
+
+
+class TestBuildNativeContentParts:
+    def test_text_then_image(self, tmp_path: Path):
+        img = tmp_path / "cat.png"
+        img.write_bytes(_png_bytes())
+        parts, skipped = build_native_content_parts("hello", [str(img)])
+        assert skipped == []
+        assert len(parts) == 2
+        assert parts[0] == {"type": "text", "text": "hello"}
+        assert parts[1]["type"] == "image_url"
+        assert parts[1]["image_url"]["url"].startswith("data:image/png;base64,")
+
+    def test_empty_text_inserts_default_prompt(self, tmp_path: Path):
+        img = tmp_path / "cat.jpg"
+        img.write_bytes(_png_bytes())
+        parts, skipped = build_native_content_parts("", [str(img)])
+        assert skipped == []
+        # Even with empty user text, we insert a neutral prompt so the turn
+        # isn't just pixels.
+        assert parts[0]["type"] == "text"
+        assert parts[0]["text"] == "What do you see in this image?"
+        assert parts[1]["type"] == "image_url"
+
+    def test_missing_file_is_skipped(self, tmp_path: Path):
+        parts, skipped = build_native_content_parts("hi", [str(tmp_path / "missing.png")])
+        assert skipped == [str(tmp_path / "missing.png")]
+        # Only text remains.
+        assert parts == [{"type": "text", "text": "hi"}]
+
+    def test_multiple_images(self, tmp_path: Path):
+        img1 = tmp_path / "a.png"
+        img2 = tmp_path / "b.png"
+        img1.write_bytes(_png_bytes())
+        img2.write_bytes(_png_bytes())
+        parts, skipped = build_native_content_parts("compare these", [str(img1), str(img2)])
+        assert skipped == []
+        image_parts = [p for p in parts if p.get("type") == "image_url"]
+        assert len(image_parts) == 2
+
+    def test_mime_inference_jpg(self, tmp_path: Path):
+        img = tmp_path / "photo.jpg"
+        img.write_bytes(_png_bytes())  # bytes are PNG but extension is jpg
+        parts, _ = build_native_content_parts("x", [str(img)])
+        url = parts[1]["image_url"]["url"]
+        assert url.startswith("data:image/jpeg;base64,")
+
+    def test_mime_inference_webp(self, tmp_path: Path):
+        img = tmp_path / "pic.webp"
+        img.write_bytes(_png_bytes())
+        parts, _ = build_native_content_parts("", [str(img)])
+        url = parts[1]["image_url"]["url"]
+        assert url.startswith("data:image/webp;base64,")
+
+
+# ─── Oversize handling ───────────────────────────────────────────────────────
+
+
+class TestLargeImageHandling:
+    """Large images attach at native size; shrink is handled reactively at
+    retry time in ``run_agent._try_shrink_image_parts_in_messages`` rather
+    than proactively here.
+    """
+
+    def test_large_image_passes_through_unchanged(self, tmp_path: Path):
+        """A multi-MB image is attached as-is — no resize, no skip."""
+        from agent import image_routing as _ir
+
+        img = tmp_path / "medium.png"
+        # 200 KB of real bytes; not huge but enough to verify no size gate fires.
+        img.write_bytes(b"\x89PNG\r\n\x1a\n" + b"X" * 200_000)
+        url = _ir._file_to_data_url(img)
+        assert url is not None
+        assert url.startswith("data:image/png;base64,")
+        # Base64 expansion means output is ~4/3 of input, plus header.
+        assert len(url) > 200_000
+
+    def test_missing_file_returns_none(self, tmp_path: Path):
+        from agent import image_routing as _ir
+        missing = tmp_path / "does_not_exist.png"
+        assert _ir._file_to_data_url(missing) is None
+
+    def test_build_native_parts_no_provider_kwarg(self, tmp_path: Path):
+        """build_native_content_parts takes text + paths, no provider kwarg."""
+        from agent import image_routing as _ir
+
+        img = tmp_path / "cat.png"
+        img.write_bytes(_png_bytes())
+        parts, skipped = _ir.build_native_content_parts("hi", [str(img)])
+        assert skipped == []
+        assert len(parts) == 2
+        assert parts[0]["type"] == "text"
+        assert parts[1]["type"] == "image_url"

tests/run_agent/test_image_shrink_recovery.py

@@ -0,0 +1,277 @@
+"""Tests for reactive image-shrink recovery.
+
+Covers the full chain for Anthropic's 5 MB per-image ceiling (and any
+future provider that returns an image-too-large error):
+
+  1. agent/error_classifier.py: 400 with "image exceeds 5 MB maximum"
+     gets FailoverReason.image_too_large, not context_overflow.
+  2. run_agent._try_shrink_image_parts_in_messages mutates the API
+     payload in-place, re-encoding native data: URL image parts to fit
+     under 4 MB using vision_tools._resize_image_for_vision.
+
+The end-to-end wiring in the retry loop is not unit-tested here — it's
+covered by the live E2E in the PR description. These tests lock in the
+two pieces that matter independently: the classifier signal and the
+payload rewriter.
+"""
+
+from __future__ import annotations
+
+import base64
+from pathlib import Path
+
+import pytest
+
+from agent.error_classifier import FailoverReason, classify_api_error
+
+
+class _FakeApiError(Exception):
+    """Stand-in for an openai.BadRequestError with status_code + body."""
+
+    def __init__(self, status_code: int, message: str, body: dict | None = None):
+        super().__init__(message)
+        self.status_code = status_code
+        self.body = body or {"error": {"message": message}}
+        self.response = None  # required by some code paths
+
+
+# ─── Classifier ──────────────────────────────────────────────────────────────
+
+
+class TestImageTooLargeClassification:
+    def test_anthropic_400_image_exceeds_message(self):
+        """Anthropic's exact wording must classify as image_too_large, not context."""
+        err = _FakeApiError(
+            status_code=400,
+            message=(
+                "messages.0.content.1.image.source.base64: image exceeds 5 MB "
+                "maximum: 12966600 bytes > 5242880 bytes"
+            ),
+        )
+        result = classify_api_error(err, provider="anthropic", model="claude-sonnet-4-6")
+        assert result.reason == FailoverReason.image_too_large
+        assert result.retryable is True
+
+    def test_generic_image_too_large_no_status(self):
+        """No status_code path: message text alone triggers classification."""
+        err = Exception("image too large for this endpoint")
+        result = classify_api_error(err, provider="some-provider", model="some-model")
+        assert result.reason == FailoverReason.image_too_large
+        assert result.retryable is True
+
+    def test_image_too_large_not_confused_with_context_overflow(self):
+        """'image exceeds' must NOT be mis-classified as context_overflow.
+
+        The context_overflow patterns include 'exceeds the limit' which is a
+        superstring risk — verify the image-too-large check fires first.
+        """
+        err = _FakeApiError(
+            status_code=400,
+            message="image exceeds the limit for this model",
+        )
+        result = classify_api_error(err, provider="anthropic", model="claude-sonnet-4-6")
+        assert result.reason == FailoverReason.image_too_large
+
+    def test_regular_context_overflow_unaffected(self):
+        """Context-overflow errors without image keywords still classify correctly."""
+        err = _FakeApiError(
+            status_code=400,
+            message="prompt is too long: context length 300000 exceeds max of 200000",
+        )
+        result = classify_api_error(err, provider="anthropic", model="claude-sonnet-4-6")
+        assert result.reason == FailoverReason.context_overflow
+
+
+# ─── Shrink helper ───────────────────────────────────────────────────────────
+
+
+def _big_png_data_url(size_kb: int) -> str:
+    """Build a data URL with a plausible large base64 payload."""
+    # Use real PNG header so MIME detection works; fill to target size.
+    raw = b"\x89PNG\r\n\x1a\n" + b"X" * (size_kb * 1024)
+    return "data:image/png;base64," + base64.b64encode(raw).decode("ascii")
+
+
+def _make_agent():
+    """Build a bare AIAgent for method-level testing, no provider setup."""
+    from run_agent import AIAgent
+    agent = object.__new__(AIAgent)
+    agent.provider = "anthropic"
+    agent.model = "claude-sonnet-4-6"
+    return agent
+
+
+class TestShrinkImagePartsHelper:
+    def test_no_messages_returns_false(self):
+        agent = _make_agent()
+        assert agent._try_shrink_image_parts_in_messages([]) is False
+        assert agent._try_shrink_image_parts_in_messages(None) is False
+
+    def test_no_image_parts_returns_false(self):
+        agent = _make_agent()
+        msgs = [
+            {"role": "user", "content": "plain text"},
+            {"role": "assistant", "content": "ack"},
+        ]
+        assert agent._try_shrink_image_parts_in_messages(msgs) is False
+
+    def test_small_image_part_not_shrunk(self, monkeypatch):
+        """An image under 4 MB is left alone — shrink helper only touches oversized ones."""
+        agent = _make_agent()
+        small_url = _big_png_data_url(100)  # ~100 KB + b64 overhead
+
+        resize_hits = {"count": 0}
+        monkeypatch.setattr(
+            "tools.vision_tools._resize_image_for_vision",
+            lambda *a, **kw: resize_hits.__setitem__("count", resize_hits["count"] + 1) or small_url,
+            raising=False,
+        )
+
+        msgs = [{
+            "role": "user",
+            "content": [
+                {"type": "text", "text": "hi"},
+                {"type": "image_url", "image_url": {"url": small_url}},
+            ],
+        }]
+        assert agent._try_shrink_image_parts_in_messages(msgs) is False
+        assert resize_hits["count"] == 0
+        # URL unchanged.
+        assert msgs[0]["content"][1]["image_url"]["url"] == small_url
+
+    def test_oversized_image_url_dict_shape_rewritten(self, monkeypatch):
+        """OpenAI chat.completions shape: {image_url: {url: data:...}}."""
+        agent = _make_agent()
+        oversized_url = _big_png_data_url(5000)  # ~5 MB raw → ~6.7 MB b64
+        shrunk = "data:image/jpeg;base64," + "A" * 1000  # small
+
+        def _fake_resize(path, mime_type=None, max_base64_bytes=None):
+            return shrunk
+
+        monkeypatch.setattr(
+            "tools.vision_tools._resize_image_for_vision",
+            _fake_resize,
+            raising=False,
+        )
+
+        msgs = [{
+            "role": "user",
+            "content": [
+                {"type": "text", "text": "look"},
+                {"type": "image_url", "image_url": {"url": oversized_url}},
+            ],
+        }]
+        changed = agent._try_shrink_image_parts_in_messages(msgs)
+        assert changed is True
+        assert msgs[0]["content"][1]["image_url"]["url"] == shrunk
+
+    def test_oversized_input_image_string_shape_rewritten(self, monkeypatch):
+        """OpenAI Responses shape: {type: input_image, image_url: "data:..."}."""
+        agent = _make_agent()
+        oversized_url = _big_png_data_url(5000)
+        shrunk = "data:image/jpeg;base64," + "B" * 1000
+
+        monkeypatch.setattr(
+            "tools.vision_tools._resize_image_for_vision",
+            lambda *a, **kw: shrunk,
+            raising=False,
+        )
+
+        msgs = [{
+            "role": "user",
+            "content": [
+                {"type": "input_text", "text": "look"},
+                {"type": "input_image", "image_url": oversized_url},
+            ],
+        }]
+        changed = agent._try_shrink_image_parts_in_messages(msgs)
+        assert changed is True
+        assert msgs[0]["content"][1]["image_url"] == shrunk
+
+    def test_multiple_images_all_shrunk(self, monkeypatch):
+        agent = _make_agent()
+        big1 = _big_png_data_url(5000)
+        big2 = _big_png_data_url(6000)
+        shrunk = "data:image/jpeg;base64," + "C" * 500
+
+        monkeypatch.setattr(
+            "tools.vision_tools._resize_image_for_vision",
+            lambda *a, **kw: shrunk,
+            raising=False,
+        )
+
+        msgs = [{
+            "role": "user",
+            "content": [
+                {"type": "text", "text": "compare"},
+                {"type": "image_url", "image_url": {"url": big1}},
+                {"type": "image_url", "image_url": {"url": big2}},
+            ],
+        }]
+        changed = agent._try_shrink_image_parts_in_messages(msgs)
+        assert changed is True
+        assert msgs[0]["content"][1]["image_url"]["url"] == shrunk
+        assert msgs[0]["content"][2]["image_url"]["url"] == shrunk
+
+    def test_http_url_images_not_touched(self, monkeypatch):
+        """Only data: URLs are candidates — http URLs are server-fetched."""
+        agent = _make_agent()
+
+        resize_hits = {"count": 0}
+        monkeypatch.setattr(
+            "tools.vision_tools._resize_image_for_vision",
+            lambda *a, **kw: resize_hits.__setitem__("count", resize_hits["count"] + 1) or "shrunk",
+            raising=False,
+        )
+
+        msgs = [{
+            "role": "user",
+            "content": [
+                {"type": "text", "text": "at this url"},
+                {"type": "image_url", "image_url": {"url": "https://example.com/big.png"}},
+            ],
+        }]
+        assert agent._try_shrink_image_parts_in_messages(msgs) is False
+        assert resize_hits["count"] == 0
+
+    def test_shrink_failure_returns_false_and_leaves_url_intact(self, monkeypatch):
+        """If re-encode fails, leave the URL alone so the caller surfaces the original error."""
+        agent = _make_agent()
+        oversized_url = _big_png_data_url(5000)
+
+        monkeypatch.setattr(
+            "tools.vision_tools._resize_image_for_vision",
+            lambda *a, **kw: None,  # resize returned nothing usable
+            raising=False,
+        )
+
+        msgs = [{
+            "role": "user",
+            "content": [
+                {"type": "image_url", "image_url": {"url": oversized_url}},
+            ],
+        }]
+        assert agent._try_shrink_image_parts_in_messages(msgs) is False
+        assert msgs[0]["content"][0]["image_url"]["url"] == oversized_url
+
+    def test_shrink_that_makes_it_bigger_rejected(self, monkeypatch):
+        """If the 'shrink' somehow produces a larger payload, skip it."""
+        agent = _make_agent()
+        oversized_url = _big_png_data_url(5000)
+        even_bigger = "data:image/png;base64," + "Z" * (10 * 1024 * 1024)
+
+        monkeypatch.setattr(
+            "tools.vision_tools._resize_image_for_vision",
+            lambda *a, **kw: even_bigger,
+            raising=False,
+        )
+
+        msgs = [{
+            "role": "user",
+            "content": [
+                {"type": "image_url", "image_url": {"url": oversized_url}},
+            ],
+        }]
+        assert agent._try_shrink_image_parts_in_messages(msgs) is False
+        # Original URL still in place, not replaced by the bigger one.
+        assert msgs[0]["content"][0]["image_url"]["url"] == oversized_url

tests/run_agent/test_vision_aware_preprocessing.py

@@ -0,0 +1,170 @@
+"""Tests for the vision-aware image preprocessing in run_agent.py.
+
+Covers:
+
+* ``_prepare_anthropic_messages_for_api`` — passes image parts through
+  unchanged when the active model reports ``supports_vision=True`` (the
+  adapter handles them natively), and falls back to text-description
+  replacement when the model lacks vision.
+
+* ``_prepare_messages_for_non_vision_model`` — the mirror method for the
+  chat.completions / codex_responses paths. Same contract.
+"""
+
+from __future__ import annotations
+
+from unittest.mock import MagicMock, patch
+
+import pytest
+
+from run_agent import AIAgent
+
+
+def _make_agent() -> AIAgent:
+    """Build a bare-bones AIAgent instance without running __init__.
+
+    Avoids the heavy provider/credential setup for these pure-method tests.
+    """
+    agent = object.__new__(AIAgent)
+    agent.provider = "anthropic"
+    agent.model = "claude-sonnet-4"
+    agent._anthropic_image_fallback_cache = {}
+    return agent
+
+
+IMG_PARTS_USER_MSG = {
+    "role": "user",
+    "content": [
+        {"type": "text", "text": "What's in this image?"},
+        {"type": "image_url", "image_url": {"url": "data:image/png;base64,AAAA"}},
+    ],
+}
+
+PLAIN_USER_MSG = {"role": "user", "content": "hello, no images here"}
+
+
+# ─── _prepare_anthropic_messages_for_api ─────────────────────────────────────
+
+
+class TestPrepareAnthropicMessages:
+    def test_no_images_passes_through(self):
+        agent = _make_agent()
+        msgs = [PLAIN_USER_MSG]
+        out = agent._prepare_anthropic_messages_for_api(msgs)
+        assert out is msgs  # unchanged reference
+
+    def test_vision_capable_passes_images_through(self):
+        """The Anthropic adapter handles image_url/input_image natively."""
+        agent = _make_agent()
+        with patch.object(agent, "_model_supports_vision", return_value=True):
+            out = agent._prepare_anthropic_messages_for_api([IMG_PARTS_USER_MSG])
+        # Passes through unchanged — image_url parts still present.
+        assert out[0]["content"][1]["type"] == "image_url"
+
+    def test_non_vision_replaces_images_with_text(self):
+        agent = _make_agent()
+        with patch.object(agent, "_model_supports_vision", return_value=False), \
+             patch.object(
+                 agent,
+                 "_describe_image_for_anthropic_fallback",
+                 return_value="[Image description: a cat]",
+             ):
+            out = agent._prepare_anthropic_messages_for_api([IMG_PARTS_USER_MSG])
+        # Content collapsed to a string containing the description + user text.
+        content = out[0]["content"]
+        assert isinstance(content, str)
+        assert "[Image description: a cat]" in content
+        assert "What's in this image?" in content
+        # No more image parts.
+        assert "image_url" not in content
+
+
+# ─── _prepare_messages_for_non_vision_model ──────────────────────────────────
+
+
+class TestPrepareMessagesForNonVision:
+    def test_no_images_passes_through(self):
+        agent = _make_agent()
+        msgs = [PLAIN_USER_MSG]
+        out = agent._prepare_messages_for_non_vision_model(msgs)
+        assert out is msgs
+
+    def test_vision_capable_passes_through(self):
+        """For vision-capable models on chat.completions path, provider handles pixels."""
+        agent = _make_agent()
+        agent.provider = "openrouter"
+        agent.model = "anthropic/claude-sonnet-4"
+        with patch.object(agent, "_model_supports_vision", return_value=True):
+            out = agent._prepare_messages_for_non_vision_model([IMG_PARTS_USER_MSG])
+        assert out[0]["content"][1]["type"] == "image_url"
+
+    def test_non_vision_strips_images(self):
+        agent = _make_agent()
+        agent.provider = "openrouter"
+        agent.model = "qwen/qwen3-235b-a22b"
+        with patch.object(agent, "_model_supports_vision", return_value=False), \
+             patch.object(
+                 agent,
+                 "_describe_image_for_anthropic_fallback",
+                 return_value="[Image description: a dog]",
+             ):
+            out = agent._prepare_messages_for_non_vision_model([IMG_PARTS_USER_MSG])
+        content = out[0]["content"]
+        assert isinstance(content, str)
+        assert "[Image description: a dog]" in content
+        assert "image_url" not in content
+
+    def test_multiple_messages_with_mixed_content(self):
+        agent = _make_agent()
+        agent.model = "qwen/qwen3-235b"
+        msgs = [
+            {"role": "user", "content": "first turn"},
+            {"role": "assistant", "content": "ack"},
+            IMG_PARTS_USER_MSG,
+        ]
+        with patch.object(agent, "_model_supports_vision", return_value=False), \
+             patch.object(
+                 agent,
+                 "_describe_image_for_anthropic_fallback",
+                 return_value="[Image: thing]",
+             ):
+            out = agent._prepare_messages_for_non_vision_model(msgs)
+        # First two messages unchanged (no images), third stripped.
+        assert out[0]["content"] == "first turn"
+        assert out[1]["content"] == "ack"
+        assert isinstance(out[2]["content"], str)
+        assert "[Image: thing]" in out[2]["content"]
+
+
+# ─── _model_supports_vision ──────────────────────────────────────────────────
+
+
+class TestModelSupportsVision:
+    def test_missing_provider_or_model_returns_false(self):
+        agent = _make_agent()
+        agent.provider = ""
+        agent.model = "claude-sonnet-4"
+        assert agent._model_supports_vision() is False
+        agent.provider = "anthropic"
+        agent.model = ""
+        assert agent._model_supports_vision() is False
+
+    def test_uses_get_model_capabilities(self):
+        agent = _make_agent()
+        fake_caps = MagicMock()
+        fake_caps.supports_vision = True
+        with patch("agent.models_dev.get_model_capabilities", return_value=fake_caps):
+            assert agent._model_supports_vision() is True
+        fake_caps.supports_vision = False
+        with patch("agent.models_dev.get_model_capabilities", return_value=fake_caps):
+            assert agent._model_supports_vision() is False
+
+    def test_none_caps_returns_false(self):
+        agent = _make_agent()
+        with patch("agent.models_dev.get_model_capabilities", return_value=None):
+            assert agent._model_supports_vision() is False
+
+    def test_exception_returns_false(self):
+        agent = _make_agent()
+        with patch("agent.models_dev.get_model_capabilities", side_effect=RuntimeError("boom")):
+            assert agent._model_supports_vision() is False