| 1 | # `read_media` Operator Guide |
| 2 | |
| 3 | `read_media` is a safe, first-class image reading and preprocessing tool for Codewhale v0.9.10. It allows vision-capable coding models to inspect visual assets (diagrams, UI mockups, screenshots, rendered graphs) with strict security, memory bounds, and privacy guards. |
| 4 | |
| 5 | --- |
| 6 | |
| 7 | ## 1. Overview and Scope |
| 8 | |
| 9 | - **Supported Formats:** PNG, JPEG, GIF, WebP. |
| 10 | - **Out of Scope:** Video files, audio streams, and background/automatic screenshot watching are deliberately excluded. |
| 11 | - **Provider-Neutral Wiring:** Decoded and normalized images are converted into native image parts across all supported providers (OpenAI Chat Completions, Anthropic Messages, and OpenAI Responses API). |
| 12 | |
| 13 | ### Show the agent a screenshot |
| 14 | |
| 15 | - Paste a clipboard image into the composer with the normal terminal paste |
| 16 | shortcut, or run `/attach <path>` for an existing PNG, JPEG, GIF, or WebP. |
| 17 | - A visible attachment row appears above the composer before the turn is sent. |
| 18 | Temporary macOS `NSIRD_screencaptureui` paths are copied into Codewhale's |
| 19 | stable attachment store when ingested. |
| 20 | - Ask the agent to inspect the screenshot. The image is sent only as part of |
| 21 | that explicit turn action; merely having a screenshot path or artifact does |
| 22 | not trigger background analysis. |
| 23 | |
| 24 | `read_media` is the corresponding agent-side path for inspecting another |
| 25 | image later in the task without requiring the operator to attach it again. |
| 26 | |
| 27 | --- |
| 28 | |
| 29 | ## 2. Activation and Catalog Policy |
| 30 | |
| 31 | - **Default-Off / Deferred Loading:** To preserve model context budgets, `read_media` is registered as a deferred tool (`defer_loading = true`) rather than occupying active slots in the default core catalog. |
| 32 | - **Explicit Model Invocation:** The tool is invoked explicitly by name when the model or user requests media inspection. |
| 33 | - **Eager Configuration:** Operators who want `read_media` always pre-loaded in the model tool catalog can configure: |
| 34 | |
| 35 | ```toml |
| 36 | [tools] |
| 37 | always_load = ["read_media"] |
| 38 | ``` |
| 39 | |
| 40 | --- |
| 41 | |
| 42 | ## 3. Tool Parameters and Schema |
| 43 | |
| 44 | ```json |
| 45 | { |
| 46 | "path": "docs/architecture.png", |
| 47 | "crop": { |
| 48 | "x": 100, |
| 49 | "y": 50, |
| 50 | "width": 800, |
| 51 | "height": 600 |
| 52 | }, |
| 53 | "detail": "auto" |
| 54 | } |
| 55 | ``` |
| 56 | |
| 57 | | Parameter | Type | Required | Description | |
| 58 | |-----------|------|----------|-------------| |
| 59 | | `path` | string | **Yes** | Workspace-relative or trusted external path to the image file. | |
| 60 | | `crop` | object | No | Optional pixel bounding box `{ "x": u32, "y": u32, "width": u32, "height": u32 }` (0-indexed). | |
| 61 | | `detail` | string | No | Resolution target: `"auto"` (max 2048px, default), `"low"` (max 1024px), `"high"` / `"original"` (up to 4096px). | |
| 62 | |
| 63 | --- |
| 64 | |
| 65 | ## 4. Safety, Privacy, and Guardrails |
| 66 | |
| 67 | ### 4.1. Workspace Boundary and Credential Protection |
| 68 | - **Workspace Containment:** Paths must resolve within the workspace or user-approved trusted external paths (`/trust`). Symlink escapes outside trusted roots are rejected. |
| 69 | - **Credential Protection:** Codewhale configuration (`config.toml`, `.codewhale/`, `.deepseek/`, secrets directory) cannot be read via `read_media` and will fail with `PermissionDenied`. |
| 70 | |
| 71 | ### 4.2. Decompression-Bomb and Memory Limits |
| 72 | - **Source Byte Limit:** Maximum source file size before decoding is **20 MiB** (`MAX_SOURCE_IMAGE_BYTES`). Oversized files are rejected before allocation. |
| 73 | - **Dimension Guards:** Maximum permitted image width and height is **8192 px** (`MAX_IMAGE_DIMENSION`). |
| 74 | - **Pixel Budget:** Maximum total pixel count is **33,554,432 pixels** (~33.5 megapixels, `MAX_IMAGE_PIXELS`). |
| 75 | - **Memory Ceiling:** Safe memory allocation during decode is capped at **64 MiB** (`MAX_DECODE_ALLOC_BYTES`). |
| 76 | - **Wire Payload Limit:** Re-encoded image payload is capped at **5 MiB** (`MAX_WIRE_IMAGE_BYTES`), matching provider constraints. |
| 77 | |
| 78 | ### 4.3. Active Route Vision Checks |
| 79 | - Before reading an image, `read_media` inspects `context.route_capabilities.image_input`. |
| 80 | - If the active model route explicitly lacks vision support (`CapabilityState::Unsupported`), the tool returns an actionable error directing the operator to switch to a vision model: |
| 81 | |
| 82 | ```text |
| 83 | read_media: the active model route does not support image input. Switch to a route marked vision-capable with /model, or configure the route's image_input capability, then try again. |
| 84 | ``` |
| 85 | |
| 86 | Only a known `Unsupported` capability blocks the explicit tool call. An |
| 87 | `Unknown` capability is admitted deliberately, matching normal attachment |
| 88 | routing: custom and self-hosted providers often do not publish modality |
| 89 | metadata, so their provider response remains authoritative. Operators who |
| 90 | need a fail-closed route can set its `image_input` capability explicitly. |
| 91 | |
| 92 | --- |
| 93 | |
| 94 | ## 5. Typed Receipts and Wire Integration |
| 95 | |
| 96 | Each successful execution yields: |
| 97 | 1. **Human-Readable Receipt (`content`):** Summarizes original format, source and final dimensions, crop details, and byte sizes. |
| 98 | 2. **Typed JSON Metadata (`metadata`):** Contains structured dimension, crop, and byte information without exposing credentials or internal tokens. |
| 99 | 3. **Rich Image Content Block (`content_blocks`):** Attaches a standardized `ToolResultContentBlock::Image` which provider adapters wire into outbound requests: |
| 100 | - **OpenAI Chat Completions:** `image_url` block in following user message. |
| 101 | - **Anthropic Messages:** `image` base64 source inside `tool_result` block. |
| 102 | - **OpenAI Responses:** `input_image` block inside function call output. |
| 103 |