| 1 | # Computer Use |
| 2 | |
| 3 | This is the Computer Use plugin included in Codewhale. The Engine embeds the |
| 4 | runtime bundle, discovers it through the existing plugin registry, and runs |
| 5 | only the copy the user has reviewed and enabled. Codewhale Apps uses that same |
| 6 | Engine inventory and approval flow. |
| 7 | |
| 8 | The bundle provides the current consolidated MCP tools for application and window observation, |
| 9 | accessibility actions, screenshots and zoom, keyboard and pointer input, |
| 10 | clipboard access, recording, and switching between registered computers. |
| 11 | Implementation exists for macOS, Windows, Linux and HarmonyOS target devices; |
| 12 | platform support still depends on the tools, OS grants and actual device |
| 13 | verification described by the upstream project. A source build is not a |
| 14 | published or certified release. |
| 15 | The bundled plugin is enabled on macOS only while Windows and Linux ports |
| 16 | are being qualified. Their raw input uses the shared desktop; they do not |
| 17 | yet provide equivalent background control or qualified native installers. |
| 18 | HarmonyOS and SSH also require separate device and workflow evidence. |
| 19 | |
| 20 | ## Included runtime |
| 21 | |
| 22 | macOS Codewhale builds carry the compiled native helper. Using the included |
| 23 | plugin needs neither a separate Computer Use app nor a compiler. It uses the |
| 24 | permission identity of its hosting Codewhale app or terminal. Accessibility |
| 25 | and Screen Recording grants remain controlled by the user in System Settings. |
| 26 | Use `request_access` to inspect readiness; a loaded plugin alone does not prove |
| 27 | its OS permissions work. |
| 28 | |
| 29 | When the standalone Computer Use helper is registered, it owns local input |
| 30 | even when Codewhale carries an embedded native helper. Version 0.11.2 keeps its |
| 31 | whale menu, permission setup, disposable background check and human |
| 32 | Pause/Stop controls, and retires the daemon when its native owner disappears. A registered helper that cannot start causes a clear |
| 33 | error; the client does not silently bypass its controls. Without a registered |
| 34 | standalone app, the included helper remains available under the host's |
| 35 | permission identity. |
| 36 | |
| 37 | The MCP server requires Node.js 20 or newer. Codewhale Apps packages its own |
| 38 | Node runtime; the CLI uses Node on PATH. Homebrew declares the dependency; |
| 39 | Cargo and direct binary users can install Node from <https://nodejs.org/>. |
| 40 | Linux also needs the appropriate X11 or Wayland utilities and AT-SPI bindings. |
| 41 | Windows uses PowerShell and UI Automation. Linux and Windows recording is |
| 42 | currently unavailable until recorder ownership and shutdown cleanup are built. |
| 43 | HarmonyOS targets require a connected device and hdc. |
| 44 | |
| 45 | Persistent holds and drags on Linux and Windows currently require the separate |
| 46 | session-aware Computer Use helper. Their direct bundled path refuses these |
| 47 | operations before sending input. macOS carries its native input owner in the |
| 48 | included bundle. Real Windows, Wayland and mixed-display validation is still |
| 49 | required before claiming equivalent platform readiness. |
| 50 | SSH and Docker use persistent session transports; real remote workflows still |
| 51 | need target-specific acceptance. The embedded Docker build context supports |
| 52 | first-use creation of a task-owned Linux desktop when a compatible daemon is available. |
| 53 | |
| 54 | ## Control and session ownership |
| 55 | |
| 56 | Select an application before sending input. On macOS, background selection |
| 57 | (`activate:false`) supports process-directed typing and accessibility actions. |
| 58 | It refuses gestures and keyboard shortcuts that would borrow the user's keyboard focus or move the shared pointer. Some Unicode and hosted-panel |
| 59 | typing also refuses rather than taking a focus lease. Explicit |
| 60 | foreground selection (`activate:true`) enables guarded shared-desktop input |
| 61 | when the user has authorized exclusive desktop use. Neither mode is an isolated |
| 62 | computer; cursor restoration does not make concurrent pointer control safe. |
| 63 | Screenshots and zoom return actual image content to compatible vision models. |
| 64 | The nonactivating preview is on by default after binding; recording is explicit. |
| 65 | Application observations return a concise default summary; request full detail |
| 66 | when needed. Text-only models can use element roles, values and advertised |
| 67 | actions. On macOS, optional local OCR enriches the selected window observation |
| 68 | with text and raster bounds; it requires Screen Recording permission and does |
| 69 | not invent accessibility elements or actions. |
| 70 | The Engine permits one inline image up to 5 MiB per tool result; use a scoped |
| 71 | capture or zoom when a larger image receives an omission receipt. |
| 72 | |
| 73 | Each task owns its MCP connection and computer selection, observations and |
| 74 | held input. Subagents within that task share the task's Computer Use session. |
| 75 | Stopping control or closing the task releases that session's input. Stale |
| 76 | observations, unexpected foreground changes and unavailable capabilities fail |
| 77 | closed with a receipt; successful dispatch still needs application-state |
| 78 | verification. |
| 79 | |
| 80 | ## Development |
| 81 | |
| 82 | The exact upstream source revision is recorded beside this directory in |
| 83 | `computer-use.upstream-sha`. This tree contains the runtime and its tests; |
| 84 | standalone app installers and release tooling belong to the upstream project. |
| 85 | |
| 86 | Run `npm test` here for unit and protocol coverage. Those tests do not type or |
| 87 | click in the user's applications. `npm run smoke` is a separate legacy live |
| 88 | check: it captures and records the selected display, so run it only when that |
| 89 | capture is intended. The upstream parity suite contains scoped application |
| 90 | fixtures for interactive verification. |
| 91 | |
| 92 | On macOS, ordinary observations follow the selected background app. Field |
| 93 | focus, selection, context menus and scrolling use supported accessibility |
| 94 | operations; raw mouse gestures stop if the user changes foreground apps. |
| 95 | Arbitrary background dragging remains unavailable. Rebuild Core to include |
| 96 | the updated native helper; updating a separate marketplace checkout alone |
| 97 | does not update an already-installed Core binary. |
| 98 | |
| 99 | This embedded source matches canonical 9a261c4. Windows controlled-desktop |
| 100 | acceptance passed in upstream CI; signed Windows distribution, mixed-DPI/raw |
| 101 | input and continuous keyboard coexistence remain unqualified. Core discovery |
| 102 | and materialization tests do not constitute an installed model-driven trial. |
| 103 |