| 1 | # Desktop browser |
| 2 | |
| 3 | [简体中文](DESKTOP_BROWSER.zh-CN.md) |
| 4 | |
| 5 | The desktop browser is a native Chromium surface inside the Reasonix window |
| 6 | that the user and the agent operate together. Websites render in Electron |
| 7 | `WebContentsView`s owned by the shell; every agent capability goes through the |
| 8 | Go desktop service so that local and remote agents, approvals, cancellation, |
| 9 | evidence and operation records share one implementation. This document is the |
| 10 | contract between the browser panel, the shell's surface manager, the Go |
| 11 | `BrowserExecutor` and the tools the agent sees. Sessions with no shell behind |
| 12 | them get the same tools from the [CDP backend](BROWSER_CDP.md). |
| 13 | |
| 14 | ```text |
| 15 | agent tool call ─▶ Go BrowserExecutor ─▶ ledger.reserve ─▶ host/browser.* ─▶ WebContentsView |
| 16 | ▲ │ │ |
| 17 | └── result/evidence ┴──────────── ledger.settle ◀─────────┘ |
| 18 | user input on the page ─▶ guest preload ─▶ shell: epoch++ ─▶ desktop/event browser:takeover |
| 19 | ``` |
| 20 | |
| 21 | ## Surfaces and trust |
| 22 | |
| 23 | - The application window is trusted. Website views are not: sandbox on, |
| 24 | context isolation on, no Node integration, no application preload, no |
| 25 | `reasonix://` access. Their only preload observes trusted user input to |
| 26 | request a take-over and exposes nothing to the page. |
| 27 | - Partitions: `persist:browser` is the shared login partition for one |
| 28 | Reasonix data home; `temp:<id>` partitions are in-memory and discarded when |
| 29 | their last tab closes. Remote Serve windows and MCP App frames use their |
| 30 | own partitions and never share the browser partition. |
| 31 | - The `BrowserSurfaceManager` in the shell owns creation, visibility, bounds, |
| 32 | focus and destruction. The React panel submits a layout rectangle; the shell |
| 33 | validates it against the window and applies it. Any application overlay |
| 34 | (dialogs, menus, command palette) sets a single overlay state that hides |
| 35 | every native website view so a page can never paint over the app. |
| 36 | - One task owns its tabs. A tab carries `{tabID, taskID, sessionID, epoch, |
| 37 | partition}`; switching the visible tab never retargets a running plan. |
| 38 | |
| 39 | ## Panel |
| 40 | |
| 41 | The right workspace gains a browser panel: tab strip per task, address bar, |
| 42 | back/forward, reload, zoom, load errors with retry, download list, DevTools |
| 43 | toggle. Restored tabs keep only `{url, title}` for safe navigation entries; |
| 44 | no form state, credentials or replayable submissions are persisted. Tab |
| 45 | metadata and the operation log are new versioned files under the desktop |
| 46 | state directory (`browser/tabs-v1.json`, `browser/operations-v1.json`). |
| 47 | |
| 48 | ## Browser control settings |
| 49 | |
| 50 | The Settings Centre page "Browser control" owns the switches below, all stored |
| 51 | in `browser-control.json` inside the shell's userData profile (`desktop-shell/`). |
| 52 | |
| 53 | - **Built-in browser control** (`controlEnabled`, default on). The shell pushes |
| 54 | it to Go with `desktop/browserControl`, and Go reads it when it builds a |
| 55 | session: a new session registers no browser tool at all, while a running |
| 56 | session keeps the tool set it started with. |
| 57 | - **Ignore certificate errors** (`ignoreCertificateErrors`, default off). |
| 58 | Relaxes `setCertificateVerifyProc` for guest sessions only. It is applied when |
| 59 | the guest-view factory prepares a partition and re-applied to every live guest |
| 60 | session when the switch changes, so no restart is needed. |
| 61 | - **Clear built-in browser cache** clears the HTTP cache plus Cache Storage, |
| 62 | Service Workers and the shader cache of `persist:browser`, keeping cookies and |
| 63 | local site data. |
| 64 | - **Clear all browser data** additionally drops every storage type, which signs |
| 65 | every site in the built-in browser out. In-memory `temp:<id>` partitions are |
| 66 | unaffected. |
| 67 | - **Import Chrome sign-in state** reads the newest Chrome profile's `Cookies` |
| 68 | database, decrypts each value (macOS: the login keychain's `Chrome Safe |
| 69 | Storage` secret; Linux: the well-known `peanuts` password; Windows: the DPAPI |
| 70 | master key from `Local State`) and writes it into `persist:browser`. Expired |
| 71 | cookies, App-Bound (`v20`) values and rows that fail to decrypt are counted as |
| 72 | skipped. Passwords are never read. |
| 73 | |
| 74 | ## Agent capabilities |
| 75 | |
| 76 | Tools are registered through the existing capability registry as one |
| 77 | `browser` capability with these operations. Every write goes through the |
| 78 | normal approval policy (`ask`, `allow`, `deny`), the normal cancellation |
| 79 | context and the evidence trajectory; there is no separate browser approval |
| 80 | system. |
| 81 | |
| 82 | | Tool | Reads/Writes | Purpose | |
| 83 | | --- | --- | --- | |
| 84 | | `browser_tabs` | read | list the task's tabs with URL, title, loading state | |
| 85 | | `browser_open` | write | open a tab (shared or temporary partition) at a URL | |
| 86 | | `browser_navigate` | write | navigate the bound tab (URL, back, forward, reload) | |
| 87 | | `browser_snapshot` | read | structural snapshot with element references | |
| 88 | | `browser_screenshot` | read | PNG of the viewport or an element, returned as an image | |
| 89 | | `browser_click` | write | click a referenced element (trusted mouse events at its centre) | |
| 90 | | `browser_type` | write | type text into a referenced element with trusted key events; optional submit | |
| 91 | | `browser_press` | write | press a key or chord | |
| 92 | | `browser_scroll` | write | scroll the viewport or an element | |
| 93 | | `browser_select` | write | choose options in a select | |
| 94 | | `browser_upload` | write | attach task files to a file input | |
| 95 | | `browser_download` | read | wait for or list downloads of the tab | |
| 96 | | `browser_close` | write | close a tab | |
| 97 | |
| 98 | Snapshot format: an accessibility-style tree (`role "name" [state] ref=e12`) |
| 99 | produced in an isolated world of the main frame and each reachable frame. |
| 100 | References are bound to `{tabID, frameID, documentVersion}`; a navigation, |
| 101 | page replacement or take-over invalidates every earlier reference, and an |
| 102 | action with a stale reference returns `not_executed: stale reference` rather |
| 103 | than guessing. Inputs and clicks are dispatched as trusted input events |
| 104 | through the shell, never by assigning element values, so React-controlled |
| 105 | inputs, custom widgets and dynamic pages behave as they would for a user. |
| 106 | |
| 107 | Screenshots and downloads never travel through control frames: the shell |
| 108 | writes them into the task's temporary directory that Go names in the request |
| 109 | and returns the path; Go turns the file into an image or file result through |
| 110 | the existing channels. Uploads read only files the task owns; remote tasks |
| 111 | stage files through the existing SFTP transfer and the task temporary |
| 112 | directory, so a remote path is never treated as a local one. |
| 113 | |
| 114 | ## Ownership, take-over and unknown writes |
| 115 | |
| 116 | - A grant `{sessionID, taskID, runtimeGeneration, tabIDs, expiresAt}` is |
| 117 | minted by Go when the task starts using the browser and revoked when the |
| 118 | service restarts, the session changes, the connection generation changes or |
| 119 | the task ends. The shell rejects `host/browser.*` calls whose grant is not |
| 120 | current. |
| 121 | - The browser toolbar's **Take over** button immediately switches the tab to |
| 122 | `human` mode and increments its epoch through trusted application IPC. |
| 123 | It works even during the 750 ms window that suppresses echoed agent input; |
| 124 | automatic keyboard, mouse and touch detection is best effort during that |
| 125 | window. Pending and queued actions for the old epoch are cancelled and Go |
| 126 | receives `browser:takeover`. **Resume** hands control back to the agent with |
| 127 | another epoch change, requiring a fresh page read. Login pages, captchas and passkeys are always a user |
| 128 | hand-over: while the tab is in `human` mode the agent cannot read or act on |
| 129 | it. |
| 130 | - Every write reserves an operation `{operationID, sessionID, generation, |
| 131 | tabID, epoch, documentToken, action, digest}` in the ledger before the |
| 132 | shell executes it. The shell reports `executed` or `not_executed` with a |
| 133 | reason; a lost reply, a crash or a service restart leaves the operation |
| 134 | `unknown`. Unknown operations are shown to the user and are never replayed |
| 135 | automatically; a reused `operationID` is rejected forever. |
| 136 | - Renderer crash of a website view cancels only that tab's actions and |
| 137 | reloads the last safe URL in `human` mode. Application renderer crash |
| 138 | pauses all browser actions until the UI re-attaches. |
| 139 | |
| 140 | ## Host calls |
| 141 | |
| 142 | | Method | Purpose | |
| 143 | | --- | --- | |
| 144 | | `host/browser.grant` `revoke` | install or revoke a grant | |
| 145 | | `host/browser.tabs.list` `open` `close` `activate` `navigate` | tab lifecycle bound to a grant | |
| 146 | | `host/browser.snapshot` | structural snapshot for a tab, returns `documentToken` | |
| 147 | | `host/browser.act` | one reserved action; returns `{executed, reason, documentToken}` | |
| 148 | | `host/browser.screenshot` | capture to a task-owned file path | |
| 149 | | `host/browser.downloads` | list or wait for downloads of a tab | |
| 150 | | `host/browser.layout` | apply the panel rectangle and overlay state | |
| 151 | |
| 152 | Events from the shell: `browser:tabs` (tab list changes), `browser:takeover` |
| 153 | (`{tabID, epoch, reason}`), `browser:download` (progress), `browser:crash`. |
| 154 | |
| 155 | ## Remote agents |
| 156 | |
| 157 | A remote Reasonix agent reaches the local browser through the existing SSH |
| 158 | connection and forward manager as a restricted host RPC carrying the same |
| 159 | `BrowserExecutor` contract. Grants are bound to the remote connection |
| 160 | generation, session and task; disconnect, reconnect or session switch revokes |
| 161 | them. Browser grants and provider-proxy credentials are separate; no shared |
| 162 | token. Older remote Serve builds negotiate capabilities and simply do not |
| 163 | advertise the browser, keeping every existing remote feature. |
| 164 | |
| 165 | The wire shape is one loopback HTTP broker per desktop. The bootstrap of a |
| 166 | fresh Serve injects `REASONIX_BROWSER_BROKER` / `REASONIX_BROWSER_TOKEN` |
| 167 | (process environment only) pointing at the reverse-forwarded broker; a reused |
| 168 | Serve is re-pointed through `POST /browser/broker` after the desktop rotates |
| 169 | the route. The broker mints one random bearer token per host connection |
| 170 | generation — registering a new generation replaces the host's old token — and |
| 171 | authenticates before dispatching to `browser.Executor` over |
| 172 | `/v1/browser/<method>`. Every request carries `X-Reasonix-Browser-Session`; |
| 173 | the broker resolves it to the one desktop tab that shows that session and |
| 174 | refuses anything else with `no_grant`. Screenshots and downloads the shell |
| 175 | writes on the desktop are staged onto the remote host through the existing |
| 176 | SFTP channel into a per-workspace scratch directory |
| 177 | (`~/.reasonix/browser-relay/<workspace>/`), so the serve's tools only ever |
| 178 | read paths local to them. A Serve started with a broker advertises `browser` |
| 179 | in the `X-Reasonix-Serve-Capabilities` header of the `/auth/token` handshake. |
| 180 | |
| 181 | ## Acceptance |
| 182 | |
| 183 | Iframes, dynamic DOM, controlled inputs, popups, upload and download, |
| 184 | navigation history, temporary partitions, shared and isolated logins; |
| 185 | take-over before approval, after approval before dispatch, lost receipt after |
| 186 | execution, restart after crash, duplicate operation IDs; remote SSH drop, |
| 187 | generation change, stale grant, cross-session misrouting. The real tasks in |
| 188 | [the migration record](DESKTOP_SHELL_MIGRATION.md#acceptance-gates) close |
| 189 | the phase. |
| 190 |