| 1 | # Browser over CDP |
| 2 | |
| 3 | [简体中文](BROWSER_CDP.zh-CN.md) |
| 4 | |
| 5 | The [desktop browser](DESKTOP_BROWSER.md) gives the agent a Chromium surface |
| 6 | the user shares, served by the Electron shell. CLI, `reasonix serve`, and |
| 7 | headless sessions have no shell behind them, so the same `browser_*` tools were |
| 8 | registered and then failed closed with no host to answer them. |
| 9 | |
| 10 | This backend is the third implementation of the host-neutral `browser.Executor` |
| 11 | (after the Electron shell and the SSH broker): it drives an external Chrome over |
| 12 | the DevTools Protocol. The tools, their descriptions, and their schemas are |
| 13 | unchanged — only who answers them is new. |
| 14 | |
| 15 | ## Enabling it |
| 16 | |
| 17 | ```toml |
| 18 | [browser] |
| 19 | enabled = true |
| 20 | ``` |
| 21 | |
| 22 | That is the whole ordinary path. On the first browser tool call Reasonix |
| 23 | launches a Chrome it owns, in a throwaway profile, and kills it with the |
| 24 | session. Nothing is launched at session start: a session that never touches a |
| 25 | browser never pays for one. |
| 26 | |
| 27 | To drive a Chrome you already have running, start it with a debugging port and |
| 28 | point the endpoint at it: |
| 29 | |
| 30 | ```bash |
| 31 | "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" --remote-debugging-port=9222 |
| 32 | ``` |
| 33 | |
| 34 | ```toml |
| 35 | [browser] |
| 36 | enabled = true |
| 37 | endpoint = "http://127.0.0.1:9222" |
| 38 | ``` |
| 39 | |
| 40 | | Key | Meaning | |
| 41 | | --- | --- | |
| 42 | | `enabled` | Off by default. On means the tools get a real browser. | |
| 43 | | `endpoint` | A running Chrome's DevTools endpoint. Empty launches one. | |
| 44 | | `allow_remote_endpoint` | Permits a non-loopback endpoint. Off by default. | |
| 45 | | `chrome_path` | Browser binary. Empty searches the usual Chrome, Chromium, and Edge locations, then `REASONIX_CHROME` and `CHROME_PATH`. | |
| 46 | | `chrome_args` | Extra launch flags, for example a proxy. | |
| 47 | | `user_data_dir` | Profile for a launched browser. Empty uses a throwaway directory, so logins never outlive the session. | |
| 48 | | `headless` | Launch without a window. | |
| 49 | |
| 50 | The desktop ignores this section: it already owns a browser, and a host that |
| 51 | brought one keeps it. |
| 52 | |
| 53 | ## What the agent can reach |
| 54 | |
| 55 | Only tabs this backend opened. An attached Chrome usually holds the user's own |
| 56 | logged-in tabs, and `browser_tabs` never enumerates them, so the agent can |
| 57 | neither read nor drive a page the user did not hand it. `temporary` tabs get |
| 58 | their own browser context, which shares no cookies and is discarded with the |
| 59 | tab. |
| 60 | |
| 61 | ## The refusals this backend owns |
| 62 | |
| 63 | A raw browser keeps no record of what an agent asked it to do, so this backend |
| 64 | owns the guarantees the Electron shell's ledger provides: |
| 65 | |
| 66 | - **One use per `operationId`.** A replayed id is refused forever. The earlier |
| 67 | attempt's effect stands, including an unknown one, so the model is told to |
| 68 | re-read the page rather than try again. |
| 69 | - **`documentToken` binds refs to a document version.** Each `browser_snapshot` |
| 70 | mints a fresh opaque token; navigation, page replacement, and a user take-over |
| 71 | retire it. A write carrying a retired token is refused as stale instead of |
| 72 | being replayed against a page the model has not seen. |
| 73 | - **Unknown outcomes are rare and honest.** A failure before anything reached |
| 74 | the page is reported as not executed, which the model may plan around. Only a |
| 75 | failure *after* input already landed — the second event of a click, a page |
| 76 | script that threw halfway — reports an unknown outcome, which must never be |
| 77 | retried. |
| 78 | |
| 79 | ### Take-over is an approximation here |
| 80 | |
| 81 | The shell knows when a human touches the page because its guest preload sees |
| 82 | trusted input that the shell did not synthesise. CDP-dispatched input is |
| 83 | indistinguishable from a hand at the keyboard once it reaches the DOM, so this |
| 84 | backend marks a short window around each of its own dispatches and counts |
| 85 | trusted input outside that window as the user's. A human click that lands inside |
| 86 | the window is missed. The failure mode is a stale snapshot, never a silent |
| 87 | replay, because every write still carries a single-use `operationId`. |
| 88 | |
| 89 | Refs and the take-over counter live in a per-document isolated world, so page |
| 90 | script can neither read the agent's refs nor forge the counter. |
| 91 | |
| 92 | ## Artifacts |
| 93 | |
| 94 | Screenshots and downloads land in a private directory that is removed with the |
| 95 | session. Downloads keep their server-suggested name, sanitised so a name can |
| 96 | never escape that directory, and never overwrite a file already there. |
| 97 | |
| 98 | ## Security |
| 99 | |
| 100 | A DevTools endpoint grants full control of that browser and of every file it can |
| 101 | read. A non-loopback `endpoint` is therefore refused unless |
| 102 | `allow_remote_endpoint` is set explicitly. |
| 103 | |
| 104 | `browser_upload` may only read from the session's write roots — the workspace |
| 105 | and any additional directories — plus the executor's own artifact directory, so |
| 106 | a file the agent just downloaded stays attachable. Symlinks are resolved before |
| 107 | that check, so a link inside the workspace cannot point a file input at a key |
| 108 | outside it. Any other path is refused with a reason rather than handed to the |
| 109 | page: the page is untrusted, and a file input is an upload channel. |
| 110 | |
| 111 | ## Cache |
| 112 | |
| 113 | Nothing here is provider-visible. The tools stay registry-only and reachable |
| 114 | through `use_capability`, so the request's tool array and the system-prompt |
| 115 | prefix are byte-identical whether or not a browser is attached. The guard is |
| 116 | `TestConfiguredBrowserBackendStaysOffTheProviderSurface` in `internal/boot`. |
| 117 | |
| 118 | ## Verifying a change |
| 119 | |
| 120 | ```bash |
| 121 | go test ./internal/browser/... ./internal/boot/ |
| 122 | ``` |
| 123 | |
| 124 | The unit tests run against a scripted DevTools server. The injected page helper |
| 125 | — the snapshot walker, the ref table, the take-over listeners — is only really |
| 126 | exercised against a real browser: |
| 127 | |
| 128 | ```bash |
| 129 | REASONIX_LIVE_CHROME=1 go test ./internal/browser/cdp -run '^TestLiveChrome$' -v -count=1 |
| 130 | ``` |
| 131 | |
| 132 | That test needs a Chrome install and stays skipped otherwise. |
| 133 | |
| 134 | ## Limits |
| 135 | |
| 136 | - Chrome, Chromium, and Chromium-based Edge only. |
| 137 | - The agent's tabs are its own; there is no way to hand it one of the user's. |
| 138 | - `browser_select` drives a `<select>` through the DOM, because a native |
| 139 | dropdown is rendered by the platform and cannot be steered with synthetic |
| 140 | mouse events. Every other write uses real input events. |
| 141 | - A snapshot stops at 2000 nodes; pass `selector` to scope it to one subtree. |
| 142 |