| 1 | # Reasonix session storage v4 |
| 2 | |
| 3 | Status: production write format for `main-v2`. The implementation lives in |
| 4 | `internal/session`; older formats are migration inputs only. |
| 5 | |
| 6 | ## Identity and layout |
| 7 | |
| 8 | ```text |
| 9 | <data-root>/sessions-v4/ |
| 10 | <session-id>/ |
| 11 | manifest.json |
| 12 | events.frames |
| 13 | legacy/ # immutable migration evidence, when present |
| 14 | .content-v1/ # immutable SHA-256 addressed objects |
| 15 | .query-cache/<session-id>/history-v1.sqlite |
| 16 | .migration/ |
| 17 | .trash/ |
| 18 | ``` |
| 19 | |
| 20 | The accepted manifest tuple is: |
| 21 | |
| 22 | - `schemaVersion: 4` |
| 23 | - `codec: reasonix.session.linear/v4` |
| 24 | - `storageRevision: 1` |
| 25 | |
| 26 | A v4 manifest without `storageRevision: 1` is an unpublished draft and is |
| 27 | accepted only by the explicit migration adapter. Controllers and clients pass |
| 28 | a host-scoped `SessionRef`; paths and codec versions do not enter business |
| 29 | interfaces. |
| 30 | |
| 31 | ## Commit and frame contract |
| 32 | |
| 33 | Each logical commit is encoded as `batch/begin`, one or more `batch/event` |
| 34 | records, and `batch/end`. Every record is an independent CRC-enabled Zstandard |
| 35 | frame with the `RX4F` header, compressed length, and decoded length. The end |
| 36 | record authenticates the ordered begin/event records with SHA-256. Readers |
| 37 | publish a commit only after the complete end record validates. |
| 38 | |
| 39 | The encoder and decoder share an 8 MiB frame budget. Event payloads above 64 |
| 40 | KiB are published to the content store before their references can be |
| 41 | accepted, so user content size is not a frame or session limit. The 16 MiB |
| 42 | write-behind budget applies cancellable backpressure and charges a large batch |
| 43 | only a bounded hot-memory amount. It is not a maximum batch or history size. |
| 44 | |
| 45 | Accepted and durable watermarks are distinct. Provider calls and side-effect |
| 46 | tools must cross `Flush`; an uncertain append is compared byte-for-byte with |
| 47 | its disk staging evidence before it can be confirmed or retried. |
| 48 | |
| 49 | ## Content and history |
| 50 | |
| 51 | Content objects preserve exact bytes and use SHA-256 identity. A separate |
| 52 | 1 MiB block-integrity index supports verified range reads. A content reference |
| 53 | is readable only after the session history index proves that the reference is |
| 54 | part of that session's durable view. Export copies the complete referenced |
| 55 | object closure and rewrites the archive to a self-contained `.content-v1`. |
| 56 | |
| 57 | The SQLite history database is derived data. It is rebuilt from the log after |
| 58 | loss, version drift, or a log revision change. History pages: |
| 59 | |
| 60 | - start at the newest durable messages; |
| 61 | - move toward older messages with an opaque cursor; |
| 62 | - bind the cursor to session identity, storage revision, projection version, |
| 63 | and durable snapshot sequence; |
| 64 | - return at most 500 messages and approximately 2 MiB of encoded data; |
| 65 | - return a preview plus `ContentRef` for a large message. |
| 66 | |
| 67 | Search uses the same fixed-snapshot cursor rules and returns previews, never |
| 68 | full bodies. Service-owned runtimes retain only the accepted UI tail and the |
| 69 | current provider model projection; durable UI message bodies are read through |
| 70 | the history service. |
| 71 | |
| 72 | ## Compatibility matrix |
| 73 | |
| 74 | | Source | Browse | Continue | New writes | |
| 75 | |---|---|---|---| |
| 76 | | checkpoint / schema-1 event log | read-only adapter | streamed into v4 | v4 only | |
| 77 | | schema-2 DAG | read-only adapter | selected head imported into v4 | v4 only | |
| 78 | | prototype v3 | read-only adapter | explicit import into v4 | v4 only | |
| 79 | | linear v3 / v3.1 | read-only adapter | explicit import into v4 | v4 only | |
| 80 | | unpublished v4 revision 0 | migration only | explicit import into revision 1 | v4 revision 1 only | |
| 81 | | v4 revision 1 | native | native | native | |
| 82 | |
| 83 | Schema-1 and checkpoint messages are streamed through a one-turn normalization |
| 84 | window into a private disk spool. Replace records truncate that spool; append |
| 85 | indexes are checked against the raw source count. This removes the former 128 |
| 86 | MiB cumulative replay gate from migration without removing it from untrusted |
| 87 | interactive legacy replay. Schema-2 still uses its graph-specific compatibility |
| 88 | reader so head, patch, redaction, and fork semantics are preserved. |
| 89 | |
| 90 | ## Recovery rules |
| 91 | |
| 92 | - Final v4 at the canonical `BranchID` always wins over an older paired legacy |
| 93 | checkpoint. It must never be reinterpreted as a preview format. |
| 94 | - A partial final frame or batch is uncommitted. The writer preserves evidence |
| 95 | before truncating to the last complete batch. |
| 96 | - Unknown required events, a complete corrupt frame, a digest mismatch, or a |
| 97 | divergent legacy/sidecar pair fail closed. |
| 98 | - Cold browse, migration, import, and fork never restore approvals or arm a |
| 99 | Goal. Restart recovery closes active tools/interactions and ends the turn as |
| 100 | interrupted; it never repeats a side effect. |
| 101 | - Failed migration and import leave the source unchanged. Targets are built in |
| 102 | sibling staging directories and published by atomic rename. |
| 103 | - Shared content is intentionally not garbage-collected in revision 1. Deleting |
| 104 | a session cannot invalidate a fork or exported archive. |
| 105 | |
| 106 | ## Resource limits versus product limits |
| 107 | |
| 108 | There is no accumulated byte, message, event, or Goal-turn product limit. |
| 109 | 64 KiB, 1 MiB, 2 MiB, 8 MiB, and 16 MiB values govern placement, transfer, |
| 110 | allocation, and backpressure. Disk exhaustion, permission failure, invalid |
| 111 | input, or a single provider request that cannot fit the provider work budget |
| 112 | remain explicit errors; none permits deleting or silently truncating history. |
| 113 | |
| 114 | ## Capacity acceptance |
| 115 | |
| 116 | The production-path capacity runner emits JSON timings, disk usage, Go memory |
| 117 | high-water marks, and process peak RSS on Unix. Its default is a quick smoke |
| 118 | run. The release-scale data set is explicit and therefore cannot become a |
| 119 | runtime admission limit: |
| 120 | |
| 121 | ```sh |
| 122 | go run ./tools/sessioncapacity |
| 123 | |
| 124 | go run ./tools/sessioncapacity \ |
| 125 | -root /absolute/path/to/evidence/sessions-v4 \ |
| 126 | -history-messages 50000 \ |
| 127 | -history-bytes 1073741824 \ |
| 128 | -attachment-bytes 1073741824 \ |
| 129 | -workset-bytes 16777216 |
| 130 | ``` |
| 131 | |
| 132 | Each history message contributes a `message/complete` and bounded |
| 133 | `model/context-replace` event; the final workset event makes the full command |
| 134 | slightly exceed 100,000 events. Attachment objects use a deterministic stream |
| 135 | without allocating an attachment-sized buffer. The runner closes, cold-opens, |
| 136 | rebuilds the history index, verifies the retained model workset, and measures |
| 137 | indexed newest-page latency. Keep the stated `-root` for release evidence; an |
| 138 | omitted root is deleted after the run. |
| 139 |