| author |  clover caruso <git@paperclover.net> 2026-09-06 15:46:15-07:00 |
| committer | clover caruso <git@paperclover.net> 2026-09-06 15:46:15-07:00 |
| log | b253974a925fe86b3f2ee31e969477fe2ae43fd3 |
| tree | ce9c5aaf243c3542d3dfbba629438deaabb2d199 |
| parent | 98814e04e30acd58e15833964b0a216cf36b7a5b |
| signature | Signed by SSH key SHA256:52mNGHRsVFBDED9IAX5pe+LRWUefqTbxEReunq21QvU |
feat: offline support and real smb client
107 files changed, 17685 insertions(+), 1960 deletions(-)
- API-AUDIT.md
- Cargo.lock
- FEATURES.md
- M6-ACCEPTANCE.md
- MILESTONE6.md
- MILESTONE7.md
- MILESTONE8.md
- PROGRESS.md
- README.md
crates
onestore-diagnostic
onestore-offline
onestore-smb
onestore
- readme.md
tools
- codex_usage_report.example.json
- codex_usage_report.mjs
- concurrent_rust.py
- crash_recovery.py
diagnostic
- native_collaboration.py
- native_disconnect.py
- native_maintenance.py
- native_probe.py
- native_stress.py
- notebook_editor.py
- notebook_report.py
- offline_cache_crash.py
- offline_document_history.py
- offline_history.py
- offline_outage.py
- offline_publication_crash.py
- smb-proxy.py
- smb_faults.py
- test_concurrent_rust.py
- test_crash_recovery.py
- test_native_disconnect.py
- test_native_maintenance.py
- test_native_stress.py
- test_notebook_editor.py
- test_notebook_report.py
- test_offline_confirmation.py
- test_offline_document_history.py
- test_offline_history.py
- test_offline_outage.py
- test_offline_publication_crash.py
- test_smb_overlap.py
- test_smb_proxy.py
- test_verify_smb_faults.py
- verify-collaboration.py
- verify-reader.py
- verify-writer.py
- verify_offline.py
- verify_offline_confirmation.py
- verify_offline_recovery.py
- verify_smb_faults.py
- verify_smb_overlap.py
API-AUDIT.md deleted-69
| ... | @@ -1,69 +0,0 @@ | ||
| 1 | # Public API audit | ||
| 2 | |||
| 3 | The current API is reasonable for a Rust reader/canvas prototype and the supported | ||
| 4 | text-edit diagnostic tool. It is an experimental interoperability API, not yet a | ||
| 5 | stable application SDK. The workspace move preserves every public name and its | ||
| 6 | behavior; this audit adds contract documentation without introducing wrappers or | ||
| 7 | changing serialization. | ||
| 8 | |||
| 9 | ## Consumer boundary | ||
| 10 | |||
| 11 | ```text | ||
| 12 | owned snapshot bytes | ||
| 13 | └─ Store::parse committed storage; checksum diagnostics | ||
| 14 | └─ RevisionIndex::parse revision identities and dependencies | ||
| 15 | └─ Document::parse semantic views; checksum damage rejected | ||
| 16 | ├─ pages() active section pages in stored order | ||
| 17 | └─ text_runs() text with inherited formatting and links | ||
| 18 | |||
| 19 | snapshot + typed identities + UTF-16 range + replacement | ||
| 20 | └─ commit_file_text lock → compare snapshot → append → flush | ||
| 21 | └─ Result<(), CommitError> success or explicit publication state | ||
| 22 | ``` | ||
| 23 | |||
| 24 | A document represents one `.one` or `.onetoc2` file, not a notebook directory. | ||
| 25 | The directory, section ordering, native references and report assets are currently | ||
| 26 | assembled by the tooling. A second crate can use the semantic model directly and | ||
| 27 | keep its layout/cache state separate. Mutating the public model changes only the | ||
| 28 | inspection view; writers reparse the supplied snapshot and enforce their own | ||
| 29 | invariants. | ||
| 30 | |||
| 31 | ## Findings and decisions | ||
| 32 | |||
| 33 | | Area | Assessment | Recommendation | | ||
| 34 | | --- | --- | --- | | ||
| 35 | | Read/write separation | Good: parsed views cannot accidentally save themselves. Snapshot comparison rejects stale writes before publication. | Keep explicit commit operations; do not add a mutable document plus generic `save()`. | | ||
| 36 | | Commit outcomes | Good: `NotCommitted`, `Unknown` and `Committed` distinguish retry behavior. An error can still mean the edit persisted. | Keep this distinction at every UI/FFI boundary. Variant documentation now states the caller action. | | ||
| 37 | | Borrowing and lifetimes | The model owns decoded strings but borrows payloads. Consumers must retain its source snapshot/store. An application object holding both bytes and borrowed views would require a different ownership design. | Let the canvas prototype determine whether its actual access pattern needs an owned model. Do not add a self-referential owner or duplicate document DTO in advance. | | ||
| 38 | | Semantic error classification | `Error` exposes diagnostic text and an offset, with no typed reason. I/O failures already have `ErrorKind`; semantic distinctions such as unsupported edits and temporarily missing contexts require text matching today. | Before adding automatic semantic recovery, choose a small typed classification based on the recovery actions it needs. A detailed variant for every parser message would enlarge the compatibility burden. | | ||
| 39 | | Editable text discovery | Readable text is broader than editable text: hidden fields, hyperlinks, equations, generated content, conflicts and protected content can be readable but rejected for writing. Currently callers can try `replace_text` and inspect its result; candidate discovery duplicates some checks in the random-edit example. | For the diagnostic tool, obtain eligibility from the writer's actual validation. If a public eligibility API is added, share that validation rather than maintain a second list of rules. | | ||
| 40 | | Identity transport | `ExGuid` is typed, ordered, hashable and serializes to its display string. It has no `FromStr` or `Deserialize`. Existing tools retain or look up typed IDs instead of parsing them. | A Rust canvas can keep typed IDs. A JSON editing endpoint should either map strings back to IDs from its snapshot or justify a canonical parser with round-trip tests. | | ||
| 41 | | Raw storage exports | `Header`, nodes, references, property arenas and both revision layers are public. They are useful for diagnostics, but expose implementation details to consumers. | Keep them available during interoperability work; have the UI depend on `onestore::document` plus edit functions. Moving them into a separate module now would create churn without an established consumer requirement. | | ||
| 42 | | Public fields and enums | Inspection fields are mutable and enums are exhaustive. Downstream code can depend on the exact shape. | Treat the current Rust and JSON shapes as experimental. Whether to restrict construction or allow future enum variants is an API-evolution tradeoff to decide with the first consumer, not an assumed stability promise. | | ||
| 43 | | Document boundaries | `pages()` excludes conflicts/history, returns no visible pages for encryption, and rejects TOC files. All referenced contexts remain in the model. | These boundaries are now documented on the method. Readers must inspect the root kind when distinguishing a locked section from an empty section. | | ||
| 44 | | Units and defaults | Coordinates are points, edit/run offsets are UTF-16 units, Time32 values use the 1980 epoch, and `Format` preserves absence separately from explicit false. | Keep the native distinctions; renderers should use resolved runs and convert offsets explicitly when crossing into UTF-8 or browser selection APIs. | | ||
| 45 | | In-memory replacements | `replace_text` and `replace_property_bytes` return complete byte images without locking or persistence. Overwriting a live file with those bytes would bypass the commit protocol. | Their documentation now directs existing-file updates to the corresponding commit functions. Raw scalar editing still requires the caller to maintain document semantics. | | ||
| 46 | | Scope and dependencies | The library stays native and synchronous, with no network runtime, renderer, process management or platform UI dependency. Internal writer helpers remain crate-private. | Keep the core boundary. Put diagnostic serving and rendering in consumers; expose a C ABI only against concrete embedding needs. | | ||
| 47 | |||
| 48 | The two decisions worth reviewing before the diagnostic tool are semantic error | ||
| 49 | classification and edit eligibility. Neither requires a broad API redesign. | ||
| 50 | The lifetime/ownership question can be informed by the separate canvas prototype. | ||
| 51 | |||
| 52 | ## Verification | ||
| 53 | |||
| 54 | - Root workspace tests: `evidence/m8/workspace-tests.log`. | ||
| 55 | - Clippy and formatting: `workspace-clippy.log`, `workspace-format.log`. | ||
| 56 | - Rustdoc and compiled documentation example: `workspace-docs.log`, | ||
| 57 | `workspace-doctests.log`. | ||
| 58 | - All eight cargo-fuzz targets build against the relocated library: | ||
| 59 | `workspace-fuzz-build.log`. | ||
| 60 | - Python verification tools: 36 tests in `workspace-python.log`. | ||
| 61 | - An independent crate under `evidence/m8/api-consumer` uses only public APIs to | ||
| 62 | create a section, traverse pages/runs, serialize identities, commit a Unicode | ||
| 63 | edit, reject a stale snapshot through typed commit/I/O state, and reopen the | ||
| 64 | exact resulting text. Output: `evidence/m8/api-consumer.log`. | ||
| 65 | |||
| 66 | The audit examined exported declarations and their implementations in storage, | ||
| 67 | revision resolution, document interpretation, creation, editing and persistence. | ||
| 68 | It does not freeze the API or turn the earlier native compatibility evidence into | ||
| 69 | a new platform guarantee. | ||
Cargo.lock+843-1
| ... | @@ -2,12 +2,101 @@ | ... | @@ -2,12 +2,101 @@ |
| 2 | # It is not intended for manual editing. | 2 | # It is not intended for manual editing. |
| 3 | version = 4 | 3 | version = 4 |
| 4 | 4 | ||
| 5 | [[package]] | ||
| 6 | name = "aead" | ||
| 7 | version = "0.6.1" | ||
| 8 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 9 | checksum = "1973cfbc1a2daf9cf550e74e1f088c28e7f7d8c1e1418fb6c9dc5184b7e84c99" | ||
| 10 | dependencies = [ | ||
| 11 | "crypto-common", | ||
| 12 | "inout", | ||
| 13 | ] | ||
| 14 | |||
| 15 | [[package]] | ||
| 16 | name = "aes" | ||
| 17 | version = "0.9.3" | ||
| 18 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 19 | checksum = "35f0f96ce78e38c3dc6d8948aa8163d06385be74000f3c7a95bf1eef35d3ea32" | ||
| 20 | dependencies = [ | ||
| 21 | "cipher", | ||
| 22 | "cpubits", | ||
| 23 | "cpufeatures", | ||
| 24 | ] | ||
| 25 | |||
| 26 | [[package]] | ||
| 27 | name = "aes-gcm" | ||
| 28 | version = "0.11.1" | ||
| 29 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 30 | checksum = "7f2b8006a0c83f52b62ba44a97b58bf76fe2f70a329e588f67f89691d93d498f" | ||
| 31 | dependencies = [ | ||
| 32 | "aead", | ||
| 33 | "aes", | ||
| 34 | "cipher", | ||
| 35 | "ctr", | ||
| 36 | "ctutils", | ||
| 37 | "ghash", | ||
| 38 | ] | ||
| 39 | |||
| 40 | [[package]] | ||
| 41 | name = "async-trait" | ||
| 42 | version = "0.1.92" | ||
| 43 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 44 | checksum = "82f6aeea286b8eb4dd3431a1be1b59d290ace00f5bfd8e2a159bc2a05e2c1667" | ||
| 45 | dependencies = [ | ||
| 46 | "proc-macro2", | ||
| 47 | "quote", | ||
| 48 | "syn 3.0.5", | ||
| 49 | ] | ||
| 50 | |||
| 5 | [[package]] | 51 | [[package]] |
| 6 | name = "bitflags" | 52 | name = "bitflags" |
| 7 | version = "2.13.1" | 53 | version = "2.13.1" |
| 8 | source = "registry+https://github.com/rust-lang/crates.io-index" | 54 | source = "registry+https://github.com/rust-lang/crates.io-index" |
| 9 | checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" | 55 | checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" |
| 10 | 56 | ||
| 57 | [[package]] | ||
| 58 | name = "block-buffer" | ||
| 59 | version = "0.12.1" | ||
| 60 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 61 | checksum = "d2f6c7dbe95a6ed67ad9f18e57daf93a2f034c524b99fd2b76d18fdfeb6660aa" | ||
| 62 | dependencies = [ | ||
| 63 | "hybrid-array", | ||
| 64 | ] | ||
| 65 | |||
| 66 | [[package]] | ||
| 67 | name = "bumpalo" | ||
| 68 | version = "3.20.3" | ||
| 69 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 70 | checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" | ||
| 71 | |||
| 72 | [[package]] | ||
| 73 | name = "bytes" | ||
| 74 | version = "1.12.1" | ||
| 75 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 76 | checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04" | ||
| 77 | |||
| 78 | [[package]] | ||
| 79 | name = "cc" | ||
| 80 | version = "1.4.5" | ||
| 81 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 82 | checksum = "005ec2760ca554fae18df7a11195552ec576cd665632a881bc011d5bb2fd4d80" | ||
| 83 | dependencies = [ | ||
| 84 | "find-msvc-tools", | ||
| 85 | "shlex", | ||
| 86 | ] | ||
| 87 | |||
| 88 | [[package]] | ||
| 89 | name = "ccm" | ||
| 90 | version = "0.6.0-rc.3" | ||
| 91 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 92 | checksum = "4edea5ea70a1285565ac264767613d6c88351a9a0557e7af793a0942590baaed" | ||
| 93 | dependencies = [ | ||
| 94 | "aead", | ||
| 95 | "cipher", | ||
| 96 | "ctr", | ||
| 97 | "subtle", | ||
| 98 | ] | ||
| 99 | |||
| 11 | [[package]] | 100 | [[package]] |
| 12 | name = "cfg-if" | 101 | name = "cfg-if" |
| 13 | version = "1.0.4" | 102 | version = "1.0.4" |
| ... | @@ -20,6 +109,175 @@ version = "0.2.2" | ... | @@ -20,6 +109,175 @@ version = "0.2.2" |
| 20 | source = "registry+https://github.com/rust-lang/crates.io-index" | 109 | source = "registry+https://github.com/rust-lang/crates.io-index" |
| 21 | checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527" | 110 | checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527" |
| 22 | 111 | ||
| 112 | [[package]] | ||
| 113 | name = "cipher" | ||
| 114 | version = "0.5.2" | ||
| 115 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 116 | checksum = "e8cf2a2c93cd704877c0858356ed03480ff301ee950b43f1cbe4573b088bfa6c" | ||
| 117 | dependencies = [ | ||
| 118 | "block-buffer", | ||
| 119 | "crypto-common", | ||
| 120 | "inout", | ||
| 121 | ] | ||
| 122 | |||
| 123 | [[package]] | ||
| 124 | name = "cmac" | ||
| 125 | version = "0.8.0" | ||
| 126 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 127 | checksum = "ac78aa94ce13e432b332a4d1bf2eff167d3a2520188ee05b337180a42fd2e62e" | ||
| 128 | dependencies = [ | ||
| 129 | "cipher", | ||
| 130 | "dbl", | ||
| 131 | "digest", | ||
| 132 | ] | ||
| 133 | |||
| 134 | [[package]] | ||
| 135 | name = "cmov" | ||
| 136 | version = "0.5.4" | ||
| 137 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 138 | checksum = "0c9ea0ac24bc397ab3c98583a3c9ba74fa56b09a4449bbe172b9b1ddb016027a" | ||
| 139 | |||
| 140 | [[package]] | ||
| 141 | name = "const-oid" | ||
| 142 | version = "0.10.2" | ||
| 143 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 144 | checksum = "a6ef517f0926dd24a1582492c791b6a4818a4d94e789a334894aa15b0d12f55c" | ||
| 145 | |||
| 146 | [[package]] | ||
| 147 | name = "cpubits" | ||
| 148 | version = "0.1.1" | ||
| 149 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 150 | checksum = "15b85f9c39137c3a891689859392b1bd49812121d0d61c9caf00d46ed5ce06ae" | ||
| 151 | |||
| 152 | [[package]] | ||
| 153 | name = "cpufeatures" | ||
| 154 | version = "0.3.1" | ||
| 155 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 156 | checksum = "5ca28b0ae3115b884660db4118d803791fd6756b6e88f39c0f3f7859060d7566" | ||
| 157 | dependencies = [ | ||
| 158 | "libc", | ||
| 159 | ] | ||
| 160 | |||
| 161 | [[package]] | ||
| 162 | name = "crypto-common" | ||
| 163 | version = "0.2.2" | ||
| 164 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 165 | checksum = "ce6e4c961d6cd6c9a86db418387425e8bdeaf05b3c8bc1411e6dca4c252f1453" | ||
| 166 | dependencies = [ | ||
| 167 | "getrandom", | ||
| 168 | "hybrid-array", | ||
| 169 | "rand_core", | ||
| 170 | ] | ||
| 171 | |||
| 172 | [[package]] | ||
| 173 | name = "ctr" | ||
| 174 | version = "0.10.1" | ||
| 175 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 176 | checksum = "baaca1c4b237092596f64d571e9db6ce4109c4ef9742e27590f1709594461f21" | ||
| 177 | dependencies = [ | ||
| 178 | "cipher", | ||
| 179 | ] | ||
| 180 | |||
| 181 | [[package]] | ||
| 182 | name = "ctutils" | ||
| 183 | version = "0.4.2" | ||
| 184 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 185 | checksum = "7d5515a3834141de9eafb9717ad39eea8247b5674e6066c404e8c4b365d2a29e" | ||
| 186 | dependencies = [ | ||
| 187 | "cmov", | ||
| 188 | ] | ||
| 189 | |||
| 190 | [[package]] | ||
| 191 | name = "dbl" | ||
| 192 | version = "0.5.0" | ||
| 193 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 194 | checksum = "f0d7a944e61df464668c5f51f56cc667396a8821434273112948ea0b66e405d7" | ||
| 195 | dependencies = [ | ||
| 196 | "hybrid-array", | ||
| 197 | ] | ||
| 198 | |||
| 199 | [[package]] | ||
| 200 | name = "digest" | ||
| 201 | version = "0.11.3" | ||
| 202 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 203 | checksum = "f1dd6dbb5841937940781866fa1281a1ff7bd3bf827091440879f9994983d5c2" | ||
| 204 | dependencies = [ | ||
| 205 | "block-buffer", | ||
| 206 | "const-oid", | ||
| 207 | "crypto-common", | ||
| 208 | "ctutils", | ||
| 209 | ] | ||
| 210 | |||
| 211 | [[package]] | ||
| 212 | name = "equivalent" | ||
| 213 | version = "1.0.2" | ||
| 214 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 215 | checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" | ||
| 216 | |||
| 217 | [[package]] | ||
| 218 | name = "errno" | ||
| 219 | version = "0.3.14" | ||
| 220 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 221 | checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" | ||
| 222 | dependencies = [ | ||
| 223 | "libc", | ||
| 224 | "windows-sys", | ||
| 225 | ] | ||
| 226 | |||
| 227 | [[package]] | ||
| 228 | name = "fallible-iterator" | ||
| 229 | version = "0.3.0" | ||
| 230 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 231 | checksum = "2acce4a10f12dc2fb14a218589d4f1f62ef011b2d0cc4b3cb1bba8e94da14649" | ||
| 232 | |||
| 233 | [[package]] | ||
| 234 | name = "fallible-streaming-iterator" | ||
| 235 | version = "0.1.9" | ||
| 236 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 237 | checksum = "7360491ce676a36bf9bb3c56c1aa791658183a54d2744120f27285738d90465a" | ||
| 238 | |||
| 239 | [[package]] | ||
| 240 | name = "fastrand" | ||
| 241 | version = "2.5.0" | ||
| 242 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 243 | checksum = "da7c62ceae207dd37ea5b845da6a0696c799f85e97da1ab5b7910be3c1c80223" | ||
| 244 | |||
| 245 | [[package]] | ||
| 246 | name = "find-msvc-tools" | ||
| 247 | version = "0.1.12" | ||
| 248 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 249 | checksum = "3e0f1c7c3a72c66fd80abe965175f7523475c0489a87d3ff9d6e8c87d87a9d2d" | ||
| 250 | |||
| 251 | [[package]] | ||
| 252 | name = "foldhash" | ||
| 253 | version = "0.2.0" | ||
| 254 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 255 | checksum = "77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb" | ||
| 256 | |||
| 257 | [[package]] | ||
| 258 | name = "futures-core" | ||
| 259 | version = "0.3.34" | ||
| 260 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 261 | checksum = "92d699e522242e69e3003b94ecc1f960f3a5e015aa7c5d7486e65ad01dd94f5e" | ||
| 262 | |||
| 263 | [[package]] | ||
| 264 | name = "futures-task" | ||
| 265 | version = "0.3.34" | ||
| 266 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 267 | checksum = "cd417de3d1d015fc3bfd2b1ea46dfc7bab72ef86f1cc7cc9c78e728b34a6d1fd" | ||
| 268 | |||
| 269 | [[package]] | ||
| 270 | name = "futures-util" | ||
| 271 | version = "0.3.34" | ||
| 272 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 273 | checksum = "0d50a92467f8ba5dd6e3ee5d4bd04d73ab2e4e1c44474a0674821dfce14b79bc" | ||
| 274 | dependencies = [ | ||
| 275 | "futures-core", | ||
| 276 | "futures-task", | ||
| 277 | "pin-project-lite", | ||
| 278 | "slab", | ||
| 279 | ] | ||
| 280 | |||
| 23 | [[package]] | 281 | [[package]] |
| 24 | name = "getrandom" | 282 | name = "getrandom" |
| 25 | version = "0.4.3" | 283 | version = "0.4.3" |
| ... | @@ -29,6 +287,80 @@ dependencies = [ | ... | @@ -29,6 +287,80 @@ dependencies = [ |
| 29 | "cfg-if", | 287 | "cfg-if", |
| 30 | "libc", | 288 | "libc", |
| 31 | "r-efi", | 289 | "r-efi", |
| 290 | "rand_core", | ||
| 291 | ] | ||
| 292 | |||
| 293 | [[package]] | ||
| 294 | name = "ghash" | ||
| 295 | version = "0.6.0" | ||
| 296 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 297 | checksum = "2eecf2d5dc9b66b732b97707a0210906b1d30523eb773193ab777c0c84b3e8d5" | ||
| 298 | dependencies = [ | ||
| 299 | "polyval", | ||
| 300 | ] | ||
| 301 | |||
| 302 | [[package]] | ||
| 303 | name = "hashbrown" | ||
| 304 | version = "0.16.1" | ||
| 305 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 306 | checksum = "841d1cc9bed7f9236f321df977030373f4a4163ae1a7dbfe1a51a2c1a51d9100" | ||
| 307 | dependencies = [ | ||
| 308 | "foldhash", | ||
| 309 | ] | ||
| 310 | |||
| 311 | [[package]] | ||
| 312 | name = "hashbrown" | ||
| 313 | version = "0.17.1" | ||
| 314 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 315 | checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a" | ||
| 316 | dependencies = [ | ||
| 317 | "foldhash", | ||
| 318 | ] | ||
| 319 | |||
| 320 | [[package]] | ||
| 321 | name = "hashlink" | ||
| 322 | version = "0.12.1" | ||
| 323 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 324 | checksum = "32069d97bb81e38fa67eab65e3393bf804bb85969f2bc06bf13f64aef5aba248" | ||
| 325 | dependencies = [ | ||
| 326 | "hashbrown 0.17.1", | ||
| 327 | ] | ||
| 328 | |||
| 329 | [[package]] | ||
| 330 | name = "hmac" | ||
| 331 | version = "0.13.0" | ||
| 332 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 333 | checksum = "6303bc9732ae41b04cb554b844a762b4115a61bfaa81e3e83050991eeb56863f" | ||
| 334 | dependencies = [ | ||
| 335 | "digest", | ||
| 336 | ] | ||
| 337 | |||
| 338 | [[package]] | ||
| 339 | name = "hybrid-array" | ||
| 340 | version = "0.4.14" | ||
| 341 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 342 | checksum = "707114b52a152fa7bdb290cd7cd5912d9467273b6d74e21b8d81aca1f8533f6b" | ||
| 343 | dependencies = [ | ||
| 344 | "typenum", | ||
| 345 | ] | ||
| 346 | |||
| 347 | [[package]] | ||
| 348 | name = "indexmap" | ||
| 349 | version = "2.14.2" | ||
| 350 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 351 | checksum = "cc4e190f5d26ca7051642629da2c52fc03bde85a03197c99408dcd291734c855" | ||
| 352 | dependencies = [ | ||
| 353 | "equivalent", | ||
| 354 | "hashbrown 0.17.1", | ||
| 355 | ] | ||
| 356 | |||
| 357 | [[package]] | ||
| 358 | name = "inout" | ||
| 359 | version = "0.2.2" | ||
| 360 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 361 | checksum = "4250ce6452e92010fdf7268ccc5d14faa80bb12fc741938534c58f16804e03c7" | ||
| 362 | dependencies = [ | ||
| 363 | "hybrid-array", | ||
| 32 | ] | 364 | ] |
| 33 | 365 | ||
| 34 | [[package]] | 366 | [[package]] |
| ... | @@ -37,12 +369,73 @@ version = "1.0.18" | ... | @@ -37,12 +369,73 @@ version = "1.0.18" |
| 37 | source = "registry+https://github.com/rust-lang/crates.io-index" | 369 | source = "registry+https://github.com/rust-lang/crates.io-index" |
| 38 | checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" | 370 | checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" |
| 39 | 371 | ||
| 372 | [[package]] | ||
| 373 | name = "js-sys" | ||
| 374 | version = "0.3.105" | ||
| 375 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 376 | checksum = "ce57d20d1ea864ce2ac172ab472d409214f4fd359f0b2a2775abdf522e2af99e" | ||
| 377 | dependencies = [ | ||
| 378 | "cfg-if", | ||
| 379 | "wasm-bindgen", | ||
| 380 | ] | ||
| 381 | |||
| 40 | [[package]] | 382 | [[package]] |
| 41 | name = "libc" | 383 | name = "libc" |
| 42 | version = "0.2.189" | 384 | version = "0.2.189" |
| 43 | source = "registry+https://github.com/rust-lang/crates.io-index" | 385 | source = "registry+https://github.com/rust-lang/crates.io-index" |
| 44 | checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2" | 386 | checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2" |
| 45 | 387 | ||
| 388 | [[package]] | ||
| 389 | name = "libsqlite3-sys" | ||
| 390 | version = "0.38.2" | ||
| 391 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 392 | checksum = "f1d20bef17f513b9b3004532233187769cd072d790971f4e4da0e346eb6401e8" | ||
| 393 | dependencies = [ | ||
| 394 | "cc", | ||
| 395 | "pkg-config", | ||
| 396 | "vcpkg", | ||
| 397 | ] | ||
| 398 | |||
| 399 | [[package]] | ||
| 400 | name = "linux-raw-sys" | ||
| 401 | version = "0.12.1" | ||
| 402 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 403 | checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53" | ||
| 404 | |||
| 405 | [[package]] | ||
| 406 | name = "log" | ||
| 407 | version = "0.4.34" | ||
| 408 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 409 | checksum = "f9f8bd3e56ce4dfc153cf470fffbfa98c7620958b312ca5c3a4b8d5181fd13c6" | ||
| 410 | |||
| 411 | [[package]] | ||
| 412 | name = "lz4_flex" | ||
| 413 | version = "0.13.1" | ||
| 414 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 415 | checksum = "7ef0d4ed8669f8f8826eb00dc878084aa8f253506c4fd5e8f58f5bce72ddb97e" | ||
| 416 | dependencies = [ | ||
| 417 | "twox-hash", | ||
| 418 | ] | ||
| 419 | |||
| 420 | [[package]] | ||
| 421 | name = "md-5" | ||
| 422 | version = "0.11.0" | ||
| 423 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 424 | checksum = "69b6441f590336821bb897fb28fc622898ccceb1d6cea3fde5ea86b090c4de98" | ||
| 425 | dependencies = [ | ||
| 426 | "cfg-if", | ||
| 427 | "digest", | ||
| 428 | ] | ||
| 429 | |||
| 430 | [[package]] | ||
| 431 | name = "md4" | ||
| 432 | version = "0.11.0" | ||
| 433 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 434 | checksum = "bd76fb0fd6b2e4be62a73f8e0858ca97f81babcb1af322dcaca196f735f17f80" | ||
| 435 | dependencies = [ | ||
| 436 | "digest", | ||
| 437 | ] | ||
| 438 | |||
| 46 | [[package]] | 439 | [[package]] |
| 47 | name = "md5" | 440 | name = "md5" |
| 48 | version = "0.8.1" | 441 | version = "0.8.1" |
| ... | @@ -55,6 +448,17 @@ version = "2.8.3" | ... | @@ -55,6 +448,17 @@ version = "2.8.3" |
| 55 | source = "registry+https://github.com/rust-lang/crates.io-index" | 448 | source = "registry+https://github.com/rust-lang/crates.io-index" |
| 56 | checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" | 449 | checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" |
| 57 | 450 | ||
| 451 | [[package]] | ||
| 452 | name = "mio" | ||
| 453 | version = "1.2.3" | ||
| 454 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 455 | checksum = "4b18443e9c262bfe8fa82f51666e2642c53393f7e5c27b3e1aeab922cff5b9d8" | ||
| 456 | dependencies = [ | ||
| 457 | "libc", | ||
| 458 | "wasi", | ||
| 459 | "windows-sys", | ||
| 460 | ] | ||
| 461 | |||
| 58 | [[package]] | 462 | [[package]] |
| 59 | name = "nix" | 463 | name = "nix" |
| 60 | version = "0.31.3" | 464 | version = "0.31.3" |
| ... | @@ -67,6 +471,34 @@ dependencies = [ | ... | @@ -67,6 +471,34 @@ dependencies = [ |
| 67 | "libc", | 471 | "libc", |
| 68 | ] | 472 | ] |
| 69 | 473 | ||
| 474 | [[package]] | ||
| 475 | name = "num_enum" | ||
| 476 | version = "0.7.6" | ||
| 477 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 478 | checksum = "5d0bca838442ec211fa11de3a8b0e0e8f3a4522575b5c4c06ed722e005036f26" | ||
| 479 | dependencies = [ | ||
| 480 | "num_enum_derive", | ||
| 481 | "rustversion", | ||
| 482 | ] | ||
| 483 | |||
| 484 | [[package]] | ||
| 485 | name = "num_enum_derive" | ||
| 486 | version = "0.7.6" | ||
| 487 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 488 | checksum = "680998035259dcfcafe653688bf2aa6d3e2dc05e98be6ab46afb089dc84f1df8" | ||
| 489 | dependencies = [ | ||
| 490 | "proc-macro-crate", | ||
| 491 | "proc-macro2", | ||
| 492 | "quote", | ||
| 493 | "syn 2.0.119", | ||
| 494 | ] | ||
| 495 | |||
| 496 | [[package]] | ||
| 497 | name = "once_cell" | ||
| 498 | version = "1.21.4" | ||
| 499 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 500 | checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" | ||
| 501 | |||
| 70 | [[package]] | 502 | [[package]] |
| 71 | name = "onestore" | 503 | name = "onestore" |
| 72 | version = "0.1.0" | 504 | version = "0.1.0" |
| ... | @@ -78,6 +510,80 @@ dependencies = [ | ... | @@ -78,6 +510,80 @@ dependencies = [ |
| 78 | "serde_json", | 510 | "serde_json", |
| 79 | ] | 511 | ] |
| 80 | 512 | ||
| 513 | [[package]] | ||
| 514 | name = "onestore-diagnostic" | ||
| 515 | version = "0.1.0" | ||
| 516 | dependencies = [ | ||
| 517 | "onestore", | ||
| 518 | "serde", | ||
| 519 | "serde_json", | ||
| 520 | ] | ||
| 521 | |||
| 522 | [[package]] | ||
| 523 | name = "onestore-offline" | ||
| 524 | version = "0.1.0" | ||
| 525 | dependencies = [ | ||
| 526 | "onestore", | ||
| 527 | "onestore-smb", | ||
| 528 | "rusqlite", | ||
| 529 | "serde", | ||
| 530 | "serde_json", | ||
| 531 | "tempfile", | ||
| 532 | "thiserror", | ||
| 533 | ] | ||
| 534 | |||
| 535 | [[package]] | ||
| 536 | name = "onestore-smb" | ||
| 537 | version = "0.1.0" | ||
| 538 | dependencies = [ | ||
| 539 | "onestore", | ||
| 540 | "serde_json", | ||
| 541 | "smb2", | ||
| 542 | "tokio", | ||
| 543 | ] | ||
| 544 | |||
| 545 | [[package]] | ||
| 546 | name = "pbkdf2" | ||
| 547 | version = "0.13.0" | ||
| 548 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 549 | checksum = "112d82ceb8c5bf524d9af484d4e4970c9fd5a0cc15ba14ad93dccd28873b0629" | ||
| 550 | dependencies = [ | ||
| 551 | "digest", | ||
| 552 | "hmac", | ||
| 553 | ] | ||
| 554 | |||
| 555 | [[package]] | ||
| 556 | name = "pin-project-lite" | ||
| 557 | version = "0.2.17" | ||
| 558 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 559 | checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" | ||
| 560 | |||
| 561 | [[package]] | ||
| 562 | name = "pkg-config" | ||
| 563 | version = "0.3.34" | ||
| 564 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 565 | checksum = "f6b464fbc74e149a392436b17d523f769e057cb6877f6a5c4618bc6f11800548" | ||
| 566 | |||
| 567 | [[package]] | ||
| 568 | name = "polyval" | ||
| 569 | version = "0.7.3" | ||
| 570 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 571 | checksum = "f0fa31d631f2b2cb2a544d0aa321ce847a94764d701ca2becc411138b93d49cd" | ||
| 572 | dependencies = [ | ||
| 573 | "cpubits", | ||
| 574 | "cpufeatures", | ||
| 575 | "universal-hash", | ||
| 576 | ] | ||
| 577 | |||
| 578 | [[package]] | ||
| 579 | name = "proc-macro-crate" | ||
| 580 | version = "3.5.0" | ||
| 581 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 582 | checksum = "e67ba7e9b2b56446f1d419b1d807906278ffa1a658a8a5d8a39dcb1f5a78614f" | ||
| 583 | dependencies = [ | ||
| 584 | "toml_edit", | ||
| 585 | ] | ||
| 586 | |||
| 81 | [[package]] | 587 | [[package]] |
| 82 | name = "proc-macro2" | 588 | name = "proc-macro2" |
| 83 | version = "1.0.107" | 589 | version = "1.0.107" |
| ... | @@ -102,6 +608,56 @@ version = "6.0.0" | ... | @@ -102,6 +608,56 @@ version = "6.0.0" |
| 102 | source = "registry+https://github.com/rust-lang/crates.io-index" | 608 | source = "registry+https://github.com/rust-lang/crates.io-index" |
| 103 | checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" | 609 | checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" |
| 104 | 610 | ||
| 611 | [[package]] | ||
| 612 | name = "rand_core" | ||
| 613 | version = "0.10.1" | ||
| 614 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 615 | checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69" | ||
| 616 | |||
| 617 | [[package]] | ||
| 618 | name = "rsqlite-vfs" | ||
| 619 | version = "0.1.1" | ||
| 620 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 621 | checksum = "c51c9ae4df8a7fba42103df5c621fa3c37eccf3a3c650879e90fc48b11cc192c" | ||
| 622 | dependencies = [ | ||
| 623 | "hashbrown 0.16.1", | ||
| 624 | "thiserror", | ||
| 625 | ] | ||
| 626 | |||
| 627 | [[package]] | ||
| 628 | name = "rusqlite" | ||
| 629 | version = "0.40.2" | ||
| 630 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 631 | checksum = "23f2a97da3e3873c73cb2a2e71b35c40ff95e0b1eefa8d72d8499a6928c3b5b3" | ||
| 632 | dependencies = [ | ||
| 633 | "bitflags", | ||
| 634 | "fallible-iterator", | ||
| 635 | "fallible-streaming-iterator", | ||
| 636 | "hashlink", | ||
| 637 | "libsqlite3-sys", | ||
| 638 | "smallvec", | ||
| 639 | "sqlite-wasm-rs", | ||
| 640 | ] | ||
| 641 | |||
| 642 | [[package]] | ||
| 643 | name = "rustix" | ||
| 644 | version = "1.1.4" | ||
| 645 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 646 | checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190" | ||
| 647 | dependencies = [ | ||
| 648 | "bitflags", | ||
| 649 | "errno", | ||
| 650 | "libc", | ||
| 651 | "linux-raw-sys", | ||
| 652 | "windows-sys", | ||
| 653 | ] | ||
| 654 | |||
| 655 | [[package]] | ||
| 656 | name = "rustversion" | ||
| 657 | version = "1.0.23" | ||
| 658 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 659 | checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f" | ||
| 660 | |||
| 105 | [[package]] | 661 | [[package]] |
| 106 | name = "serde" | 662 | name = "serde" |
| 107 | version = "1.0.229" | 663 | version = "1.0.229" |
| ... | @@ -129,7 +685,7 @@ checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348" | ... | @@ -129,7 +685,7 @@ checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348" |
| 129 | dependencies = [ | 685 | dependencies = [ |
| 130 | "proc-macro2", | 686 | "proc-macro2", |
| 131 | "quote", | 687 | "quote", |
| 132 | "syn", | 688 | "syn 3.0.5", |
| 133 | ] | 689 | ] |
| 134 | 690 | ||
| 135 | [[package]] | 691 | [[package]] |
| ... | @@ -145,6 +701,112 @@ dependencies = [ | ... | @@ -145,6 +701,112 @@ dependencies = [ |
| 145 | "zmij", | 701 | "zmij", |
| 146 | ] | 702 | ] |
| 147 | 703 | ||
| 704 | [[package]] | ||
| 705 | name = "sha1" | ||
| 706 | version = "0.11.0" | ||
| 707 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 708 | checksum = "aacc4cc499359472b4abe1bf11d0b12e688af9a805fa5e3016f9a386dc2d0214" | ||
| 709 | dependencies = [ | ||
| 710 | "cfg-if", | ||
| 711 | "cpufeatures", | ||
| 712 | "digest", | ||
| 713 | ] | ||
| 714 | |||
| 715 | [[package]] | ||
| 716 | name = "sha2" | ||
| 717 | version = "0.11.0" | ||
| 718 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 719 | checksum = "446ba717509524cb3f22f17ecc096f10f4822d76ab5c0b9822c5f9c284e825f4" | ||
| 720 | dependencies = [ | ||
| 721 | "cfg-if", | ||
| 722 | "cpufeatures", | ||
| 723 | "digest", | ||
| 724 | ] | ||
| 725 | |||
| 726 | [[package]] | ||
| 727 | name = "shlex" | ||
| 728 | version = "2.0.1" | ||
| 729 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 730 | checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" | ||
| 731 | |||
| 732 | [[package]] | ||
| 733 | name = "slab" | ||
| 734 | version = "0.4.12" | ||
| 735 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 736 | checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" | ||
| 737 | |||
| 738 | [[package]] | ||
| 739 | name = "smallvec" | ||
| 740 | version = "1.16.0" | ||
| 741 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 742 | checksum = "b9be42f50aa861c555654aa3a37f52f4b1074bacf4e48fe0ef7fa584e80f1f0f" | ||
| 743 | |||
| 744 | [[package]] | ||
| 745 | name = "smb2" | ||
| 746 | version = "0.21.0" | ||
| 747 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 748 | checksum = "9ef4f20cff3d39a131335d17df6146eada7851d7705252f2765b8717a0f52db8" | ||
| 749 | dependencies = [ | ||
| 750 | "aes", | ||
| 751 | "aes-gcm", | ||
| 752 | "async-trait", | ||
| 753 | "ccm", | ||
| 754 | "cmac", | ||
| 755 | "digest", | ||
| 756 | "futures-util", | ||
| 757 | "getrandom", | ||
| 758 | "hmac", | ||
| 759 | "log", | ||
| 760 | "lz4_flex", | ||
| 761 | "md-5", | ||
| 762 | "md4", | ||
| 763 | "num_enum", | ||
| 764 | "pbkdf2", | ||
| 765 | "sha1", | ||
| 766 | "sha2", | ||
| 767 | "thiserror", | ||
| 768 | "tokio", | ||
| 769 | ] | ||
| 770 | |||
| 771 | [[package]] | ||
| 772 | name = "socket2" | ||
| 773 | version = "0.6.5" | ||
| 774 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 775 | checksum = "c3d1e2c7f27f8d4cb10542a02c49005dbd6e93095799d6f3be745fae9f8fedd4" | ||
| 776 | dependencies = [ | ||
| 777 | "libc", | ||
| 778 | "windows-sys", | ||
| 779 | ] | ||
| 780 | |||
| 781 | [[package]] | ||
| 782 | name = "sqlite-wasm-rs" | ||
| 783 | version = "0.5.5" | ||
| 784 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 785 | checksum = "dc3efc0da82635d7e1ced0053bbbfa8c7ab9645d0bf36ceb4f7127bb85315d75" | ||
| 786 | dependencies = [ | ||
| 787 | "cc", | ||
| 788 | "js-sys", | ||
| 789 | "rsqlite-vfs", | ||
| 790 | "wasm-bindgen", | ||
| 791 | ] | ||
| 792 | |||
| 793 | [[package]] | ||
| 794 | name = "subtle" | ||
| 795 | version = "2.6.1" | ||
| 796 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 797 | checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292" | ||
| 798 | |||
| 799 | [[package]] | ||
| 800 | name = "syn" | ||
| 801 | version = "2.0.119" | ||
| 802 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 803 | checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297" | ||
| 804 | dependencies = [ | ||
| 805 | "proc-macro2", | ||
| 806 | "quote", | ||
| 807 | "unicode-ident", | ||
| 808 | ] | ||
| 809 | |||
| 148 | [[package]] | 810 | [[package]] |
| 149 | name = "syn" | 811 | name = "syn" |
| 150 | version = "3.0.5" | 812 | version = "3.0.5" |
| ... | @@ -156,12 +818,192 @@ dependencies = [ | ... | @@ -156,12 +818,192 @@ dependencies = [ |
| 156 | "unicode-ident", | 818 | "unicode-ident", |
| 157 | ] | 819 | ] |
| 158 | 820 | ||
| 821 | [[package]] | ||
| 822 | name = "tempfile" | ||
| 823 | version = "3.27.0" | ||
| 824 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 825 | checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd" | ||
| 826 | dependencies = [ | ||
| 827 | "fastrand", | ||
| 828 | "getrandom", | ||
| 829 | "once_cell", | ||
| 830 | "rustix", | ||
| 831 | "windows-sys", | ||
| 832 | ] | ||
| 833 | |||
| 834 | [[package]] | ||
| 835 | name = "thiserror" | ||
| 836 | version = "2.0.20" | ||
| 837 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 838 | checksum = "ec86235f5fcc2a73650310756d2ac5b138a5780bbbdfae3eeccec992c435ba4f" | ||
| 839 | dependencies = [ | ||
| 840 | "thiserror-impl", | ||
| 841 | ] | ||
| 842 | |||
| 843 | [[package]] | ||
| 844 | name = "thiserror-impl" | ||
| 845 | version = "2.0.20" | ||
| 846 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 847 | checksum = "bc04cd3e1236dd4a98afca4569f2deb3f120e5422a4023be2cb683f8486292af" | ||
| 848 | dependencies = [ | ||
| 849 | "proc-macro2", | ||
| 850 | "quote", | ||
| 851 | "syn 3.0.5", | ||
| 852 | ] | ||
| 853 | |||
| 854 | [[package]] | ||
| 855 | name = "tokio" | ||
| 856 | version = "1.53.1" | ||
| 857 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 858 | checksum = "202caea871b69668250d242070849eb495be178ed697a3e98aebce5bc81a0bed" | ||
| 859 | dependencies = [ | ||
| 860 | "bytes", | ||
| 861 | "libc", | ||
| 862 | "mio", | ||
| 863 | "pin-project-lite", | ||
| 864 | "socket2", | ||
| 865 | "windows-sys", | ||
| 866 | ] | ||
| 867 | |||
| 868 | [[package]] | ||
| 869 | name = "toml_datetime" | ||
| 870 | version = "1.1.1+spec-1.1.0" | ||
| 871 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 872 | checksum = "3165f65f62e28e0115a00b2ebdd37eb6f3b641855f9d636d3cd4103767159ad7" | ||
| 873 | dependencies = [ | ||
| 874 | "serde_core", | ||
| 875 | ] | ||
| 876 | |||
| 877 | [[package]] | ||
| 878 | name = "toml_edit" | ||
| 879 | version = "0.25.13+spec-1.1.0" | ||
| 880 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 881 | checksum = "6975367e4d2ef766d86af01ffad14b622fecc8d4357a998fbc4deb6e9bacaf9b" | ||
| 882 | dependencies = [ | ||
| 883 | "indexmap", | ||
| 884 | "toml_datetime", | ||
| 885 | "toml_parser", | ||
| 886 | "winnow", | ||
| 887 | ] | ||
| 888 | |||
| 889 | [[package]] | ||
| 890 | name = "toml_parser" | ||
| 891 | version = "1.1.3+spec-1.1.0" | ||
| 892 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 893 | checksum = "1d38ac1cf9b95face32296c0a3ede1fdc270627c9d9c02a7274dd6d960dc4d56" | ||
| 894 | dependencies = [ | ||
| 895 | "winnow", | ||
| 896 | ] | ||
| 897 | |||
| 898 | [[package]] | ||
| 899 | name = "twox-hash" | ||
| 900 | version = "2.1.4" | ||
| 901 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 902 | checksum = "5283634e518fe9e82c7b20520bb4bc209009fd16c82077c802f8111ecbb0117a" | ||
| 903 | |||
| 904 | [[package]] | ||
| 905 | name = "typenum" | ||
| 906 | version = "1.20.1" | ||
| 907 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 908 | checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20" | ||
| 909 | |||
| 159 | [[package]] | 910 | [[package]] |
| 160 | name = "unicode-ident" | 911 | name = "unicode-ident" |
| 161 | version = "1.0.24" | 912 | version = "1.0.24" |
| 162 | source = "registry+https://github.com/rust-lang/crates.io-index" | 913 | source = "registry+https://github.com/rust-lang/crates.io-index" |
| 163 | checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" | 914 | checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" |
| 164 | 915 | ||
| 916 | [[package]] | ||
| 917 | name = "universal-hash" | ||
| 918 | version = "0.6.1" | ||
| 919 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 920 | checksum = "f4987bdc12753382e0bec4a65c50738ffaabc998b9cdd1f952fb5f39b0048a96" | ||
| 921 | dependencies = [ | ||
| 922 | "crypto-common", | ||
| 923 | "ctutils", | ||
| 924 | ] | ||
| 925 | |||
| 926 | [[package]] | ||
| 927 | name = "vcpkg" | ||
| 928 | version = "0.2.15" | ||
| 929 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 930 | checksum = "accd4ea62f7bb7a82fe23066fb0957d48ef677f6eeb8215f372f52e48bb32426" | ||
| 931 | |||
| 932 | [[package]] | ||
| 933 | name = "wasi" | ||
| 934 | version = "0.11.1+wasi-snapshot-preview1" | ||
| 935 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 936 | checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" | ||
| 937 | |||
| 938 | [[package]] | ||
| 939 | name = "wasm-bindgen" | ||
| 940 | version = "0.2.128" | ||
| 941 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 942 | checksum = "aecb87a33d3b0c5e3b7aa46336eaf486cffafbd281b195e4c8b80d50df2351bf" | ||
| 943 | dependencies = [ | ||
| 944 | "cfg-if", | ||
| 945 | "once_cell", | ||
| 946 | "rustversion", | ||
| 947 | "wasm-bindgen-macro", | ||
| 948 | "wasm-bindgen-shared", | ||
| 949 | ] | ||
| 950 | |||
| 951 | [[package]] | ||
| 952 | name = "wasm-bindgen-macro" | ||
| 953 | version = "0.2.128" | ||
| 954 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 955 | checksum = "a690d511e3c1a8b3a55e33511e3c2c00c78415cd23650f32b808627f5696b9ed" | ||
| 956 | dependencies = [ | ||
| 957 | "quote", | ||
| 958 | "wasm-bindgen-macro-support", | ||
| 959 | ] | ||
| 960 | |||
| 961 | [[package]] | ||
| 962 | name = "wasm-bindgen-macro-support" | ||
| 963 | version = "0.2.128" | ||
| 964 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 965 | checksum = "411e4887f0071ef2d2164a9d5fdf2d20efbef78fccd3a78b0c10a1dc5295e48a" | ||
| 966 | dependencies = [ | ||
| 967 | "bumpalo", | ||
| 968 | "proc-macro2", | ||
| 969 | "quote", | ||
| 970 | "syn 3.0.5", | ||
| 971 | "wasm-bindgen-shared", | ||
| 972 | ] | ||
| 973 | |||
| 974 | [[package]] | ||
| 975 | name = "wasm-bindgen-shared" | ||
| 976 | version = "0.2.128" | ||
| 977 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 978 | checksum = "81941cd78d0c92026c33e5e01312845a4cb1e9af3407f9134b100dd03144103e" | ||
| 979 | dependencies = [ | ||
| 980 | "unicode-ident", | ||
| 981 | ] | ||
| 982 | |||
| 983 | [[package]] | ||
| 984 | name = "windows-link" | ||
| 985 | version = "0.2.1" | ||
| 986 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 987 | checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" | ||
| 988 | |||
| 989 | [[package]] | ||
| 990 | name = "windows-sys" | ||
| 991 | version = "0.61.2" | ||
| 992 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 993 | checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" | ||
| 994 | dependencies = [ | ||
| 995 | "windows-link", | ||
| 996 | ] | ||
| 997 | |||
| 998 | [[package]] | ||
| 999 | name = "winnow" | ||
| 1000 | version = "1.0.4" | ||
| 1001 | source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| 1002 | checksum = "23b97319f7b8343df12cc98938e5c3eb436064524c8d2b4e30a1d3a36eecdf81" | ||
| 1003 | dependencies = [ | ||
| 1004 | "memchr", | ||
| 1005 | ] | ||
| 1006 | |||
| 165 | [[package]] | 1007 | [[package]] |
| 166 | name = "zmij" | 1008 | name = "zmij" |
| 167 | version = "1.0.23" | 1009 | version = "1.0.23" |
FEATURES.md deleted-29
| ... | @@ -1,29 +0,0 @@ | ||
| 1 | # Milestone 6 feature matrix | ||
| 2 | |||
| 3 | The matrix follows MS-ONE's document families, including features absent from the | ||
| 4 | personal corpus. Each row requires an independently authored native fixture, | ||
| 5 | Rust interpretation checks, and a review-report inspection before acceptance. | ||
| 6 | A retained opaque payload is evidence of preservation, not interpreted coverage. | ||
| 7 | |||
| 8 | | Family | Required cases | Specification Evidence | | ||
| 9 | | --- | --- | --- --- | | ||
| 10 | | Notebook structure | Multiple notebooks, section groups, section ordering/colors, duplicate names, empty sections | 2.2.14–18, 2.2.91–96 [Native feature controls](corpus/m6/native-features-01/ORACLE.md); [Rust scale](evidence/m6/rust-scale-final-01.log) | | ||
| 11 | | Pages | Titles/alternate titles, order, subpages, duplicate text/titles, authors/timestamps, RTL | 2.2.19, 2.2.29–31, 2.2.58, 2.3.74 [Direction/origin controls](corpus/m6/native-page-direction-03); [private review](evidence/m6/private-current-report-05/qa/review.json) | | ||
| 12 | | Outline hierarchy | Positioned outlines, nesting, outline groups, indentation, collapsed/hidden content | 2.2.20–23, 2.3.8, 2.3.18–19 [Structure](corpus/m6/native-structure-01/ORACLE.md); [saved collapse](corpus/m6/rust-collapse-control-01/ORACLE.md) | | ||
| 13 | | Text | ASCII/Unicode preference, surrogate pairs, combining marks, mixed scripts, empty/long paragraphs | 2.1.4, 2.2.5, 2.2.23, 2.2.89 [100 native histories](evidence/m6/native-histories-independent-03.log); [break controls](corpus/m6/native-break-controls-02) | | ||
| 14 | | Character formatting | Mixed runs, font/size/color/highlight, bold/italic/underline/strike, super/subscript, language | 2.2.43–45, 2.2.76–77, 2.3.9–16 [Native probes](corpus/m6/native-probes-01/ORACLE.md); [private native comparison](evidence/m6/private-current-compare-01.log) | | ||
| 15 | | Paragraph formatting | Named styles, alignment, spacing, RTL, style inheritance | 2.2.44, 2.2.80, 2.2.83, 2.3.81–83 [Structure](corpus/m6/native-structure-01/ORACLE.md); [feature report inspection](evidence/m6/features-report-05/qa/review.json) | | ||
| 16 | | Lists | Bullets, numbering, restarts, mixed indentation, custom fonts/formats | 2.2.25, 2.2.57, 2.3.20, 2.3.43 [Structure](corpus/m6/native-structure-01/ORACLE.md); [numbering controls](corpus/m6/native-features-01/ORACLE.md) | | ||
| 17 | | Tables | Multiple rows/columns, nested content, widths, shading, borders, locked columns | 2.2.26–28, 2.2.66, 2.2.70, 2.2.97 [Widths/locks](corpus/m6/native-structure-01/ORACLE.md); [RTL](corpus/m6/native-page-direction-03); [shading limitation](corpus/m6/cell-shading-control-01/ORACLE.md) | | ||
| 18 | | Links | External/internal links, formatted labels, embedded text-run data | 2.2.78, 2.2.82, 2.2.90, 2.3.75–77 [Native link controls](corpus/m6/native-link-controls-01); [empty-label regression](corpus/m6/native-empty-link-01) | | ||
| 19 | | Images | Multiple formats, sizes, alt text, filenames, internal/external containers, background/printout images | 2.2.24, 2.2.36, 2.2.59, 2.2.75, 2.2.79 [Native formats/roles](corpus/m6/native-features-01/ORACLE.md); [browser inspection](evidence/m6/features-report-05/qa/review.json) | | ||
| 20 | | Files and media | Attachments, original filenames, recording identifiers, audio/video payloads and associated text | 2.2.32–33, 2.2.60–61, 2.2.71–72 [Native attachment/recording controls](corpus/m6/native-features-01/ORACLE.md) | | ||
| 21 | | Ink and embedded objects | Strokes, placement, native payload/export comparison, math and embedded text-run objects | Native ink corpus; 2.3.79–80 [Native ink](corpus/native-ink/cold-ui-ink); [math](corpus/m6/native-math-01/ORACLE.md): payload/run preservation, opaque rendering | | ||
| 22 | | Tags and tasks | Standard/custom tags, multiple tags, checked/unchecked state, task status/dates | 2.1.9, 2.2.41–42, 2.2.84–88, 2.3.85–96 [Native task controls](corpus/m6/native-features-01/ORACLE.md); [private tags](evidence/m6/private-current-compare-01.log) | | ||
| 23 | | History and recovery | Version pages/contexts, recycle-bin pages, conflicts and their source relationships | 2.1.1–2, 2.1.17, 2.2.34–40, 2.2.86 [Private history review](evidence/m6/private-current-report-05/qa/review.json); [live conflicts/recovery](corpus/m6/live-collaboration-15/ORACLE.md) | | ||
| 24 | | Protection and unknowns | Locked section identity/ciphertext, unknown JCIDs/properties, malformed structures | MS-ONESTORE encryption; MS-ONE 2.1.5, 2.1.12 [Existing corpus checks](evidence/m6/corpus-regression-01.log); [document fuzzing](evidence/m6/document-public-fuzz-08.log): ciphertext/raw data preserved | | ||
| 25 | | Scale and interactions | Rust/native large notebooks, many pages/objects/assets, repeated identities/text, mixed feature histories | All applicable rows [Independent native histories](evidence/m6/native-histories-independent-03.log); [Rust scale](evidence/m6/rust-scale-final-01.log) | | ||
| 26 | |||
| 27 | [M6-ACCEPTANCE.md](M6-ACCEPTANCE.md) records the native builds, comparisons, | ||
| 28 | report inspections, regression campaigns and interpretation boundaries. `evidence/m6/spec-objects.json` | ||
| 29 | is the current object-definition inventory extracted from the supplied markdown. | ||
M6-ACCEPTANCE.md deleted-102
| ... | @@ -1,102 +0,0 @@ | ||
| 1 | # Milestone 6 verification | ||
| 2 | |||
| 3 | The deliverable is a read-only document model and reading report. Its acceptance | ||
| 4 | boundary is the copied personal notebook: automated native comparisons plus a | ||
| 5 | page-by-page review with no known easily spottable content failures. Native PDF | ||
| 6 | references preserve access to the original canvas arrangement. | ||
| 7 | |||
| 8 | ## Personal notebook | ||
| 9 | |||
| 10 | [The current report](evidence/m6/private-current-report-05/index.html) includes the | ||
| 11 | user-approved newer `Video.one`. [Its source manifest](corpus/private/current-source-manifest.json) | ||
| 12 | identifies that snapshot; [the original frozen manifest](corpus/private/source-manifest.json) | ||
| 13 | remains separate. The original files are never edited by the library tests. | ||
| 14 | |||
| 15 | | Check | Evidence | | ||
| 16 | | --- | --- | | ||
| 17 | | 26 current/recycle pages, 109 tags, 188,560 explicit character-format comparisons | [Fresh-cache comparison](evidence/m6/private-current-compare-01.log) | | ||
| 18 | | 18 historical pages, including all three versions of the updated Video page | [Album](evidence/m6/private-history-final-01.log), [Video](evidence/m6/private-video-current-history-cold-compare-01.log), [deleted page](evidence/m6/private-deleted-history-final-01.log) | | ||
| 19 | | All 45 report pages reviewed: ordinary, historical, recycle-bin and default template | [Review record](evidence/m6/private-current-report-05/qa/review.json) | | ||
| 20 | | Frozen copy, approved current copy and live sources match their respective manifests | [Integrity check](evidence/m6/source-integrity-current-01.json) | | ||
| 21 | |||
| 22 | The updated current page was inspected through eight browser viewports and two | ||
| 23 | native PDF pages. Unchanged page bodies inherit their earlier review only after | ||
| 24 | an exact rendered-content comparison. All exported image payloads decode. | ||
| 25 | History labels use America/Los_Angeles. Seventeen historical dates match native | ||
| 26 | UI evidence; copying the sole deleted-page version changes its native displayed | ||
| 27 | date, so the report retains the source revision timestamp and records that | ||
| 28 | normalization separately. | ||
| 29 | |||
| 30 | ## Independent and adversarial checks | ||
| 31 | |||
| 32 | | Campaign | Result and evidence | | ||
| 33 | | --- | --- | | ||
| 34 | | Native edit histories | [100 histories / 2,000 independently specified operations](evidence/m6/native-histories-independent-03.log); [101 resulting pages compared](evidence/m6/native-histories-final-02.log) | | ||
| 35 | | Shrinking | [Seed 21](evidence/m6/history-shrink-21/minimal.json) reduced from 20 operations to 3 in 37 fresh-clone replays; [empty-link regression](corpus/m6/native-empty-link-01) | | ||
| 36 | | Rust scale generation | [128 sections / 2,097,152 Unicode scalars](evidence/m6/rust-scale-final-01.log), including reverse section ordering after native import | | ||
| 37 | | Document fuzzing | [65,736 runs / 121 seconds](evidence/m6/document-public-fuzz-08.log) after the collapse-field change; prior [100,398-run campaign](evidence/m6/document-public-fuzz-07.log) and [11,423 private-seed runs](evidence/m6/document-private-fuzz-03.log) | | ||
| 38 | | Rust checks | [All targets](evidence/m6/rust-regression-06.log), [clippy](evidence/m6/clippy-final-04.log) | | ||
| 39 | | Native/report tooling | [18 tests](evidence/m6/tool-regression-15.log), including deliberate oracle failures and TIFF/native pixel parity | | ||
| 40 | | Existing writer and fault cases | [Writer](evidence/m6/writer-regression-01.log), [collaboration/FLUSH regression](evidence/m6/collaboration-regression-02.log) | | ||
| 41 | | Expanded live matrix | [Seven passing cases](corpus/m6/live-collaboration-15/ORACLE.md), [fresh-cache comparison](evidence/m6/live-collaboration-cold-compare-15.log), [native conflict view](corpus/m6/live-collaboration-cold-15/conflict.png) | | ||
| 42 | |||
| 43 | The document fuzzer mutates property streams inside native files and repairs | ||
| 44 | object checksums, reaching document interpretation beyond header rejection. | ||
| 45 | It traverses referenced historical revisions and resolves text runs. Native | ||
| 46 | histories have a separate expected-operation model; matching two views derived | ||
| 47 | from the Rust decoder is not counted as independent verification. | ||
| 48 | |||
| 49 | The live matrix covers disjoint edits, competing edits, Samba restart, per-client | ||
| 50 | transport loss, whole-file lock contention, OneNote process termination and abrupt | ||
| 51 | VM termination. The process/VM cases recover already server-persisted edits. | ||
| 52 | They do not establish unsynchronized-cache durability or physical power-loss | ||
| 53 | safety. Earlier stage-5 cases separately cover lost successful SMB FLUSH replies. | ||
| 54 | |||
| 55 | ## Interpretation and oracle boundaries | ||
| 56 | |||
| 57 | - Native XML omits partial black highlights. Native PDF rectangles supply a | ||
| 58 | separate check; the reader retains the stored highlight. | ||
| 59 | - RTL native XML column order differs from physical left-to-right storage order. | ||
| 60 | Independent PDF coordinates establish the conversion. Page-origin translation, | ||
| 61 | locked/unlocked column widths and printout borders have isolated controls. | ||
| 62 | - A saved paragraph collapse default differs from a client's transient expanded | ||
| 63 | view. [The native UI control](corpus/m6/native-expanded-control-ui-01/ORACLE.md) | ||
| 64 | and [Rust-authored control](corpus/m6/rust-collapse-control-01/ORACLE.md) distinguish them. | ||
| 65 | - Ink, structured equations, encrypted content and unknown properties retain | ||
| 66 | payloads or source identities. Preservation is not decryption or interpretation. | ||
| 67 | Structured equation runs expose a native-reference placeholder; ordinary inline | ||
| 68 | math text remains readable. | ||
| 69 | - TIFF browser previews match OneNote's exported pixels while original TIFF bytes | ||
| 70 | remain available. Native image conversion is not a license to replace source data. | ||
| 71 | - [Cell shading](corpus/m6/cell-shading-control-01/ORACLE.md) follows the documented | ||
| 72 | stored COLORREF value. A fresh OneNote 2010 cache opens the control but omits | ||
| 73 | shading from XML and renders the cell white; native rendering parity for that | ||
| 74 | property is not claimed. | ||
| 75 | |||
| 76 | Verification uses OneNote 2010 build 14.0.4763.1000 in disposable Windows 7 clones. | ||
| 77 | The earlier storage/collaboration corpus used build 14.0.7015.1000 on the physical | ||
| 78 | machine. Each run records its environment and cleanup. These finite campaigns | ||
| 79 | support this milestone's review boundary, not a claim of universal compatibility | ||
| 80 | or absence of bugs. | ||
| 81 | |||
| 82 | ## Reproduce | ||
| 83 | |||
| 84 | Use a new output directory for each native capture or report: | ||
| 85 | |||
| 86 | ```sh | ||
| 87 | cargo test --all-targets | ||
| 88 | cargo clippy --all-targets -- -D warnings | ||
| 89 | cargo build --example document | ||
| 90 | PYTHONPATH=tools python3 -m unittest discover -s tools -p 'test_*.py' | ||
| 91 | python3 tools/native_runner.py COPIED_NOTEBOOK NEW_CAPTURE --pdf | ||
| 92 | python3 tools/verify-document.py COPIED_NOTEBOOK NEW_CAPTURE/read | ||
| 93 | python3 tools/notebook_report.py COPIED_NOTEBOOK NEW_REPORT \ | ||
| 94 | --native NEW_CAPTURE/read --timezone America/Los_Angeles | ||
| 95 | cargo +nightly fuzz run document -- -max_total_time=120 -max_len=512 -rss_limit_mb=2048 | ||
| 96 | python3 tools/native_collaboration.py NEW_CAPTURE --linux OWNED_LAB_NAME | ||
| 97 | ``` | ||
| 98 | |||
| 99 | Python native comparison/report tests require Pillow and pdfplumber. Historical | ||
| 100 | report references use repeated `--versions CAPTURE_DIRECTORY` arguments after | ||
| 101 | `verify-document.py --versions CAPTURE_DIRECTORY/version-ui.json` has checked | ||
| 102 | source hashes and associated the native copies with stored revisions. | ||
MILESTONE6.md deleted-183
| ... | @@ -1,183 +0,0 @@ | ||
| 1 | # Milestone 6: a reviewable OneNote document model | ||
| 2 | |||
| 3 | Completed milestone; verification and review artifacts are recorded in | ||
| 4 | [M6-ACCEPTANCE.md](M6-ACCEPTANCE.md). Clover chose | ||
| 5 | extracting a real notebook into a readable document model as the first hands-on | ||
| 6 | result. This milestone combines document-format work with repeatable native | ||
| 7 | verification; completion requires no known easily spottable bugs in her notebook | ||
| 8 | after automated native comparisons and an independent page-by-page inspection. | ||
| 9 | Her review should find subtle design issues, not serve as the basic QA pass. | ||
| 10 | |||
| 11 | ## What Clover receives | ||
| 12 | |||
| 13 | A local, read-only review report generated from the copied personal notebook, | ||
| 14 | with notebook/section/page navigation, readable content, extracted media, and a | ||
| 15 | document-structure view. The report exposes paragraph nesting and ordering, | ||
| 16 | text-run styles, outline coordinates, tables, links, tags, and source identities. | ||
| 17 | It accompanies a machine-readable model and an asset directory, so the result is | ||
| 18 | useful to future applications as well as inspectable by a person. | ||
| 19 | |||
| 20 | The report presents a readable interpretation of the document. It does not claim | ||
| 21 | to reproduce OneNote's layout engine. Coordinate values and native reference | ||
| 22 | captures let Clover assess spatial information without depending on a new canvas | ||
| 23 | renderer. Ink, recording data, encrypted sections, and unrecognized structures | ||
| 24 | must remain explicitly accounted for, with retained payloads or source references | ||
| 25 | as appropriate. Unsupported interpretation must never turn into silent omission. | ||
| 26 | |||
| 27 | Feedback should be attached to identifiable pages/objects and answer: | ||
| 28 | |||
| 29 | - Does the section/page hierarchy match the notebook, including subpages and order? | ||
| 30 | - Does the representation preserve meaningful paragraph/list/table structure? | ||
| 31 | - Are text, formatting, links, images, attachments, and tags correctly associated? | ||
| 32 | - Where does spatial arrangement communicate something the readable view loses? | ||
| 33 | - Which opaque content prevents this model from supporting a useful reader? | ||
| 34 | |||
| 35 | The review ends with concrete corrections or priorities for the document model. | ||
| 36 | Editing API design follows that feedback; no GUI editor is part of this milestone. | ||
| 37 | |||
| 38 | ## Sequence and gates | ||
| 39 | |||
| 40 | | Step | Deliverable | Gate before proceeding | | ||
| 41 | | --- | --- | --- | | ||
| 42 | | 1. Repeatable native oracle | Automated fresh-clone runs using the existing VM controllers, named disposable notebooks, bounded captures, and unconditional teardown | Existing reader/writer native cases replay successfully; intentional content corruption causes a failed comparison; a failed run retains a reproducible artifact bundle and leaves no owned VM running | | ||
| 43 | | 2. Document semantics | A typed view over the resolved store graph, independent of COM and filesystem transport | Every interpreted feature has a native fixture and assertions for its semantic invariants; unknown and encrypted content is represented explicitly | | ||
| 44 | | 3. Real-notebook review | Machine-readable export, assets, and a human-readable report derived from the same model | All copied sections/pages are accounted for, source hashes remain unchanged, and supported content matches newly captured native evidence | | ||
| 45 | | 4. Adversarial validation | Seeded native edit histories, parser fuzzing, and replay of existing commit/concurrency cases | The feature/failure matrix passes, discrepancies have regression cases, and the acceptance report distinguishes exact matches, allowed native normalization, and opaque content | | ||
| 46 | |||
| 47 | ## Document scope | ||
| 48 | |||
| 49 | Start with notebook and section structure, ordered pages/subpages, titles, outlines, | ||
| 50 | paragraphs, and Unicode text runs. Add character/paragraph formatting, list and | ||
| 51 | indentation semantics, tables, hyperlinks, images, attachments, and note tags. | ||
| 52 | Expose conflict pages and recycle-bin membership separately from ordinary pages; | ||
| 53 | retain their relationship to their source page/section. Account for observed | ||
| 54 | unrecognized page-like metadata instead of filtering it out of the report. | ||
| 55 | |||
| 56 | MS-ONE text-run boundaries use character positions with rules distinct from Rust | ||
| 57 | UTF-8 byte indices. Fixtures must cover surrogate pairs, combining marks, RTL text, | ||
| 58 | mixed formatting, empty paragraphs, repeated text, and embedded objects. Titles, | ||
| 59 | cached title strings, styles, and layout metadata need explicit interpretation; | ||
| 60 | generic scalar decoding alone cannot establish those relationships. | ||
| 61 | |||
| 62 | Keep document interpretation above `onestore`'s revision/property graph. Retain | ||
| 63 | access to raw identities and values for unknown properties. Add types only where | ||
| 64 | they express document semantics or prevent invalid interpretation; keep the initial | ||
| 65 | export/API provisional until Clover has reviewed actual notebook content. | ||
| 66 | |||
| 67 | ## Independent verification | ||
| 68 | |||
| 69 | ```text | ||
| 70 | Authored operation history ──→ expected document semantics | ||
| 71 | │ | ||
| 72 | └─→ real OneNote edit/save → captured .one + native XML/payloads | ||
| 73 | │ | ||
| 74 | └─→ Rust document model | ||
| 75 | │ | ||
| 76 | compare all three ─┘ | ||
| 77 | |||
| 78 | Existing Rust scalar edit → fresh native read/edit/save → Rust reread | ||
| 79 | ``` | ||
| 80 | |||
| 81 | Native capture runs use a fresh clone/cache for each independent acceptance case. | ||
| 82 | Tests of a continuing collaboration session deliberately retain that session's | ||
| 83 | cache, then use a separate fresh verifier after synchronization. Completion must | ||
| 84 | be observed through file/content state, not inferred from an arbitrary sleep or a | ||
| 85 | successful asynchronous COM call. COM IDs are cache identities, not persistent | ||
| 86 | notebook identities. | ||
| 87 | |||
| 88 | Compare page relationships/order, paragraph boundaries, text runs/styles, table | ||
| 89 | cells, positions, links, and payload bytes. Hashes establish artifact integrity; | ||
| 90 | they do not establish semantic correctness. Native XML supplies an independent | ||
| 91 | view of supported content. Selected UI captures cover facts COM omits, especially | ||
| 92 | conflicts and visual interpretation. Whole-file byte equality after a native save | ||
| 93 | is not the oracle because OneNote rewrites metadata and representation. | ||
| 94 | |||
| 95 | Build native-only control cases for observed normalization, including the previous | ||
| 96 | table-width and ink z-order changes. Comparison exceptions require specific | ||
| 97 | evidence and a narrow assertion; broad removal of timestamps, IDs, or geometry | ||
| 98 | must not conceal meaningful changes. | ||
| 99 | |||
| 100 | Use a small independent test model for generated operations; expected values must | ||
| 101 | not be reconstructed by the same Rust decoder being checked. Shrink a failed | ||
| 102 | history by removing operations and simplifying their parameters, replaying each | ||
| 103 | candidate from a clean starting state. Preserve seed, operation history, input and | ||
| 104 | output files, model diff, native XML/payloads, app/server versions, and relevant | ||
| 105 | trace excerpts. A regression must reproduce the observed discrepancy before its | ||
| 106 | fix can be accepted. | ||
| 107 | |||
| 108 | Build a specification-derived feature matrix, including features absent from the | ||
| 109 | personal and stock notebooks. Produce large synthetic notebooks both through Rust | ||
| 110 | and through real OneNote; compare independent generation paths and exercise size, | ||
| 111 | ordering, repetition, and mixed-feature interactions. Any unsupported feature must | ||
| 112 | be explicitly accounted for and investigated, not silently omitted from coverage. | ||
| 113 | |||
| 114 | Initial randomized target: 100 reproducible native histories of roughly 20 | ||
| 115 | operations, spanning the supported feature matrix. This is a campaign budget, | ||
| 116 | not a statistical safety claim. Native-generated files then seed fast local | ||
| 117 | reader fuzzing. Continue the existing stateful short-I/O/partial-persistence | ||
| 118 | commit tests; run the existing collaboration scenarios with two Windows clients | ||
| 119 | and the disposable Linux server. Process termination, Samba restart, transport | ||
| 120 | loss, and abrupt VM termination are distinct fault cases and must be reported as | ||
| 121 | such. VM termination does not simulate physical host power loss. | ||
| 122 | |||
| 123 | ## Tooling findings and setup work | ||
| 124 | |||
| 125 | The implementation is in `tools/w7`. Windows instances use sparse qcow2 overlays | ||
| 126 | on a sealed base, unique hostnames/MACs/control targets, and an authenticated agent. | ||
| 127 | Linux instances also use overlays. A VDE network connects Windows clients to one | ||
| 128 | Linux Samba appliance at `192.168.77.1`; the Mac has forwarded SSH and SMB ports. | ||
| 129 | The MCP's up/down/status operations are enough for ordinary fresh-clone tests. | ||
| 130 | Reuse those controllers from the test runner rather than implementing another | ||
| 131 | QEMU lifecycle. Add a narrow controlled-crash facility only when fault cases need | ||
| 132 | it, with ownership checks and restart/recovery of the same overlay. | ||
| 133 | |||
| 134 | Start with two Windows clients and one Linux server. Windows uses x86 CPU | ||
| 135 | emulation on this host; the suggested 25-client capacity has not been measured. | ||
| 136 | Increase concurrency from measured test throughput and resource use. One Linux | ||
| 137 | lab address means server restart/configuration tests require exclusive ownership | ||
| 138 | of that server; ordinary cases can use separate notebook directories. | ||
| 139 | |||
| 140 | The scout confirmed: | ||
| 141 | |||
| 142 | - Windows 7 clone `m6scout`, PowerShell 5.1.14409.1005, OneNote 14.0.4763.1000. | ||
| 143 | The earlier physical-machine corpus used OneNote 14.0.7015.1000. Record both | ||
| 144 | builds' evidence separately; success on one must not be silently attributed to | ||
| 145 | the other. | ||
| 146 | - The clone desktop was 800×600, so existing 1280×720 AHK coordinates cannot be | ||
| 147 | reused without setting and verifying display configuration. | ||
| 148 | - Z: mapped to `\\10.0.0.1\agent` (zenith). The disposable Linux share is a | ||
| 149 | different destination, `\\192.168.77.1\agent`. Parameterize shared paths rather | ||
| 150 | than reusing the old A: convention or remapping Z:. | ||
| 151 | - Linux reported Samba 4.22.10 and an ext4 data filesystem. Windows read a | ||
| 152 | Linux-created marker and Linux read a Windows-created marker through that share. | ||
| 153 | - A fresh native cache opened the template-free Rust notebook and returned its | ||
| 154 | expected paragraph through COM. The captured XML is under | ||
| 155 | `evidence/milestone6-scout/page.xml`. Cache reset remains necessary when reusing | ||
| 156 | file identities within a clone. | ||
| 157 | - Procmon started through `win7_spawn` and wrote a PML capture. The unfiltered | ||
| 158 | short capture grew to about 126 MB; filter by test process/path and bound capture | ||
| 159 | duration before using it in a campaign. Trace contents were not analyzed in this | ||
| 160 | scout, and the transient PML was removed with the clone. | ||
| 161 | - All six VM-tooling tests passed with the scout running. One test assumed the | ||
| 162 | first control port was free; it now checks that the registered target points to | ||
| 163 | the allocated port, while retaining the separate uniqueness assertion. | ||
| 164 | - Both scout VMs were stopped and deleted after the smoke checks. Neither the | ||
| 165 | sealed images nor the personal notebook were edited. | ||
| 166 | |||
| 167 | Existing native scripts assume a parked personal Windows profile, fixed A: paths, | ||
| 168 | and specific display coordinates. Adapt their orchestration for disposable clones | ||
| 169 | while retaining the guards used for the physical laptop. Windows base identity, | ||
| 170 | Linux package/server versions, and relevant configuration must accompany every | ||
| 171 | run; cloud-init currently installs packages at clone creation time. | ||
| 172 | |||
| 173 | ## Completion and review boundary | ||
| 174 | |||
| 175 | The milestone is ready after the copied notebook passes automated comparison and | ||
| 176 | an independent page-by-page inspection with no known easily spottable bugs. | ||
| 177 | Corrections found during that inspection need reproductions and regression checks | ||
| 178 | before handoff. Clover's subsequent review addresses subtle representation and | ||
| 179 | design decisions. Deliver the report, model, | ||
| 180 | assets, acceptance matrix, and reproduction command together. Preserve originals | ||
| 181 | and export evidence before automatically deleting all machines created by the | ||
| 182 | run. The next decision is which document operations to expose for editing based | ||
| 183 | on that review, rather than another general approval to continue infrastructure. | ||
MILESTONE7.md deleted-429
| ... | @@ -1,429 +0,0 @@ | ||
| 1 | # Milestone 7: concurrent editing | ||
| 2 | |||
| 3 | Acceptance gates passed for the tested macOS Rust, OneNote 2010 and Linux Samba | ||
| 4 | configuration. The edited notebook has fresh native references and an HTML | ||
| 5 | review; independent operation histories verify overlapping clients, retained | ||
| 6 | conflicts and the final notebook after native application closure. | ||
| 7 | |||
| 8 | ## Gates | ||
| 9 | |||
| 10 | 1. Document edits: publish text and dependent run boundaries atomically; check | ||
| 11 | Unicode boundaries, preserve untouched objects and opaque properties, reject | ||
| 12 | unsupported edits before writing. Validate native round trips before racing. | ||
| 13 | 2. Random-edit CLI: select a page and seed, record the exact intended operation, | ||
| 14 | source identity and commit outcome. Operate on disposable notebook copies. | ||
| 15 | 3. Local concurrency: independent reader/writer processes, synchronized starts, | ||
| 16 | stale snapshots, lock contention, randomized delays and interrupted commits. | ||
| 17 | Verify every successful observation and explain every accepted edit. | ||
| 18 | 4. Native concurrency: grow from three native clients plus multiple Rust | ||
| 19 | processes to larger measured workloads. Mix disjoint and same-paragraph | ||
| 20 | changes, ordinary reads, synchronization, disconnects and restarts. Prove | ||
| 21 | operation overlap from recorded intervals; do not count open idle clients. | ||
| 22 | 5. Adversarial replay: retain seeds, operation logs and snapshots; shrink failures | ||
| 23 | and add regressions. Compare the independent operation model with current | ||
| 24 | native content and explicitly reachable conflicts, then cold-reopen results. | ||
| 25 | 6. Review and cleanup: inspect the edited report and native references, rerun | ||
| 26 | prior gates, verify original source hashes, delete owned machines, and record | ||
| 27 | observed client counts, operations, faults and the limits of the evidence. | ||
| 28 | |||
| 29 | Disjoint acknowledged changes must survive. Competing changes must be accounted | ||
| 30 | for by their observed ordering or accessible conflict content; historical bytes | ||
| 31 | alone do not prove recovery. Unknown commit outcomes require rereading before a | ||
| 32 | retry. Native COM page snapshots must not replay untouched stale containers. | ||
| 33 | |||
| 34 | Crash tests distinguish application cache state, server-persisted state and | ||
| 35 | storage durability. No finite campaign proves all schedules or hardware | ||
| 36 | power-loss behavior. | ||
| 37 | |||
| 38 | ## Replay | ||
| 39 | |||
| 40 | Use fresh output directories and Linux VM names; the native harness deletes its | ||
| 41 | owned machines on exit. The Linux lab address allows one server run at a time. | ||
| 42 | |||
| 43 | ```sh | ||
| 44 | python3 tools/native_collaboration.py evidence/m7/new-stress --linux new-stress --stress-clients 3 --stress-operations 150 --sync-every 0 --rust-writers 4 --rust-readers 3 --edit --seed 913 | ||
| 45 | python3 tools/native_runner.py evidence/m7/new-stress/stress-closed/notebook evidence/m7/new-cold --expected-pages 1 | ||
| 46 | python3 tools/verify-document.py evidence/m7/new-stress/stress-closed/notebook evidence/m7/new-cold/read | ||
| 47 | python3 tools/native_stress.py evidence/m7/new-stress evidence/m7/new-cold/read | ||
| 48 | python3 tools/native_collaboration.py evidence/m7/new-conflict --linux new-conflict --conflict-clients 3 --rust-writers 3 --rust-readers 2 --stress-operations 5 --seed 917 | ||
| 49 | cargo +nightly fuzz run edit_text -- -max_total_time=300 -max_len=128 -rss_limit_mb=2048 | ||
| 50 | ``` | ||
| 51 | |||
| 52 | ## Evidence | ||
| 53 | |||
| 54 | - `evidence/m7/text-edit-native-02`: a length-changing Unicode edit reopened in | ||
| 55 | a fresh OneNote VM; 297 explicit formatting comparisons passed on one page. | ||
| 56 | - `evidence/m7/local-concurrency-02`: ten Rust writers and six readers; | ||
| 57 | 600 acknowledged commits and 8,101 checked observations. The final notebook | ||
| 58 | reopened in a fresh native cache (`local-concurrency-native-02`). | ||
| 59 | - `evidence/m7/edit-text-multi-fuzz-03.log`: 31,394 stateful executions with | ||
| 60 | six independently stale writer snapshots and interrupted publication. This | ||
| 61 | simulates interleavings; native concurrency is a separate gate. | ||
| 62 | - `evidence/m7/native-concurrency-01` and `native-concurrency-02`: three native | ||
| 63 | clients, four Rust writers and three Rust readers exposed a macOS SMB lock | ||
| 64 | failure. Native COM acknowledgments were not durable synchronization: the | ||
| 65 | stalled server copy did not contain those cached native edits. These runs | ||
| 66 | failed acceptance and all of their machines were deleted. | ||
| 67 | - `evidence/m7/lock-race-default`: a notebook-free, four-process reproducer | ||
| 68 | detected overlapping exclusive holders using an independent local marker, | ||
| 69 | plus stranded locks when switching read-only and read/write opens. Explicit | ||
| 70 | unlock alone did not fix it. Uniform access modes still violated exclusion. | ||
| 71 | - `evidence/m7/lock-race-atomic`: atomic open-and-lock passed all three access | ||
| 72 | modes without overlap. The macOS file adapter now uses that operation. | ||
| 73 | - `evidence/m7/smb-rust-only-04`: the formerly stalled workload completed all | ||
| 74 | 120 commits, 940 verified reads and 203 overlapping writer calls after that | ||
| 75 | change. Broader acceptance remains active; this does not establish a native | ||
| 76 | multi-client pass. | ||
| 77 | |||
| 78 | - `evidence/m7/lock-race-atomic-16`: sixteen processes, 4,516 acquired locks | ||
| 79 | across read-only, read/write and alternating access; no marker overlap. | ||
| 80 | - `evidence/m7/smb-rust-only-05`: ten writers and six readers completed 1,000 | ||
| 81 | commits and 14,913 verified observations. The saved result subsequently | ||
| 82 | failed fresh-cache native acceptance (`smb-concurrency-native-05`): OneNote | ||
| 83 | returned zero pages after five minutes. The dependency-checkpoint fix and subsequent native checks below explain | ||
| 84 | and resolve this failure; Rust parsing alone was insufficient acceptance. | ||
| 85 | - `evidence/m7/native-concurrency-03`: all 90 native edits and 120 Rust edits | ||
| 86 | converged, but the overlap gate correctly rejected the run. Native calls | ||
| 87 | began 4.5–5.4 seconds after the first Rust commit; Rust finished at 2.7 | ||
| 88 | seconds. The shared-file start marker was delayed. The harness now waits | ||
| 89 | for each native client's first-edit acknowledgment before releasing Rust. | ||
| 90 | - `evidence/m7/local-stateful-01`: eight writers and five readers, 800 random | ||
| 91 | length-changing Unicode replacements, 9,820 verified observations. The | ||
| 92 | independent oracle applies recorded UTF-16 edit intents to each committed | ||
| 93 | state and checks all reader observations against the resulting history. | ||
| 94 | - `evidence/m7/edit-text-multi-fuzz-04.log`: 26,377 renewed stateful fuzz | ||
| 95 | executions completed without a failure. | ||
| 96 | - `evidence/m7/native-concurrency-04`: all 90 native edits and 120 Rust edits | ||
| 97 | converged, but overlap remained zero. Rust's first successful commit occurred | ||
| 98 | 10.21 seconds after its start signal, following thousands of busy reads while | ||
| 99 | native clients synchronized after every edit. Atomic open requests deny-all | ||
| 100 | sharing on this mount; native open handles can postpone Rust admission. | ||
| 101 | - `evidence/m7/native-threshold-02` and `native-threshold-03`: a fresh section | ||
| 102 | containing the same final text opened normally. The long-history section | ||
| 103 | opened after shortening revision dependencies while preserving its resolved | ||
| 104 | graph; coalescing file-node fragments alone did not help. Tested histories | ||
| 105 | through 676 transactions opened; 701 and longer failed. This bounds an | ||
| 106 | observed compatibility failure, not a documented format limit. | ||
| 107 | - The writer now appends an independent current-object snapshot before dependency | ||
| 108 | depth would exceed 512, retaining all older revisions. Section snapshots reuse | ||
| 109 | immutable property and attachment declarations; TOC snapshots remap CompactIDs | ||
| 110 | into a complete table. `checkpoint-boundaries-01.log` covers two boundaries in | ||
| 111 | both native section and TOC fixtures, with every historical revision checked. | ||
| 112 | `checkpoint-publication-01.log` checks interrupted text publication at the | ||
| 113 | boundary against complete old/new text and formatting states. | ||
| 114 | - `evidence/m7/local-checkpoint-01`: ten writers and six readers completed | ||
| 115 | 1,500 randomized Unicode replacements, 22,995 checked observations and 11,966 | ||
| 116 | overlapping writer calls. `checkpoint-native-01` cold-opened its saved result; | ||
| 117 | native text exactly matched the independently replayed edit history. | ||
| 118 | - `evidence/m7/edit-text-checkpoint-fuzz-05.log`: 4,558 stateful executions in | ||
| 119 | 302 seconds, including cached snapshots immediately before a full checkpoint, | ||
| 120 | with interrupted publication and character/formatting oracles. No failure. | ||
| 121 | `checkpoint-regression-02.log`, `checkpoint-clippy-01.log`, and | ||
| 122 | `tool-regression-03.log` record passing Rust checks and 22 Python tests. | ||
| 123 | - `evidence/m7/checkpoint-attachment-native-01`: a native section retained its | ||
| 124 | text, images, attachment and formatting through 2,052 scalar edits and repeated | ||
| 125 | snapshots. Fresh-cache reading passed 483 explicit character-format checks; | ||
| 126 | all native page XML matched the earlier reference except page modification | ||
| 127 | time, recorded separately in `retention.json`. | ||
| 128 | - `evidence/m7/checkpoint-toc-native-01`: the native two-page notebook reopened | ||
| 129 | after 1,026 TOC edits crossing two dependency checkpoints. The generator and | ||
| 130 | input/output hashes are retained in `evidence/m7/checkpoint-fixtures`. | ||
| 131 | - `checkpoint-toc-native-02` repeated the TOC check with native PDFs: two pages, | ||
| 132 | two tags and 3,562 explicit character-format comparisons passed. PDF geometry | ||
| 133 | verified the existing black-highlight case that native XML omits. | ||
| 134 | - `native-concurrency-05` established 633 clock-bounded native/Rust call overlaps | ||
| 135 | under background synchronization. Its original oracle incorrectly treated | ||
| 136 | transaction numbers as permanent across native renumbering. Replaying intent | ||
| 137 | content validated all 800 Rust commits and 6,370 reads through one counter | ||
| 138 | decrease. The run stopped before explicit native convergence; after cleanup, | ||
| 139 | one native paragraph lacked its final three cache-acknowledged edits. It is | ||
| 140 | not an acceptance pass. The content-history oracle now rejects branches, | ||
| 141 | missing acknowledgements, partial observations and reads inconsistent with | ||
| 142 | recorded real-time bounds; its regressions are in `test_native_stress.py`. | ||
| 143 | - `native-concurrency-06`: three native writers, four Rust writers and three | ||
| 144 | Rust readers passed the overlapping background-sync gate. All 600 native edits | ||
| 145 | and 800 Rust commits converged in every native client and the shared file; | ||
| 146 | 7,252 Rust reads matched the intent history, with 609 conservatively | ||
| 147 | clock-bounded native/Rust call overlaps. A separate cold-cache capture follows | ||
| 148 | in `native-concurrency-cold-06`. | ||
| 149 | - `random-edit-smb-01.json` and `random-edit-smb-01.one`: the CLI created a | ||
| 150 | separate edited file on Samba, with byte equality verified directly at the | ||
| 151 | server. Examples share the commit adapter's standard-fsync fallback when | ||
| 152 | macOS full-sync is unsupported. Rust commit/edit regressions and clippy passed | ||
| 153 | (`flush-regression-01.log`, `flush-clippy-01.log`). | ||
| 154 | - `native-concurrency-cold-06`: a separate fresh OneNote cache retained all | ||
| 155 | 1,400 intended edits from the successful mixed run. Native comparison passed | ||
| 156 | 54,330 explicit character-format checks, and exact paragraph text matched | ||
| 157 | the independently replayed operations. All associated machines were deleted. | ||
| 158 | |||
| 159 | - `native-concurrency-07`: five native writers, eight Rust writers and five Rust | ||
| 160 | readers completed 1,000 native edits and 1,600 Rust commits, with 21,152 checked | ||
| 161 | reads and 296 clock-bounded native/Rust call overlaps. All six paragraphs | ||
| 162 | converged. `native-concurrency-cold-07` independently retained all 2,600 intended | ||
| 163 | edits and passed 95,034 explicit format comparisons. Every owned VM was deleted. | ||
| 164 | - `edit-text-checkpoint-fuzz-06.log`: 10,596 stateful executions in 901 seconds, | ||
| 165 | without failure. `edit-text-insertion-fuzz-07.log`: 4,639 executions in 301 | ||
| 166 | seconds including legacy and empty text properties, without failure. | ||
| 167 | - Random edits on the private corpus exposed unsupported legacy Unicode | ||
| 168 | promotion and absent initial text properties. The writer now adds Unicode | ||
| 169 | text while preserving the original legacy property, as MS-ONE 2.2.23 permits. | ||
| 170 | `text-insertion-boundaries-02.log` checks native fixtures, unrelated objects, | ||
| 171 | historical revisions, short writes and interrupted publication. | ||
| 172 | - A title edit previously left navigation caches stale. Text and cached titles | ||
| 173 | now publish in one revision and transaction; `title-atomicity-04.log` and | ||
| 174 | `title-regression-02.log` check rename/clear, history and interrupted short | ||
| 175 | writes. `private-random-native-01/title-verification.log` demonstrates that | ||
| 176 | the strengthened independent verifier rejects the earlier faulty copy. | ||
| 177 | CachedTitleStringFromPage's mandatory empty value applies to nonempty Unicode | ||
| 178 | titles; original legacy titles may retain it. The writer's modified-title | ||
| 179 | checks enforce that condition without rejecting untouched legacy content. | ||
| 180 | - `edit-text-title-fuzz-08.log`: 7,383 executions in 301 seconds, adding a native | ||
| 181 | title fixture and assertions relating title text, metadata and alternate title | ||
| 182 | after every persisted observation. No failure. The later single-title-object | ||
| 183 | admission guard was separately covered by `title-regression-02.log`. | ||
| 184 | - `title-empty-native-01`: a cleared title cold-opened successfully; two pages, | ||
| 185 | two tags and 3,394 explicit format checks passed, with four native-PDF highlight | ||
| 186 | checks. OneNote regenerated its empty page name from body text during opening; | ||
| 187 | this check establishes text/format retention, not stored automatic-title parity. | ||
| 188 | - `private-random-campaign-04`: five seeded edits on each of 26 current pages, | ||
| 189 | 130 total. An independent UTF-16 splice model checks all resolved styles, | ||
| 190 | unrelated nodes, metadata, contexts, historical revisions and payload hashes | ||
| 191 | after every edit. All ten frozen source files remained byte-identical. | ||
| 192 | `private-random-native-02/verification-02.log` records a fresh-cache pass on | ||
| 193 | all 26 pages, 109 tags and 186,325 explicit format comparisons, with five native | ||
| 194 | PDF highlight checks. The capture VM was deleted. | ||
| 195 | - `title-all-targets-01.log` records passing Rust all-target tests; | ||
| 196 | `title-clippy-01.log` has no warnings. `random-native-oracles-01.log` records | ||
| 197 | 30 passing Python tests, including content-chain replacement histories and | ||
| 198 | deliberate title-cache damage. | ||
| 199 | - `private-random-report-04`: the edited copy is rendered with all 45 stored | ||
| 200 | pages and 26 current native references. Browser inspection covered the edited | ||
| 201 | Video page; all 2,357 local links resolve (`link-check.json`). The original | ||
| 202 | source remains untouched. | ||
| 203 | - `native-random-08`: randomized replacements and bold/italic changes exposed | ||
| 204 | a test-driver error. On its second edit, Win7's framework left OneNote's | ||
| 205 | `🦀` entity undecoded, so the script treated entity spelling as text. | ||
| 206 | The independent intent oracle rejected that history. This run is not a pass; | ||
| 207 | all four machines were deleted. `tools/native/text.ps1` now decodes supplementary | ||
| 208 | numeric entities before the framework decoder, preserving escaped literals. | ||
| 209 | `native-text-test-01/failure-artifacts/text-result.json` records six passing | ||
| 210 | Windows decoder regressions. Its unrelated native-capture phase was correctly | ||
| 211 | stopped by the profile-isolation guard because this text-only author script | ||
| 212 | had not initialized a test profile; the VM was deleted. | ||
| 213 | - `automatic-title-native-01`: 18 application-authored automatic-title cases | ||
| 214 | captured trimming, empty paragraphs, formatting, entities and long text. | ||
| 215 | Input and stored metadata are retained with the native page names. Additional | ||
| 216 | Unicode-boundary and outline-order cases are being measured before extending | ||
| 217 | automatic-title updates. | ||
| 218 | - `automatic-title-native-02` captured ten more application-authored cases. | ||
| 219 | Automatic names select an outline by position (top before bottom, then left), | ||
| 220 | use the first text line, and trim whitespace. The 255-UTF-16-unit boundary | ||
| 221 | includes a complete supplementary character when it starts at unit 254, | ||
| 222 | yielding 256 units. Both fixture sets pass native comparison: 19/11 pages and | ||
| 223 | 11,680/5,304 explicit format checks. The first set also establishes OneNote's | ||
| 224 | XML omission of an otherwise empty outline containing only ASCII spaces; | ||
| 225 | the oracle permits that case while still rejecting omitted text, lists or tags. | ||
| 226 | These are measured compatibility rules, not a completed automatic-title writer. | ||
| 227 | - `title-checkpoint-atomicity-02.log` passes interrupted multi-object title | ||
| 228 | publication both on the native fixture and at a full dependency checkpoint, | ||
| 229 | using 17/257-byte short writes respectively. | ||
| 230 | - `native-random-09` replayed seed 912 after the native decoder fix: three | ||
| 231 | original OneNote writers, four Rust writers and three Rust readers completed | ||
| 232 | 450 native replacements with bold/italic changes and 600 Rust Unicode | ||
| 233 | replacements. All four paragraphs converged; 8,564 reads matched the | ||
| 234 | independently chained replacement intents, with 260 clock-bounded native/Rust | ||
| 235 | call overlaps. The final native formatting matched each actor's last recorded | ||
| 236 | intent. All three Windows VMs and the Linux server were deleted. | ||
| 237 | - `native-random-cold-09`: a separate fresh cache retained the exact final | ||
| 238 | text produced by all 1,050 replacement intents. Native comparison passed | ||
| 239 | 1,974 explicit format checks; an independent replay also checked 456 character | ||
| 240 | bold/italic values against the recorded native editing intent. The cold VM | ||
| 241 | was deleted. | ||
| 242 | - `random-noop-01` records seed 301 replacing ` café ` with itself and producing | ||
| 243 | identical output. The CLI now turns an identical replacement into an insertion. | ||
| 244 | `test_random_edit_campaign.py` verifies both new-file and in-place operation; | ||
| 245 | `tool-regression-06.log` records 31 passing Python tests and | ||
| 246 | `final-clippy-04.log` is clean. | ||
| 247 | |||
| 248 | - `automatic-title-native-03`: twelve additional native cases distinguish | ||
| 249 | explicit titles (full length, leading whitespace removed, trailing whitespace | ||
| 250 | retained, first line) from automatic body summaries (trimmed and bounded). | ||
| 251 | Empty lines/paragraphs/outlines fall through to later text; table cells supply | ||
| 252 | automatic titles. All 13 captured pages pass 9,744 native format comparisons. | ||
| 253 | - The writer now publishes automatic navigation metadata when body text changes | ||
| 254 | and when an explicit title is cleared, choosing body outlines by position. | ||
| 255 | New sections use the same bounded automatic-title encoding. No public API was | ||
| 256 | added. `automatic-title-regression-03.log` passes edit/writer regressions; | ||
| 257 | `automatic-title-edit-tests-01.log` covers native line/UTF-16 boundaries and | ||
| 258 | multi-object interrupted publication. Historical objects and all non-title | ||
| 259 | metadata fields remain checked independently. | ||
| 260 | - `automatic-title-edits-01` applies 26 independently checked edits to the 13-page | ||
| 261 | native fixture. `automatic-title-edits-native-01` then passes fresh-cache text, | ||
| 262 | cached navigation title and 6,992 character-format comparisons. Its VM was | ||
| 263 | deleted. `title-empty-native-02` independently confirms the corrected cleared | ||
| 264 | title's navigation label, two pages, two tags and 3,394 format checks, with four | ||
| 265 | PDF black-highlight checks; its VM was deleted. | ||
| 266 | - `edit-text-automatic-title-fuzz-09.log`: 6,550 stateful executions in 301 seconds | ||
| 267 | after automatic-title changes, with no failure. `automatic-title-all-targets-01.log` | ||
| 268 | passes all Rust targets; `automatic-title-clippy-01.log` has no warnings. | ||
| 269 | - `private-random-campaign-05` passes another 130 edits against the whole-document | ||
| 270 | and payload-preservation oracle. All ten frozen source files remained unchanged. | ||
| 271 | - `private-random-native-03` cold-opens that edited copy: 26 pages, 109 tags, | ||
| 272 | 186,325 format checks, five PDF highlight checks, and cached navigation titles | ||
| 273 | pass. Its VM was deleted. `private-random-report-05` contains the edited pages | ||
| 274 | and fresh native references; all 2,357 local links in 46 HTML files resolve. | ||
| 275 | - `automatic-title-native-04/05` establish RTL outline ordering by descending | ||
| 276 | x anchor (width does not affect it), reversed RTL table-cell order, skipped | ||
| 277 | attachments, and whitespace trimming after UTF-16 truncation. The two public | ||
| 278 | fixtures in `corpus/m7/automatic-titles` retain native provenance. Ten direct | ||
| 279 | body-edit regressions cover the RTL/attachment selection rules. | ||
| 280 | - `automatic-title-edits-02/03` apply 21/18 independently checked edits to these | ||
| 281 | fixtures. Fresh native captures pass seven/six pages, cached navigation labels, | ||
| 282 | and 2,064/6,288 format checks. Both VMs were deleted. All Rust targets and | ||
| 283 | clippy pass in `automatic-title-all-targets-02.log` and | ||
| 284 | `automatic-title-clippy-03.log`. | ||
| 285 | - `native-random-10` repeats the ten-client replacement workload with seed 913: | ||
| 286 | 450 native edits, 600 Rust commits, 5,595 reads, and 466 clock-bounded native/Rust | ||
| 287 | call overlaps. All four paragraphs converge. `native-random-cold-10` passes | ||
| 288 | 1,740 format comparisons; intent replay independently checks all 1,050 edits | ||
| 289 | and 392 native bold/italic character values. All associated VMs were deleted. | ||
| 290 | - `tools/native_stress.py RUN CAPTURE/read` now replays saved editing intents | ||
| 291 | directly against fresh native XML, including exact paragraph multiplicity and | ||
| 292 | native formatting intent. `tool-regression-08.log` records 33 passing tests, | ||
| 293 | including rejection of missing/extra content, incorrect formatting, lost edits | ||
| 294 | and split surrogate pairs. | ||
| 295 | - `recovery-01` repeats all seven shared-notebook scenarios after the writer | ||
| 296 | changes: disjoint edits, reachable competing edits, server restart, transport | ||
| 297 | reconnect, lock contention, application restart and VM restart. Both Windows | ||
| 298 | clients and the Linux server were deleted. `recovery-cold-01` opens the final | ||
| 299 | saved notebook in a new cache and passes text, navigation and 641 character | ||
| 300 | format checks; its VM was deleted. | ||
| 301 | - The conflict oracle follows current page-manifest references only. | ||
| 302 | `conflict-oracle-regressions-01.log` checks the native offline-edit fixture and | ||
| 303 | rejects treating unreferenced history or detached object spaces as recovery. | ||
| 304 | `tool-regression-09.log` records 34 passing Python tests. | ||
| 305 | - `edit-text-rtl-fuzz-10.log` adds the native RTL title and non-title body | ||
| 306 | candidates to the stateful interrupted-edit workload: 4,531 executions in | ||
| 307 | 301 seconds, no failure. | ||
| 308 | - `mixed-conflict-01` FAILS acceptance. Three native clients edited the same | ||
| 309 | paragraph offline while three Rust writers committed 15 appends and two Rust | ||
| 310 | readers observed them. After reconnection, all three native alternatives were | ||
| 311 | reachable, but the final Rust text was absent from the current page and its | ||
| 312 | conflict pages. The pre-reconnect notebook and every convergence snapshot are | ||
| 313 | retained. All three Windows VMs and the Linux server were deleted. Rust edits | ||
| 314 | changed the text object's modification time while ancestor paragraph/outline/ | ||
| 315 | page times stayed unchanged; native edits changed those ancestors too. That | ||
| 316 | difference is a hypothesis for investigation, not an established cause. | ||
| 317 | - `mixed-conflict-02` reduces the workload to one native client, two Rust writers, | ||
| 318 | one reader and two Rust commits. Both competing results remain reachable; this | ||
| 319 | reduction does not reproduce the failure. Its Windows and Linux VMs were deleted. | ||
| 320 | - `mixed-conflict-cold-01` independently opens the failed final notebook in a | ||
| 321 | fresh cache. The native conflict UI shows only the three native alternatives; | ||
| 322 | clipboard captures recover the exact Unicode text of both conflict pages. | ||
| 323 | The Rust result is also absent from all exported stored revision objects in | ||
| 324 | the last convergence snapshot (`mixed-conflict-01/lost-edit.json`). Native | ||
| 325 | comparison passes 200 format checks on the surviving main page; that reader | ||
| 326 | agreement does not validate retention of the missing edits. The VM was deleted. | ||
| 327 | - `mixed-conflict-03` tests ancestor modification timestamps with the original | ||
| 328 | three-native/three-Rust-writer workload, using the separate | ||
| 329 | `evidence/m7/ancestor-timestamp-probe.py`. The first reconnect capture referenced | ||
| 330 | three conflict spaces without default revisions, so the strict reader stopped | ||
| 331 | the run. After client teardown, two references remained unresolved. All VMs | ||
| 332 | were deleted. `mixed-conflict-cold-03` nevertheless opens the saved file in a | ||
| 333 | fresh OneNote cache with the full Rust text; its VM was deleted. This is an | ||
| 334 | inconclusive experiment, not acceptance of a timestamp fix. | ||
| 335 | - The native checkpoint observer now preserves each distinct incomplete snapshot | ||
| 336 | and retries missing document contexts for at most two minutes. Other parse | ||
| 337 | errors still fail immediately, and retention still requires every competing | ||
| 338 | result to be reachable. This allows the next experiment to distinguish an | ||
| 339 | intermediate cross-space save from a persistent missing conflict. | ||
| 340 | - `mixed-conflict-04` repeats the timestamp experiment with that observer. One | ||
| 341 | incomplete snapshot resolves, then all three native alternatives and the full | ||
| 342 | 15-commit Rust result are reachable. All Windows and Linux VMs were deleted. | ||
| 343 | - Text edits now publish existing ancestor modification timestamps in the same | ||
| 344 | transaction as text, run boundaries and navigation caches. Preservation tests | ||
| 345 | independently follow raw object references and compare every unrelated field; | ||
| 346 | interrupted title/checkpoint publication also checks all modification times. | ||
| 347 | `ancestor-edit-regressions-04.log`, `ancestor-all-targets-01.log`, | ||
| 348 | `ancestor-clippy-01.log` and `ancestor-tool-regressions-01.log` pass. | ||
| 349 | Native acceptance of the atomic implementation is recorded below, separately | ||
| 350 | from the timestamp experiment. | ||
| 351 | - `mixed-conflict-05` stopped before any Rust commit: disconnecting native NICs | ||
| 352 | stranded an existing SMB handle and correctly excluded Rust. The setup now | ||
| 353 | holds the file's exclusive lock while disconnecting clients. Rust readers | ||
| 354 | back off during contention. All owned VMs were deleted; the final stopped | ||
| 355 | clone's removal is recorded in `cleanup-confirmed.json`. | ||
| 356 | - `private-random-campaign-06` applies 130 edits with seed 918 and verifies all | ||
| 357 | ten frozen source files unchanged. Fresh native capture | ||
| 358 | `private-random-native-04/verification-02.log` passes 26 pages, 109 tags, | ||
| 359 | 188,956 character-format comparisons and five black-highlight paragraphs. | ||
| 360 | Its PDF maps a rendered combining-accent glyph to a space. The PDF oracle | ||
| 361 | checks the uniquely located paragraph and the unhighlighted glyph's inline | ||
| 362 | region; missing ordinary text, ambiguous matches and black rectangles in that | ||
| 363 | region fail. `tool-regression-14.log` passes all 35 Python tests. | ||
| 364 | `private-random-report-06` associates all 26 current pages with the fresh | ||
| 365 | native references; all 2,357 local links resolve. Its VM was deleted. | ||
| 366 | - `edit-text-ancestor-fuzz-11.log` exercises stateful edits and interrupted | ||
| 367 | publication after ancestor timestamp propagation: 4,236 runs in 301 seconds, | ||
| 368 | no failure. | ||
| 369 | - `mixed-conflict-06` passes with the atomic implementation: three native | ||
| 370 | alternatives and the full 15-commit Rust result remain reachable after all | ||
| 371 | three native clients close. A transient incomplete save resolves before | ||
| 372 | acceptance. All Windows and Linux VMs were deleted. `mixed-conflict-cold-06` | ||
| 373 | opens the closed result in a fresh cache; native conflict UI clipboard | ||
| 374 | captures independently match all four intended results exactly. That VM was | ||
| 375 | deleted too. The ordinary page comparison passes separately; its zero | ||
| 376 | explicit format checks do not substitute for the four intent comparisons. | ||
| 377 | - The native clone cleanup now handles the VM helper's `SystemExit` on shutdown | ||
| 378 | timeout, terminates the owned VM and deletes its overlay. The regression | ||
| 379 | checks deletion and the teardown record; `tool-regression-17.log` passes all | ||
| 380 | 36 Python tests, and the private native comparison still passes. | ||
| 381 | - `native-random-11` never reached notebook editing: YAML parsed its all-digit | ||
| 382 | WAN MAC address as a sexagesimal integer, so cloud-init could not configure | ||
| 383 | the interface. The failed boot log is preserved and its VM was deleted. | ||
| 384 | Quoting both MAC addresses fixes the seed. `linux-lab-regression-01.log` | ||
| 385 | passes the two lifecycle tests; `native-random-12` reuses the same VM name | ||
| 386 | and deterministic MAC after deletion and reaches ready Windows clients. | ||
| 387 | - `native-random-12` passes seed 918 with the ancestor fix: three native | ||
| 388 | writers, four Rust writers and three Rust readers; 450 native edits, 600 | ||
| 389 | Rust commits, 2,471 checked reads and 428 clock-bounded overlapping native/Rust | ||
| 390 | calls. All four intended paragraphs converge and native formatting matches | ||
| 391 | the recorded operations. All task-owned Windows and Linux VMs were deleted. | ||
| 392 | - `native-random-cold-12` independently reopens that result and compares every | ||
| 393 | recorded native/Rust edit with the native XML. Its VM was deleted. | ||
| 394 | - `same-second-build` is an isolated controlled-clock build with one recorded | ||
| 395 | source substitution: a text edit uses its baseline element timestamp. | ||
| 396 | `same-second-equivalence/result.json` confirms that its resolved revision | ||
| 397 | exactly matches a production-library edit completed in the same actual | ||
| 398 | second. The production clock and library are unchanged. `mixed-conflict-07` | ||
| 399 | uses that build to test baseline-equal timestamps with three native writers, | ||
| 400 | three Rust writers and two Rust readers. | ||
| 401 | - `mixed-conflict-07` passes: the four page-content element timestamps remain | ||
| 402 | exactly equal to the cached baseline across all 15 Rust commits, and all | ||
| 403 | four competing results survive reconnect and application closure. The three | ||
| 404 | Windows VMs and Linux server were deleted. This distinguishes valid equal | ||
| 405 | timestamps from the inconsistent descendant/ancestor times in the original | ||
| 406 | failing writer; it does not require inventing future timestamps. | ||
| 407 | - `mixed-conflict-cold-07` independently opens the same-second result and | ||
| 408 | captures exact Unicode clipboard text from the main page and all three | ||
| 409 | native conflict pages. All four results match the recorded intent, and the | ||
| 410 | VM was deleted. Conflict retention now counts duplicate text instead of | ||
| 411 | collapsing it into a set: `exact-retention.json` rechecks both successful | ||
| 412 | closed notebooks against the exact four-result multiset. | ||
| 413 | - `native-random-13` passes seed 919 with twelve clients: four native writers, | ||
| 414 | five Rust writers and three Rust readers; 600 native edits, 750 Rust commits, | ||
| 415 | 1,903 checked reads and 456 clock-bounded overlapping native/Rust calls. | ||
| 416 | All five intended paragraphs converge. The harness now captures | ||
| 417 | `stress-closed` after every native client closes; fresh-cache acceptance uses | ||
| 418 | this snapshot instead of the earlier live checkpoint. | ||
| 419 | - `native-random-cold-13` opens that closed snapshot in a fresh native cache: | ||
| 420 | all 1,350 recorded edits match exactly across five paragraphs, with 1,672 | ||
| 421 | explicit character-format comparisons and 390 checks against the native | ||
| 422 | writers' formatting intents. Its VM and all campaign VMs were deleted. | ||
| 423 | - Final cleanup records are `final-teardown-check.json` and | ||
| 424 | `final-source-check.json`: 21 owned Windows clone identities and five Linux | ||
| 425 | VM identities are absent, and all ten frozen private source files retain | ||
| 426 | their hashes. `tool-regression-19.log` passes 36 Python tests. The HTML review | ||
| 427 | is `private-random-report-06/index.html`; its 26 current pages link to the | ||
| 428 | verified native references. Physical power-loss and unsynchronized native | ||
| 429 | cache durability remain outside these guarantees. | ||
MILESTONE8.md deleted-84
| ... | @@ -1,84 +0,0 @@ | ||
| 1 | # Workspace, crash recovery and diagnostic editing | ||
| 2 | |||
| 3 | Goal resumed after the user accepted the workspace split and public API audit. | ||
| 4 | API boundary changes remain authorized when justified by implementation needs. | ||
| 5 | |||
| 6 | ## Acceptance gates | ||
| 7 | |||
| 8 | 1. **Workspace:** move the existing library, tests and examples into | ||
| 9 | `crates/onestore`, preserve the root corpus and example executable paths, | ||
| 10 | and pass Rust, Python and fuzz build checks. A sibling crate can consume | ||
| 11 | the library without reaching into its source directory. | ||
| 12 | 2. **Public API audit:** review exported types, ownership, validation, edit | ||
| 13 | contracts and consumer ergonomics; verify a separate crate can read and edit | ||
| 14 | through public APIs; present [API-AUDIT.md](API-AUDIT.md) before continuing. | ||
| 15 | 3. **Storage interruption:** verify durable images after every write/flush | ||
| 16 | boundary, including dropped unflushed writes, partial persistence, counter | ||
| 17 | rollover and revision checkpoints. Reopen the persisted image, continue | ||
| 18 | editing it, retain reproducible failures, and independently validate a | ||
| 19 | representative image matrix with OneNote. | ||
| 20 | 4. **Abrupt VM stops:** stop disposable Windows clients and the Samba server | ||
| 21 | without guest shutdown during normal concurrent editing. Preserve disks, | ||
| 22 | restart the same machines, compare acknowledged operations and reachable | ||
| 23 | conflicts, and cold-open the recovered server notebook in fresh OneNote. | ||
| 24 | Track native cache acceptance separately from server durability. | ||
| 25 | 5. **Diagnostic editor:** extend the HTML reading report with supported text | ||
| 26 | edits on a task-owned notebook copy. Edits use the Rust library, reject stale | ||
| 27 | snapshots, preserve unrelated content and expose ambiguous commit outcomes | ||
| 28 | without automatic replay. Verify browser interaction, two-reader stale-edit | ||
| 29 | handling, Unicode and native round trips. This is a diagnostic interface; | ||
| 30 | canvas rendering and product UI remain separate work. | ||
| 31 | 6. **Closure:** rerun affected checks, review the implementation for unnecessary | ||
| 32 | state and abstractions, verify source hashes, retain evidence and replay | ||
| 33 | commands, delete owned VMs, and provide the running diagnostic tool. | ||
| 34 | |||
| 35 | Abrupt VM stops discard guest memory while the host remains powered. Simulated | ||
| 36 | storage loss exercises the library's ordered-flush contract; neither establishes | ||
| 37 | the physical drive's behavior during actual host power loss. | ||
| 38 | |||
| 39 | ## Evidence | ||
| 40 | |||
| 41 | Evidence belongs under `evidence/m8/`. Original notebooks are never edited. | ||
| 42 | |||
| 43 | ### Workspace and API review checkpoint | ||
| 44 | |||
| 45 | The workspace gate passed: 56 Rust integration tests (one intentional local | ||
| 46 | fuzz-seed generator ignored), 36 Python tests, all example builds, all eight fuzz | ||
| 47 | target builds, Clippy, formatting, rustdoc and its compiled example. The external | ||
| 48 | API consumer also passes. Logs are listed in [API-AUDIT.md](API-AUDIT.md). | ||
| 49 | |||
| 50 | The user accepted the public API audit and authorized continuation. The checkpoint | ||
| 51 | changed source locations and documented existing contracts, preserving signatures | ||
| 52 | and runtime behavior. No VMs were started for that checkpoint. | ||
| 53 | |||
| 54 | ### Storage interruption campaign | ||
| 55 | |||
| 56 | `power-loss-02` passed 2,508 persisted-image checks and subsequent edits across | ||
| 57 | native Unicode text, 255→256 and 65535→65536 counter rollover, an attachment page, | ||
| 58 | and a pending 512-revision checkpoint. Each write/flush boundary is exercised with | ||
| 59 | six persistence policies, including complete loss of unflushed writes and partial, | ||
| 60 | reordered persistence. Comparisons cover every resolved current object, including | ||
| 61 | raw properties and payloads. Acknowledged publication must select the complete | ||
| 62 | new state; earlier cuts may select only the complete old or new state. | ||
| 63 | |||
| 64 | `power-loss-03` repeats the same checks with exact source and failure-image | ||
| 65 | retention added to the harness. `edit-text-fuzz-01.log` records 4,327 stateful | ||
| 66 | inputs over 302 seconds without failure. The 22 retained boundary images from | ||
| 67 | `power-loss-02` are undergoing fresh OneNote captures in `power-native-01`. | ||
| 68 | |||
| 69 | Replay: | ||
| 70 | |||
| 71 | ```sh | ||
| 72 | cargo run --release -p onestore --example power_loss -- /new/matrix-directory | ||
| 73 | python3 tools/power_loss_native.py /new/matrix-directory /new/native-directory | ||
| 74 | cargo +nightly fuzz run edit_text -- -max_total_time=300 -timeout=60 -max_len=128 | ||
| 75 | ``` | ||
| 76 | |||
| 77 | The TOC extension (`power-toc-01`) passes another 918 persisted images and | ||
| 78 | subsequent color edits, including counter rollover and a pending checkpoint. | ||
| 79 | `power-loss-04` passes the combined 3,426-image matrix with exact source/failure | ||
| 80 | retention after the harness refactor. The native gate uses the 22 retained images | ||
| 81 | from `power-loss-02` and 13 from `power-toc-01`; captures verify those exact source | ||
| 82 | hashes. All 22 section captures in `power-native-01` passed. The TOC captures in | ||
| 83 | `power-toc-native-01` also compare OneNote's notebook color to the persisted value | ||
| 84 | and are still running at this checkpoint. | ||
PROGRESS.md deleted-227
| ... | @@ -1,227 +0,0 @@ | ||
| 1 | # OneNote interoperability | ||
| 2 | |||
| 3 | Target: an embeddable Rust library that reads and writes revision stores, | ||
| 4 | preserves unedited content, and collaborates with OneNote 2010 over SMB. | ||
| 5 | |||
| 6 | | Stage | Acceptance evidence | State | | ||
| 7 | | --- | --- | --- | | ||
| 8 | | 1. Corpus | Native single-operation fixtures, private corpus integrity, independent cold reopen, semantic assertions, reproducible provenance | Passed current corpus gate | | ||
| 9 | | 2. Storage | Committed revision/object resolution, opaque preservation, malformed-input tests and fuzzing against native fixtures | Passed current corpus gate | | ||
| 10 | | 3. Writer | Create and edit through the native open/edit/save/read loop without collateral changes | Passed current corpus gate | | ||
| 11 | | 4. Durability | Injected write/flush failures, interrupted commits, acknowledged persistence, native recovery | Passed implemented scalar-commit gate | | ||
| 12 | | 5. Collaboration | Observed locks and I/O, deliberate contention, competing edits and reconnect convergence | Passed tested scalar collaboration gate | | ||
| 13 | |||
| 14 | The personal source is `/Volumes/clover/Documents/OneNote`. It is read only. | ||
| 15 | `corpus/private/original` holds the copied baseline; `source-manifest.json` records | ||
| 16 | SHA-256 hashes and source modification times. Native automation must operate on | ||
| 17 | disposable copies, with the personal Windows registry and cache parked first. | ||
| 18 | |||
| 19 | Wayback already contained `C:\one-tests\profile-original.reg` and | ||
| 20 | `C:\one-tests\profile-original-cache` at task start. Native tests used an isolated | ||
| 21 | profile with UnfiledNotesSection at `C:\one-tests\Loose.one`. After the stage-5 | ||
| 22 | captures, Restore returned the original registry and cache to their active paths; | ||
| 23 | Status confirmed restoration, with test backups parked and OneNote closed. | ||
| 24 | The primary native run is `corpus/native/20260905-05`, generated by OneNote | ||
| 25 | 14.0.7015.1000. Its nine snapshots were independently opened from fresh caches | ||
| 26 | under `cold-05-*`. Text, styles, outline position, table cells, image bytes, and | ||
| 27 | attachment bytes pass `python3 tools/verify-corpus.py`. Transactions in the | ||
| 28 | synthetic section grow from 4 to 28 across the captured operations. | ||
| 29 | |||
| 30 | `corpus/native-ink` records two native mouse-drawn strokes and their cold-open | ||
| 31 | binary XML payloads. `evidence/stage1/native-ink.png` shows the page. | ||
| 32 | `corpus/native-delete` records a page before deletion and its cold-open recovery | ||
| 33 | from the notebook recycle bin. The AHK ink recipe requires the displayed | ||
| 34 | 1280x720 maximized OneNote layout; semantic XML verifies that strokes survived. | ||
| 35 | |||
| 36 | The private corpus has 26 pages including two recycle-bin pages. A fresh read | ||
| 37 | can return partial page content while loading; the reader now checks the final | ||
| 38 | hierarchy against all captured page IDs. `corpus/private/exact-native` passed an | ||
| 39 | independent cold read with 26 expected pages. It includes images, ink, tags, | ||
| 40 | lists, OCR, and media. All ten original source hashes and modification times | ||
| 41 | were rechecked unchanged after native testing. COM IDs change across cache | ||
| 42 | resets and must not serve as the file-identity oracle. | ||
| 43 | |||
| 44 | `corpus/native-encrypted` records native password protection and a fresh-cache | ||
| 45 | unlock with password `fictitious-only`. The cold XML and attachment bytes pass | ||
| 46 | the corpus verifier. OneNote's UI introduced one leading empty paragraph; | ||
| 47 | that paragraph is retained in the fixture oracle. `read.ps1 -UseCurrentCache` | ||
| 48 | captures the manually unlocked test session; the caller closes its UI afterward. | ||
| 49 | |||
| 50 | The corpus is a starting interoperability gate, not a complete feature matrix. | ||
| 51 | Shared-file conflicts now have native evidence in the stage-5 corpus; ordinary | ||
| 52 | user-visible page version history is outside this corpus gate. | ||
| 53 | |||
| 54 | The Rust reader follows committed transactions, resolves revision dependencies, | ||
| 55 | global IDs, roles, contexts, roots, reference-count overrides, and file payloads. | ||
| 56 | Checks cover object and object-space cycles, missing targets, immutable data, | ||
| 57 | MD5 hashes, override CRCs, and reference counts. All revisions in the ten private | ||
| 58 | files pass the current unencrypted checks. The encrypted fixture retains opaque | ||
| 59 | ciphertext and explicitly refuses property traversal. | ||
| 60 | |||
| 61 | Native compatibility findings: | ||
| 62 | - Transaction CRCs accumulate across prior sentinel entries and fragment links | ||
| 63 | are excluded. Native transaction fragments can leave four bytes after the link. | ||
| 64 | - Reference counts include repeated references from the same source object. | ||
| 65 | - Table-of-contents override CRCs include preceding object declaration counts, | ||
| 66 | although that format does not use object groups. | ||
| 67 | - Encryption-key containers can have zero padding after the footer when their | ||
| 68 | compressed chunk references round the size to eight bytes. | ||
| 69 | |||
| 70 | Coverage-guided runs recorded under `evidence/stage2` completed 5,529,107 storage | ||
| 71 | inputs, 6,537,029 property inputs, and 2,998,472 revision inputs without crashes. | ||
| 72 | These are bounded initial runs, not durability or interoperability proofs. | ||
| 73 | An independent 100,000-level property nesting test checks parse and drop safety. | ||
| 74 | The extended revision run completed 20,718,981 inputs in 601 seconds with a | ||
| 75 | 262,144-byte input limit and no crashes. `tools/verify-reader.py` independently | ||
| 76 | matches 26 private pages and 581 nonempty text objects by page, three hyperlink | ||
| 77 | targets, 18 image payloads byte for byte, and three native GIF-to-PNG conversions | ||
| 78 | pixel for pixel. It requires Pillow. Whitespace-only paragraphs and boilerplate | ||
| 79 | date/time text remain in the raw graph but are outside this text comparison. | ||
| 80 | One page-like object space has undocumented metadata JCID 0x0002003E; it remains | ||
| 81 | in the raw graph and is excluded from the ordinary-page inventory. | ||
| 82 | |||
| 83 | Specs in `resources/md` are the implementation references. File-format conformance, | ||
| 84 | actual OneNote acceptance, and SMB durability are separate checks. | ||
| 85 | |||
| 86 | The first stage-3 encoding of `replace_property_bytes` appended new chunks, | ||
| 87 | replaced the affected list path with fresh list identities, copied committed | ||
| 88 | transaction entries, and published new header references. Every original byte | ||
| 89 | outside the header remained unchanged. The stage-4 append protocol below | ||
| 90 | superseded that encoding; the stage-3 fixtures retain its native acceptance. | ||
| 91 | |||
| 92 | `corpus/writer` records the first native cycle: Rust replaces the plain-text | ||
| 93 | fixture with `Portable plain text...`; OneNote reads that text from a fresh cache; | ||
| 94 | OneNote changes it back to `Fictitious plain text.` and saves; Rust reads and edits | ||
| 95 | that file again; a second fresh native read returns `Portable plain text...`. | ||
| 96 | The native test uses equal-length text. Storage tests also replace the scalar | ||
| 97 | with lengths 0 through 40 and re-resolve every prior revision unchanged. | ||
| 98 | |||
| 99 | Stage 3 now includes template-free `create_section` and `create_table_of_contents`. | ||
| 100 | The first creates one page with one plain-text paragraph and an explicit author; | ||
| 101 | the second records ordered section filenames and their file identities. Native | ||
| 102 | fresh-cache reads verify Unicode including a surrogate pair, both newly created | ||
| 103 | notebook files, native editing/saving of the created section, and a subsequent | ||
| 104 | longer Rust text replacement. `replace_property_bytes` also edits `.onetoc2` | ||
| 105 | scalars; OneNote independently confirms the requested notebook color. | ||
| 106 | `tools/verify-writer.py` verifies the captured native semantics and artifact hashes. | ||
| 107 | |||
| 108 | The writer preserves unknown data and every prior revision. Native XML comparison | ||
| 109 | of the ink fixture retains formatting, positions, tables, image data, two ink | ||
| 110 | strokes, and attachment bytes; the changed outline height is allowed to reflow. | ||
| 111 | Creation initially exposed two malformed sequences: an object group needs its | ||
| 112 | following dependency-override node, and a root-space selector must follow that | ||
| 113 | space's declaration. The reader now rejects both. An isolated byte-order probe | ||
| 114 | changed a rejected empty section into a native-readable one. Failed creation | ||
| 115 | experiments live under evidence rather than the accepted writer corpus. | ||
| 116 | |||
| 117 | The native capture script now filters exact `.one` extensions: Windows wildcard | ||
| 118 | matching had also matched `.onetoc2` and created an extra empty section. Old | ||
| 119 | captures retain that provenance; newer reads verify one section and one page. | ||
| 120 | Writer fuzzing completed 2,386,509 scalar inputs and 2,900,483 template-free | ||
| 121 | section inputs without crashes. A subsequent combined section/TOC run covers | ||
| 122 | variable section counts and reference graphs (statistics in evidence/stage3). | ||
| 123 | The corpus gates describe these tested operations, not every MS-ONE feature. | ||
| 124 | |||
| 125 | Stage 4 begins with a concrete failure: directly copying the standalone writer's | ||
| 126 | header over a live file leaves 82 of 1025 prefix-tear states invalid on the small | ||
| 127 | native text fixture. `examples/header_faults.rs` reproduces this. Standalone | ||
| 128 | serialization must not be mistaken for an in-place commit protocol. | ||
| 129 | |||
| 130 | The stage-4 writer now extends the existing revision-manifest list and transaction | ||
| 131 | log instead of replacing their header pointers. Original committed nodes, property | ||
| 132 | blobs, and prior revisions remain unchanged; unused fragment tails and log capacity | ||
| 133 | are populated. Restoring the previous header resolves the previous graph. This | ||
| 134 | encoding now passes independent native reads; the earlier stage-3 evidence used | ||
| 135 | the standalone encoding. | ||
| 136 | |||
| 137 | `commit_property_bytes` accepts a storage implementation through `CommitIo`. | ||
| 138 | Its caller must supply OneNote-compatible exclusion and durable ordered flushes. | ||
| 139 | A locked snapshot comparison precedes writes. Data and metadata flush before a | ||
| 140 | one-byte publication write. At counter-byte rollover the log includes empty | ||
| 141 | transactions up to the largest intermediate count; a flush of the highest changed | ||
| 142 | counter byte precedes cleanup of lower bytes. An interrupted cleanup still resolves | ||
| 143 | the new revision. Failures distinguish NotCommitted, Unknown, and Committed. | ||
| 144 | The in-memory crash model exercises short reads/writes, every I/O failure point, | ||
| 145 | and arbitrary subsets of unflushed byte changes for ordinary commits and 255→256. | ||
| 146 | It passes, including log-fragment rollover; this is not yet a native crash test. | ||
| 147 | Updated append scalar fuzzing completed 465,817 inputs in 181 seconds, no crashes. | ||
| 148 | The combined section/TOC creation fuzz run completed 350,293 inputs in 181 seconds. | ||
| 149 | |||
| 150 | The Windows laptop became reachable again with no Procmon or OneNote process; | ||
| 151 | the user confirmed its battery had run out. The personal profile and cache | ||
| 152 | remained isolated. Procmon 4.1 produced no trace, and | ||
| 153 | the native agent is not elevated. Shared testing continued without that driver. | ||
| 154 | `corpus/append/round-01` records independent cold native reads of plain and complex | ||
| 155 | appended edits, TOC color, counter 255→256 and 65535→65536, their interrupted | ||
| 156 | cleanup states, and all eight combinations of the tested numeric metadata tears. | ||
| 157 | OneNote reads the expected old/new text, preserves the complex fixture's full XML | ||
| 158 | and attachment bytes, and can edit/save after an interrupted counter cleanup. | ||
| 159 | The manifest records artifact hashes; `tools/verify-writer.py` checks the captures. | ||
| 160 | |||
| 161 | The crash model now includes read failures and has a shared implementation used by | ||
| 162 | `fuzz/fuzz_targets/commit.rs`. Stateful fuzzing completed 21,506 inputs in 302 | ||
| 163 | seconds with no crash, retrying from partially persisted files at ordinary and | ||
| 164 | 255→256 commits. Full Rust tests, clippy, original corpus integrity, the private | ||
| 165 | reader oracle, and native writer comparisons pass. | ||
| 166 | |||
| 167 | `commit_file_property` acquires a whole-file filesystem lock and serializes calls | ||
| 168 | inside the process because macOS SMB flock is reentrant within a process. POSIX | ||
| 169 | byte-range locking returned ENOTSUP on this mount. Flock produced a server-visible | ||
| 170 | exclusive 0+UINT64_MAX lock, blocked a Windows read, and rejected a second Mac | ||
| 171 | process's lock. Rust sync_all uses F_FULLFSYNC on macOS; when that extension is | ||
| 172 | unsupported, the adapter uses standard fsync. Apple's SMB source routes it through | ||
| 173 | the ordinary SMB flush path. A real committed edit through the mount passed a | ||
| 174 | subsequent independent cold OneNote read. This depends on the filesystem/server | ||
| 175 | honoring flush and exclusion; it does not claim physical server power-loss testing. | ||
| 176 | Native shared activity exposed read locks at 0xFFFFFFFB and write locks at | ||
| 177 | 0xFFFFFFFD in `evidence/stage4/shared-native-locks-02.log`. | ||
| 178 | |||
| 179 | Stage 5 acceptance is captured in `corpus/collaboration/round-01`, with artifact | ||
| 180 | hashes and `tools/verify-collaboration.py`. Each final notebook was independently | ||
| 181 | reopened from a fresh native cache on OneNote 2010 14.0.7015.1000. | ||
| 182 | |||
| 183 | | Scenario | Observed result | | ||
| 184 | | --- | --- | | ||
| 185 | | Whole-file lock contention | The native edit/sync calls completed while the file remained byte-identical under the lock; the saved edit appeared after release and survived cold reopen | | ||
| 186 | | Different paragraphs | The main page contained both Rust's Unicode edit and the native outline edit | | ||
| 187 | | Same paragraph | Rust text remained on the main page; native competing text survived in a conflict object space | | ||
| 188 | | Native offline edit | With A: disconnected, the server retained the old text; another SMB session committed Rust text; native reconnect preserved both versions | | ||
| 189 | | Lost preparation FLUSH reply | `NotCommitted`; native cold read retained the old paragraph | | ||
| 190 | | Lost publication FLUSH reply | `Unknown`; native cold read recovered the new paragraph | | ||
| 191 | | Lost counter-cleanup FLUSH reply | `Committed`; native cold read recovered the new paragraph after 255→511→256 publication | | ||
| 192 | |||
| 193 | The dedicated loopback SMB session recorded server-visible whole-file locks, | ||
| 194 | actual write ranges/counter values, and successful FLUSH replies withheld before | ||
| 195 | client delivery. The trace verifies a successful flush of counter 511 before | ||
| 196 | cleanup to 256. macOS revoked the disconnected file handles; the adapter returned | ||
| 197 | errors rather than treating reconnection as acknowledgement. The proxy was stopped | ||
| 198 | and its temporary mount directory removed after capture. A: was restored to its | ||
| 199 | existing `\\zenith.miku-sun.ts.net\agent` mapping; the unrelated Z: mapping and | ||
| 200 | personal macOS mounts were left in place. | ||
| 201 | |||
| 202 | OneNote's COM hierarchy omits conflict pages. The reader resolves their metadata | ||
| 203 | JCID 0x20038 and retained text; the offline fixture additionally includes a native | ||
| 204 | UI screenshot showing the conflicting edit. The full-page COM update harness | ||
| 205 | produced a redundant conflict copy even for different paragraphs. Its first merge | ||
| 206 | also changed one table column from 39.14614105224609 to 38.61000061035156 points | ||
| 207 | and one ink z-order from 7 to 6. The verifier binds those exact changes and checks | ||
| 208 | retained image/ink payloads, table contents/styles, attachment bytes, quick styles, | ||
| 209 | and outline positions; it does not claim pixel-identical native layout. | ||
| 210 | |||
| 211 | An unlocked snapshot caught during native saving referenced an object space that | ||
| 212 | had not yet been written. It is retained as `corpus/malformed/native-inflight.one`. | ||
| 213 | `read_file` now shares the commit exclusion, and writer preflight validates the | ||
| 214 | whole current graph. A regression verifies that even a no-op request against this | ||
| 215 | intermediate graph fails before any storage I/O. Diagnostic inventory and edit | ||
| 216 | selection also validate the graph before traversing it. | ||
| 217 | |||
| 218 | Final coverage-guided revision fuzzing, seeded with native collaboration and | ||
| 219 | transport-recovery files, completed 1,453,528 inputs in 122 seconds without a crash | ||
| 220 | (608 MB peak RSS). The final stateful commit run completed 4,992 inputs in 91 | ||
| 221 | seconds without a crash (483 MB peak RSS), following the earlier 21,506-input run. | ||
| 222 | All Rust targets, clippy, the corpus/private reader oracles, writer comparisons, | ||
| 223 | and collaboration verification pass. The ten personal source files still match | ||
| 224 | their original hashes, sizes, and modification times. This completes the bounded | ||
| 225 | stage-5 scalar collaboration gate on the tested Windows/macOS/Samba combination; | ||
| 226 | physical power-loss durability and other client/server implementations are not | ||
| 227 | established by these protocol-failure captures. | ||
README.md deleted-208
| ... | @@ -1,208 +0,0 @@ | ||
| 1 | # onestore | ||
| 2 | |||
| 3 | An experimental native Rust library for OneNote revision stores (`.one` and | ||
| 4 | `.onetoc2`). It reads committed object graphs, creates a small notebook without | ||
| 5 | a template, appends scalar-property and text edits with a recoverable commit protocol, | ||
| 6 | and interprets MS-ONE document structure, formatting, media, and historical pages. | ||
| 7 | The storage gates are recorded in [PROGRESS.md](PROGRESS.md); document-model | ||
| 8 | verification is recorded in [M6-ACCEPTANCE.md](M6-ACCEPTANCE.md). Concurrent-editing | ||
| 9 | verification is recorded in [MILESTONE7.md](MILESTONE7.md). | ||
| 10 | |||
| 11 | Text edits publish ancestor modification timestamps with the changed content. | ||
| 12 | Omitting those timestamps caused acknowledged edits to disappear during native | ||
| 13 | conflict merging. The repaired eight-client replay retains all four competing | ||
| 14 | results after reconnection, application closure and fresh-cache native inspection; | ||
| 15 | see `mixed-conflict-06` in the milestone evidence. Use disposable copies for | ||
| 16 | notebook editing. | ||
| 17 | |||
| 18 | ## Workspace | ||
| 19 | |||
| 20 | `crates/onestore` contains the library, examples and integration tests. New Rust | ||
| 21 | prototypes belong in sibling directories under `crates/` and depend on | ||
| 22 | `onestore = { path = "../onestore" }`. The root manifest discovers these crates. | ||
| 23 | The consumer boundary and API tradeoffs are recorded in [API-AUDIT.md](API-AUDIT.md). | ||
| 24 | Shared native fixtures, specifications, evidence and Python/VM tools stay at the | ||
| 25 | repository root; `fuzz/` remains an independent cargo-fuzz workspace. | ||
| 26 | |||
| 27 | Run Cargo commands from the root. Select `-p onestore` when working only on the | ||
| 28 | library, or `--workspace` for checks across all crates. Example binary paths | ||
| 29 | remain `target/debug/examples/…` for the native verification tools. | ||
| 30 | |||
| 31 | ## Supported surface | ||
| 32 | |||
| 33 | | API | Contract | | ||
| 34 | | --- | --- | | ||
| 35 | | `Store`, `RevisionIndex`, `ResolvedRevision` | Parse storage, resolve revisions and reference graphs, expose roots and objects | | ||
| 36 | | `PropertySets`, `Object::references` | Decode properties and ID streams while retaining raw values | | ||
| 37 | | `Object::file_reference`, `Store::file_data` | Identify internal/external payloads and read internal payload bytes | | ||
| 38 | | `document::Document`, `Revision::text_runs` | Interpret document objects and inherited text formatting while retaining unknown properties and revision identities | | ||
| 39 | | `create_section` | Create one page containing one plain-text paragraph and an author, including Unicode | | ||
| 40 | | `create_table_of_contents` | Create ordered section entries from filenames and file identities | | ||
| 41 | | `replace_property_bytes` | Append one scalar-property revision; preserve prior revisions and unrelated property values and references | | ||
| 42 | | `replace_text`, `commit_text`, `commit_file_text` | Replace a UTF-16 range within one ordinary text run; publish text, run boundaries and modification time together | | ||
| 43 | | `read_file` | Read a snapshot under whole-file exclusion | | ||
| 44 | | `commit_file_property` | Lock, compare the source snapshot, append and flush, then publish the revision | | ||
| 45 | | `CommitIo`, `commit_property_bytes` | Supply another storage backend with equivalent exclusion and ordered durability | | ||
| 46 | |||
| 47 | Scalar edits accept encoded values and require the caller to maintain MS-ONE | ||
| 48 | semantics. Text edits maintain run boundaries, inherit the insertion run's formatting, | ||
| 49 | and promote legacy text to Unicode when needed. Explicit and body-derived | ||
| 50 | navigation titles update in the same transaction; unsupported fields, | ||
| 51 | protected objects and split surrogate pairs are | ||
| 52 | rejected before writing. Appended snapshots cap revision dependency depth at 512 | ||
| 53 | while retaining historical revisions. TOC snapshots can remap encoded CompactIDs | ||
| 54 | without changing their resolved references. Password-protected | ||
| 55 | sections retain their encrypted structure and payloads; the library does not | ||
| 56 | derive password keys or decrypt their pages. | ||
| 57 | External `.onebin` references identify payloads for the caller to obtain. Cloud | ||
| 58 | FSSHTTP synchronization and a C ABI are outside the implemented surface. | ||
| 59 | |||
| 60 | ## Try it | ||
| 61 | |||
| 62 | Requires Rust 1.97 or later for the verified build. Examples create new destinations | ||
| 63 | and refuse to overwrite them. The Python report requires Pillow. | ||
| 64 | |||
| 65 | ```sh | ||
| 66 | cargo run --example create_notebook -- /tmp/one-demo 'Hello from Rust.' 'Example Author' | ||
| 67 | cargo run --example inventory -- /tmp/one-demo/synthetic.one | ||
| 68 | cargo run --example inspect -- /tmp/one-demo/synthetic.one | ||
| 69 | cargo run --example document -- /tmp/one-demo/synthetic.one /tmp/one-model | ||
| 70 | python3 tools/notebook_report.py /tmp/one-demo /tmp/one-report --timezone America/Los_Angeles | ||
| 71 | ``` | ||
| 72 | |||
| 73 | A seeded text edit on a disposable copy records its page, UTF-16 range, | ||
| 74 | replacement and outcome as JSON: | ||
| 75 | |||
| 76 | ```sh | ||
| 77 | cargo run --example random_edit -- /tmp/one-demo/synthetic.one /tmp/edited.one 42 | ||
| 78 | ``` | ||
| 79 | |||
| 80 | The report contains readable pages, document JSON, assets, source identities and | ||
| 81 | coordinates. It preserves paragraph nesting, lists, tables, links and tags. | ||
| 82 | Historical contexts, recycle-bin pages and default templates are represented | ||
| 83 | separately. Native ink and structured equations retain their source data and | ||
| 84 | appear explicitly as opaque content. The report is a reading view; its native | ||
| 85 | PDF references supply the original canvas layout. | ||
| 86 | |||
| 87 | Read and validate a snapshot before interpreting its graph: | ||
| 88 | |||
| 89 | ```rust,no_run | ||
| 90 | use onestore::{read_file, RevisionIndex, Store}; | ||
| 91 | |||
| 92 | fn main() -> Result<(), Box<dyn std::error::Error>> { | ||
| 93 | let bytes = read_file("notebook/synthetic.one")?; | ||
| 94 | let store = Store::parse(&bytes)?; | ||
| 95 | if !store.checksum_mismatches.is_empty() { | ||
| 96 | return Err("Transaction checksum damage".into()); | ||
| 97 | } | ||
| 98 | let index = RevisionIndex::parse(&store)?; | ||
| 99 | index.validate_current()?; | ||
| 100 | Ok(()) | ||
| 101 | } | ||
| 102 | ``` | ||
| 103 | |||
| 104 | `Store::parse` exposes checksum mismatches for diagnostic readers; the writer | ||
| 105 | rejects them. The resolved graph borrows the snapshot. Select the object-space ID, | ||
| 106 | object ID, and property from that graph, then pass those IDs, the same snapshot, | ||
| 107 | and the replacement's encoded bytes to `commit_file_property`. The | ||
| 108 | `edit_property` example demonstrates selection by JCID/property/expected bytes, | ||
| 109 | with optional explicit IDs when conflict copies contain identical text. | ||
| 110 | |||
| 111 | ## Commit behavior | ||
| 112 | |||
| 113 | ```text | ||
| 114 | exclusive lock → exact snapshot comparison | ||
| 115 | → append data → flush | ||
| 116 | → prepare header metadata → flush | ||
| 117 | → publish transaction counter → flush | ||
| 118 | → finish counter rollover → flush → unlock | ||
| 119 | ``` | ||
| 120 | |||
| 121 | Stale snapshots fail before writing. Live readers must use equivalent exclusion. | ||
| 122 | Native conflict creation can still expose cross-space references before their | ||
| 123 | targets are saved; such snapshots must be rejected and reread while synchronization | ||
| 124 | proceeds. `read_file` and `commit_file_property` serialize within the process because | ||
| 125 | macOS SMB locks can be reentrant. The lock is nonblocking across processes; | ||
| 126 | contention requires a fresh read and a later retry. | ||
| 127 | |||
| 128 | | Error state | Meaning and caller action | | ||
| 129 | | --- | --- | | ||
| 130 | | `NotCommitted` | This edit was not published. Preparation bytes may exist. Reread before retrying. | | ||
| 131 | | `Unknown` | Publication may have persisted despite the error. Reread and resolve the intended edit before retrying. | | ||
| 132 | | `Committed` | Publication was durably acknowledged; counter cleanup or lock release failed. Reopen the committed result instead of replaying the edit. | | ||
| 133 | |||
| 134 | At counter rollover, empty transactions make intermediate published counts valid. | ||
| 135 | The highest changed byte is flushed before lower bytes are cleaned up. The | ||
| 136 | 255→256 and 65535→65536 boundaries and interrupted cleanup states have independent | ||
| 137 | native acceptance captures. A no-op still flushes; a failed flush has an unknown | ||
| 138 | durability outcome. | ||
| 139 | |||
| 140 | The filesystem adapter uses whole-file locking. On macOS it acquires the lock | ||
| 141 | as part of opening the file (`O_EXLOCK | O_NONBLOCK`): separate open and `flock` | ||
| 142 | calls allowed overlapping exclusive holders and stranded server locks under | ||
| 143 | multi-process SMB contention. `tools/smb_lock_race.py` reproduces that failure | ||
| 144 | without notebook parsing or writing. On the tested macOS SMB mount, | ||
| 145 | POSIX byte-range locks returned `ENOTSUP`; whole-file locks excluded native | ||
| 146 | OneNote's lock ranges. `sync_all` falls back to `fsync` on macOS only when | ||
| 147 | `F_FULLFSYNC` is unsupported. Successful SMB FLUSH replies were observed on the | ||
| 148 | wire. Correctness requires the backend to honor exclusion and ordered flushes. | ||
| 149 | The evidence covers transport failures, not physical server power loss or every | ||
| 150 | filesystem's lock implementation. | ||
| 151 | |||
| 152 | ## Verification | ||
| 153 | |||
| 154 | ```sh | ||
| 155 | cargo test --all-targets | ||
| 156 | cargo clippy --all-targets -- -D warnings | ||
| 157 | python3 tools/verify-corpus.py | ||
| 158 | python3 tools/verify-reader.py # requires Pillow and the local private corpus | ||
| 159 | python3 tools/verify-writer.py | ||
| 160 | python3 tools/verify-collaboration.py | ||
| 161 | python3 tools/verify-document.py /path/to/copied/notebook /path/to/native/read | ||
| 162 | ``` | ||
| 163 | |||
| 164 | The frozen private corpus is excluded from version control. Its verifier checks 26 pages, | ||
| 165 | 581 text objects, hyperlink targets, exact image bytes, and native image conversions. | ||
| 166 | Synthetic corpus manifests bind binary fixtures to independent native XML and | ||
| 167 | attachment captures. `verify-corpus.py` also requires that private corpus. | ||
| 168 | |||
| 169 | Storage tests exercise malformed references, deep properties, historical revision | ||
| 170 | preservation, short I/O, stale snapshots, counter tears, and every injected I/O | ||
| 171 | failure point at ordinary and rollover commits. The crash model persists arbitrary | ||
| 172 | subsets of unflushed bytes and is shared with the stateful commit fuzzer. | ||
| 173 | |||
| 174 | ```sh | ||
| 175 | cargo +nightly fuzz run revisions -- -max_total_time=120 -max_len=262144 -rss_limit_mb=2048 | ||
| 176 | cargo +nightly fuzz run commit -- -max_total_time=300 -max_len=4096 -rss_limit_mb=2048 | ||
| 177 | ``` | ||
| 178 | |||
| 179 | Fuzz targets cover storage, properties, revisions, scalar edits, creation, and | ||
| 180 | multi-edit interrupted commits. Seed the revision target with native `.one` files | ||
| 181 | using symlinks under `fuzz/corpus/revisions`; empty seed directories mostly exercise | ||
| 182 | header rejection. Bounded run counts and native findings live in `PROGRESS.md`. | ||
| 183 | |||
| 184 | The document fuzzer mutates native property streams, repairs their checksums, and | ||
| 185 | traverses every retained revision and resolved text run. Its public seeds live in | ||
| 186 | the source target; private seeds are supplied only at runtime. Native edit-history | ||
| 187 | tests compare independently generated operations, OneNote XML and the Rust model; | ||
| 188 | failed histories can be replayed and shrunk in fresh disposable clones. The | ||
| 189 | document feature matrix is in [FEATURES.md](FEATURES.md), and the milestone's | ||
| 190 | acceptance contract is in [MILESTONE6.md](MILESTONE6.md). | ||
| 191 | |||
| 192 | The stage-5 gate uses OneNote 2010 build 14.0.7015.1000 on Windows 7, a macOS SMB | ||
| 193 | mount, and Samba on zenith. [The collaboration corpus](corpus/collaboration/round-01) | ||
| 194 | captures native lock contention, different-paragraph merging, same-paragraph | ||
| 195 | conflicts, offline editing/reconnection, and lost successful FLUSH replies at | ||
| 196 | preparation, publication, and counter cleanup. Each final notebook was reopened | ||
| 197 | from a fresh native cache. Competing text survives as native conflict pages; | ||
| 198 | OneNote's COM hierarchy omits those pages, so the offline case also includes a | ||
| 199 | native UI capture. Full-page COM updates produced an extra conflict copy during | ||
| 200 | the disjoint case and two recorded geometry changes; the verifier checks those | ||
| 201 | exact changes as well as retained image, ink, table, and attachment content. | ||
| 202 | |||
| 203 | `tools/native/profile.ps1` parks/restores the personal native profile and cache; | ||
| 204 | `cold.ps1`, `read.ps1`, and `collaborate.ps1` operate on disposable test roots. | ||
| 205 | `tools/zenith-locks` observes server locks. `tools/smb-proxy.py` traces and interrupts | ||
| 206 | a dedicated loopback test session; its control JSON selects the successful response | ||
| 207 | and occurrence to withhold. The captured trace and result files are the regression | ||
| 208 | oracle; replaying the native experiments requires the supplied Windows/share setup. | ||
crates/onestore-diagnostic/Cargo.toml created+10
| ... | @@ -0,0 +1,10 @@ | ||
| 1 | [package] | ||
| 2 | name = "onestore-diagnostic" | ||
| 3 | version = "0.1.0" | ||
| 4 | edition = "2024" | ||
| 5 | publish = false | ||
| 6 | |||
| 7 | [dependencies] | ||
| 8 | onestore = { path = "../onestore" } | ||
| 9 | serde = { version = "1.0.229", features = ["derive"] } | ||
| 10 | serde_json = "1.0.151" | ||
crates/onestore-diagnostic/src/main.rs created+107
| ... | @@ -0,0 +1,107 @@ | ||
| 1 | use onestore::{ExGuid, Insertion, PreparedEdit, TextAttribute}; | ||
| 2 | use serde::Deserialize; | ||
| 3 | use serde_json::{Value, json}; | ||
| 4 | use std::{ | ||
| 5 | env, fs, | ||
| 6 | io::{self, Write}, | ||
| 7 | }; | ||
| 8 | |||
| 9 | #[derive(Deserialize)] | ||
| 10 | #[serde(deny_unknown_fields)] | ||
| 11 | struct Edit { | ||
| 12 | space: ExGuid, | ||
| 13 | object: ExGuid, | ||
| 14 | action: Action, | ||
| 15 | } | ||
| 16 | |||
| 17 | #[derive(Deserialize)] | ||
| 18 | #[serde(tag = "type", deny_unknown_fields)] | ||
| 19 | enum Action { | ||
| 20 | Text { | ||
| 21 | start: u32, | ||
| 22 | end: u32, | ||
| 23 | replacement: String, | ||
| 24 | }, | ||
| 25 | Format { | ||
| 26 | start: u32, | ||
| 27 | end: u32, | ||
| 28 | attributes: Vec<TextAttribute>, | ||
| 29 | }, | ||
| 30 | Paragraph { | ||
| 31 | before: Option<ExGuid>, | ||
| 32 | text: String, | ||
| 33 | author: String, | ||
| 34 | }, | ||
| 35 | Outline { | ||
| 36 | x: f32, | ||
| 37 | y: f32, | ||
| 38 | text: String, | ||
| 39 | author: String, | ||
| 40 | }, | ||
| 41 | } | ||
| 42 | |||
| 43 | fn run() -> Result<Value, Box<dyn std::error::Error>> { | ||
| 44 | let args: Vec<_> = env::args_os().skip(1).collect(); | ||
| 45 | if args.len() != 3 | ||
| 46 | || !["snapshot", "check", "commit"] | ||
| 47 | .iter() | ||
| 48 | .any(|mode| args[0] == *mode) | ||
| 49 | { | ||
| 50 | return Err("Usage: onestore-diagnostic snapshot FILE NEW_SNAPSHOT | check SNAPSHOT - | commit FILE SNAPSHOT; edits arrive as JSON on stdin".into()); | ||
| 51 | } | ||
| 52 | if args[0] == "snapshot" { | ||
| 53 | let bytes = onestore::read_file(&args[1])?; | ||
| 54 | let mut file = fs::OpenOptions::new() | ||
| 55 | .write(true) | ||
| 56 | .create_new(true) | ||
| 57 | .open(&args[2])?; | ||
| 58 | file.write_all(&bytes)?; | ||
| 59 | file.sync_all()?; | ||
| 60 | return Ok(json!({"ok": true, "bytes": bytes.len()})); | ||
| 61 | } | ||
| 62 | let check = args[0] == "check"; | ||
| 63 | if check && args[2] != "-" { | ||
| 64 | return Err("The check command requires '-' as its final argument".into()); | ||
| 65 | } | ||
| 66 | let bytes = fs::read(if check { &args[1] } else { &args[2] })?; | ||
| 67 | let edit: Edit = serde_json::from_reader(io::stdin().lock())?; | ||
| 68 | let sid = edit.space; | ||
| 69 | let oid = edit.object; | ||
| 70 | let prepared = match edit.action { | ||
| 71 | Action::Text { | ||
| 72 | start, | ||
| 73 | end, | ||
| 74 | replacement, | ||
| 75 | } => PreparedEdit::text(&bytes, sid, oid, start..end, &replacement)?, | ||
| 76 | Action::Format { | ||
| 77 | start, | ||
| 78 | end, | ||
| 79 | attributes, | ||
| 80 | } => PreparedEdit::format(&bytes, sid, oid, start..end, &attributes)?, | ||
| 81 | Action::Paragraph { | ||
| 82 | before, | ||
| 83 | text, | ||
| 84 | author, | ||
| 85 | } => PreparedEdit::insert( | ||
| 86 | &bytes, | ||
| 87 | sid, | ||
| 88 | &Insertion::paragraph(oid, before, &text, &author)?, | ||
| 89 | )?, | ||
| 90 | Action::Outline { x, y, text, author } => { | ||
| 91 | PreparedEdit::insert(&bytes, sid, &Insertion::outline(oid, x, y, &text, &author)?)? | ||
| 92 | } | ||
| 93 | }; | ||
| 94 | if check { | ||
| 95 | return Ok(json!({"ok": true})); | ||
| 96 | } | ||
| 97 | Ok(match prepared.commit_file(&args[1]) { | ||
| 98 | Ok(()) => json!({"ok": true, "state": "Committed"}), | ||
| 99 | Err(error) => json!({"ok": false, "state": format!("{:?}", error.state), | ||
| 100 | "kind": format!("{:?}", error.error.kind()), "error": error.error.to_string()}), | ||
| 101 | }) | ||
| 102 | } | ||
| 103 | |||
| 104 | fn main() { | ||
| 105 | let result = run().unwrap_or_else(|error| json!({"ok": false, "state": "NotCommitted", "kind": "Input", "error": error.to_string()})); | ||
| 106 | println!("{result}"); | ||
| 107 | } | ||
crates/onestore-offline/Cargo.toml created+23
| ... | @@ -0,0 +1,23 @@ | ||
| 1 | [package] | ||
| 2 | name = "onestore-offline" | ||
| 3 | version = "0.1.0" | ||
| 4 | edition = "2024" | ||
| 5 | publish = false | ||
| 6 | |||
| 7 | [features] | ||
| 8 | smb = ["dep:onestore-smb"] | ||
| 9 | |||
| 10 | [dependencies] | ||
| 11 | onestore = { path = "../onestore" } | ||
| 12 | onestore-smb = { path = "../onestore-smb", optional = true } | ||
| 13 | rusqlite = { version = "=0.40.2", features = ["bundled"] } | ||
| 14 | thiserror = "2" | ||
| 15 | serde = { version = "1", features = ["derive"] } | ||
| 16 | serde_json = "1" | ||
| 17 | |||
| 18 | [dev-dependencies] | ||
| 19 | tempfile = "3" | ||
| 20 | |||
| 21 | [[example]] | ||
| 22 | name = "smb_offline_client" | ||
| 23 | required-features = ["smb"] | ||
crates/onestore-offline/README.md created+191
| ... | @@ -0,0 +1,191 @@ | ||
| 1 | # onestore-offline | ||
| 2 | |||
| 3 | Durable local editing for a OneNote file, in an optional sibling crate. The current | ||
| 4 | foundation stores a complete working image and typed text, insertion and formatting intents in a local | ||
| 5 | SQLite database. `sync_once` provides a reconciliation step and `start_sync` owns | ||
| 6 | automatic polling and reconnects. Local success does not | ||
| 7 | acknowledge publication to a shared notebook. | ||
| 8 | |||
| 9 | ```no_run | ||
| 10 | use onestore::ExGuid; | ||
| 11 | use onestore_offline::Replica; | ||
| 12 | # fn example(path: &std::path::Path, source: &[u8], space: ExGuid, text: ExGuid) | ||
| 13 | # -> Result<(), Box<dyn std::error::Error>> { | ||
| 14 | // `space` and `text` are identities from the supplied section's document model. | ||
| 15 | let cache = Replica::create(path, source)?; | ||
| 16 | let snapshot = cache.snapshot()?; | ||
| 17 | let local_id = cache.edit_text(&snapshot, space, text, 0..0, "Offline edit ")?; | ||
| 18 | drop(cache); | ||
| 19 | |||
| 20 | let reopened = Replica::open(path)?; | ||
| 21 | let current = reopened.snapshot()?; | ||
| 22 | let pending = reopened.pending()?; | ||
| 23 | assert_eq!(pending.last().map(|edit| edit.id), local_id); | ||
| 24 | # Ok(()) | ||
| 25 | # } | ||
| 26 | ``` | ||
| 27 | |||
| 28 | `insert` accepts the core library's `Insertion` value and durably retains its | ||
| 29 | object identities. Keep that value across retries; its `text_object()` identifies | ||
| 30 | the new text for subsequent offline edits. Pending entries expose | ||
| 31 | `Operation::Text(TextEdit)`, `Operation::Insert(Insertion)` or | ||
| 32 | `Operation::Format(FormatEdit)` through their | ||
| 33 | `operation` field. Synchronization applies these in queue order, so an inserted | ||
| 34 | outline can precede its paragraphs and their later edits. Missing anchors or | ||
| 35 | existing insertion identities preserve a conflict and the complete local image. | ||
| 36 | |||
| 37 | `rebase_paragraph_conflict(id, local, remote, parent, before)` and | ||
| 38 | `rebase_outline_conflict(id, local, remote, page, x, y)` accept reviewed replacement | ||
| 39 | placements for the oldest insertion conflict. They preserve creation time, text, | ||
| 40 | author and object identities, keeping later edits attached to their original targets. | ||
| 41 | The images must still match the cache; the new placement must be valid in the remote | ||
| 42 | image. Paragraph intents cannot become outlines or vice versa. Pending edits and | ||
| 43 | uncertain publication attempts cannot be repositioned through conflict review. | ||
| 44 | |||
| 45 | ```no_run | ||
| 46 | use onestore::{ExGuid, Insertion, TextAttribute}; | ||
| 47 | use onestore_offline::{EditStatus, Replica}; | ||
| 48 | # fn add_outline(cache: &Replica, space: ExGuid, page: ExGuid) | ||
| 49 | # -> Result<(), Box<dyn std::error::Error>> { | ||
| 50 | let outline = Insertion::outline(page, 144.0, 216.0, "Offline outline", "Author")?; | ||
| 51 | let id = cache.insert(&cache.snapshot()?, space, &outline)?; | ||
| 52 | cache.format( | ||
| 53 | &cache.snapshot()?, space, outline.text_object(), 0..7, | ||
| 54 | &[TextAttribute::Bold(true)], | ||
| 55 | )?; | ||
| 56 | if let Some(id) = id { | ||
| 57 | // A running worker may already have advanced this state. | ||
| 58 | match cache.status(id)? { | ||
| 59 | Some(EditStatus::Published { revision }) => println!("{revision}"), | ||
| 60 | state => println!("{state:?}"), | ||
| 61 | } | ||
| 62 | } | ||
| 63 | # Ok(()) | ||
| 64 | # } | ||
| 65 | ``` | ||
| 66 | |||
| 67 | `format` accepts the core `TextAttribute` slice and a UTF-16 range. Its durable intent | ||
| 68 | retains the observed text and selected attribute values. Remote text changes must | ||
| 69 | leave an unambiguous range; independent remote attributes merge, while competing | ||
| 70 | values preserve `FormattingChanged`. A remote value that already matches the | ||
| 71 | requested value is accepted. Enabling superscript or subscript also checks the | ||
| 72 | opposite attribute that the operation clears. If the remote image already satisfies | ||
| 73 | the whole operation, guarded confirmation still precedes a durable receipt. | ||
| 74 | |||
| 75 | Recognized version-one through version-three caches migrate transactionally to the typed | ||
| 76 | queue. The migration retains images, local IDs, publication attempts, conflicts, | ||
| 77 | receipts and the autoincrement sequence; it does not reuse acknowledged IDs when | ||
| 78 | the pending queue is empty. | ||
| 79 | |||
| 80 | Share one `Replica` between application threads. Each edit compares its supplied | ||
| 81 | snapshot under the cache transaction; stale snapshots return `Io(ResourceBusy)`. | ||
| 82 | The intent and its resulting image commit together. No-op edits return `None`. | ||
| 83 | Keep the cache on a local filesystem: the connection holds exclusive ownership | ||
| 84 | between transactions, and a second open fails busy. No network wait occurs in a | ||
| 85 | local edit. After a database error, reopen and inspect the durable state before | ||
| 86 | retrying. | ||
| 87 | |||
| 88 | `sync_once(&mut remote)` processes the oldest pending edit through a `Remote` | ||
| 89 | implementation, returning its ID and `EditStatus`. A durable `Published` receipt | ||
| 90 | survives reopening. Publication attempts are recorded before network I/O; a retained | ||
| 91 | attempted revision requires comparison, flushing and refreshed header version | ||
| 92 | metadata before acknowledgement. If a formatting attempt's revision is missing, | ||
| 93 | the complete requested effect can instead be confirmed on a uniquely aligned | ||
| 94 | range; its receipt identifies that confirmed current revision. Otherwise the | ||
| 95 | missing attempt remains `AwaitingConfirmation`. Neither path replays an uncertain | ||
| 96 | publication. Overlapping or ambiguous edits retain `Conflict` status, their complete | ||
| 97 | local image and the last observed remote image returned by `remote_snapshot`. | ||
| 98 | Transport errors return `Error::Remote` or `Error::RemoteIo`; inspect `status(id)` | ||
| 99 | after the error to distinguish a retained attempt from a pending edit or receipt. | ||
| 100 | The error return does not roll back a locally acknowledged intent. | ||
| 101 | |||
| 102 | Rebasing accepts only character mappings shared by every minimum insertion/deletion | ||
| 103 | alignment. UTF-16 ranges must preserve Unicode scalar boundaries. A bounded | ||
| 104 | alignment search also leaves a conflict when it cannot establish a unique mapping. | ||
| 105 | The current operation processes one queue head; while edits remain, `snapshot` | ||
| 106 | preserves the complete local working image. An empty queue can refresh from the | ||
| 107 | remote image. Synchronization holds a separate owner lock, so local edits can | ||
| 108 | continue during network waits; competing synchronization calls return `WouldBlock`. | ||
| 109 | |||
| 110 | `rebase_conflict(id, local, remote, range)` lets a caller explicitly place the | ||
| 111 | oldest conflicting text or formatting intent at a reviewed UTF-16 range in the remote image. | ||
| 112 | The supplied images must still match `snapshot()` and `remote_snapshot()`. | ||
| 113 | It preserves the original replacement or requested attributes, intent ID, complete local working image | ||
| 114 | and every later intent; the selected range and remote paragraph become the | ||
| 115 | intent's new comparison base in one local transaction. Formatting also captures | ||
| 116 | the reviewed attribute values as its new precondition. This operation performs | ||
| 117 | no network I/O, clears the conflict to `Pending`, and wakes the worker. | ||
| 118 | Publication still reads the latest remote image and uses exact guarded comparison; | ||
| 119 | another overlapping remote edit can produce a new conflict. An uncertain attempt | ||
| 120 | cannot be rebased, and selecting text that already equals the replacement does | ||
| 121 | not create a publication acknowledgement. | ||
| 122 | |||
| 123 | With the optional `smb` feature, `SmbRemote::new(client, path, limit)` binds an | ||
| 124 | `onestore_smb::Client` to one share-relative file and snapshot limit. Remote identity uses the logical root | ||
| 125 | object space, which survives the tested native compaction that replaces the file ID. | ||
| 126 | |||
| 127 | An `Arc<Replica>` can own one background worker. Supply a connection factory, poll | ||
| 128 | interval and observer; successful publications drain immediately, durable local | ||
| 129 | edits wake the worker, and `wake()` requests an immediate reachability retry. | ||
| 130 | Transport failures discard the old connection and retry through the factory; | ||
| 131 | Read contention and `NotCommitted` operations with `WouldBlock` or `ResourceBusy` | ||
| 132 | reuse the connection. Contended `NotCommitted` operations use randomized backoff, | ||
| 133 | capped at one second, to separate competing retry cycles. Cancellation interrupts this delay; local wake notifications | ||
| 134 | remain coalesced until its end. | ||
| 135 | `RemoteIo` distinguishes connection/read failures from | ||
| 136 | local `Io` errors. Cache/document errors stop the worker. Observers receive every | ||
| 137 | attempt's result on the worker thread, including unchanged conflict/uncertain | ||
| 138 | statuses; durable edit state remains available through `status`. | ||
| 139 | |||
| 140 | ```no_run | ||
| 141 | # #[cfg(feature = "smb")] | ||
| 142 | # fn example(cache: std::sync::Arc<onestore_offline::Replica>, username: String, password: String) | ||
| 143 | # -> Result<(), Box<dyn std::error::Error>> { | ||
| 144 | use onestore_offline::SmbRemote; | ||
| 145 | use onestore_smb::{Client, Credentials}; | ||
| 146 | use std::time::Duration; | ||
| 147 | |||
| 148 | let worker = cache.start_sync( | ||
| 149 | Duration::from_secs(2), | ||
| 150 | move || { | ||
| 151 | Client::connect( | ||
| 152 | "server:445", "notes", | ||
| 153 | Credentials { username: &username, password: &password, domain: "" }, | ||
| 154 | Duration::from_secs(5), | ||
| 155 | ).map(|client| SmbRemote::new(client, "Personal/Video.one", 64 * 1024 * 1024)) | ||
| 156 | }, | ||
| 157 | |result| { | ||
| 158 | if let Err(error) = result { eprintln!("{error}"); } | ||
| 159 | }, | ||
| 160 | )?; | ||
| 161 | // Retain `worker` while synchronization should run; local edits wake it automatically. | ||
| 162 | worker.stop()?; | ||
| 163 | # Ok(()) | ||
| 164 | # } | ||
| 165 | ``` | ||
| 166 | |||
| 167 | `stop()` cancels future steps and joins the worker, returning a fatal cache error | ||
| 168 | or worker panic. Dropping it requests cancellation without waiting. An in-flight | ||
| 169 | step completes before releasing ownership; a replacement worker cannot start | ||
| 170 | while the old one still owns the replica. Remote operations and callbacks must | ||
| 171 | have bounded execution times if shutdown latency matters. A dropped worker can | ||
| 172 | briefly retain the cache; use `stop()` before requiring an immediate reopen. | ||
| 173 | Cancellation and application restart retain pending edits and publication attempts. | ||
| 174 | Credentials belong to the factory, not the cache database. | ||
| 175 | |||
| 176 | Creation refuses existing paths. Opening recognizes the application identity and | ||
| 177 | schema version, validates database integrity and both notebook images, and rejects | ||
| 178 | unsupported journal modes without converting them. Failed initialization preserves | ||
| 179 | the file for inspection. SQLite uses DELETE journaling, EXTRA synchronization and | ||
| 180 | fullfsync; each required setting is queried back. | ||
| 181 | |||
| 182 | The cache and its pending intents contain notebook content. The core `onestore` | ||
| 183 | crate stays independent of SQLite and network runtimes. Device and simulator | ||
| 184 | examples compile and link for iOS; recorded process-interruption tests do not | ||
| 185 | establish physical power-loss durability. Evidence is tracked in [Milestone 9](../../evidence/MILESTONE9.md). | ||
| 186 | |||
| 187 | The SMB-enabled `smb_offline_client` example is an owned-lab workload for | ||
| 188 | `tools/native_collaboration.py --offline --embedded-smb`. It separates local | ||
| 189 | acknowledgements, remote publication attempts, persisted receipts and cache reopen | ||
| 190 | checks. Its append-specific conflict review policy lives in the test client; | ||
| 191 | the library continues to preserve conflicts requiring an explicit decision. | ||
crates/onestore-offline/examples/cache_probe.rs created+230
| ... | @@ -0,0 +1,230 @@ | ||
| 1 | use onestore::{ | ||
| 2 | ExGuid, Insertion, RevisionIndex, Store, TextAttribute, | ||
| 3 | document::{Document, Kind}, | ||
| 4 | }; | ||
| 5 | use onestore_offline::Replica; | ||
| 6 | use std::{ | ||
| 7 | io::{self, BufRead, Write}, | ||
| 8 | path::Path, | ||
| 9 | }; | ||
| 10 | |||
| 11 | fn content(bytes: &[u8]) -> (ExGuid, ExGuid, String) { | ||
| 12 | let store = Store::parse(bytes).unwrap(); | ||
| 13 | assert!(store.checksum_mismatches.is_empty()); | ||
| 14 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 15 | index.validate_current().unwrap(); | ||
| 16 | let document = Document::parse(&index).unwrap(); | ||
| 17 | document | ||
| 18 | .spaces | ||
| 19 | .iter() | ||
| 20 | .find_map(|(sid, space)| { | ||
| 21 | let revision = &space.revisions[&space.contexts[&ExGuid::default()]]; | ||
| 22 | revision | ||
| 23 | .nodes | ||
| 24 | .iter() | ||
| 25 | .find_map(|(oid, node)| match &node.kind { | ||
| 26 | Kind::RichText { text, .. } => Some((*sid, *oid, text.clone())), | ||
| 27 | _ => None, | ||
| 28 | }) | ||
| 29 | }) | ||
| 30 | .unwrap() | ||
| 31 | } | ||
| 32 | |||
| 33 | fn payload(operation: u64, size: usize) -> String { | ||
| 34 | format!("{operation}:🦀{}", "x".repeat(size)) | ||
| 35 | } | ||
| 36 | |||
| 37 | fn main() -> Result<(), Box<dyn std::error::Error>> { | ||
| 38 | let args: Vec<_> = std::env::args().collect(); | ||
| 39 | let mode = &args[1]; | ||
| 40 | let path = Path::new(&args[2]); | ||
| 41 | let operation_kind = | ||
| 42 | std::env::var("ONESTORE_CACHE_PROBE_OPERATION").unwrap_or_else(|_| "text".into()); | ||
| 43 | assert!(matches!( | ||
| 44 | operation_kind.as_str(), | ||
| 45 | "text" | "insert" | "format" | ||
| 46 | )); | ||
| 47 | let size = match std::env::var("ONESTORE_CACHE_PROBE_BYTES") { | ||
| 48 | Ok(value) => value.parse::<usize>()?, | ||
| 49 | Err(std::env::VarError::NotPresent) => 2 * 1024 * 1024, | ||
| 50 | Err(error) => return Err(error.into()), | ||
| 51 | }; | ||
| 52 | assert!(size > 0 && size <= 2 * 1024 * 1024); | ||
| 53 | let seed = std::env::var_os("ONESTORE_CACHE_PROBE_SOURCE") | ||
| 54 | .map(std::fs::read) | ||
| 55 | .transpose()?; | ||
| 56 | if mode == "init" { | ||
| 57 | let source = match seed { | ||
| 58 | Some(source) => source, | ||
| 59 | None => onestore::create_section( | ||
| 60 | "cache.one", | ||
| 61 | &if operation_kind == "format" { | ||
| 62 | payload(0, size) | ||
| 63 | } else { | ||
| 64 | "Base".into() | ||
| 65 | }, | ||
| 66 | "Fixture", | ||
| 67 | )?, | ||
| 68 | }; | ||
| 69 | Replica::create(path, &source)?; | ||
| 70 | return Ok(()); | ||
| 71 | } | ||
| 72 | let cache = Replica::open(path)?; | ||
| 73 | if mode == "read" { | ||
| 74 | let snapshot = cache.snapshot()?; | ||
| 75 | let base = cache.remote_snapshot()?; | ||
| 76 | let (sid, target, mut expected) = content(&base); | ||
| 77 | let store = Store::parse(&snapshot)?; | ||
| 78 | assert!(store.checksum_mismatches.is_empty()); | ||
| 79 | let index = RevisionIndex::parse(&store)?; | ||
| 80 | index.validate_current()?; | ||
| 81 | let document = Document::parse(&index)?; | ||
| 82 | let space = &document.spaces[&sid]; | ||
| 83 | let revision = &space.revisions[&space.contexts[&ExGuid::default()]]; | ||
| 84 | let mut operations = Vec::new(); | ||
| 85 | let mut ids = Vec::new(); | ||
| 86 | let mut font_size = None; | ||
| 87 | for pending in cache.pending()? { | ||
| 88 | let operation = match pending.operation { | ||
| 89 | onestore_offline::Operation::Text(edit) => { | ||
| 90 | assert_eq!(operation_kind, "text"); | ||
| 91 | assert_eq!(edit.object, target); | ||
| 92 | assert_eq!(edit.before, expected); | ||
| 93 | assert_eq!( | ||
| 94 | edit.range, | ||
| 95 | 0..u32::try_from(expected.encode_utf16().count())? | ||
| 96 | ); | ||
| 97 | let operation: u64 = edit.replacement.split_once(':').unwrap().0.parse()?; | ||
| 98 | expected = payload(operation, size); | ||
| 99 | assert_eq!(edit.replacement, expected); | ||
| 100 | operation | ||
| 101 | } | ||
| 102 | onestore_offline::Operation::Insert(insertion) => { | ||
| 103 | assert_eq!(operation_kind, "insert"); | ||
| 104 | let Kind::RichText { text, .. } = | ||
| 105 | &revision.nodes[&insertion.text_object()].kind | ||
| 106 | else { | ||
| 107 | panic!("Missing inserted text") | ||
| 108 | }; | ||
| 109 | let operation: u64 = text.split_once(':').unwrap().0.parse()?; | ||
| 110 | assert_eq!(*text, payload(operation, size)); | ||
| 111 | assert_eq!(serde_json::to_value(&insertion)?["text"], *text); | ||
| 112 | let outline = &revision.nodes[&insertion.object()]; | ||
| 113 | assert!(matches!(outline.kind, Kind::Outline { .. })); | ||
| 114 | assert_eq!( | ||
| 115 | (outline.layout.x, outline.layout.y), | ||
| 116 | (Some(144.0), Some(operation as f32 * 72.0)) | ||
| 117 | ); | ||
| 118 | let (_, page) = document | ||
| 119 | .pages()? | ||
| 120 | .into_iter() | ||
| 121 | .find(|(space, _)| *space == sid) | ||
| 122 | .unwrap(); | ||
| 123 | assert!(revision.nodes[&page].children.contains(&insertion.object())); | ||
| 124 | operation | ||
| 125 | } | ||
| 126 | onestore_offline::Operation::Format(edit) => { | ||
| 127 | assert_eq!(operation_kind, "format"); | ||
| 128 | assert_eq!(edit.object, target); | ||
| 129 | assert_eq!(edit.before, expected); | ||
| 130 | assert_eq!( | ||
| 131 | edit.range, | ||
| 132 | 0..u32::try_from(expected.encode_utf16().count())? | ||
| 133 | ); | ||
| 134 | let [TextAttribute::FontSize(value)] = edit.attributes.as_slice() else { | ||
| 135 | panic!("Unexpected formatting intent") | ||
| 136 | }; | ||
| 137 | assert!((7.0..=130.0).contains(value) && value.fract() == 0.0); | ||
| 138 | font_size = Some(*value); | ||
| 139 | *value as u64 - 6 | ||
| 140 | } | ||
| 141 | }; | ||
| 142 | assert!(operations.last().is_none_or(|last| *last < operation)); | ||
| 143 | assert!(ids.last().is_none_or(|last| *last < pending.id)); | ||
| 144 | operations.push(operation); | ||
| 145 | ids.push(pending.id); | ||
| 146 | } | ||
| 147 | assert!(matches!(&revision.nodes[&target].kind,Kind::RichText{text,..} if *text==expected)); | ||
| 148 | if let Some(font_size) = font_size { | ||
| 149 | assert!( | ||
| 150 | revision | ||
| 151 | .text_runs(target)? | ||
| 152 | .iter() | ||
| 153 | .all(|run| run.format.font_size == Some(font_size)) | ||
| 154 | ); | ||
| 155 | } | ||
| 156 | if operations.is_empty() { | ||
| 157 | assert!( | ||
| 158 | snapshot == base, | ||
| 159 | "An empty local queue changed its working image" | ||
| 160 | ); | ||
| 161 | } | ||
| 162 | if let Some(output) = args.get(3) { | ||
| 163 | std::fs::write(output, &snapshot)?; | ||
| 164 | } | ||
| 165 | println!( | ||
| 166 | "{}", | ||
| 167 | serde_json::json!({"operations": operations, "ids": ids, "section_bytes": snapshot.len(), "complete_payloads": true}) | ||
| 168 | ); | ||
| 169 | return Ok(()); | ||
| 170 | } | ||
| 171 | assert_eq!(mode, "edit"); | ||
| 172 | assert!( | ||
| 173 | matches!(Replica::open(path), Err(onestore_offline::Error::Database(error)) if error.sqlite_error_code() == Some(rusqlite::ErrorCode::DatabaseBusy)) | ||
| 174 | ); | ||
| 175 | println!("ready"); | ||
| 176 | io::stdout().flush()?; | ||
| 177 | let input = io::stdin(); | ||
| 178 | let mut lines = input.lock().lines(); | ||
| 179 | let instruction = lines.next().unwrap()?; | ||
| 180 | let (operation, acknowledgement) = instruction.split_once(' ').unwrap(); | ||
| 181 | let operation: u64 = operation.parse()?; | ||
| 182 | let source = cache.snapshot()?; | ||
| 183 | let (sid, oid, text) = content(&source); | ||
| 184 | let replacement = payload(operation, size); | ||
| 185 | println!("editing {operation}"); | ||
| 186 | io::stdout().flush()?; | ||
| 187 | let id = match operation_kind.as_str() { | ||
| 188 | "text" => cache.edit_text( | ||
| 189 | &source, | ||
| 190 | sid, | ||
| 191 | oid, | ||
| 192 | 0..text.encode_utf16().count().try_into()?, | ||
| 193 | &replacement, | ||
| 194 | )?, | ||
| 195 | "insert" => { | ||
| 196 | let store = Store::parse(&source)?; | ||
| 197 | let index = RevisionIndex::parse(&store)?; | ||
| 198 | let document = Document::parse(&index)?; | ||
| 199 | let (sid, page) = document.pages()?[0]; | ||
| 200 | let insertion = Insertion::outline( | ||
| 201 | page, | ||
| 202 | 144.0, | ||
| 203 | operation as f32 * 72.0, | ||
| 204 | &replacement, | ||
| 205 | "Fixture", | ||
| 206 | )?; | ||
| 207 | cache.insert(&source, sid, &insertion)? | ||
| 208 | } | ||
| 209 | "format" => { | ||
| 210 | assert!((1..=124).contains(&operation)); | ||
| 211 | cache.format( | ||
| 212 | &source, | ||
| 213 | sid, | ||
| 214 | oid, | ||
| 215 | 0..text.encode_utf16().count().try_into()?, | ||
| 216 | &[TextAttribute::FontSize(6.0 + operation as f32)], | ||
| 217 | )? | ||
| 218 | } | ||
| 219 | _ => unreachable!(), | ||
| 220 | } | ||
| 221 | .unwrap(); | ||
| 222 | if acknowledgement == "unack" { | ||
| 223 | println!("durable {operation} {id}"); | ||
| 224 | } else { | ||
| 225 | println!("ack {operation} {id}"); | ||
| 226 | } | ||
| 227 | io::stdout().flush()?; | ||
| 228 | lines.next().transpose()?; | ||
| 229 | Ok(()) | ||
| 230 | } | ||
crates/onestore-offline/examples/recovery_probe.rs created+160
| ... | @@ -0,0 +1,160 @@ | ||
| 1 | mod support { | ||
| 2 | pub mod view; | ||
| 3 | } | ||
| 4 | use support::view::view; | ||
| 5 | |||
| 6 | use onestore::{CommitError, CommitIo, PreparedEdit}; | ||
| 7 | use onestore_offline::{EditStatus, Remote, Replica}; | ||
| 8 | use serde_json::json; | ||
| 9 | use std::{ | ||
| 10 | env, | ||
| 11 | fs::{self, File, OpenOptions}, | ||
| 12 | io::{self, Write}, | ||
| 13 | os::unix::fs::FileExt, | ||
| 14 | path::Path, | ||
| 15 | }; | ||
| 16 | |||
| 17 | fn phase(name: &str) { | ||
| 18 | println!("{}", json!({"event":"phase", "name":name})); | ||
| 19 | io::stdout().flush().unwrap(); | ||
| 20 | if env::var("ONESTORE_RECOVERY_PAUSE").ok().as_deref() == Some(name) { | ||
| 21 | let mut line = String::new(); | ||
| 22 | assert!( | ||
| 23 | io::stdin().read_line(&mut line).unwrap() > 0, | ||
| 24 | "Controller closed a paused operation" | ||
| 25 | ); | ||
| 26 | } | ||
| 27 | } | ||
| 28 | |||
| 29 | struct Disk { | ||
| 30 | file: File, | ||
| 31 | writes: usize, | ||
| 32 | flushes: usize, | ||
| 33 | } | ||
| 34 | |||
| 35 | impl CommitIo for Disk { | ||
| 36 | fn read_at(&mut self, offset: u64, output: &mut [u8]) -> io::Result<usize> { | ||
| 37 | self.file.read_at(output, offset) | ||
| 38 | } | ||
| 39 | fn write_at(&mut self, offset: u64, data: &[u8]) -> io::Result<usize> { | ||
| 40 | self.writes += 1; | ||
| 41 | println!( | ||
| 42 | "{}", | ||
| 43 | json!({"event":"write", "number":self.writes, "offset":offset, "bytes":data.len()}) | ||
| 44 | ); | ||
| 45 | phase(&format!("write-{}-before", self.writes)); | ||
| 46 | let result = self.file.write_at(data, offset); | ||
| 47 | phase(&format!("write-{}-after", self.writes)); | ||
| 48 | result | ||
| 49 | } | ||
| 50 | fn flush(&mut self) -> io::Result<()> { | ||
| 51 | self.flushes += 1; | ||
| 52 | phase(&format!("flush-{}-before", self.flushes)); | ||
| 53 | let result = self.file.sync_all(); | ||
| 54 | phase(&format!("flush-{}-after", self.flushes)); | ||
| 55 | result | ||
| 56 | } | ||
| 57 | } | ||
| 58 | |||
| 59 | impl Remote for Disk { | ||
| 60 | fn read(&mut self) -> io::Result<Vec<u8>> { | ||
| 61 | phase("read-before"); | ||
| 62 | let result = onestore::read_snapshot( | ||
| 63 | |offset, output| self.file.read_at(output, offset), | ||
| 64 | 256 * 1024 * 1024, | ||
| 65 | ) | ||
| 66 | .and_then(|snapshot| snapshot.ok_or_else(|| io::ErrorKind::WouldBlock.into())); | ||
| 67 | phase("read-after"); | ||
| 68 | result | ||
| 69 | } | ||
| 70 | fn publish(&mut self, edit: &PreparedEdit<'_>) -> Result<(), CommitError> { | ||
| 71 | phase("publish-before"); | ||
| 72 | let result = edit.commit(self); | ||
| 73 | phase("publish-after"); | ||
| 74 | result | ||
| 75 | } | ||
| 76 | fn confirm(&mut self, snapshot: &[u8]) -> Result<(), CommitError> { | ||
| 77 | phase("confirm-before"); | ||
| 78 | let result = onestore::confirm_snapshot(self, snapshot); | ||
| 79 | phase("confirm-after"); | ||
| 80 | result | ||
| 81 | } | ||
| 82 | } | ||
| 83 | |||
| 84 | fn report(cache: &Replica, root: &Path) -> Result<(), Box<dyn std::error::Error>> { | ||
| 85 | let local = view(&cache.snapshot()?)?; | ||
| 86 | let remote = view(&fs::read(root.join("remote.one"))?)?; | ||
| 87 | let (status, revision) = match cache.status(1)? { | ||
| 88 | Some(EditStatus::Pending) => ("pending", None), | ||
| 89 | Some(EditStatus::AwaitingConfirmation { revision }) => { | ||
| 90 | ("uncertain", Some(revision.to_string())) | ||
| 91 | } | ||
| 92 | Some(EditStatus::Published { revision }) => ("published", Some(revision.to_string())), | ||
| 93 | Some(EditStatus::Conflict(_)) => ("conflict", None), | ||
| 94 | None => ("missing", None), | ||
| 95 | }; | ||
| 96 | println!( | ||
| 97 | "{}", | ||
| 98 | json!({"event":"state", "status":status, "revision":revision, | ||
| 99 | "local_text":local.text, "remote_text":remote.text, "remote_revision":remote.revision.to_string(), | ||
| 100 | "pending":cache.pending()?.iter().map(|pending|match &pending.operation { | ||
| 101 | onestore_offline::Operation::Text(edit) => json!({"id":pending.id,"before":edit.before,"replacement":edit.replacement,"range":[edit.range.start,edit.range.end]}), | ||
| 102 | operation => json!({"id":pending.id,"operation":operation}), | ||
| 103 | }).collect::<Vec<_>>() }) | ||
| 104 | ); | ||
| 105 | Ok(()) | ||
| 106 | } | ||
| 107 | |||
| 108 | fn main() -> Result<(), Box<dyn std::error::Error>> { | ||
| 109 | let args: Vec<_> = env::args().skip(1).collect(); | ||
| 110 | let mode = args | ||
| 111 | .first() | ||
| 112 | .ok_or("Expected init|inspect|sync DIRECTORY [SOURCE]")?; | ||
| 113 | let root = Path::new(args.get(1).ok_or("Missing owned directory")?); | ||
| 114 | if mode == "init" && args.len() == 3 { | ||
| 115 | let source = fs::read(&args[2])?; | ||
| 116 | let target = view(&source)?; | ||
| 117 | fs::create_dir(root)?; | ||
| 118 | let mut remote = OpenOptions::new() | ||
| 119 | .write(true) | ||
| 120 | .create_new(true) | ||
| 121 | .open(root.join("remote.one"))?; | ||
| 122 | remote.write_all(&source)?; | ||
| 123 | remote.sync_all()?; | ||
| 124 | drop(remote); | ||
| 125 | let cache = Replica::create(root.join("cache.sqlite"), &source)?; | ||
| 126 | let at = u32::try_from(target.text.encode_utf16().count())?; | ||
| 127 | assert_eq!( | ||
| 128 | cache.edit_text( | ||
| 129 | &source, | ||
| 130 | target.space, | ||
| 131 | target.object, | ||
| 132 | at..at, | ||
| 133 | " [offline-recovery]" | ||
| 134 | )?, | ||
| 135 | Some(1) | ||
| 136 | ); | ||
| 137 | phase("local-after"); | ||
| 138 | report(&cache, root)?; | ||
| 139 | } else if args.len() == 2 && ["inspect", "sync"].contains(&mode.as_str()) { | ||
| 140 | let cache = Replica::open(root.join("cache.sqlite"))?; | ||
| 141 | if mode == "sync" { | ||
| 142 | let mut disk = Disk { | ||
| 143 | file: OpenOptions::new() | ||
| 144 | .read(true) | ||
| 145 | .write(true) | ||
| 146 | .open(root.join("remote.one"))?, | ||
| 147 | writes: 0, | ||
| 148 | flushes: 0, | ||
| 149 | }; | ||
| 150 | phase("sync-before"); | ||
| 151 | let result = cache.sync_once(&mut disk); | ||
| 152 | phase("sync-after"); | ||
| 153 | result?; | ||
| 154 | } | ||
| 155 | report(&cache, root)?; | ||
| 156 | } else { | ||
| 157 | return Err("Expected init|inspect|sync DIRECTORY [SOURCE]".into()); | ||
| 158 | } | ||
| 159 | Ok(()) | ||
| 160 | } | ||
crates/onestore-offline/examples/smb_offline_client.rs created+593
| ... | @@ -0,0 +1,593 @@ | ||
| 1 | #[path = "../../onestore/examples/support/concurrent.rs"] | ||
| 2 | mod concurrent; | ||
| 3 | |||
| 4 | use onestore::{ | ||
| 5 | CommitError, CommitState, ExGuid, Insertion, PreparedEdit, RevisionIndex, Store, TextAttribute, | ||
| 6 | document::Document, | ||
| 7 | }; | ||
| 8 | use onestore_offline::{EditStatus, Error, Remote, Replica, SmbRemote}; | ||
| 9 | use onestore_smb::{Client, Credentials}; | ||
| 10 | use serde_json::json; | ||
| 11 | use std::{ | ||
| 12 | env, | ||
| 13 | io::{self, Write}, | ||
| 14 | path::{Path, PathBuf}, | ||
| 15 | sync::Arc, | ||
| 16 | thread, | ||
| 17 | time::{Duration, Instant, SystemTime, UNIX_EPOCH}, | ||
| 18 | }; | ||
| 19 | |||
| 20 | mod support { | ||
| 21 | pub mod view; | ||
| 22 | } | ||
| 23 | use concurrent::document_view; | ||
| 24 | use support::view::view; | ||
| 25 | |||
| 26 | fn now() -> u128 { | ||
| 27 | SystemTime::now() | ||
| 28 | .duration_since(UNIX_EPOCH) | ||
| 29 | .unwrap() | ||
| 30 | .as_micros() | ||
| 31 | } | ||
| 32 | |||
| 33 | #[derive(Clone)] | ||
| 34 | enum Pause { | ||
| 35 | Outage(PathBuf), | ||
| 36 | FormatReply(PathBuf), | ||
| 37 | } | ||
| 38 | |||
| 39 | struct Traced { | ||
| 40 | remote: SmbRemote, | ||
| 41 | before: Option<(String, Option<serde_json::Value>)>, | ||
| 42 | pause: Option<Pause>, | ||
| 43 | documents: bool, | ||
| 44 | } | ||
| 45 | |||
| 46 | impl Remote for Traced { | ||
| 47 | fn read(&mut self) -> io::Result<Vec<u8>> { | ||
| 48 | let started = now(); | ||
| 49 | let bytes = self.remote.read()?; | ||
| 50 | let observed = view(&bytes).map_err(|error| io::Error::other(error.to_string()))?; | ||
| 51 | let documents = if self.documents { | ||
| 52 | Some(document_view(&bytes).map_err(io::Error::other)?) | ||
| 53 | } else { | ||
| 54 | None | ||
| 55 | }; | ||
| 56 | println!( | ||
| 57 | "{}", | ||
| 58 | json!({"event":"read", "started_us":started, "finished_us":now(), "text":observed.text, "documents":documents}) | ||
| 59 | ); | ||
| 60 | self.before = Some((observed.text, documents)); | ||
| 61 | Ok(bytes) | ||
| 62 | } | ||
| 63 | fn publish(&mut self, edit: &PreparedEdit<'_>) -> Result<(), CommitError> { | ||
| 64 | let after = view(edit.as_bytes()).map_err(|error| CommitError { | ||
| 65 | state: CommitState::NotCommitted, | ||
| 66 | error: io::Error::other(error.to_string()), | ||
| 67 | })?; | ||
| 68 | let documents = if self.documents { | ||
| 69 | Some(document_view(edit.as_bytes()).map_err(|error| CommitError { | ||
| 70 | state: CommitState::NotCommitted, | ||
| 71 | error: io::Error::other(error), | ||
| 72 | })?) | ||
| 73 | } else { | ||
| 74 | None | ||
| 75 | }; | ||
| 76 | let changes = if let Some(after) = &documents { | ||
| 77 | let before = self | ||
| 78 | .before | ||
| 79 | .as_ref() | ||
| 80 | .and_then(|(_, documents)| documents.as_ref()) | ||
| 81 | .ok_or_else(|| CommitError { | ||
| 82 | state: CommitState::NotCommitted, | ||
| 83 | error: io::Error::other("Document publication has no observed source"), | ||
| 84 | })?; | ||
| 85 | Some( | ||
| 86 | after | ||
| 87 | .as_object() | ||
| 88 | .unwrap() | ||
| 89 | .iter() | ||
| 90 | .filter(|(id, value)| before.get(*id) != Some(*value)) | ||
| 91 | .map(|(id, value)| (id.clone(), value.clone())) | ||
| 92 | .collect::<serde_json::Map<_, _>>(), | ||
| 93 | ) | ||
| 94 | } else { | ||
| 95 | None | ||
| 96 | }; | ||
| 97 | let pause = match &self.pause { | ||
| 98 | Some(Pause::Outage(marker)) => Some(( | ||
| 99 | marker, | ||
| 100 | marker.parent().unwrap().join("offline-outage-resumed"), | ||
| 101 | "outage", | ||
| 102 | )), | ||
| 103 | Some(Pause::FormatReply(marker)) | ||
| 104 | if changes.as_ref().is_some_and(|changes| { | ||
| 105 | changes.len() == 1 | ||
| 106 | && self | ||
| 107 | .before | ||
| 108 | .as_ref() | ||
| 109 | .and_then(|(_, before)| before.as_ref()) | ||
| 110 | .is_some_and(|before| changes.keys().all(|id| before.get(id).is_some())) | ||
| 111 | }) => | ||
| 112 | { | ||
| 113 | Some((marker, marker.with_extension("resume"), "format")) | ||
| 114 | } | ||
| 115 | _ => None, | ||
| 116 | }; | ||
| 117 | if let Some((marker, resumed, kind)) = pause | ||
| 118 | && !resumed.exists() | ||
| 119 | { | ||
| 120 | std::fs::write(marker, after.revision.to_string()).map_err(|error| CommitError { | ||
| 121 | state: CommitState::NotCommitted, | ||
| 122 | error, | ||
| 123 | })?; | ||
| 124 | println!( | ||
| 125 | "{}", | ||
| 126 | json!({"event":"publication_paused", "revision":after.revision.to_string(), "kind":kind, "at_us":now()}) | ||
| 127 | ); | ||
| 128 | let deadline = Instant::now() + Duration::from_secs(120); | ||
| 129 | while !resumed.exists() { | ||
| 130 | if Instant::now() >= deadline { | ||
| 131 | return Err(CommitError { | ||
| 132 | state: CommitState::NotCommitted, | ||
| 133 | error: io::Error::new( | ||
| 134 | io::ErrorKind::TimedOut, | ||
| 135 | "Offline publication barrier timed out", | ||
| 136 | ), | ||
| 137 | }); | ||
| 138 | } | ||
| 139 | thread::sleep(Duration::from_millis(10)); | ||
| 140 | } | ||
| 141 | } | ||
| 142 | let started = now(); | ||
| 143 | let result = self.remote.publish(edit); | ||
| 144 | let finished = now(); | ||
| 145 | println!( | ||
| 146 | "{}", | ||
| 147 | json!({"event":"remote_attempt", "started_us":started, "finished_us":finished, | ||
| 148 | "revision":after.revision.to_string(), "space":after.space.to_string(), "object":after.object.to_string(), "before":self.before.as_ref().map(|(text,_)|text), "after":after.text, | ||
| 149 | "state":format!("{:?}", result.as_ref().map_or_else(|error| error.state, |_| CommitState::Committed)), | ||
| 150 | "documents":documents, "document_changes":changes}) | ||
| 151 | ); | ||
| 152 | if result | ||
| 153 | .as_ref() | ||
| 154 | .is_err_and(|error| error.state == CommitState::Unknown) | ||
| 155 | && let Some(Pause::FormatReply(marker)) = &self.pause | ||
| 156 | && marker.with_extension("isolate").exists() | ||
| 157 | { | ||
| 158 | std::fs::write(marker.with_extension("isolate"), serde_json::to_vec(&json!({"space":after.space.to_string(), "revision":after.revision.to_string(), "after_us":finished})).unwrap()) | ||
| 159 | .map_err(|error| CommitError { state:CommitState::Unknown, error })?; | ||
| 160 | println!( | ||
| 161 | "{}", | ||
| 162 | json!({"event":"confirmation_paused", "revision":after.revision.to_string(), "at_us":now()}) | ||
| 163 | ); | ||
| 164 | let deadline = Instant::now() + Duration::from_secs(120); | ||
| 165 | while !marker.with_extension("confirmation-resume").exists() { | ||
| 166 | if Instant::now() >= deadline { | ||
| 167 | return Err(CommitError { | ||
| 168 | state: CommitState::Unknown, | ||
| 169 | error: io::Error::new( | ||
| 170 | io::ErrorKind::TimedOut, | ||
| 171 | "Offline confirmation barrier timed out", | ||
| 172 | ), | ||
| 173 | }); | ||
| 174 | } | ||
| 175 | thread::sleep(Duration::from_millis(10)); | ||
| 176 | } | ||
| 177 | } | ||
| 178 | result | ||
| 179 | } | ||
| 180 | fn confirm(&mut self, snapshot: &[u8]) -> Result<(), CommitError> { | ||
| 181 | let captured = (|| -> Result<_, Box<dyn std::error::Error>> { | ||
| 182 | let store = Store::parse(snapshot)?; | ||
| 183 | let index = RevisionIndex::parse(&store)?; | ||
| 184 | let revisions = index | ||
| 185 | .spaces | ||
| 186 | .iter() | ||
| 187 | .map(|(id, space)| { | ||
| 188 | ( | ||
| 189 | id.to_string(), | ||
| 190 | space | ||
| 191 | .revisions | ||
| 192 | .keys() | ||
| 193 | .map(ToString::to_string) | ||
| 194 | .collect::<Vec<_>>(), | ||
| 195 | ) | ||
| 196 | }) | ||
| 197 | .collect::<std::collections::BTreeMap<_, _>>(); | ||
| 198 | let current = index | ||
| 199 | .spaces | ||
| 200 | .iter() | ||
| 201 | .filter_map(|(id, space)| { | ||
| 202 | space | ||
| 203 | .labels | ||
| 204 | .get(&(ExGuid::default(), 1)) | ||
| 205 | .map(|revision| (id.to_string(), revision.to_string())) | ||
| 206 | }) | ||
| 207 | .collect::<std::collections::BTreeMap<_, _>>(); | ||
| 208 | let path = env::var_os("ONESTORE_OFFLINE_CONFIRM_DIR").map(|directory| { | ||
| 209 | PathBuf::from(directory).join(format!("{}-{}.one", std::process::id(), now())) | ||
| 210 | }); | ||
| 211 | if let Some(path) = &path { | ||
| 212 | std::fs::OpenOptions::new() | ||
| 213 | .write(true) | ||
| 214 | .create_new(true) | ||
| 215 | .open(path)? | ||
| 216 | .write_all(snapshot)?; | ||
| 217 | } | ||
| 218 | Ok((revisions, current, path)) | ||
| 219 | })(); | ||
| 220 | let (revisions, current, capture) = captured.map_err(|error| CommitError { | ||
| 221 | state: CommitState::NotCommitted, | ||
| 222 | error: io::Error::other(error.to_string()), | ||
| 223 | })?; | ||
| 224 | let started = now(); | ||
| 225 | let result = self.remote.confirm(snapshot); | ||
| 226 | println!( | ||
| 227 | "{}", | ||
| 228 | json!({"event":"remote_confirm", "started_us":started, "finished_us":now(), "revisions":revisions, "current_revisions":current, "capture":capture.as_ref().and_then(|path| path.file_name()).map(|name| name.to_string_lossy()), "text":self.before.as_ref().map(|(text,_)|text), | ||
| 229 | "state":format!("{:?}", result.as_ref().map_or_else(|error| error.state, |_| CommitState::Committed)), "error":result.as_ref().err().map(|error|error.error.to_string())}) | ||
| 230 | ); | ||
| 231 | result | ||
| 232 | } | ||
| 233 | } | ||
| 234 | |||
| 235 | fn tokens(text: &str) -> Option<Vec<&str>> { | ||
| 236 | let mut remaining = text.strip_prefix("Concurrent edits:")?; | ||
| 237 | let mut tokens = Vec::new(); | ||
| 238 | while !remaining.is_empty() { | ||
| 239 | let body = remaining.strip_prefix(" [w")?; | ||
| 240 | let (token, tail) = body.split_once(']')?; | ||
| 241 | let (actor, operation) = token.split_once(':')?; | ||
| 242 | if actor.is_empty() | ||
| 243 | || operation.is_empty() | ||
| 244 | || !actor | ||
| 245 | .bytes() | ||
| 246 | .chain(operation.bytes()) | ||
| 247 | .all(|b| b.is_ascii_digit()) | ||
| 248 | || tokens.contains(&token) | ||
| 249 | { | ||
| 250 | return None; | ||
| 251 | } | ||
| 252 | tokens.push(token); | ||
| 253 | remaining = tail; | ||
| 254 | } | ||
| 255 | Some(tokens) | ||
| 256 | } | ||
| 257 | |||
| 258 | // This owned workload explicitly resolves append conflicts after all retained tokens. | ||
| 259 | fn append_position(before: &str, current: &str, token: &str) -> Option<u32> { | ||
| 260 | let original = tokens(before)?; | ||
| 261 | let present = tokens(current)?; | ||
| 262 | let mut retained = present.iter(); | ||
| 263 | for token in original { | ||
| 264 | retained.find(|&&candidate| candidate == token)?; | ||
| 265 | } | ||
| 266 | if current.contains(token) { | ||
| 267 | return None; | ||
| 268 | } | ||
| 269 | u32::try_from(current.encode_utf16().count()).ok() | ||
| 270 | } | ||
| 271 | |||
| 272 | fn queue_document( | ||
| 273 | cache: &Replica, | ||
| 274 | actor: &str, | ||
| 275 | operation: usize, | ||
| 276 | parent: Option<ExGuid>, | ||
| 277 | deadline: Instant, | ||
| 278 | ) -> Result<(ExGuid, [u64; 2]), Box<dyn std::error::Error>> { | ||
| 279 | let source = cache.snapshot()?; | ||
| 280 | let store = Store::parse(&source)?; | ||
| 281 | let index = RevisionIndex::parse(&store)?; | ||
| 282 | let document = Document::parse(&index)?; | ||
| 283 | let (space, page) = document.pages()?[0]; | ||
| 284 | let text = format!("Document {actor}:{operation} 🦀"); | ||
| 285 | let insertion = if operation.is_multiple_of(2) { | ||
| 286 | let column: u32 = actor | ||
| 287 | .strip_prefix('w') | ||
| 288 | .ok_or("Missing writer number")? | ||
| 289 | .parse()?; | ||
| 290 | Insertion::outline( | ||
| 291 | page, | ||
| 292 | 144.0 + column as f32 * 240.0, | ||
| 293 | 144.0 + operation as f32 * 72.0, | ||
| 294 | &text, | ||
| 295 | "Offline document writer", | ||
| 296 | )? | ||
| 297 | } else { | ||
| 298 | Insertion::paragraph( | ||
| 299 | parent.ok_or("Missing prior outline")?, | ||
| 300 | None, | ||
| 301 | &text, | ||
| 302 | "Offline document writer", | ||
| 303 | )? | ||
| 304 | }; | ||
| 305 | let parent = if operation.is_multiple_of(2) { | ||
| 306 | insertion.object() | ||
| 307 | } else { | ||
| 308 | parent.unwrap() | ||
| 309 | }; | ||
| 310 | let range = 1..u32::try_from(text.encode_utf16().count())? - 2; | ||
| 311 | let attributes = [ | ||
| 312 | TextAttribute::Bold(true), | ||
| 313 | TextAttribute::FontSize(18.0 + (operation % 9) as f32), | ||
| 314 | TextAttribute::Color(Some([0x12, 0x34, 0x56])), | ||
| 315 | ]; | ||
| 316 | let mut ids = [0; 2]; | ||
| 317 | for (step, id) in ids.iter_mut().enumerate() { | ||
| 318 | loop { | ||
| 319 | if Instant::now() >= deadline { | ||
| 320 | return Err("Document queue timed out; cache retained".into()); | ||
| 321 | } | ||
| 322 | let source = cache.snapshot()?; | ||
| 323 | let started = now(); | ||
| 324 | let result = if step == 0 { | ||
| 325 | cache.insert(&source, space, &insertion) | ||
| 326 | } else { | ||
| 327 | cache.format( | ||
| 328 | &source, | ||
| 329 | space, | ||
| 330 | insertion.text_object(), | ||
| 331 | range.clone(), | ||
| 332 | &attributes, | ||
| 333 | ) | ||
| 334 | }; | ||
| 335 | match result { | ||
| 336 | Ok(Some(acknowledged)) => { | ||
| 337 | *id = acknowledged; | ||
| 338 | println!( | ||
| 339 | "{}", | ||
| 340 | json!({"event":"local_document_commit","id":acknowledged,"operation":operation,"kind":if step==0 {"insert"} else {"format"},"space":space.to_string(),"object":insertion.text_object().to_string(),"text":text,"insertion":if step==0 {Some(&insertion)} else {None},"range":[range.start,range.end],"attributes":attributes,"started_us":started,"finished_us":now()}) | ||
| 341 | ); | ||
| 342 | break; | ||
| 343 | } | ||
| 344 | Err(Error::Io(error)) if error.kind() == io::ErrorKind::ResourceBusy => {} | ||
| 345 | other => return Err(format!("Unexpected document queue result: {other:?}").into()), | ||
| 346 | } | ||
| 347 | } | ||
| 348 | } | ||
| 349 | Ok((parent, ids)) | ||
| 350 | } | ||
| 351 | |||
| 352 | fn main() -> Result<(), Box<dyn std::error::Error>> { | ||
| 353 | let args: Vec<_> = env::args().skip(1).collect(); | ||
| 354 | if args.len() != 7 | ||
| 355 | || !["read", "write"].contains(&args[0].as_str()) | ||
| 356 | || !args[2].bytes().all(|b| b.is_ascii_alphanumeric()) | ||
| 357 | { | ||
| 358 | return Err("Expected read|write FILE ACTOR OPERATIONS START_FILE STOP_FILE SEED".into()); | ||
| 359 | } | ||
| 360 | let documents = env::var_os("ONESTORE_OFFLINE_DOCUMENTS").is_some(); | ||
| 361 | let address = env::var("ONESTORE_SMB_LAB")?; | ||
| 362 | let share = env::var("ONESTORE_SMB_SHARE")?; | ||
| 363 | let initial = Client::connect( | ||
| 364 | &address, | ||
| 365 | &share, | ||
| 366 | Credentials::default(), | ||
| 367 | Duration::from_secs(5), | ||
| 368 | )?; | ||
| 369 | if args[0] == "read" { | ||
| 370 | return concurrent::run( | ||
| 371 | &args, | ||
| 372 | |path| initial.read(path, 256 * 1024 * 1024), | ||
| 373 | |_, _, _, _, _, _| unreachable!(), | ||
| 374 | ); | ||
| 375 | } | ||
| 376 | let timeout: u64 = env::var("ONESTORE_CLIENT_TIMEOUT_MS") | ||
| 377 | .unwrap_or_else(|_| "600000".into()) | ||
| 378 | .parse()?; | ||
| 379 | let deadline = Instant::now() | ||
| 380 | .checked_add(Duration::from_millis(timeout)) | ||
| 381 | .ok_or("Invalid timeout")?; | ||
| 382 | let operations: usize = args[3].parse()?; | ||
| 383 | let mut seed: u64 = args[6].parse()?; | ||
| 384 | if operations == 0 { | ||
| 385 | return Err("Expected positive operations".into()); | ||
| 386 | } | ||
| 387 | let source = initial.read(&args[1], 256 * 1024 * 1024)?; | ||
| 388 | drop(initial); | ||
| 389 | let cache_path = Path::new(&args[4]) | ||
| 390 | .parent() | ||
| 391 | .ok_or("Missing workload directory")? | ||
| 392 | .join(format!("{}.sqlite", args[2])); | ||
| 393 | let cache = Arc::new(Replica::create(&cache_path, &source)?); | ||
| 394 | println!( | ||
| 395 | "{}", | ||
| 396 | json!({"event":"ready", "pid":std::process::id(), "actor":args[2], "offline":true, "document_operations":documents}) | ||
| 397 | ); | ||
| 398 | while !Path::new(&args[4]).exists() { | ||
| 399 | if Instant::now() >= deadline { | ||
| 400 | return Err("Start barrier timed out".into()); | ||
| 401 | } | ||
| 402 | thread::sleep(Duration::from_millis(5)); | ||
| 403 | } | ||
| 404 | let path = args[1].clone(); | ||
| 405 | let outage = env::var_os("ONESTORE_OFFLINE_OUTAGE_DIR").map(PathBuf::from); | ||
| 406 | let pause = outage | ||
| 407 | .as_ref() | ||
| 408 | .map(|directory| Pause::Outage(directory.join(format!("offline-paused-{}", args[2])))) | ||
| 409 | .or_else(|| { | ||
| 410 | env::var_os("ONESTORE_OFFLINE_FORMAT_REPLY_DIR").map(|directory| { | ||
| 411 | Pause::FormatReply( | ||
| 412 | PathBuf::from(directory).join(format!("offline-paused-{}", args[2])), | ||
| 413 | ) | ||
| 414 | }) | ||
| 415 | }); | ||
| 416 | let (fatal_tx, fatal_rx) = std::sync::mpsc::channel(); | ||
| 417 | let worker = cache.start_sync(Duration::from_millis(50), move || { | ||
| 418 | let client = Client::connect(&address, &share, Credentials::default(), Duration::from_secs(5))?; | ||
| 419 | println!("{}", json!({"event":"transport_connected", "at_us":now()})); | ||
| 420 | Ok(Traced { remote: SmbRemote::new(client, &path, 256 * 1024 * 1024), before:None, pause:pause.clone(), documents }) | ||
| 421 | }, move |result| { | ||
| 422 | if let Err(error) = result { | ||
| 423 | println!("{}", json!({"event":"sync_error", "error":error.to_string(), "at_us":now()})); | ||
| 424 | if !matches!(error, Error::RemoteIo(_) | Error::Remote(_)) && !matches!(error, Error::Io(error) if error.kind() == io::ErrorKind::WouldBlock) { | ||
| 425 | let _ = fatal_tx.send(error.to_string()); | ||
| 426 | } | ||
| 427 | } | ||
| 428 | })?; | ||
| 429 | let mut ids = Vec::new(); | ||
| 430 | let mut document_ids = std::collections::BTreeSet::new(); | ||
| 431 | let mut document_parent = None; | ||
| 432 | let result = (|| -> Result<(), Box<dyn std::error::Error>> { | ||
| 433 | let mut generated = 0; | ||
| 434 | let mut received = 0; | ||
| 435 | loop { | ||
| 436 | match fatal_rx.try_recv() { | ||
| 437 | Ok(error) => return Err(error.into()), | ||
| 438 | Err(std::sync::mpsc::TryRecvError::Disconnected) => { | ||
| 439 | return Err("Worker exited before completion".into()); | ||
| 440 | } | ||
| 441 | Err(std::sync::mpsc::TryRecvError::Empty) => {} | ||
| 442 | } | ||
| 443 | while received < ids.len() { | ||
| 444 | match cache.status(ids[received])? { | ||
| 445 | Some(EditStatus::Published { revision }) => { | ||
| 446 | println!( | ||
| 447 | "{}", | ||
| 448 | json!({"event":if document_ids.contains(&ids[received]) {"document_receipt"} else {"remote_receipt"}, "id":ids[received], "revision":revision.to_string(), "at_us":now()}) | ||
| 449 | ); | ||
| 450 | received += 1; | ||
| 451 | } | ||
| 452 | Some(_) => break, | ||
| 453 | None => return Err("Local intent disappeared".into()), | ||
| 454 | } | ||
| 455 | } | ||
| 456 | if Instant::now() >= deadline { | ||
| 457 | return Err("Offline workload timed out; cache retained".into()); | ||
| 458 | } | ||
| 459 | let pending = cache.pending()?; | ||
| 460 | let capacity = if outage | ||
| 461 | .as_ref() | ||
| 462 | .is_some_and(|directory| !directory.join("offline-outage-down").exists()) | ||
| 463 | { | ||
| 464 | 1 | ||
| 465 | } else if documents && outage.is_some() { | ||
| 466 | 24 | ||
| 467 | } else { | ||
| 468 | 8 | ||
| 469 | }; | ||
| 470 | if generated < operations && pending.len() < capacity { | ||
| 471 | let source = cache.snapshot()?; | ||
| 472 | let target = view(&source)?; | ||
| 473 | let at = u32::try_from(target.text.encode_utf16().count())?; | ||
| 474 | let token = format!(" [{}:{}]", args[2], generated); | ||
| 475 | let started = now(); | ||
| 476 | match cache.edit_text(&source, target.space, target.object, at..at, &token) { | ||
| 477 | Ok(Some(id)) => { | ||
| 478 | println!( | ||
| 479 | "{}", | ||
| 480 | json!({"event":"local_commit", "id":id, "operation":generated, "space":target.space.to_string(), "object":target.object.to_string(), "before":target.text, "token":token, "started_us":started, "finished_us":now()}) | ||
| 481 | ); | ||
| 482 | ids.push(id); | ||
| 483 | if documents { | ||
| 484 | let (parent, added) = queue_document( | ||
| 485 | &cache, | ||
| 486 | &args[2], | ||
| 487 | generated, | ||
| 488 | document_parent, | ||
| 489 | deadline, | ||
| 490 | )?; | ||
| 491 | document_parent = Some(parent); | ||
| 492 | document_ids.extend(added); | ||
| 493 | ids.extend(added); | ||
| 494 | } | ||
| 495 | generated += 1; | ||
| 496 | } | ||
| 497 | Err(Error::Io(error)) if error.kind() == io::ErrorKind::ResourceBusy => {} | ||
| 498 | other => return Err(format!("Unexpected local result: {other:?}").into()), | ||
| 499 | } | ||
| 500 | } | ||
| 501 | if let Some(intent) = pending.first() | ||
| 502 | && matches!(cache.status(intent.id)?, Some(EditStatus::Conflict(_))) | ||
| 503 | { | ||
| 504 | let local = cache.snapshot()?; | ||
| 505 | let remote = cache.remote_snapshot()?; | ||
| 506 | let current = view(&remote)?; | ||
| 507 | let onestore_offline::Operation::Text(edit) = &intent.operation else { | ||
| 508 | return Err("Expected text probe intents".into()); | ||
| 509 | }; | ||
| 510 | let at = append_position(&edit.before, ¤t.text, &edit.replacement) | ||
| 511 | .ok_or("Append model disagrees with retained history")?; | ||
| 512 | match cache.rebase_conflict(intent.id, &local, &remote, at..at) { | ||
| 513 | Ok(()) => println!( | ||
| 514 | "{}", | ||
| 515 | json!({"event":"reviewed_append", "id":intent.id, "before":edit.before, "remote":current.text, "token":edit.replacement, "at_us":now()}) | ||
| 516 | ), | ||
| 517 | Err(Error::Io(error)) | ||
| 518 | if [ | ||
| 519 | io::ErrorKind::ResourceBusy, | ||
| 520 | io::ErrorKind::WouldBlock, | ||
| 521 | io::ErrorKind::InvalidInput, | ||
| 522 | ] | ||
| 523 | .contains(&error.kind()) => {} | ||
| 524 | Err(error) => return Err(error.into()), | ||
| 525 | } | ||
| 526 | } | ||
| 527 | if generated == operations && received == ids.len() { | ||
| 528 | if !cache.pending()?.is_empty() { | ||
| 529 | return Err("Acknowledged queue did not drain".into()); | ||
| 530 | } | ||
| 531 | break; | ||
| 532 | } | ||
| 533 | seed = seed | ||
| 534 | .wrapping_mul(6364136223846793005) | ||
| 535 | .wrapping_add(1442695040888963407); | ||
| 536 | thread::sleep(Duration::from_millis(1 + (seed >> 32) % 7)); | ||
| 537 | } | ||
| 538 | Ok(()) | ||
| 539 | })(); | ||
| 540 | let stopped = worker.stop(); | ||
| 541 | result?; | ||
| 542 | stopped?; | ||
| 543 | drop(cache); | ||
| 544 | let reopened = Replica::open(&cache_path)?; | ||
| 545 | if !reopened.pending()?.is_empty() { | ||
| 546 | return Err("Pending edits reappeared after reopen".into()); | ||
| 547 | } | ||
| 548 | for id in ids { | ||
| 549 | let Some(EditStatus::Published { revision }) = reopened.status(id)? else { | ||
| 550 | return Err("Receipt did not survive reopen".into()); | ||
| 551 | }; | ||
| 552 | println!( | ||
| 553 | "{}", | ||
| 554 | json!({"event":if document_ids.contains(&id) {"reopened_document_receipt"} else {"reopened_receipt"}, "id":id, "revision":revision.to_string()}) | ||
| 555 | ); | ||
| 556 | } | ||
| 557 | println!( | ||
| 558 | "{}", | ||
| 559 | json!({"event":"done", "operations":operations, "at_us":now()}) | ||
| 560 | ); | ||
| 561 | Ok(()) | ||
| 562 | } | ||
| 563 | |||
| 564 | #[test] | ||
| 565 | fn append_review_requires_a_unique_ordered_history_and_an_absent_new_token() { | ||
| 566 | assert_eq!( | ||
| 567 | append_position( | ||
| 568 | "Concurrent edits: [w0:0]", | ||
| 569 | "Concurrent edits: [w1:0] [w0:0]", | ||
| 570 | " [w0:1]" | ||
| 571 | ), | ||
| 572 | Some(31) | ||
| 573 | ); | ||
| 574 | for current in [ | ||
| 575 | "Concurrent edits:", | ||
| 576 | "Concurrent edits: [w0:0] [w0:0]", | ||
| 577 | "Concurrent edits: [w0:1] [w0:0]", | ||
| 578 | "Concurrent edits: changed [w0:0]", | ||
| 579 | ] { | ||
| 580 | assert_eq!( | ||
| 581 | append_position("Concurrent edits: [w0:0]", current, " [w0:1]"), | ||
| 582 | None | ||
| 583 | ); | ||
| 584 | } | ||
| 585 | assert_eq!( | ||
| 586 | append_position( | ||
| 587 | "Concurrent edits: [w0:0] [w1:0]", | ||
| 588 | "Concurrent edits: [w1:0] [w0:0]", | ||
| 589 | " [w0:1]" | ||
| 590 | ), | ||
| 591 | None | ||
| 592 | ); | ||
| 593 | } | ||
crates/onestore-offline/examples/support/view.rs created+57
| ... | @@ -0,0 +1,57 @@ | ||
| 1 | use onestore::{ | ||
| 2 | ExGuid, RevisionIndex, Store, | ||
| 3 | document::{Document, Kind}, | ||
| 4 | }; | ||
| 5 | |||
| 6 | pub struct View { | ||
| 7 | pub space: ExGuid, | ||
| 8 | pub object: ExGuid, | ||
| 9 | pub revision: ExGuid, | ||
| 10 | pub text: String, | ||
| 11 | } | ||
| 12 | |||
| 13 | pub fn view(bytes: &[u8]) -> Result<View, Box<dyn std::error::Error>> { | ||
| 14 | let store = Store::parse(bytes)?; | ||
| 15 | if !store.checksum_mismatches.is_empty() { | ||
| 16 | return Err("Transaction checksum damage".into()); | ||
| 17 | } | ||
| 18 | let index = RevisionIndex::parse(&store)?; | ||
| 19 | index.validate_current()?; | ||
| 20 | let document = Document::parse(&index)?; | ||
| 21 | let mut found = Vec::new(); | ||
| 22 | for (sid, page) in document.pages()? { | ||
| 23 | let space = &document.spaces[&sid]; | ||
| 24 | let rid = space.contexts[&ExGuid::default()]; | ||
| 25 | let revision = &space.revisions[&rid]; | ||
| 26 | let mut pending = vec![page]; | ||
| 27 | let mut seen = std::collections::BTreeSet::new(); | ||
| 28 | while let Some(oid) = pending.pop() { | ||
| 29 | if !seen.insert(oid) { | ||
| 30 | continue; | ||
| 31 | } | ||
| 32 | let node = &revision.nodes[&oid]; | ||
| 33 | pending.extend( | ||
| 34 | node.children | ||
| 35 | .iter() | ||
| 36 | .chain(&node.content) | ||
| 37 | .chain(&node.structure) | ||
| 38 | .copied(), | ||
| 39 | ); | ||
| 40 | if let Kind::RichText { text, .. } = &node.kind | ||
| 41 | && text.starts_with("Concurrent edits:") | ||
| 42 | { | ||
| 43 | revision.text_runs(oid)?; | ||
| 44 | found.push(View { | ||
| 45 | space: sid, | ||
| 46 | object: oid, | ||
| 47 | revision: rid, | ||
| 48 | text: text.clone(), | ||
| 49 | }); | ||
| 50 | } | ||
| 51 | } | ||
| 52 | } | ||
| 53 | if found.len() != 1 { | ||
| 54 | return Err("Expected one concurrent-edit paragraph".into()); | ||
| 55 | } | ||
| 56 | Ok(found.pop().unwrap()) | ||
| 57 | } | ||
crates/onestore-offline/src/formatting.rs created+204
| ... | @@ -0,0 +1,204 @@ | ||
| 1 | use super::*; | ||
| 2 | use onestore::TextAttribute; | ||
| 3 | use serde::{Deserialize, Serialize}; | ||
| 4 | use serde_json::{Value, json}; | ||
| 5 | use std::collections::BTreeMap; | ||
| 6 | |||
| 7 | #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] | ||
| 8 | #[serde(deny_unknown_fields)] | ||
| 9 | struct Span { | ||
| 10 | end: u32, | ||
| 11 | values: Vec<Value>, | ||
| 12 | } | ||
| 13 | |||
| 14 | /// A formatting intent and the text/attribute values observed before local publication. | ||
| 15 | #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] | ||
| 16 | #[serde(deny_unknown_fields)] | ||
| 17 | pub struct FormatEdit { | ||
| 18 | pub object: ExGuid, | ||
| 19 | pub before: String, | ||
| 20 | pub range: Range<u32>, | ||
| 21 | pub attributes: Vec<TextAttribute>, | ||
| 22 | observed: Vec<Span>, | ||
| 23 | } | ||
| 24 | |||
| 25 | fn fields(attributes: &[TextAttribute]) -> BTreeMap<&'static str, Value> { | ||
| 26 | let mut fields = BTreeMap::new(); | ||
| 27 | for attribute in attributes { | ||
| 28 | let (name, value) = match attribute { | ||
| 29 | TextAttribute::Bold(v) => ("bold", json!(v)), | ||
| 30 | TextAttribute::Italic(v) => ("italic", json!(v)), | ||
| 31 | TextAttribute::Underline(v) => ("underline", json!(v)), | ||
| 32 | TextAttribute::Strike(v) => ("strike", json!(v)), | ||
| 33 | TextAttribute::Superscript(v) => { | ||
| 34 | if *v { | ||
| 35 | fields.insert("subscript", json!(false)); | ||
| 36 | } | ||
| 37 | ("superscript", json!(v)) | ||
| 38 | } | ||
| 39 | TextAttribute::Subscript(v) => { | ||
| 40 | if *v { | ||
| 41 | fields.insert("superscript", json!(false)); | ||
| 42 | } | ||
| 43 | ("subscript", json!(v)) | ||
| 44 | } | ||
| 45 | TextAttribute::Font(v) => ("font", json!(v)), | ||
| 46 | TextAttribute::FontSize(v) => ("font_size", json!(v)), | ||
| 47 | TextAttribute::Color(v) | TextAttribute::Highlight(v) => ( | ||
| 48 | if matches!(attribute, TextAttribute::Color(_)) { | ||
| 49 | "color" | ||
| 50 | } else { | ||
| 51 | "highlight" | ||
| 52 | }, | ||
| 53 | json!(v.map_or(0xff000000, |[r, g, b]| u32::from_le_bytes([r, g, b, 0]))), | ||
| 54 | ), | ||
| 55 | }; | ||
| 56 | fields.insert(name, value); | ||
| 57 | } | ||
| 58 | fields | ||
| 59 | } | ||
| 60 | |||
| 61 | fn observe( | ||
| 62 | source: &[u8], | ||
| 63 | space: ExGuid, | ||
| 64 | object: ExGuid, | ||
| 65 | range: Range<u32>, | ||
| 66 | fields: &BTreeMap<&str, Value>, | ||
| 67 | ) -> Result<Vec<Span>> { | ||
| 68 | let store = Store::parse(source)?; | ||
| 69 | let index = RevisionIndex::parse(&store)?; | ||
| 70 | let document = Document::parse(&index)?; | ||
| 71 | let space = &document.spaces[&space]; | ||
| 72 | let revision = &space.revisions[&space.contexts[&ExGuid::default()]]; | ||
| 73 | let mut spans: Vec<Span> = Vec::new(); | ||
| 74 | let mut start = 0; | ||
| 75 | for run in revision.text_runs(object)? { | ||
| 76 | let end = | ||
| 77 | start + u32::try_from(run.text.encode_utf16().count()).map_err(io::Error::other)?; | ||
| 78 | if (start < range.end && range.start < end) || (start == 0 && end == 0 && range == (0..0)) { | ||
| 79 | let format = serde_json::to_value(run.format).map_err(io::Error::other)?; | ||
| 80 | let values = fields | ||
| 81 | .iter() | ||
| 82 | .map(|(name, desired)| { | ||
| 83 | let value = &format[*name]; | ||
| 84 | if value.is_null() && desired.is_boolean() { | ||
| 85 | json!(false) | ||
| 86 | } else if value.is_null() && matches!(*name, "color" | "highlight") { | ||
| 87 | json!(0xff000000_u32) | ||
| 88 | } else { | ||
| 89 | value.clone() | ||
| 90 | } | ||
| 91 | }) | ||
| 92 | .collect::<Vec<_>>(); | ||
| 93 | let end = end.min(range.end) - range.start; | ||
| 94 | if let Some(last) = spans.last_mut().filter(|s| s.values == values) { | ||
| 95 | last.end = end; | ||
| 96 | } else { | ||
| 97 | spans.push(Span { end, values }); | ||
| 98 | } | ||
| 99 | } | ||
| 100 | start = end; | ||
| 101 | } | ||
| 102 | Ok(spans) | ||
| 103 | } | ||
| 104 | |||
| 105 | impl Replica { | ||
| 106 | /// Durably records a visual-formatting change and its observed attribute values. | ||
| 107 | /// Independent remote attributes can merge; competing values preserve a conflict. | ||
| 108 | pub fn format( | ||
| 109 | &self, | ||
| 110 | source: &[u8], | ||
| 111 | space: ExGuid, | ||
| 112 | object: ExGuid, | ||
| 113 | range: Range<u32>, | ||
| 114 | attributes: &[TextAttribute], | ||
| 115 | ) -> Result<Option<u64>> { | ||
| 116 | let (edit, prepared) = FormatEdit::capture(source, space, object, range, attributes)?; | ||
| 117 | self.record(source, space, Operation::Format(edit), &prepared) | ||
| 118 | } | ||
| 119 | } | ||
| 120 | |||
| 121 | impl FormatEdit { | ||
| 122 | pub(crate) fn capture<'a>( | ||
| 123 | source: &'a [u8], | ||
| 124 | space: ExGuid, | ||
| 125 | object: ExGuid, | ||
| 126 | range: Range<u32>, | ||
| 127 | attributes: &[TextAttribute], | ||
| 128 | ) -> Result<(Self, PreparedEdit<'a>)> { | ||
| 129 | let prepared = PreparedEdit::format(source, space, object, range.clone(), attributes)?; | ||
| 130 | let before = paragraph(source, space, object)?.ok_or_else(|| { | ||
| 131 | io::Error::new( | ||
| 132 | io::ErrorKind::InvalidData, | ||
| 133 | "Prepared formatting has no text target", | ||
| 134 | ) | ||
| 135 | })?; | ||
| 136 | let observed = observe(source, space, object, range.clone(), &fields(attributes))?; | ||
| 137 | let edit = Self { | ||
| 138 | object, | ||
| 139 | before, | ||
| 140 | range, | ||
| 141 | attributes: attributes.to_vec(), | ||
| 142 | observed, | ||
| 143 | }; | ||
| 144 | Ok((edit, prepared)) | ||
| 145 | } | ||
| 146 | |||
| 147 | pub(crate) fn prepare<'a>( | ||
| 148 | &self, | ||
| 149 | snapshot: &'a [u8], | ||
| 150 | space: ExGuid, | ||
| 151 | ) -> Result<std::result::Result<PreparedEdit<'a>, ConflictKind>> { | ||
| 152 | let Some(text) = paragraph(snapshot, space, self.object)? else { | ||
| 153 | return Ok(Err(ConflictKind::TargetUnavailable)); | ||
| 154 | }; | ||
| 155 | let Some(range) = rebase::rebase(&self.before, &text, self.range.clone()) else { | ||
| 156 | return Ok(Err(ConflictKind::TextChanged)); | ||
| 157 | }; | ||
| 158 | let prepared = match PreparedEdit::format( | ||
| 159 | snapshot, | ||
| 160 | space, | ||
| 161 | self.object, | ||
| 162 | range.clone(), | ||
| 163 | &self.attributes, | ||
| 164 | ) { | ||
| 165 | Ok(prepared) => prepared, | ||
| 166 | Err(_) => return Ok(Err(ConflictKind::UnsupportedEdit)), | ||
| 167 | }; | ||
| 168 | let desired = fields(&self.attributes); | ||
| 169 | let observed = observe(snapshot, space, self.object, range.clone(), &desired)?; | ||
| 170 | let wanted: Vec<_> = desired.values().collect(); | ||
| 171 | for spans in [&self.observed, &observed] { | ||
| 172 | if spans.is_empty() | ||
| 173 | || (!range.is_empty() && spans[0].end == 0) | ||
| 174 | || spans.last().unwrap().end != range.end - range.start | ||
| 175 | || spans.windows(2).any(|s| s[0].end >= s[1].end) | ||
| 176 | || spans.iter().any(|s| s.values.len() != wanted.len()) | ||
| 177 | { | ||
| 178 | return Ok(Err(ConflictKind::UnsupportedEdit)); | ||
| 179 | } | ||
| 180 | } | ||
| 181 | let (mut before, mut after) = (0, 0); | ||
| 182 | while before < self.observed.len() && after < observed.len() { | ||
| 183 | let old = &self.observed[before]; | ||
| 184 | let current = &observed[after]; | ||
| 185 | if old | ||
| 186 | .values | ||
| 187 | .iter() | ||
| 188 | .zip(¤t.values) | ||
| 189 | .zip(&wanted) | ||
| 190 | .any(|((old, new), wanted)| new != old && new != *wanted) | ||
| 191 | { | ||
| 192 | return Ok(Err(ConflictKind::FormattingChanged)); | ||
| 193 | } | ||
| 194 | let end = old.end.min(current.end); | ||
| 195 | if old.end == end { | ||
| 196 | before += 1; | ||
| 197 | } | ||
| 198 | if current.end == end { | ||
| 199 | after += 1; | ||
| 200 | } | ||
| 201 | } | ||
| 202 | Ok(Ok(prepared)) | ||
| 203 | } | ||
| 204 | } | ||
crates/onestore-offline/src/lib.rs created+362
| ... | @@ -0,0 +1,362 @@ | ||
| 1 | #![forbid(unsafe_code)] | ||
| 2 | #![doc = include_str!("../README.md")] | ||
| 3 | |||
| 4 | use onestore::{ | ||
| 5 | ExGuid, Insertion, PreparedEdit, RevisionIndex, Store, | ||
| 6 | document::{Document, Kind}, | ||
| 7 | }; | ||
| 8 | use rusqlite::{Connection, OpenFlags, TransactionBehavior, params}; | ||
| 9 | use std::{fs::OpenOptions, io, ops::Range, path::Path, sync::Mutex, time::Duration}; | ||
| 10 | |||
| 11 | mod formatting; | ||
| 12 | mod rebase; | ||
| 13 | mod schema; | ||
| 14 | pub use formatting::FormatEdit; | ||
| 15 | mod sync; | ||
| 16 | pub use sync::{ConflictKind, EditStatus, Remote}; | ||
| 17 | mod worker; | ||
| 18 | pub use worker::SyncWorker; | ||
| 19 | #[cfg(feature = "smb")] | ||
| 20 | mod smb; | ||
| 21 | #[cfg(feature = "smb")] | ||
| 22 | pub use smb::SmbRemote; | ||
| 23 | |||
| 24 | #[derive(Debug, thiserror::Error)] | ||
| 25 | pub enum Error { | ||
| 26 | #[error(transparent)] | ||
| 27 | Database(#[from] rusqlite::Error), | ||
| 28 | #[error(transparent)] | ||
| 29 | Io(#[from] io::Error), | ||
| 30 | #[error(transparent)] | ||
| 31 | Document(#[from] onestore::Error), | ||
| 32 | #[error(transparent)] | ||
| 33 | Remote(#[from] onestore::CommitError), | ||
| 34 | #[error(transparent)] | ||
| 35 | RemoteIo(io::Error), | ||
| 36 | } | ||
| 37 | |||
| 38 | type Result<T> = std::result::Result<T, Error>; | ||
| 39 | |||
| 40 | const APPLICATION_ID: u32 = 0x4f4e454f; | ||
| 41 | const SCHEMA_VERSION: u32 = 4; | ||
| 42 | |||
| 43 | /// Text and its observed precondition, retained across cache reopen and rebasing. | ||
| 44 | #[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)] | ||
| 45 | #[serde(deny_unknown_fields)] | ||
| 46 | pub struct TextEdit { | ||
| 47 | pub object: ExGuid, | ||
| 48 | pub before: String, | ||
| 49 | pub range: Range<u32>, | ||
| 50 | pub replacement: String, | ||
| 51 | } | ||
| 52 | |||
| 53 | #[derive(Debug, Clone, PartialEq, serde::Serialize, serde::Deserialize)] | ||
| 54 | #[serde(deny_unknown_fields)] | ||
| 55 | pub enum Operation { | ||
| 56 | Text(TextEdit), | ||
| 57 | Insert(Insertion), | ||
| 58 | Format(FormatEdit), | ||
| 59 | } | ||
| 60 | |||
| 61 | /// A locally acknowledged intent; its ID remains stable across cache reopen. | ||
| 62 | #[derive(Debug, Clone, PartialEq)] | ||
| 63 | pub struct PendingEdit { | ||
| 64 | pub id: u64, | ||
| 65 | pub space: ExGuid, | ||
| 66 | pub operation: Operation, | ||
| 67 | } | ||
| 68 | |||
| 69 | /// Owns one local cache. Share this handle between threads; a second open fails busy. | ||
| 70 | /// SQLite's exclusive connection retains ownership between local transactions. | ||
| 71 | pub struct Replica { | ||
| 72 | connection: Mutex<Connection>, | ||
| 73 | synchronization: Mutex<()>, | ||
| 74 | worker: Mutex<std::sync::Weak<worker::Signal>>, | ||
| 75 | } | ||
| 76 | |||
| 77 | impl Replica { | ||
| 78 | /// Seeds a new cache from a validated notebook image, refusing any existing path. | ||
| 79 | /// An initialization error preserves the created file for inspection. | ||
| 80 | pub fn create(path: impl AsRef<Path>, source: &[u8]) -> Result<Self> { | ||
| 81 | validate(source)?; | ||
| 82 | let mut options = OpenOptions::new(); | ||
| 83 | options.read(true).write(true).create_new(true); | ||
| 84 | #[cfg(unix)] | ||
| 85 | { | ||
| 86 | use std::os::unix::fs::OpenOptionsExt; | ||
| 87 | options.mode(0o600); | ||
| 88 | } | ||
| 89 | drop(options.open(path.as_ref())?); | ||
| 90 | Self::connect(path.as_ref(), Some(source)) | ||
| 91 | } | ||
| 92 | |||
| 93 | /// Reopens an existing cache and its durable pending edits without network access. | ||
| 94 | /// Unrecognized databases and unsupported journal modes are rejected without conversion. | ||
| 95 | pub fn open(path: impl AsRef<Path>) -> Result<Self> { | ||
| 96 | Self::connect(path.as_ref(), None) | ||
| 97 | } | ||
| 98 | |||
| 99 | fn connect(path: &Path, source: Option<&[u8]>) -> Result<Self> { | ||
| 100 | let mut connection = Connection::open_with_flags(path, OpenFlags::SQLITE_OPEN_READ_WRITE)?; | ||
| 101 | connection.busy_timeout(Duration::ZERO)?; | ||
| 102 | connection.execute_batch( | ||
| 103 | "PRAGMA locking_mode=EXCLUSIVE; PRAGMA synchronous=EXTRA; PRAGMA fullfsync=ON; PRAGMA foreign_keys=ON;", | ||
| 104 | )?; | ||
| 105 | for (name, expected) in [("locking_mode", "exclusive"), ("journal_mode", "delete")] { | ||
| 106 | let actual: String = connection.pragma_query_value(None, name, |row| row.get(0))?; | ||
| 107 | if actual != expected { | ||
| 108 | return Err(io::Error::new( | ||
| 109 | io::ErrorKind::InvalidData, | ||
| 110 | "Unsupported cache locking or journal mode", | ||
| 111 | ) | ||
| 112 | .into()); | ||
| 113 | } | ||
| 114 | } | ||
| 115 | for (name, expected) in [("synchronous", 3), ("fullfsync", 1), ("foreign_keys", 1)] { | ||
| 116 | let actual: i64 = connection.pragma_query_value(None, name, |row| row.get(0))?; | ||
| 117 | if actual != expected { | ||
| 118 | return Err(io::Error::new( | ||
| 119 | io::ErrorKind::Unsupported, | ||
| 120 | "Required cache synchronization is unavailable", | ||
| 121 | ) | ||
| 122 | .into()); | ||
| 123 | } | ||
| 124 | } | ||
| 125 | let transaction = connection.transaction_with_behavior(TransactionBehavior::Exclusive)?; | ||
| 126 | let application: u32 = | ||
| 127 | transaction.pragma_query_value(None, "application_id", |row| row.get(0))?; | ||
| 128 | let version: u32 = | ||
| 129 | transaction.pragma_query_value(None, "user_version", |row| row.get(0))?; | ||
| 130 | if let Some(source) = source { | ||
| 131 | let tables: i64 = | ||
| 132 | transaction | ||
| 133 | .query_row("SELECT count(*) FROM sqlite_schema", [], |row| row.get(0))?; | ||
| 134 | if application != 0 || version != 0 || tables != 0 { | ||
| 135 | return Err(io::Error::new( | ||
| 136 | io::ErrorKind::AlreadyExists, | ||
| 137 | "Cache initialization found an existing database", | ||
| 138 | ) | ||
| 139 | .into()); | ||
| 140 | } | ||
| 141 | transaction.pragma_update(None, "application_id", APPLICATION_ID)?; | ||
| 142 | transaction.pragma_update(None, "user_version", SCHEMA_VERSION)?; | ||
| 143 | transaction.execute_batch( | ||
| 144 | " | ||
| 145 | CREATE TABLE replica ( | ||
| 146 | id INTEGER PRIMARY KEY CHECK(id=1), | ||
| 147 | base BLOB NOT NULL, | ||
| 148 | working BLOB NOT NULL | ||
| 149 | ) STRICT; | ||
| 150 | ", | ||
| 151 | )?; | ||
| 152 | schema::create(&transaction)?; | ||
| 153 | transaction.execute("INSERT INTO replica VALUES (1, ?1, ?1)", [source])?; | ||
| 154 | } else { | ||
| 155 | if application != APPLICATION_ID || !(1..=SCHEMA_VERSION).contains(&version) { | ||
| 156 | return Err(io::Error::new( | ||
| 157 | io::ErrorKind::InvalidData, | ||
| 158 | "Unrecognized cache or unsupported schema version", | ||
| 159 | ) | ||
| 160 | .into()); | ||
| 161 | } | ||
| 162 | let integrity: String = | ||
| 163 | transaction.query_row("PRAGMA quick_check", [], |row| row.get(0))?; | ||
| 164 | if integrity != "ok" { | ||
| 165 | return Err(io::Error::new( | ||
| 166 | io::ErrorKind::InvalidData, | ||
| 167 | "Cache integrity check failed", | ||
| 168 | ) | ||
| 169 | .into()); | ||
| 170 | } | ||
| 171 | let (base, working): (Vec<u8>, Vec<u8>) = transaction.query_row( | ||
| 172 | "SELECT base, working FROM replica WHERE id=1", | ||
| 173 | [], | ||
| 174 | |row| Ok((row.get(0)?, row.get(1)?)), | ||
| 175 | )?; | ||
| 176 | if validate(&base)? != validate(&working)? { | ||
| 177 | return Err(io::Error::new( | ||
| 178 | io::ErrorKind::InvalidData, | ||
| 179 | "Cache images belong to different documents", | ||
| 180 | ) | ||
| 181 | .into()); | ||
| 182 | } | ||
| 183 | if version < SCHEMA_VERSION { | ||
| 184 | schema::migrate(&transaction, version)?; | ||
| 185 | transaction.pragma_update(None, "user_version", SCHEMA_VERSION)?; | ||
| 186 | } | ||
| 187 | pending(&transaction)?; | ||
| 188 | } | ||
| 189 | transaction.commit()?; | ||
| 190 | Ok(Self { | ||
| 191 | connection: Mutex::new(connection), | ||
| 192 | synchronization: Mutex::new(()), | ||
| 193 | worker: Mutex::new(std::sync::Weak::new()), | ||
| 194 | }) | ||
| 195 | } | ||
| 196 | |||
| 197 | /// Returns the latest complete locally committed image, including pending edits. | ||
| 198 | pub fn snapshot(&self) -> Result<Vec<u8>> { | ||
| 199 | let connection = self | ||
| 200 | .connection | ||
| 201 | .lock() | ||
| 202 | .map_err(|_| io::Error::other("Cache owner panicked"))?; | ||
| 203 | Ok( | ||
| 204 | connection.query_row("SELECT working FROM replica WHERE id=1", [], |row| { | ||
| 205 | row.get(0) | ||
| 206 | })?, | ||
| 207 | ) | ||
| 208 | } | ||
| 209 | |||
| 210 | pub fn pending(&self) -> Result<Vec<PendingEdit>> { | ||
| 211 | let connection = self | ||
| 212 | .connection | ||
| 213 | .lock() | ||
| 214 | .map_err(|_| io::Error::other("Cache owner panicked"))?; | ||
| 215 | pending(&connection) | ||
| 216 | } | ||
| 217 | |||
| 218 | /// Atomically records an intent and its resulting local image; returns its durable ID. | ||
| 219 | /// Unchanged text returns `None`. A stale image returns `Io(ResourceBusy)`. | ||
| 220 | /// After a database error, reopen and inspect the cache before retrying the edit. | ||
| 221 | pub fn edit_text( | ||
| 222 | &self, | ||
| 223 | source: &[u8], | ||
| 224 | space: ExGuid, | ||
| 225 | object: ExGuid, | ||
| 226 | range: Range<u32>, | ||
| 227 | replacement: &str, | ||
| 228 | ) -> Result<Option<u64>> { | ||
| 229 | let edit = PreparedEdit::text(source, space, object, range.clone(), replacement)?; | ||
| 230 | let before = paragraph(source, space, object)?.ok_or_else(|| { | ||
| 231 | io::Error::new( | ||
| 232 | io::ErrorKind::InvalidData, | ||
| 233 | "Prepared edit has no text target", | ||
| 234 | ) | ||
| 235 | })?; | ||
| 236 | self.record( | ||
| 237 | source, | ||
| 238 | space, | ||
| 239 | Operation::Text(TextEdit { | ||
| 240 | object, | ||
| 241 | before, | ||
| 242 | range, | ||
| 243 | replacement: replacement.to_owned(), | ||
| 244 | }), | ||
| 245 | &edit, | ||
| 246 | ) | ||
| 247 | } | ||
| 248 | |||
| 249 | /// Durably queues a validated insertion with its stable object identities. | ||
| 250 | /// Uses the same snapshot and local-acknowledgement contract as `edit_text`. | ||
| 251 | pub fn insert( | ||
| 252 | &self, | ||
| 253 | source: &[u8], | ||
| 254 | space: ExGuid, | ||
| 255 | insertion: &Insertion, | ||
| 256 | ) -> Result<Option<u64>> { | ||
| 257 | let edit = PreparedEdit::insert(source, space, insertion)?; | ||
| 258 | self.record(source, space, Operation::Insert(insertion.clone()), &edit) | ||
| 259 | } | ||
| 260 | |||
| 261 | fn record( | ||
| 262 | &self, | ||
| 263 | source: &[u8], | ||
| 264 | space: ExGuid, | ||
| 265 | operation: Operation, | ||
| 266 | edit: &PreparedEdit<'_>, | ||
| 267 | ) -> Result<Option<u64>> { | ||
| 268 | let mut connection = self | ||
| 269 | .connection | ||
| 270 | .lock() | ||
| 271 | .map_err(|_| io::Error::other("Cache owner panicked"))?; | ||
| 272 | let transaction = connection.transaction_with_behavior(TransactionBehavior::Immediate)?; | ||
| 273 | let current: Vec<u8> = | ||
| 274 | transaction.query_row("SELECT working FROM replica WHERE id=1", [], |row| { | ||
| 275 | row.get(0) | ||
| 276 | })?; | ||
| 277 | if current != source { | ||
| 278 | return Err(io::Error::new( | ||
| 279 | io::ErrorKind::ResourceBusy, | ||
| 280 | "The local snapshot changed before this edit", | ||
| 281 | ) | ||
| 282 | .into()); | ||
| 283 | } | ||
| 284 | if edit.as_bytes() == source { | ||
| 285 | return Ok(None); | ||
| 286 | } | ||
| 287 | transaction.execute( | ||
| 288 | "INSERT INTO edits(space, operation) VALUES (?1, ?2)", | ||
| 289 | params![ | ||
| 290 | space.to_string(), | ||
| 291 | serde_json::to_string(&operation).map_err(io::Error::other)? | ||
| 292 | ], | ||
| 293 | )?; | ||
| 294 | let id = u64::try_from(transaction.last_insert_rowid()).map_err(io::Error::other)?; | ||
| 295 | transaction.execute( | ||
| 296 | "UPDATE replica SET working=?1 WHERE id=1", | ||
| 297 | [edit.as_bytes()], | ||
| 298 | )?; | ||
| 299 | transaction.commit()?; | ||
| 300 | drop(connection); | ||
| 301 | self.wake_sync(); | ||
| 302 | Ok(Some(id)) | ||
| 303 | } | ||
| 304 | |||
| 305 | fn wake_sync(&self) { | ||
| 306 | if let Ok(worker) = self.worker.lock() | ||
| 307 | && let Some(worker) = worker.upgrade() | ||
| 308 | { | ||
| 309 | worker.wake(); | ||
| 310 | } | ||
| 311 | } | ||
| 312 | } | ||
| 313 | |||
| 314 | fn paragraph(source: &[u8], space: ExGuid, object: ExGuid) -> Result<Option<String>> { | ||
| 315 | let store = Store::parse(source)?; | ||
| 316 | let index = RevisionIndex::parse(&store)?; | ||
| 317 | let document = Document::parse(&index)?; | ||
| 318 | let node = document | ||
| 319 | .spaces | ||
| 320 | .get(&space) | ||
| 321 | .and_then(|space| { | ||
| 322 | space | ||
| 323 | .contexts | ||
| 324 | .get(&ExGuid::default()) | ||
| 325 | .and_then(|revision| space.revisions.get(revision)) | ||
| 326 | }) | ||
| 327 | .and_then(|revision| revision.nodes.get(&object)); | ||
| 328 | Ok(match node.map(|node| &node.kind) { | ||
| 329 | Some(Kind::RichText { text, .. }) => Some(text.clone()), | ||
| 330 | _ => None, | ||
| 331 | }) | ||
| 332 | } | ||
| 333 | |||
| 334 | fn validate(source: &[u8]) -> Result<ExGuid> { | ||
| 335 | let store = Store::parse(source)?; | ||
| 336 | if !store.checksum_mismatches.is_empty() { | ||
| 337 | return Err(io::Error::new( | ||
| 338 | io::ErrorKind::InvalidData, | ||
| 339 | "Notebook transaction checksum damage", | ||
| 340 | ) | ||
| 341 | .into()); | ||
| 342 | } | ||
| 343 | let index = RevisionIndex::parse(&store)?; | ||
| 344 | index.validate_current()?; | ||
| 345 | Document::parse(&index)?; | ||
| 346 | Ok(index.root) | ||
| 347 | } | ||
| 348 | |||
| 349 | fn pending(connection: &Connection) -> Result<Vec<PendingEdit>> { | ||
| 350 | let mut query = connection.prepare("SELECT id, space, operation FROM edits ORDER BY id")?; | ||
| 351 | let mut rows = query.query([])?; | ||
| 352 | let mut edits = Vec::new(); | ||
| 353 | while let Some(row) = rows.next()? { | ||
| 354 | edits.push(PendingEdit { | ||
| 355 | id: u64::try_from(row.get::<_, i64>(0)?).map_err(io::Error::other)?, | ||
| 356 | space: row.get::<_, String>(1)?.parse()?, | ||
| 357 | operation: serde_json::from_str(&row.get::<_, String>(2)?) | ||
| 358 | .map_err(|error| io::Error::new(io::ErrorKind::InvalidData, error))?, | ||
| 359 | }); | ||
| 360 | } | ||
| 361 | Ok(edits) | ||
| 362 | } | ||
crates/onestore-offline/src/rebase.rs created+297
| ... | @@ -0,0 +1,297 @@ | ||
| 1 | use std::ops::Range; | ||
| 2 | |||
| 3 | const INFINITY: u32 = 1 << 30; | ||
| 4 | const MAX_CELLS: usize = 4_000_000; | ||
| 5 | |||
| 6 | struct Matrix { | ||
| 7 | band: usize, | ||
| 8 | values: Vec<u32>, | ||
| 9 | } | ||
| 10 | |||
| 11 | impl Matrix { | ||
| 12 | fn get(&self, row: usize, column: usize) -> u32 { | ||
| 13 | let Some(column) = column | ||
| 14 | .checked_add(self.band) | ||
| 15 | .and_then(|at| at.checked_sub(row)) | ||
| 16 | else { | ||
| 17 | return INFINITY; | ||
| 18 | }; | ||
| 19 | let width = self.band * 2 + 1; | ||
| 20 | if column >= width { | ||
| 21 | return INFINITY; | ||
| 22 | } | ||
| 23 | self.values | ||
| 24 | .get(row * width + column) | ||
| 25 | .copied() | ||
| 26 | .unwrap_or(INFINITY) | ||
| 27 | } | ||
| 28 | |||
| 29 | fn build(before: &[char], after: &[char], band: usize) -> Self { | ||
| 30 | let width = band * 2 + 1; | ||
| 31 | let mut matrix = Self { | ||
| 32 | band, | ||
| 33 | values: vec![INFINITY; (before.len() + 1) * width], | ||
| 34 | }; | ||
| 35 | for row in 0..=before.len() { | ||
| 36 | for column in row.saturating_sub(band)..=after.len().min(row + band) { | ||
| 37 | let mut cost = if row == 0 && column == 0 { 0 } else { INFINITY }; | ||
| 38 | if row > 0 { | ||
| 39 | cost = cost.min(matrix.get(row - 1, column) + 1); | ||
| 40 | } | ||
| 41 | if column > 0 { | ||
| 42 | cost = cost.min(matrix.get(row, column - 1) + 1); | ||
| 43 | } | ||
| 44 | if row > 0 && column > 0 && before[row - 1] == after[column - 1] { | ||
| 45 | cost = cost.min(matrix.get(row - 1, column - 1)); | ||
| 46 | } | ||
| 47 | matrix.values[row * width + column + band - row] = cost; | ||
| 48 | } | ||
| 49 | } | ||
| 50 | matrix | ||
| 51 | } | ||
| 52 | } | ||
| 53 | |||
| 54 | /// Requires the same mapping in every minimum insertion/deletion alignment. | ||
| 55 | pub(crate) fn rebase(before: &str, after: &str, range: Range<u32>) -> Option<Range<u32>> { | ||
| 56 | if range.start > range.end { | ||
| 57 | return None; | ||
| 58 | } | ||
| 59 | let mut a: Vec<_> = before.chars().collect(); | ||
| 60 | let offsets: Vec<_> = std::iter::once(0) | ||
| 61 | .chain(a.iter().scan(0_u32, |at, character| { | ||
| 62 | *at = at.checked_add(character.len_utf16() as u32)?; | ||
| 63 | Some(*at) | ||
| 64 | })) | ||
| 65 | .collect(); | ||
| 66 | let local = | ||
| 67 | offsets.binary_search(&range.start).ok()?..offsets.binary_search(&range.end).ok()?; | ||
| 68 | if before == after { | ||
| 69 | return Some(range); | ||
| 70 | } | ||
| 71 | let mut b: Vec<_> = after.chars().collect(); | ||
| 72 | let (n, m) = (a.len(), b.len()); | ||
| 73 | let mut band = n.abs_diff(m).max(1); | ||
| 74 | let forward = loop { | ||
| 75 | let width = band.checked_mul(2)?.checked_add(1)?; | ||
| 76 | if (n + 1).checked_mul(width)? > MAX_CELLS { | ||
| 77 | return None; | ||
| 78 | } | ||
| 79 | let matrix = Matrix::build(&a, &b, band); | ||
| 80 | if matrix.get(n, m) <= band as u32 { | ||
| 81 | break matrix; | ||
| 82 | } | ||
| 83 | band *= 2; | ||
| 84 | }; | ||
| 85 | let distance = forward.get(n, m); | ||
| 86 | a.reverse(); | ||
| 87 | b.reverse(); | ||
| 88 | let backward = Matrix::build(&a, &b, band); | ||
| 89 | a.reverse(); | ||
| 90 | b.reverse(); | ||
| 91 | let position = |index: usize| { | ||
| 92 | let mut matched = None; | ||
| 93 | for column in index.saturating_sub(band)..=m.min(index + band) { | ||
| 94 | let cost = forward.get(index, column); | ||
| 95 | if cost + 1 + backward.get(n - index - 1, m - column) == distance { | ||
| 96 | return None; | ||
| 97 | } | ||
| 98 | if b.get(column) == Some(&a[index]) | ||
| 99 | && cost + backward.get(n - index - 1, m - column - 1) == distance | ||
| 100 | { | ||
| 101 | if matched.is_some() { | ||
| 102 | return None; | ||
| 103 | } | ||
| 104 | matched = Some(column); | ||
| 105 | } | ||
| 106 | } | ||
| 107 | matched | ||
| 108 | }; | ||
| 109 | let mapped = if local.is_empty() { | ||
| 110 | if local.start > 0 { | ||
| 111 | position(local.start - 1)?; | ||
| 112 | } | ||
| 113 | if local.start < n { | ||
| 114 | position(local.start)?; | ||
| 115 | } | ||
| 116 | let mut boundary = None; | ||
| 117 | for column in local.start.saturating_sub(band)..=m.min(local.start + band) { | ||
| 118 | if forward.get(local.start, column) + backward.get(n - local.start, m - column) | ||
| 119 | == distance | ||
| 120 | { | ||
| 121 | if boundary.is_some() { | ||
| 122 | return None; | ||
| 123 | } | ||
| 124 | boundary = Some(column); | ||
| 125 | } | ||
| 126 | } | ||
| 127 | let at = boundary?; | ||
| 128 | at..at | ||
| 129 | } else { | ||
| 130 | let start = position(local.start)?; | ||
| 131 | for index in local.start + 1..local.end { | ||
| 132 | if position(index)? != start + index - local.start { | ||
| 133 | return None; | ||
| 134 | } | ||
| 135 | } | ||
| 136 | start..start + local.len() | ||
| 137 | }; | ||
| 138 | let start = b[..mapped.start] | ||
| 139 | .iter() | ||
| 140 | .map(|character| character.len_utf16()) | ||
| 141 | .sum::<usize>(); | ||
| 142 | let end = start | ||
| 143 | + b[mapped] | ||
| 144 | .iter() | ||
| 145 | .map(|character| character.len_utf16()) | ||
| 146 | .sum::<usize>(); | ||
| 147 | Some(u32::try_from(start).ok()?..u32::try_from(end).ok()?) | ||
| 148 | } | ||
| 149 | |||
| 150 | #[cfg(test)] | ||
| 151 | mod tests { | ||
| 152 | use super::*; | ||
| 153 | |||
| 154 | fn optimal_paths(a: &[char], b: &[char]) -> Vec<Vec<(usize, usize)>> { | ||
| 155 | fn visit( | ||
| 156 | a: &[char], | ||
| 157 | b: &[char], | ||
| 158 | path: &mut Vec<(usize, usize)>, | ||
| 159 | cost: usize, | ||
| 160 | best: &mut usize, | ||
| 161 | found: &mut Vec<Vec<(usize, usize)>>, | ||
| 162 | ) { | ||
| 163 | if cost > *best { | ||
| 164 | return; | ||
| 165 | } | ||
| 166 | let (i, j) = *path.last().unwrap(); | ||
| 167 | if (i, j) == (a.len(), b.len()) { | ||
| 168 | if cost < *best { | ||
| 169 | found.clear(); | ||
| 170 | *best = cost; | ||
| 171 | } | ||
| 172 | found.push(path.clone()); | ||
| 173 | return; | ||
| 174 | } | ||
| 175 | for (next_i, next_j, charge) in [(i + 1, j + 1, 0), (i + 1, j, 1), (i, j + 1, 1)] { | ||
| 176 | if next_i > a.len() || next_j > b.len() || (charge == 0 && a[i] != b[j]) { | ||
| 177 | continue; | ||
| 178 | } | ||
| 179 | path.push((next_i, next_j)); | ||
| 180 | visit(a, b, path, cost + charge, best, found); | ||
| 181 | path.pop(); | ||
| 182 | } | ||
| 183 | } | ||
| 184 | let mut found = Vec::new(); | ||
| 185 | let mut best = usize::MAX; | ||
| 186 | visit(a, b, &mut vec![(0, 0)], 0, &mut best, &mut found); | ||
| 187 | found | ||
| 188 | } | ||
| 189 | |||
| 190 | #[test] | ||
| 191 | fn bounded_alignment_agrees_with_exhaustive_paths_for_every_small_unicode_edit() { | ||
| 192 | let mut words = vec![String::new()]; | ||
| 193 | for length in 1..=4 { | ||
| 194 | for bits in 0..1 << length { | ||
| 195 | words.push( | ||
| 196 | (0..length) | ||
| 197 | .map(|bit| if bits & (1 << bit) == 0 { 'a' } else { '🦀' }) | ||
| 198 | .collect(), | ||
| 199 | ); | ||
| 200 | } | ||
| 201 | } | ||
| 202 | let mut cases = 0; | ||
| 203 | for before in &words { | ||
| 204 | let a: Vec<_> = before.chars().collect(); | ||
| 205 | for after in &words { | ||
| 206 | let b: Vec<_> = after.chars().collect(); | ||
| 207 | let paths = optimal_paths(&a, &b); | ||
| 208 | for start in 0..=a.len() { | ||
| 209 | for end in start..=a.len() { | ||
| 210 | let mut expected = None; | ||
| 211 | let mut valid = true; | ||
| 212 | for path in &paths { | ||
| 213 | let mapping: Vec<_> = (0..a.len()) | ||
| 214 | .map(|i| { | ||
| 215 | path.windows(2).find_map(|edge| { | ||
| 216 | (edge[0].0 == i && edge[1] == (i + 1, edge[0].1 + 1)) | ||
| 217 | .then_some(edge[0].1) | ||
| 218 | }) | ||
| 219 | }) | ||
| 220 | .collect(); | ||
| 221 | let candidate = if start == end { | ||
| 222 | let vertices: Vec<_> = path | ||
| 223 | .iter() | ||
| 224 | .filter(|(i, _)| *i == start) | ||
| 225 | .map(|(_, j)| *j) | ||
| 226 | .collect(); | ||
| 227 | if (start > 0 && mapping[start - 1].is_none()) | ||
| 228 | || (start < a.len() && mapping[start].is_none()) | ||
| 229 | || vertices.len() != 1 | ||
| 230 | { | ||
| 231 | None | ||
| 232 | } else { | ||
| 233 | Some(vertices[0]..vertices[0]) | ||
| 234 | } | ||
| 235 | } else if let Some(first) = mapping[start] { | ||
| 236 | (start..end) | ||
| 237 | .all(|i| mapping[i] == Some(first + i - start)) | ||
| 238 | .then_some(first..first + end - start) | ||
| 239 | } else { | ||
| 240 | None | ||
| 241 | }; | ||
| 242 | let Some(candidate) = candidate else { | ||
| 243 | valid = false; | ||
| 244 | break; | ||
| 245 | }; | ||
| 246 | if expected.as_ref().is_some_and(|old| *old != candidate) { | ||
| 247 | valid = false; | ||
| 248 | break; | ||
| 249 | } | ||
| 250 | expected = Some(candidate); | ||
| 251 | } | ||
| 252 | let expected = if valid { | ||
| 253 | expected.map(|range| { | ||
| 254 | b[..range.start].iter().map(|c| c.len_utf16() as u32).sum() | ||
| 255 | ..b[..range.end].iter().map(|c| c.len_utf16() as u32).sum() | ||
| 256 | }) | ||
| 257 | } else { | ||
| 258 | None | ||
| 259 | }; | ||
| 260 | let range = a[..start].iter().map(|c| c.len_utf16() as u32).sum() | ||
| 261 | ..a[..end].iter().map(|c| c.len_utf16() as u32).sum(); | ||
| 262 | assert_eq!( | ||
| 263 | rebase(before, after, range), | ||
| 264 | expected, | ||
| 265 | "{before:?} -> {after:?}, characters {start}..{end}" | ||
| 266 | ); | ||
| 267 | cases += 1; | ||
| 268 | } | ||
| 269 | } | ||
| 270 | } | ||
| 271 | } | ||
| 272 | assert_eq!(cases, 10881); | ||
| 273 | } | ||
| 274 | |||
| 275 | #[test] | ||
| 276 | fn maps_disjoint_unicode_edits_and_rejects_ambiguous_or_overlapping_changes() { | ||
| 277 | assert_eq!(rebase("ab🦀cd", "Xab🦀cYd", 2..4), Some(3..5)); | ||
| 278 | assert_eq!(rebase("abc", "XabcY", 1..2), Some(2..3)); | ||
| 279 | assert_eq!(rebase("abc", "XabcY", 1..1), Some(2..2)); | ||
| 280 | assert_eq!(rebase("abc", "abcX", 3..3), None); | ||
| 281 | assert_eq!(rebase("abc", "ac", 1..2), None); | ||
| 282 | assert_eq!(rebase("abc", "", 1..1), None); | ||
| 283 | assert_eq!(rebase("aaaa", "aaaaa", 1..2), None); | ||
| 284 | assert_eq!(rebase("ab🦀cd", "ab🦀cd", 3..4), None); | ||
| 285 | assert_eq!(rebase("abc", "abc", 4..4), None); | ||
| 286 | } | ||
| 287 | |||
| 288 | #[test] | ||
| 289 | fn long_paragraphs_with_small_remote_changes_use_a_narrow_band() { | ||
| 290 | let text = format!("{}🦀{}", "a".repeat(8192), "b".repeat(8192)); | ||
| 291 | assert_eq!( | ||
| 292 | rebase(&text, &format!("X{text}Y"), 8192..8194), | ||
| 293 | Some(8193..8195) | ||
| 294 | ); | ||
| 295 | assert_eq!(rebase(&"a".repeat(8192), &"b".repeat(8192), 1..2), None); | ||
| 296 | } | ||
| 297 | } | ||
crates/onestore-offline/src/schema.rs created+119
| ... | @@ -0,0 +1,119 @@ | ||
| 1 | use super::*; | ||
| 2 | use rusqlite::{OptionalExtension, Transaction}; | ||
| 3 | |||
| 4 | const CONFLICTS: &str = "CREATE TABLE conflicts ( | ||
| 5 | edit_id INTEGER PRIMARY KEY REFERENCES edits(id) ON DELETE CASCADE, | ||
| 6 | kind INTEGER NOT NULL CHECK(kind BETWEEN 0 AND 3) | ||
| 7 | ) STRICT;"; | ||
| 8 | |||
| 9 | pub(crate) fn create(transaction: &Transaction<'_>) -> Result<()> { | ||
| 10 | transaction.execute_batch( | ||
| 11 | "CREATE TABLE edits ( | ||
| 12 | id INTEGER PRIMARY KEY AUTOINCREMENT CHECK(id>0), | ||
| 13 | space TEXT NOT NULL, | ||
| 14 | operation TEXT NOT NULL | ||
| 15 | ) STRICT; | ||
| 16 | CREATE TABLE attempt ( | ||
| 17 | id INTEGER PRIMARY KEY CHECK(id=1), | ||
| 18 | edit_id INTEGER NOT NULL UNIQUE REFERENCES edits(id) ON DELETE CASCADE, | ||
| 19 | revision TEXT NOT NULL | ||
| 20 | ) STRICT; | ||
| 21 | CREATE TABLE receipts ( | ||
| 22 | edit_id INTEGER PRIMARY KEY CHECK(edit_id>0), | ||
| 23 | revision TEXT NOT NULL | ||
| 24 | ) STRICT;", | ||
| 25 | )?; | ||
| 26 | transaction.execute_batch(CONFLICTS)?; | ||
| 27 | Ok(()) | ||
| 28 | } | ||
| 29 | |||
| 30 | pub(crate) fn migrate(transaction: &Transaction<'_>, version: u32) -> Result<()> { | ||
| 31 | if version == 3 { | ||
| 32 | transaction.execute_batch("ALTER TABLE conflicts RENAME TO old_conflicts;")?; | ||
| 33 | transaction.execute_batch(CONFLICTS)?; | ||
| 34 | transaction.execute_batch( | ||
| 35 | "INSERT INTO conflicts SELECT * FROM old_conflicts; DROP TABLE old_conflicts;", | ||
| 36 | )?; | ||
| 37 | return Ok(()); | ||
| 38 | } | ||
| 39 | |||
| 40 | let mut query = transaction.prepare( | ||
| 41 | "SELECT id, space, object, before_text, start, end, replacement FROM edits ORDER BY id", | ||
| 42 | )?; | ||
| 43 | let mut rows = query.query([])?; | ||
| 44 | let mut edits = Vec::new(); | ||
| 45 | while let Some(row) = rows.next()? { | ||
| 46 | edits.push(PendingEdit { | ||
| 47 | id: u64::try_from(row.get::<_, i64>(0)?).map_err(io::Error::other)?, | ||
| 48 | space: row.get::<_, String>(1)?.parse()?, | ||
| 49 | operation: Operation::Text(TextEdit { | ||
| 50 | object: row.get::<_, String>(2)?.parse()?, | ||
| 51 | before: row.get(3)?, | ||
| 52 | range: row.get(4)?..row.get(5)?, | ||
| 53 | replacement: row.get(6)?, | ||
| 54 | }), | ||
| 55 | }); | ||
| 56 | } | ||
| 57 | drop(rows); | ||
| 58 | drop(query); | ||
| 59 | let sequence: i64 = transaction | ||
| 60 | .query_row( | ||
| 61 | "SELECT seq FROM sqlite_sequence WHERE name='edits'", | ||
| 62 | [], | ||
| 63 | |row| row.get(0), | ||
| 64 | ) | ||
| 65 | .optional()? | ||
| 66 | .unwrap_or(0); | ||
| 67 | let (mut attempts, mut conflicts, mut receipts) = (Vec::new(), Vec::new(), Vec::new()); | ||
| 68 | if version == 2 { | ||
| 69 | let mut query = transaction.prepare("SELECT edit_id, revision FROM attempt")?; | ||
| 70 | attempts = query | ||
| 71 | .query_map([], |row| { | ||
| 72 | Ok((row.get::<_, i64>(0)?, row.get::<_, String>(1)?)) | ||
| 73 | })? | ||
| 74 | .collect::<std::result::Result<_, _>>()?; | ||
| 75 | let mut query = transaction.prepare("SELECT edit_id, kind FROM conflicts")?; | ||
| 76 | conflicts = query | ||
| 77 | .query_map([], |row| Ok((row.get::<_, i64>(0)?, row.get::<_, i64>(1)?)))? | ||
| 78 | .collect::<std::result::Result<_, _>>()?; | ||
| 79 | let mut query = transaction.prepare("SELECT edit_id, revision FROM receipts")?; | ||
| 80 | receipts = query | ||
| 81 | .query_map([], |row| { | ||
| 82 | Ok((row.get::<_, i64>(0)?, row.get::<_, String>(1)?)) | ||
| 83 | })? | ||
| 84 | .collect::<std::result::Result<_, _>>()?; | ||
| 85 | transaction | ||
| 86 | .execute_batch("DROP TABLE attempt; DROP TABLE conflicts; DROP TABLE receipts;")?; | ||
| 87 | } | ||
| 88 | transaction.execute_batch("DROP TABLE edits;")?; | ||
| 89 | create(transaction)?; | ||
| 90 | for edit in edits { | ||
| 91 | transaction.execute( | ||
| 92 | "INSERT INTO edits(id,space,operation) VALUES (?1,?2,?3)", | ||
| 93 | params![ | ||
| 94 | i64::try_from(edit.id).map_err(io::Error::other)?, | ||
| 95 | edit.space.to_string(), | ||
| 96 | serde_json::to_string(&edit.operation).map_err(io::Error::other)? | ||
| 97 | ], | ||
| 98 | )?; | ||
| 99 | } | ||
| 100 | // A drained queue must not reuse IDs belonging to existing durable receipts. | ||
| 101 | transaction.execute("DELETE FROM sqlite_sequence WHERE name='edits'", [])?; | ||
| 102 | transaction.execute( | ||
| 103 | "INSERT INTO sqlite_sequence(name,seq) VALUES ('edits',?1)", | ||
| 104 | [sequence], | ||
| 105 | )?; | ||
| 106 | for (id, revision) in attempts { | ||
| 107 | transaction.execute( | ||
| 108 | "INSERT INTO attempt VALUES (1,?1,?2)", | ||
| 109 | params![id, revision], | ||
| 110 | )?; | ||
| 111 | } | ||
| 112 | for (id, kind) in conflicts { | ||
| 113 | transaction.execute("INSERT INTO conflicts VALUES (?1,?2)", params![id, kind])?; | ||
| 114 | } | ||
| 115 | for (id, revision) in receipts { | ||
| 116 | transaction.execute("INSERT INTO receipts VALUES (?1,?2)", params![id, revision])?; | ||
| 117 | } | ||
| 118 | Ok(()) | ||
| 119 | } | ||
crates/onestore-offline/src/smb.rs created+36
| ... | @@ -0,0 +1,36 @@ | ||
| 1 | use crate::Remote; | ||
| 2 | use onestore::{CommitError, PreparedEdit}; | ||
| 3 | use onestore_smb::Client; | ||
| 4 | use std::io; | ||
| 5 | |||
| 6 | /// Binds every reconciliation operation to one share-relative file and read limit. | ||
| 7 | /// Connection loss retires the client; reconnect before subsequent sync attempts. | ||
| 8 | pub struct SmbRemote { | ||
| 9 | client: Client, | ||
| 10 | path: String, | ||
| 11 | limit: usize, | ||
| 12 | } | ||
| 13 | |||
| 14 | impl SmbRemote { | ||
| 15 | pub fn new(client: Client, path: impl Into<String>, limit: usize) -> Self { | ||
| 16 | Self { | ||
| 17 | client, | ||
| 18 | path: path.into(), | ||
| 19 | limit, | ||
| 20 | } | ||
| 21 | } | ||
| 22 | } | ||
| 23 | |||
| 24 | impl Remote for SmbRemote { | ||
| 25 | fn read(&mut self) -> io::Result<Vec<u8>> { | ||
| 26 | self.client.read(&self.path, self.limit) | ||
| 27 | } | ||
| 28 | |||
| 29 | fn publish(&mut self, edit: &PreparedEdit<'_>) -> Result<(), CommitError> { | ||
| 30 | self.client.commit_prepared(&self.path, edit) | ||
| 31 | } | ||
| 32 | |||
| 33 | fn confirm(&mut self, snapshot: &[u8]) -> Result<(), CommitError> { | ||
| 34 | self.client.confirm_snapshot(&self.path, snapshot) | ||
| 35 | } | ||
| 36 | } | ||
crates/onestore-offline/src/sync.rs created+458
| ... | @@ -0,0 +1,458 @@ | ||
| 1 | use super::*; | ||
| 2 | use onestore::{CommitError, CommitState}; | ||
| 3 | use rusqlite::OptionalExtension; | ||
| 4 | use std::sync::{MutexGuard, TryLockError}; | ||
| 5 | |||
| 6 | /// A single remote file with fresh reads and native-compatible guarded publication. | ||
| 7 | /// Errors retain publication state; confirmation compares, flushes, and notifies cached readers. | ||
| 8 | pub trait Remote { | ||
| 9 | fn read(&mut self) -> io::Result<Vec<u8>>; | ||
| 10 | fn publish(&mut self, edit: &PreparedEdit<'_>) -> std::result::Result<(), CommitError>; | ||
| 11 | fn confirm(&mut self, snapshot: &[u8]) -> std::result::Result<(), CommitError>; | ||
| 12 | } | ||
| 13 | |||
| 14 | #[derive(Debug, Clone, Copy, PartialEq, Eq)] | ||
| 15 | #[repr(i64)] | ||
| 16 | pub enum ConflictKind { | ||
| 17 | TextChanged = 0, | ||
| 18 | TargetUnavailable = 1, | ||
| 19 | UnsupportedEdit = 2, | ||
| 20 | FormattingChanged = 3, | ||
| 21 | } | ||
| 22 | |||
| 23 | #[derive(Debug, Clone, Copy, PartialEq, Eq)] | ||
| 24 | pub enum EditStatus { | ||
| 25 | Pending, | ||
| 26 | AwaitingConfirmation { | ||
| 27 | revision: ExGuid, | ||
| 28 | }, | ||
| 29 | Conflict(ConflictKind), | ||
| 30 | /// Revision containing the confirmed effect; it can differ from a retired attempted revision. | ||
| 31 | Published { | ||
| 32 | revision: ExGuid, | ||
| 33 | }, | ||
| 34 | } | ||
| 35 | |||
| 36 | impl Replica { | ||
| 37 | /// Returns a durable receipt or the persisted state of a locally acknowledged edit. | ||
| 38 | pub fn status(&self, id: u64) -> Result<Option<EditStatus>> { | ||
| 39 | let id = i64::try_from(id).map_err(io::Error::other)?; | ||
| 40 | let connection = self | ||
| 41 | .connection | ||
| 42 | .lock() | ||
| 43 | .map_err(|_| io::Error::other("Cache owner panicked"))?; | ||
| 44 | if let Some(revision) = connection | ||
| 45 | .query_row( | ||
| 46 | "SELECT revision FROM receipts WHERE edit_id=?1", | ||
| 47 | [id], | ||
| 48 | |row| row.get::<_, String>(0), | ||
| 49 | ) | ||
| 50 | .optional()? | ||
| 51 | { | ||
| 52 | return Ok(Some(EditStatus::Published { | ||
| 53 | revision: revision.parse()?, | ||
| 54 | })); | ||
| 55 | } | ||
| 56 | let record: Option<(Option<String>, Option<i64>)> = connection.query_row( | ||
| 57 | "SELECT attempt.revision, conflicts.kind FROM edits LEFT JOIN attempt ON attempt.edit_id=edits.id LEFT JOIN conflicts ON conflicts.edit_id=edits.id WHERE edits.id=?1", [id], |row| Ok((row.get(0)?,row.get(1)?))).optional()?; | ||
| 58 | Ok(match record { | ||
| 59 | None => None, | ||
| 60 | Some((Some(revision), _)) => Some(EditStatus::AwaitingConfirmation { | ||
| 61 | revision: revision.parse()?, | ||
| 62 | }), | ||
| 63 | Some((None, Some(kind))) => Some(EditStatus::Conflict(match kind { | ||
| 64 | 0 => ConflictKind::TextChanged, | ||
| 65 | 1 => ConflictKind::TargetUnavailable, | ||
| 66 | 2 => ConflictKind::UnsupportedEdit, | ||
| 67 | 3 => ConflictKind::FormattingChanged, | ||
| 68 | _ => { | ||
| 69 | return Err(io::Error::new( | ||
| 70 | io::ErrorKind::InvalidData, | ||
| 71 | "Unknown cached conflict kind", | ||
| 72 | ) | ||
| 73 | .into()); | ||
| 74 | } | ||
| 75 | })), | ||
| 76 | Some((None, None)) => Some(EditStatus::Pending), | ||
| 77 | }) | ||
| 78 | } | ||
| 79 | |||
| 80 | /// The last observed remote image, retained alongside the complete local working image. | ||
| 81 | /// Observation alone does not acknowledge any pending edit's remote durability. | ||
| 82 | pub fn remote_snapshot(&self) -> Result<Vec<u8>> { | ||
| 83 | let connection = self | ||
| 84 | .connection | ||
| 85 | .lock() | ||
| 86 | .map_err(|_| io::Error::other("Cache owner panicked"))?; | ||
| 87 | Ok(connection.query_row("SELECT base FROM replica WHERE id=1", [], |row| row.get(0))?) | ||
| 88 | } | ||
| 89 | |||
| 90 | /// Reconciles one pending edit, or refreshes the working image when the queue is empty. | ||
| 91 | /// Network I/O holds synchronization ownership without holding the cache mutex. | ||
| 92 | /// Uncertain edits are never replayed; retired formatting may confirm its complete observed effect. | ||
| 93 | pub fn sync_once(&self, remote: &mut impl Remote) -> Result<Option<(u64, EditStatus)>> { | ||
| 94 | let _owner = self.sync_owner()?; | ||
| 95 | let snapshot = remote.read().map_err(Error::RemoteIo)?; | ||
| 96 | let identity = validate(&snapshot)?; | ||
| 97 | let (intent, attempted) = { | ||
| 98 | let mut connection = self | ||
| 99 | .connection | ||
| 100 | .lock() | ||
| 101 | .map_err(|_| io::Error::other("Cache owner panicked"))?; | ||
| 102 | let transaction = | ||
| 103 | connection.transaction_with_behavior(TransactionBehavior::Immediate)?; | ||
| 104 | let base: Vec<u8> = | ||
| 105 | transaction | ||
| 106 | .query_row("SELECT base FROM replica WHERE id=1", [], |row| row.get(0))?; | ||
| 107 | let base_store = Store::parse(&base)?; | ||
| 108 | if RevisionIndex::parse(&base_store)?.root != identity { | ||
| 109 | return Err(io::Error::new( | ||
| 110 | io::ErrorKind::InvalidInput, | ||
| 111 | "Remote snapshot belongs to another document", | ||
| 112 | ) | ||
| 113 | .into()); | ||
| 114 | } | ||
| 115 | let Some(intent) = pending(&transaction)?.into_iter().next() else { | ||
| 116 | transaction.execute( | ||
| 117 | "UPDATE replica SET base=?1, working=?1 WHERE id=1", | ||
| 118 | [&snapshot], | ||
| 119 | )?; | ||
| 120 | transaction.commit()?; | ||
| 121 | return Ok(None); | ||
| 122 | }; | ||
| 123 | let attempted = transaction | ||
| 124 | .query_row( | ||
| 125 | "SELECT revision FROM attempt WHERE edit_id=?1", | ||
| 126 | [i64::try_from(intent.id).map_err(io::Error::other)?], | ||
| 127 | |row| row.get::<_, String>(0), | ||
| 128 | ) | ||
| 129 | .optional()?; | ||
| 130 | transaction.execute("UPDATE replica SET base=?1 WHERE id=1", [&snapshot])?; | ||
| 131 | transaction.commit()?; | ||
| 132 | (intent, attempted) | ||
| 133 | }; | ||
| 134 | if let Some(revision) = attempted { | ||
| 135 | let mut revision = revision.parse::<ExGuid>()?; | ||
| 136 | let store = Store::parse(&snapshot)?; | ||
| 137 | let index = RevisionIndex::parse(&store)?; | ||
| 138 | if !index | ||
| 139 | .spaces | ||
| 140 | .get(&intent.space) | ||
| 141 | .is_some_and(|space| space.revisions.contains_key(&revision)) | ||
| 142 | { | ||
| 143 | let satisfied = match &intent.operation { | ||
| 144 | Operation::Format(edit) => edit | ||
| 145 | .prepare(&snapshot, intent.space)? | ||
| 146 | .is_ok_and(|prepared| prepared.as_bytes() == snapshot), | ||
| 147 | _ => false, | ||
| 148 | }; | ||
| 149 | if !satisfied { | ||
| 150 | return Ok(Some(( | ||
| 151 | intent.id, | ||
| 152 | EditStatus::AwaitingConfirmation { revision }, | ||
| 153 | ))); | ||
| 154 | } | ||
| 155 | revision = index.spaces[&intent.space].labels[&(ExGuid::default(), 1)]; | ||
| 156 | } | ||
| 157 | if let Err(error) = remote.confirm(&snapshot) { | ||
| 158 | if error.state == CommitState::Committed { | ||
| 159 | self.acknowledge(intent.id, revision, &snapshot)?; | ||
| 160 | } | ||
| 161 | return Err(error.into()); | ||
| 162 | } | ||
| 163 | self.acknowledge(intent.id, revision, &snapshot)?; | ||
| 164 | return Ok(Some((intent.id, EditStatus::Published { revision }))); | ||
| 165 | } | ||
| 166 | let candidate = match &intent.operation { | ||
| 167 | Operation::Format(edit) => edit.prepare(&snapshot, intent.space)?, | ||
| 168 | Operation::Insert(insertion) => { | ||
| 169 | PreparedEdit::insert(&snapshot, intent.space, insertion) | ||
| 170 | .map_err(|_| ConflictKind::UnsupportedEdit) | ||
| 171 | } | ||
| 172 | Operation::Text(edit) => paragraph(&snapshot, intent.space, edit.object)? | ||
| 173 | .ok_or(ConflictKind::TargetUnavailable) | ||
| 174 | .and_then(|text| { | ||
| 175 | crate::rebase::rebase(&edit.before, &text, edit.range.clone()) | ||
| 176 | .ok_or(ConflictKind::TextChanged) | ||
| 177 | }) | ||
| 178 | .and_then(|range| { | ||
| 179 | PreparedEdit::text( | ||
| 180 | &snapshot, | ||
| 181 | intent.space, | ||
| 182 | edit.object, | ||
| 183 | range, | ||
| 184 | &edit.replacement, | ||
| 185 | ) | ||
| 186 | .map_err(|_| ConflictKind::UnsupportedEdit) | ||
| 187 | }), | ||
| 188 | }; | ||
| 189 | let prepared = match candidate { | ||
| 190 | Ok(prepared) => prepared, | ||
| 191 | Err(kind) => { | ||
| 192 | let connection = self | ||
| 193 | .connection | ||
| 194 | .lock() | ||
| 195 | .map_err(|_| io::Error::other("Cache owner panicked"))?; | ||
| 196 | connection.execute("INSERT INTO conflicts(edit_id, kind) VALUES (?1, ?2) ON CONFLICT(edit_id) DO UPDATE SET kind=excluded.kind", params![i64::try_from(intent.id).map_err(io::Error::other)?, kind as i64])?; | ||
| 197 | return Ok(Some((intent.id, EditStatus::Conflict(kind)))); | ||
| 198 | } | ||
| 199 | }; | ||
| 200 | if prepared.as_bytes() == snapshot { | ||
| 201 | let store = Store::parse(&snapshot)?; | ||
| 202 | let index = RevisionIndex::parse(&store)?; | ||
| 203 | let revision = index.spaces[&intent.space].labels[&(ExGuid::default(), 1)]; | ||
| 204 | if let Err(error) = remote.confirm(&snapshot) { | ||
| 205 | if error.state == CommitState::Committed { | ||
| 206 | self.acknowledge(intent.id, revision, &snapshot)?; | ||
| 207 | } | ||
| 208 | return Err(error.into()); | ||
| 209 | } | ||
| 210 | self.acknowledge(intent.id, revision, &snapshot)?; | ||
| 211 | return Ok(Some((intent.id, EditStatus::Published { revision }))); | ||
| 212 | } | ||
| 213 | let store = Store::parse(prepared.as_bytes())?; | ||
| 214 | let index = RevisionIndex::parse(&store)?; | ||
| 215 | let revision = index.spaces[&intent.space].labels[&(ExGuid::default(), 1)]; | ||
| 216 | { | ||
| 217 | let mut connection = self | ||
| 218 | .connection | ||
| 219 | .lock() | ||
| 220 | .map_err(|_| io::Error::other("Cache owner panicked"))?; | ||
| 221 | let transaction = | ||
| 222 | connection.transaction_with_behavior(TransactionBehavior::Immediate)?; | ||
| 223 | transaction.execute( | ||
| 224 | "DELETE FROM conflicts WHERE edit_id=?1", | ||
| 225 | [i64::try_from(intent.id).map_err(io::Error::other)?], | ||
| 226 | )?; | ||
| 227 | transaction.execute( | ||
| 228 | "INSERT INTO attempt(id, edit_id, revision) VALUES (1, ?1, ?2)", | ||
| 229 | params![ | ||
| 230 | i64::try_from(intent.id).map_err(io::Error::other)?, | ||
| 231 | revision.to_string() | ||
| 232 | ], | ||
| 233 | )?; | ||
| 234 | transaction.commit()?; | ||
| 235 | } | ||
| 236 | match remote.publish(&prepared) { | ||
| 237 | Ok(()) => {} | ||
| 238 | Err(error) if error.state == CommitState::NotCommitted => { | ||
| 239 | let connection = self | ||
| 240 | .connection | ||
| 241 | .lock() | ||
| 242 | .map_err(|_| io::Error::other("Cache owner panicked"))?; | ||
| 243 | connection.execute( | ||
| 244 | "DELETE FROM attempt WHERE edit_id=?1", | ||
| 245 | [i64::try_from(intent.id).map_err(io::Error::other)?], | ||
| 246 | )?; | ||
| 247 | return Err(error.into()); | ||
| 248 | } | ||
| 249 | Err(error) if error.state == CommitState::Committed => { | ||
| 250 | self.acknowledge(intent.id, revision, prepared.as_bytes())?; | ||
| 251 | return Err(error.into()); | ||
| 252 | } | ||
| 253 | Err(error) => return Err(error.into()), | ||
| 254 | } | ||
| 255 | self.acknowledge(intent.id, revision, prepared.as_bytes())?; | ||
| 256 | Ok(Some((intent.id, EditStatus::Published { revision }))) | ||
| 257 | } | ||
| 258 | |||
| 259 | /// Places the oldest text or formatting conflict at a reviewed remote UTF-16 range. | ||
| 260 | /// The requested replacement/attributes, local image, intent ID and later edits are preserved. | ||
| 261 | /// Both supplied images must match `snapshot` and `remote_snapshot`; stale review | ||
| 262 | /// returns `Io(ResourceBusy)`. Uncertain publication attempts cannot be rebased. | ||
| 263 | pub fn rebase_conflict( | ||
| 264 | &self, | ||
| 265 | id: u64, | ||
| 266 | local: &[u8], | ||
| 267 | remote: &[u8], | ||
| 268 | range: Range<u32>, | ||
| 269 | ) -> Result<()> { | ||
| 270 | self.resolve_conflict(id, local, remote, |intent| { | ||
| 271 | Ok(match intent.operation { | ||
| 272 | Operation::Text(mut edit) => { | ||
| 273 | let prepared = PreparedEdit::text( | ||
| 274 | remote, | ||
| 275 | intent.space, | ||
| 276 | edit.object, | ||
| 277 | range.clone(), | ||
| 278 | &edit.replacement, | ||
| 279 | )?; | ||
| 280 | if prepared.as_bytes() == remote { | ||
| 281 | return Err(io::Error::new( | ||
| 282 | io::ErrorKind::InvalidInput, | ||
| 283 | "The selected range already contains the replacement", | ||
| 284 | ) | ||
| 285 | .into()); | ||
| 286 | } | ||
| 287 | edit.before = | ||
| 288 | paragraph(remote, intent.space, edit.object)?.ok_or_else(|| { | ||
| 289 | io::Error::new( | ||
| 290 | io::ErrorKind::InvalidData, | ||
| 291 | "The remote text target is unavailable", | ||
| 292 | ) | ||
| 293 | })?; | ||
| 294 | edit.range = range; | ||
| 295 | Operation::Text(edit) | ||
| 296 | } | ||
| 297 | Operation::Format(edit) => Operation::Format( | ||
| 298 | FormatEdit::capture( | ||
| 299 | remote, | ||
| 300 | intent.space, | ||
| 301 | edit.object, | ||
| 302 | range, | ||
| 303 | &edit.attributes, | ||
| 304 | )? | ||
| 305 | .0, | ||
| 306 | ), | ||
| 307 | Operation::Insert(_) => { | ||
| 308 | return Err(io::Error::new( | ||
| 309 | io::ErrorKind::InvalidInput, | ||
| 310 | "Insertion conflicts require a placement, not a text range", | ||
| 311 | ) | ||
| 312 | .into()); | ||
| 313 | } | ||
| 314 | }) | ||
| 315 | }) | ||
| 316 | } | ||
| 317 | |||
| 318 | /// Repositions the oldest paragraph insertion conflict against reviewed cache images. | ||
| 319 | /// Object identities and dependent edits are retained; uncertain attempts cannot be moved. | ||
| 320 | pub fn rebase_paragraph_conflict( | ||
| 321 | &self, | ||
| 322 | id: u64, | ||
| 323 | local: &[u8], | ||
| 324 | remote: &[u8], | ||
| 325 | parent: ExGuid, | ||
| 326 | before: Option<ExGuid>, | ||
| 327 | ) -> Result<()> { | ||
| 328 | self.resolve_conflict(id, local, remote, |intent| { | ||
| 329 | let Operation::Insert(insertion) = intent.operation else { | ||
| 330 | return Err(io::Error::new( | ||
| 331 | io::ErrorKind::InvalidInput, | ||
| 332 | "Select a paragraph insertion conflict", | ||
| 333 | ) | ||
| 334 | .into()); | ||
| 335 | }; | ||
| 336 | let insertion = insertion.reposition_paragraph(parent, before)?; | ||
| 337 | PreparedEdit::insert(remote, intent.space, &insertion)?; | ||
| 338 | Ok(Operation::Insert(insertion)) | ||
| 339 | }) | ||
| 340 | } | ||
| 341 | |||
| 342 | /// Repositions the oldest outline insertion conflict against reviewed cache images. | ||
| 343 | /// Object identities and dependent edits are retained; uncertain attempts cannot be moved. | ||
| 344 | pub fn rebase_outline_conflict( | ||
| 345 | &self, | ||
| 346 | id: u64, | ||
| 347 | local: &[u8], | ||
| 348 | remote: &[u8], | ||
| 349 | page: ExGuid, | ||
| 350 | x: f32, | ||
| 351 | y: f32, | ||
| 352 | ) -> Result<()> { | ||
| 353 | self.resolve_conflict(id, local, remote, |intent| { | ||
| 354 | let Operation::Insert(insertion) = intent.operation else { | ||
| 355 | return Err(io::Error::new( | ||
| 356 | io::ErrorKind::InvalidInput, | ||
| 357 | "Select an outline insertion conflict", | ||
| 358 | ) | ||
| 359 | .into()); | ||
| 360 | }; | ||
| 361 | let insertion = insertion.reposition_outline(page, x, y)?; | ||
| 362 | PreparedEdit::insert(remote, intent.space, &insertion)?; | ||
| 363 | Ok(Operation::Insert(insertion)) | ||
| 364 | }) | ||
| 365 | } | ||
| 366 | |||
| 367 | fn resolve_conflict( | ||
| 368 | &self, | ||
| 369 | id: u64, | ||
| 370 | local: &[u8], | ||
| 371 | remote: &[u8], | ||
| 372 | update: impl FnOnce(PendingEdit) -> Result<Operation>, | ||
| 373 | ) -> Result<()> { | ||
| 374 | let owner = self.sync_owner()?; | ||
| 375 | let intent = self | ||
| 376 | .pending()? | ||
| 377 | .into_iter() | ||
| 378 | .next() | ||
| 379 | .filter(|intent| intent.id == id) | ||
| 380 | .ok_or_else(|| { | ||
| 381 | io::Error::new( | ||
| 382 | io::ErrorKind::InvalidInput, | ||
| 383 | "Only the oldest conflict can be rebased", | ||
| 384 | ) | ||
| 385 | })?; | ||
| 386 | let operation = update(intent)?; | ||
| 387 | let mut connection = self | ||
| 388 | .connection | ||
| 389 | .lock() | ||
| 390 | .map_err(|_| io::Error::other("Cache owner panicked"))?; | ||
| 391 | let transaction = connection.transaction_with_behavior(TransactionBehavior::Immediate)?; | ||
| 392 | let (base, working): (Vec<u8>, Vec<u8>) = | ||
| 393 | transaction.query_row("SELECT base, working FROM replica WHERE id=1", [], |row| { | ||
| 394 | Ok((row.get(0)?, row.get(1)?)) | ||
| 395 | })?; | ||
| 396 | if working != local || base != remote { | ||
| 397 | return Err(io::Error::new( | ||
| 398 | io::ErrorKind::ResourceBusy, | ||
| 399 | "The reviewed cache images changed", | ||
| 400 | ) | ||
| 401 | .into()); | ||
| 402 | } | ||
| 403 | let id = i64::try_from(id).map_err(io::Error::other)?; | ||
| 404 | let eligible: bool = transaction.query_row( | ||
| 405 | "SELECT EXISTS(SELECT 1 FROM conflicts WHERE edit_id=?1) AND NOT EXISTS(SELECT 1 FROM attempt)", | ||
| 406 | [id], |row| row.get(0), | ||
| 407 | )?; | ||
| 408 | if !eligible { | ||
| 409 | return Err(io::Error::new( | ||
| 410 | io::ErrorKind::InvalidInput, | ||
| 411 | "The edit is not an unattempted conflict", | ||
| 412 | ) | ||
| 413 | .into()); | ||
| 414 | } | ||
| 415 | transaction.execute( | ||
| 416 | "UPDATE edits SET operation=?1 WHERE id=?2", | ||
| 417 | params![ | ||
| 418 | serde_json::to_string(&operation).map_err(io::Error::other)?, | ||
| 419 | id | ||
| 420 | ], | ||
| 421 | )?; | ||
| 422 | transaction.execute("DELETE FROM conflicts WHERE edit_id=?1", [id])?; | ||
| 423 | transaction.commit()?; | ||
| 424 | drop(connection); | ||
| 425 | drop(owner); | ||
| 426 | self.wake_sync(); | ||
| 427 | Ok(()) | ||
| 428 | } | ||
| 429 | |||
| 430 | fn sync_owner(&self) -> Result<MutexGuard<'_, ()>> { | ||
| 431 | Ok(self | ||
| 432 | .synchronization | ||
| 433 | .try_lock() | ||
| 434 | .map_err(|error| match error { | ||
| 435 | TryLockError::WouldBlock => io::Error::from(io::ErrorKind::WouldBlock), | ||
| 436 | TryLockError::Poisoned(_) => { | ||
| 437 | io::Error::other("Synchronization owner panicked; reopen the cache") | ||
| 438 | } | ||
| 439 | })?) | ||
| 440 | } | ||
| 441 | |||
| 442 | fn acknowledge(&self, id: u64, revision: ExGuid, snapshot: &[u8]) -> Result<()> { | ||
| 443 | let id = i64::try_from(id).map_err(io::Error::other)?; | ||
| 444 | let mut connection = self | ||
| 445 | .connection | ||
| 446 | .lock() | ||
| 447 | .map_err(|_| io::Error::other("Cache owner panicked"))?; | ||
| 448 | let transaction = connection.transaction_with_behavior(TransactionBehavior::Immediate)?; | ||
| 449 | transaction.execute( | ||
| 450 | "INSERT INTO receipts(edit_id, revision) VALUES (?1, ?2)", | ||
| 451 | params![id, revision.to_string()], | ||
| 452 | )?; | ||
| 453 | transaction.execute("DELETE FROM edits WHERE id=?1", [id])?; | ||
| 454 | transaction.execute("UPDATE replica SET base=?1, working=CASE WHEN EXISTS(SELECT 1 FROM edits) THEN working ELSE ?1 END WHERE id=1", [snapshot])?; | ||
| 455 | transaction.commit()?; | ||
| 456 | Ok(()) | ||
| 457 | } | ||
| 458 | } | ||
crates/onestore-offline/src/worker.rs created+167
| ... | @@ -0,0 +1,167 @@ | ||
| 1 | use super::*; | ||
| 2 | use std::{ | ||
| 3 | collections::hash_map::RandomState, | ||
| 4 | hash::BuildHasher, | ||
| 5 | sync::{ | ||
| 6 | Arc, | ||
| 7 | atomic::{AtomicBool, Ordering}, | ||
| 8 | mpsc::{self, SyncSender}, | ||
| 9 | }, | ||
| 10 | thread::{self, JoinHandle}, | ||
| 11 | time::Instant, | ||
| 12 | }; | ||
| 13 | |||
| 14 | pub(super) struct Signal { | ||
| 15 | stopped: AtomicBool, | ||
| 16 | sender: SyncSender<()>, | ||
| 17 | } | ||
| 18 | |||
| 19 | impl Signal { | ||
| 20 | pub(super) fn wake(&self) { | ||
| 21 | // One retained notification covers edits that arrive during network I/O. | ||
| 22 | let _ = self.sender.try_send(()); | ||
| 23 | } | ||
| 24 | } | ||
| 25 | |||
| 26 | /// Owns automatic reconciliation. Dropping requests cancellation without blocking. | ||
| 27 | /// The in-flight sync step finishes before ownership is released; `stop` waits for it. | ||
| 28 | pub struct SyncWorker { | ||
| 29 | signal: Arc<Signal>, | ||
| 30 | thread: Option<JoinHandle<Result<()>>>, | ||
| 31 | } | ||
| 32 | |||
| 33 | impl SyncWorker { | ||
| 34 | /// Requests a retry, for example after a network reachability change. | ||
| 35 | /// A pending contention backoff finishes before processing the notification. | ||
| 36 | pub fn wake(&self) { | ||
| 37 | self.signal.wake(); | ||
| 38 | } | ||
| 39 | |||
| 40 | /// Cancels future steps and waits for the current step and callback to finish. | ||
| 41 | /// A stopped worker leaves pending edits and uncertain attempts in the cache. | ||
| 42 | /// Call outside the worker's own callback, which cannot join its calling thread. | ||
| 43 | pub fn stop(mut self) -> Result<()> { | ||
| 44 | self.signal.stopped.store(true, Ordering::Release); | ||
| 45 | self.signal.wake(); | ||
| 46 | self.thread | ||
| 47 | .take() | ||
| 48 | .expect("Worker owns its thread") | ||
| 49 | .join() | ||
| 50 | .map_err(|_| io::Error::other("Synchronization worker panicked"))? | ||
| 51 | } | ||
| 52 | } | ||
| 53 | |||
| 54 | impl Drop for SyncWorker { | ||
| 55 | fn drop(&mut self) { | ||
| 56 | self.signal.stopped.store(true, Ordering::Release); | ||
| 57 | self.signal.wake(); | ||
| 58 | } | ||
| 59 | } | ||
| 60 | |||
| 61 | impl Replica { | ||
| 62 | /// Starts one worker, reconnecting through `connect` after transport failures. | ||
| 63 | /// Local edits wake it; `interval` controls idle polling and transport retries. | ||
| 64 | /// Contended operations returning `NotCommitted` also back off by up to one second. | ||
| 65 | /// `observe` runs on the worker after each attempt, including connection errors. | ||
| 66 | /// Cache/document errors stop the worker; inspect them through `observe` or `stop`. | ||
| 67 | /// Remote calls and callbacks must be bounded for `stop` to have bounded latency. | ||
| 68 | pub fn start_sync<R, F, O>( | ||
| 69 | self: &Arc<Self>, | ||
| 70 | interval: Duration, | ||
| 71 | mut connect: F, | ||
| 72 | mut observe: O, | ||
| 73 | ) -> io::Result<SyncWorker> | ||
| 74 | where | ||
| 75 | R: Remote + 'static, | ||
| 76 | F: FnMut() -> io::Result<R> + Send + 'static, | ||
| 77 | O: FnMut(&Result<Option<(u64, EditStatus)>>) + Send + 'static, | ||
| 78 | { | ||
| 79 | if interval.is_zero() || Instant::now().checked_add(interval).is_none() { | ||
| 80 | return Err(io::Error::new( | ||
| 81 | io::ErrorKind::InvalidInput, | ||
| 82 | "Synchronization interval must be positive and representable", | ||
| 83 | )); | ||
| 84 | } | ||
| 85 | let mut owner = self | ||
| 86 | .worker | ||
| 87 | .lock() | ||
| 88 | .map_err(|_| io::Error::other("Synchronization worker registration panicked"))?; | ||
| 89 | if owner.upgrade().is_some() { | ||
| 90 | return Err(io::ErrorKind::WouldBlock.into()); | ||
| 91 | } | ||
| 92 | let (sender, receiver) = mpsc::sync_channel(1); | ||
| 93 | let signal = Arc::new(Signal { | ||
| 94 | stopped: AtomicBool::new(false), | ||
| 95 | sender, | ||
| 96 | }); | ||
| 97 | let replica = Arc::clone(self); | ||
| 98 | let worker_signal = Arc::clone(&signal); | ||
| 99 | let thread = thread::Builder::new() | ||
| 100 | .name("onestore-sync".into()) | ||
| 101 | .spawn(move || { | ||
| 102 | let mut remote = None; | ||
| 103 | let jitter = RandomState::new(); | ||
| 104 | let mut contention = 0_u32; | ||
| 105 | while !worker_signal.stopped.load(Ordering::Acquire) { | ||
| 106 | let result = match remote.as_mut() { | ||
| 107 | Some(remote) => replica.sync_once(remote), | ||
| 108 | None => match connect() { | ||
| 109 | Ok(connected) => { | ||
| 110 | remote = Some(connected); | ||
| 111 | continue; | ||
| 112 | } | ||
| 113 | Err(error) => Err(Error::RemoteIo(error)), | ||
| 114 | }, | ||
| 115 | }; | ||
| 116 | observe(&result); | ||
| 117 | match result { | ||
| 118 | Ok(Some((_, EditStatus::Published { .. }))) => { | ||
| 119 | contention = 0; | ||
| 120 | continue; | ||
| 121 | } | ||
| 122 | Ok(None) => contention = 0, | ||
| 123 | Ok(_) => {} | ||
| 124 | Err(Error::Remote(onestore::CommitError { | ||
| 125 | state: onestore::CommitState::NotCommitted, | ||
| 126 | ref error, | ||
| 127 | })) if matches!( | ||
| 128 | error.kind(), | ||
| 129 | io::ErrorKind::WouldBlock | io::ErrorKind::ResourceBusy | ||
| 130 | ) => | ||
| 131 | { | ||
| 132 | contention = contention.saturating_add(1); | ||
| 133 | let ceiling = (50_u64 << contention.min(5)).min(1000); | ||
| 134 | let until = Instant::now() | ||
| 135 | + Duration::from_millis(jitter.hash_one(contention) % ceiling); | ||
| 136 | // Local wakes must not keep competing writers in the same retry phase. | ||
| 137 | while !worker_signal.stopped.load(Ordering::Acquire) { | ||
| 138 | let Some(remaining) = until.checked_duration_since(Instant::now()) | ||
| 139 | else { | ||
| 140 | break; | ||
| 141 | }; | ||
| 142 | let _ = receiver.recv_timeout(remaining); | ||
| 143 | } | ||
| 144 | continue; | ||
| 145 | } | ||
| 146 | Err(Error::RemoteIo(ref error)) | ||
| 147 | if matches!( | ||
| 148 | error.kind(), | ||
| 149 | io::ErrorKind::WouldBlock | io::ErrorKind::ResourceBusy | ||
| 150 | ) => {} | ||
| 151 | Err(Error::RemoteIo(_) | Error::Remote(_)) => remote = None, | ||
| 152 | Err(Error::Io(ref error)) if error.kind() == io::ErrorKind::WouldBlock => {} | ||
| 153 | Err(error) => return Err(error), | ||
| 154 | } | ||
| 155 | if !worker_signal.stopped.load(Ordering::Acquire) { | ||
| 156 | let _ = receiver.recv_timeout(interval); | ||
| 157 | } | ||
| 158 | } | ||
| 159 | Ok(()) | ||
| 160 | })?; | ||
| 161 | *owner = Arc::downgrade(&signal); | ||
| 162 | Ok(SyncWorker { | ||
| 163 | signal, | ||
| 164 | thread: Some(thread), | ||
| 165 | }) | ||
| 166 | } | ||
| 167 | } | ||
crates/onestore-offline/tests/cache.rs created+685
| ... | @@ -0,0 +1,685 @@ | ||
| 1 | use onestore::{ | ||
| 2 | ExGuid, RevisionIndex, Store, | ||
| 3 | document::{Document, Kind}, | ||
| 4 | }; | ||
| 5 | use onestore_offline::{Error, Replica}; | ||
| 6 | use std::{ | ||
| 7 | collections::BTreeSet, | ||
| 8 | fs, | ||
| 9 | io::ErrorKind, | ||
| 10 | sync::Barrier, | ||
| 11 | time::{Duration, Instant}, | ||
| 12 | }; | ||
| 13 | |||
| 14 | fn target(source: &[u8]) -> (ExGuid, ExGuid, String) { | ||
| 15 | let store = Store::parse(source).unwrap(); | ||
| 16 | assert!(store.checksum_mismatches.is_empty()); | ||
| 17 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 18 | index.validate_current().unwrap(); | ||
| 19 | let doc = Document::parse(&index).unwrap(); | ||
| 20 | doc.spaces | ||
| 21 | .iter() | ||
| 22 | .find_map(|(sid, space)| { | ||
| 23 | let revision = &space.revisions[&space.contexts[&ExGuid::default()]]; | ||
| 24 | revision | ||
| 25 | .nodes | ||
| 26 | .iter() | ||
| 27 | .find_map(|(oid, node)| match &node.kind { | ||
| 28 | Kind::RichText { text, .. } => Some((*sid, *oid, text.clone())), | ||
| 29 | _ => None, | ||
| 30 | }) | ||
| 31 | }) | ||
| 32 | .unwrap() | ||
| 33 | } | ||
| 34 | |||
| 35 | #[test] | ||
| 36 | fn cache_reopen_preserves_exact_images_and_intents() { | ||
| 37 | let dir = tempfile::tempdir().unwrap(); | ||
| 38 | let path = dir.path().join("section.sqlite"); | ||
| 39 | let source = onestore::create_section("section.one", "café 🦀", "Fixture").unwrap(); | ||
| 40 | let replica = Replica::create(&path, &source).unwrap(); | ||
| 41 | assert_eq!(replica.snapshot().unwrap(), source); | ||
| 42 | assert!(replica.pending().unwrap().is_empty()); | ||
| 43 | let (sid, oid, _) = target(&source); | ||
| 44 | assert_eq!( | ||
| 45 | replica.edit_text(&source, sid, oid, 0..0, "").unwrap(), | ||
| 46 | None | ||
| 47 | ); | ||
| 48 | assert_eq!( | ||
| 49 | replica.edit_text(&source, sid, oid, 0..4, "café").unwrap(), | ||
| 50 | None | ||
| 51 | ); | ||
| 52 | let first = replica | ||
| 53 | .edit_text(&source, sid, oid, 5..7, "🐈 日本語") | ||
| 54 | .unwrap() | ||
| 55 | .unwrap(); | ||
| 56 | let edited = replica.snapshot().unwrap(); | ||
| 57 | assert_eq!(target(&edited).2, "café 🐈 日本語"); | ||
| 58 | let intents = replica.pending().unwrap(); | ||
| 59 | assert_eq!(intents.len(), 1); | ||
| 60 | assert_eq!(intents[0].id, first); | ||
| 61 | let onestore_offline::Operation::Text(first_edit) = &intents[0].operation else { | ||
| 62 | panic!() | ||
| 63 | }; | ||
| 64 | assert_eq!(first_edit.before, "café 🦀"); | ||
| 65 | assert_eq!(first_edit.range, 5..7); | ||
| 66 | assert_eq!(first_edit.replacement, "🐈 日本語"); | ||
| 67 | assert_eq!((intents[0].space, first_edit.object), (sid, oid)); | ||
| 68 | drop(replica); | ||
| 69 | let replica = Replica::open(&path).unwrap(); | ||
| 70 | assert_eq!(replica.snapshot().unwrap(), edited); | ||
| 71 | assert_eq!(replica.pending().unwrap(), intents); | ||
| 72 | let second = replica | ||
| 73 | .edit_text(&edited, sid, oid, 0..0, "Recovered ") | ||
| 74 | .unwrap() | ||
| 75 | .unwrap(); | ||
| 76 | assert!(second > first); | ||
| 77 | let onestore_offline::Operation::Text(second_edit) = &replica.pending().unwrap()[1].operation | ||
| 78 | else { | ||
| 79 | panic!() | ||
| 80 | }; | ||
| 81 | assert_eq!(second_edit.before, "café 🐈 日本語"); | ||
| 82 | } | ||
| 83 | |||
| 84 | #[test] | ||
| 85 | fn failed_edits_preserve_both_intent_queue_and_working_image() { | ||
| 86 | let dir = tempfile::tempdir().unwrap(); | ||
| 87 | let path = dir.path().join("section.sqlite"); | ||
| 88 | let source = onestore::create_section("section.one", "café 🦀", "Fixture").unwrap(); | ||
| 89 | let replica = Replica::create(&path, &source).unwrap(); | ||
| 90 | let (sid, oid, _) = target(&source); | ||
| 91 | for (range, replacement) in [(6..7, "X"), (0..u32::MAX, "X"), (0..1, "\n")] { | ||
| 92 | assert!( | ||
| 93 | replica | ||
| 94 | .edit_text(&source, sid, oid, range, replacement) | ||
| 95 | .is_err() | ||
| 96 | ); | ||
| 97 | assert_eq!(replica.snapshot().unwrap(), source); | ||
| 98 | assert!(replica.pending().unwrap().is_empty()); | ||
| 99 | } | ||
| 100 | drop(replica); | ||
| 101 | let connection = rusqlite::Connection::open(&path).unwrap(); | ||
| 102 | connection.execute_batch("CREATE TRIGGER fail_image BEFORE UPDATE ON replica BEGIN SELECT RAISE(ABORT, 'Injected image update failure'); END;").unwrap(); | ||
| 103 | drop(connection); | ||
| 104 | let replica = Replica::open(&path).unwrap(); | ||
| 105 | assert!(matches!( | ||
| 106 | replica.edit_text(&source, sid, oid, 0..0, "lost? "), | ||
| 107 | Err(Error::Database(_)) | ||
| 108 | )); | ||
| 109 | drop(replica); | ||
| 110 | let replica = Replica::open(&path).unwrap(); | ||
| 111 | assert_eq!(replica.snapshot().unwrap(), source); | ||
| 112 | assert!(replica.pending().unwrap().is_empty()); | ||
| 113 | } | ||
| 114 | |||
| 115 | #[test] | ||
| 116 | fn ownership_and_foreign_file_rejection_preserve_existing_data() { | ||
| 117 | let dir = tempfile::tempdir().unwrap(); | ||
| 118 | let path = dir.path().join("section.sqlite"); | ||
| 119 | assert!(Replica::open(&path).is_err()); | ||
| 120 | assert!(!path.exists()); | ||
| 121 | assert!(Replica::create(&path, b"invalid").is_err()); | ||
| 122 | assert!(!path.exists()); | ||
| 123 | let source = onestore::create_section("section.one", "Owned", "Fixture").unwrap(); | ||
| 124 | let replica = Replica::create(&path, &source).unwrap(); | ||
| 125 | for _ in 0..3 { | ||
| 126 | assert!( | ||
| 127 | matches!(Replica::open(&path), Err(Error::Database(error)) if error.sqlite_error_code() == Some(rusqlite::ErrorCode::DatabaseBusy)) | ||
| 128 | ); | ||
| 129 | assert!( | ||
| 130 | matches!(Replica::create(&path, &source), Err(Error::Io(error)) if error.kind() == ErrorKind::AlreadyExists) | ||
| 131 | ); | ||
| 132 | assert_eq!(replica.snapshot().unwrap(), source); | ||
| 133 | } | ||
| 134 | drop(replica); | ||
| 135 | for sql in [ | ||
| 136 | "PRAGMA application_id=0", | ||
| 137 | "PRAGMA application_id=1330529615; PRAGMA user_version=99", | ||
| 138 | ] { | ||
| 139 | let connection = rusqlite::Connection::open(&path).unwrap(); | ||
| 140 | connection.execute_batch(sql).unwrap(); | ||
| 141 | drop(connection); | ||
| 142 | let before = fs::read(&path).unwrap(); | ||
| 143 | assert!(Replica::open(&path).is_err()); | ||
| 144 | assert_eq!(fs::read(&path).unwrap(), before); | ||
| 145 | } | ||
| 146 | let foreign = dir.path().join("foreign.sqlite"); | ||
| 147 | let connection = rusqlite::Connection::open(&foreign).unwrap(); | ||
| 148 | connection | ||
| 149 | .execute_batch( | ||
| 150 | "CREATE TABLE unrelated (value TEXT); INSERT INTO unrelated VALUES ('preserve');", | ||
| 151 | ) | ||
| 152 | .unwrap(); | ||
| 153 | drop(connection); | ||
| 154 | let before = fs::read(&foreign).unwrap(); | ||
| 155 | assert!(Replica::open(&foreign).is_err()); | ||
| 156 | assert_eq!(fs::read(&foreign).unwrap(), before); | ||
| 157 | let incomplete = dir.path().join("incomplete.sqlite"); | ||
| 158 | fs::write(&incomplete, []).unwrap(); | ||
| 159 | assert!(Replica::open(&incomplete).is_err()); | ||
| 160 | assert_eq!(fs::read(&incomplete).unwrap(), b""); | ||
| 161 | } | ||
| 162 | |||
| 163 | #[test] | ||
| 164 | fn twelve_local_editors_reject_stale_ranges_and_preserve_every_acknowledgement() { | ||
| 165 | let dir = tempfile::tempdir().unwrap(); | ||
| 166 | let path = dir.path().join("section.sqlite"); | ||
| 167 | let source = onestore::create_section("section.one", "Shared café 🦀", "Fixture").unwrap(); | ||
| 168 | let replica = Replica::create(&path, &source).unwrap(); | ||
| 169 | let (sid, oid, _) = target(&source); | ||
| 170 | let barrier = Barrier::new(12); | ||
| 171 | let deadline = Instant::now() + Duration::from_secs(60); | ||
| 172 | let outcomes = std::thread::scope(|scope| { | ||
| 173 | let handles: Vec<_> = (0..12) | ||
| 174 | .map(|writer| { | ||
| 175 | let (replica, source, barrier) = (&replica, &source, &barrier); | ||
| 176 | scope.spawn(move || { | ||
| 177 | barrier.wait(); | ||
| 178 | let first = | ||
| 179 | replica.edit_text(source, sid, oid, 0..0, &format!("[initial-{writer}] ")); | ||
| 180 | let mut ids = Vec::new(); | ||
| 181 | let first_won = match first { | ||
| 182 | Ok(Some(id)) => { | ||
| 183 | ids.push(id); | ||
| 184 | true | ||
| 185 | } | ||
| 186 | Err(Error::Io(error)) if error.kind() == ErrorKind::ResourceBusy => false, | ||
| 187 | other => panic!("Unexpected first edit: {other:?}"), | ||
| 188 | }; | ||
| 189 | for edit in 0..20 { | ||
| 190 | loop { | ||
| 191 | assert!( | ||
| 192 | Instant::now() < deadline, | ||
| 193 | "Writer {writer} stopped progressing at {edit}" | ||
| 194 | ); | ||
| 195 | let source = replica.snapshot().unwrap(); | ||
| 196 | match replica.edit_text( | ||
| 197 | &source, | ||
| 198 | sid, | ||
| 199 | oid, | ||
| 200 | 0..0, | ||
| 201 | &format!("[{writer}-{edit}] "), | ||
| 202 | ) { | ||
| 203 | Ok(Some(id)) => { | ||
| 204 | ids.push(id); | ||
| 205 | break; | ||
| 206 | } | ||
| 207 | Err(Error::Io(error)) | ||
| 208 | if error.kind() == ErrorKind::ResourceBusy => {} | ||
| 209 | other => panic!("Unexpected edit: {other:?}"), | ||
| 210 | } | ||
| 211 | } | ||
| 212 | } | ||
| 213 | (first_won, ids) | ||
| 214 | }) | ||
| 215 | }) | ||
| 216 | .collect(); | ||
| 217 | handles | ||
| 218 | .into_iter() | ||
| 219 | .map(|handle| handle.join().unwrap()) | ||
| 220 | .collect::<Vec<_>>() | ||
| 221 | }); | ||
| 222 | assert_eq!(outcomes.iter().filter(|(won, _)| *won).count(), 1); | ||
| 223 | let ids: BTreeSet<_> = outcomes.into_iter().flat_map(|(_, ids)| ids).collect(); | ||
| 224 | assert_eq!(ids.len(), 241); | ||
| 225 | let final_bytes = replica.snapshot().unwrap(); | ||
| 226 | let content = target(&final_bytes).2; | ||
| 227 | let mut expected = "Shared café 🦀".to_owned(); | ||
| 228 | for pending in replica.pending().unwrap() { | ||
| 229 | let onestore_offline::Operation::Text(edit) = pending.operation else { | ||
| 230 | panic!() | ||
| 231 | }; | ||
| 232 | assert_eq!(edit.before, expected); | ||
| 233 | assert_eq!(edit.range, 0..0); | ||
| 234 | expected.insert_str(0, &edit.replacement); | ||
| 235 | } | ||
| 236 | assert_eq!(content, expected); | ||
| 237 | for writer in 0..12 { | ||
| 238 | for edit in 0..20 { | ||
| 239 | assert_eq!(content.matches(&format!("[{writer}-{edit}] ")).count(), 1); | ||
| 240 | } | ||
| 241 | } | ||
| 242 | assert_eq!( | ||
| 243 | replica | ||
| 244 | .pending() | ||
| 245 | .unwrap() | ||
| 246 | .iter() | ||
| 247 | .map(|edit| edit.id) | ||
| 248 | .collect::<BTreeSet<_>>(), | ||
| 249 | ids | ||
| 250 | ); | ||
| 251 | assert!( | ||
| 252 | matches!(replica.edit_text(&source, sid, oid, 0..0, ""), Err(Error::Io(error)) if error.kind() == ErrorKind::ResourceBusy) | ||
| 253 | ); | ||
| 254 | drop(replica); | ||
| 255 | let reopened = Replica::open(&path).unwrap(); | ||
| 256 | assert_eq!(reopened.snapshot().unwrap(), final_bytes); | ||
| 257 | assert_eq!(reopened.pending().unwrap().len(), 241); | ||
| 258 | } | ||
| 259 | |||
| 260 | #[test] | ||
| 261 | fn seeded_unicode_edits_and_restarts_match_an_independent_text_model() { | ||
| 262 | let dir = tempfile::tempdir().unwrap(); | ||
| 263 | let path = dir.path().join("model.sqlite"); | ||
| 264 | let mut text = "ab🚀ab🦀 é repeated repeated".to_owned(); | ||
| 265 | let mut source = onestore::create_section("model.one", &text, "Fixture").unwrap(); | ||
| 266 | let mut replica = Replica::create(&path, &source).unwrap(); | ||
| 267 | let (space, object, _) = target(&source); | ||
| 268 | let mut random = 911_u64; | ||
| 269 | let mut next = || { | ||
| 270 | random ^= random << 13; | ||
| 271 | random ^= random >> 7; | ||
| 272 | random ^= random << 17; | ||
| 273 | random | ||
| 274 | }; | ||
| 275 | let mut intents = Vec::new(); | ||
| 276 | for step in 0..1024 { | ||
| 277 | let boundaries: Vec<_> = text | ||
| 278 | .char_indices() | ||
| 279 | .map(|(at, _)| at) | ||
| 280 | .chain([text.len()]) | ||
| 281 | .collect(); | ||
| 282 | let first = boundaries[next() as usize % boundaries.len()]; | ||
| 283 | let last = boundaries[next() as usize % boundaries.len()]; | ||
| 284 | let bytes = first.min(last)..first.max(last); | ||
| 285 | let range = u32::try_from(text[..bytes.start].encode_utf16().count()).unwrap() | ||
| 286 | ..u32::try_from(text[..bytes.end].encode_utf16().count()).unwrap(); | ||
| 287 | let replacement = ["", "🐈", "日本語", "repeated", "é", "ab🦀ab"][next() as usize % 6]; | ||
| 288 | let mut expected = text.clone(); | ||
| 289 | expected.replace_range(bytes, replacement); | ||
| 290 | let acknowledgement = replica | ||
| 291 | .edit_text(&source, space, object, range.clone(), replacement) | ||
| 292 | .unwrap(); | ||
| 293 | if let Some(id) = acknowledgement { | ||
| 294 | assert_ne!(text, expected); | ||
| 295 | assert!( | ||
| 296 | intents | ||
| 297 | .last() | ||
| 298 | .is_none_or(|edit: &onestore_offline::PendingEdit| edit.id < id) | ||
| 299 | ); | ||
| 300 | intents.push(onestore_offline::PendingEdit { | ||
| 301 | id, | ||
| 302 | space, | ||
| 303 | operation: onestore_offline::Operation::Text(onestore_offline::TextEdit { | ||
| 304 | object, | ||
| 305 | before: text, | ||
| 306 | range, | ||
| 307 | replacement: replacement.into(), | ||
| 308 | }), | ||
| 309 | }); | ||
| 310 | assert!( | ||
| 311 | matches!(replica.edit_text(&source, space, object, 0..0, "stale"), Err(Error::Io(error)) if error.kind() == ErrorKind::ResourceBusy) | ||
| 312 | ); | ||
| 313 | } else { | ||
| 314 | assert_eq!(text, expected); | ||
| 315 | } | ||
| 316 | text = expected; | ||
| 317 | source = replica.snapshot().unwrap(); | ||
| 318 | assert_eq!(target(&source).2, text, "seed 911, step {step}"); | ||
| 319 | if step % 37 == 0 { | ||
| 320 | drop(replica); | ||
| 321 | replica = Replica::open(&path).unwrap(); | ||
| 322 | assert_eq!(replica.snapshot().unwrap(), source); | ||
| 323 | assert_eq!(replica.pending().unwrap(), intents); | ||
| 324 | } | ||
| 325 | } | ||
| 326 | assert!( | ||
| 327 | intents.len() > 512, | ||
| 328 | "The history checkpoint boundary was not exercised" | ||
| 329 | ); | ||
| 330 | drop(replica); | ||
| 331 | let replica = Replica::open(&path).unwrap(); | ||
| 332 | assert_eq!(replica.pending().unwrap(), intents); | ||
| 333 | assert_eq!(replica.snapshot().unwrap(), source); | ||
| 334 | } | ||
| 335 | |||
| 336 | #[test] | ||
| 337 | #[ignore = "migrates a fresh copy of a retained version-two or version-three cache"] | ||
| 338 | fn migrate_retained_cache_copy() { | ||
| 339 | use onestore_offline::{EditStatus, Operation, PendingEdit, TextEdit}; | ||
| 340 | let source = std::path::PathBuf::from(std::env::var_os("ONESTORE_MIGRATION_SOURCE").unwrap()); | ||
| 341 | let output = std::path::PathBuf::from(std::env::var_os("ONESTORE_MIGRATION_OUTPUT").unwrap()); | ||
| 342 | assert!(source.is_absolute() && output.is_absolute()); | ||
| 343 | let mut original = fs::File::open(&source).unwrap(); | ||
| 344 | let mut destination = fs::OpenOptions::new() | ||
| 345 | .write(true) | ||
| 346 | .create_new(true) | ||
| 347 | .open(&output) | ||
| 348 | .unwrap(); | ||
| 349 | std::io::copy(&mut original, &mut destination).unwrap(); | ||
| 350 | destination.sync_all().unwrap(); | ||
| 351 | drop(destination); | ||
| 352 | let db = rusqlite::Connection::open(&output).unwrap(); | ||
| 353 | let version = db | ||
| 354 | .pragma_query_value(None, "user_version", |r| r.get::<_, u32>(0)) | ||
| 355 | .unwrap(); | ||
| 356 | assert!(matches!(version, 2 | 3)); | ||
| 357 | if std::env::var_os("ONESTORE_MIGRATION_ROLLBACK").is_some() { | ||
| 358 | assert_eq!(version, 2); | ||
| 359 | db.pragma_update(None, "foreign_keys", false).unwrap(); | ||
| 360 | db.execute( | ||
| 361 | "INSERT INTO attempt VALUES (1,999999,'{00000001-0000-0000-0000-000000000000},1')", | ||
| 362 | [], | ||
| 363 | ) | ||
| 364 | .unwrap(); | ||
| 365 | drop(db); | ||
| 366 | let before = fs::read(&output).unwrap(); | ||
| 367 | assert!( | ||
| 368 | matches!(Replica::open(&output), Err(Error::Database(rusqlite::Error::SqliteFailure(error, _))) if error.extended_code == 787) | ||
| 369 | ); | ||
| 370 | assert_eq!(fs::read(&output).unwrap(), before); | ||
| 371 | let db = rusqlite::Connection::open(&output).unwrap(); | ||
| 372 | assert_eq!( | ||
| 373 | db.pragma_query_value(None, "user_version", |r| r.get::<_, u32>(0)) | ||
| 374 | .unwrap(), | ||
| 375 | 2 | ||
| 376 | ); | ||
| 377 | println!("migration: rollback preserved {} bytes", before.len()); | ||
| 378 | return; | ||
| 379 | } | ||
| 380 | if std::env::var_os("ONESTORE_MIGRATION_CONFLICT").is_some() { | ||
| 381 | db.execute("INSERT INTO conflicts SELECT min(id),0 FROM edits", []) | ||
| 382 | .unwrap(); | ||
| 383 | } | ||
| 384 | let (base, working): (Vec<u8>, Vec<u8>) = db | ||
| 385 | .query_row("SELECT base,working FROM replica", [], |r| { | ||
| 386 | Ok((r.get(0)?, r.get(1)?)) | ||
| 387 | }) | ||
| 388 | .unwrap(); | ||
| 389 | let sequence: i64 = db | ||
| 390 | .query_row( | ||
| 391 | "SELECT seq FROM sqlite_sequence WHERE name='edits'", | ||
| 392 | [], | ||
| 393 | |r| r.get(0), | ||
| 394 | ) | ||
| 395 | .unwrap(); | ||
| 396 | let pending: Vec<PendingEdit> = | ||
| 397 | if version == 2 { | ||
| 398 | let mut query = db | ||
| 399 | .prepare("SELECT id,space,object,before_text,start,end,replacement FROM edits ORDER BY id") | ||
| 400 | .unwrap(); | ||
| 401 | let pending: Vec<PendingEdit> = query | ||
| 402 | .query_map([], |r| { | ||
| 403 | Ok(PendingEdit { | ||
| 404 | id: u64::try_from(r.get::<_, i64>(0)?).unwrap(), | ||
| 405 | space: r.get::<_, String>(1)?.parse().unwrap(), | ||
| 406 | operation: Operation::Text(TextEdit { | ||
| 407 | object: r.get::<_, String>(2)?.parse().unwrap(), | ||
| 408 | before: r.get(3)?, | ||
| 409 | range: r.get(4)?..r.get(5)?, | ||
| 410 | replacement: r.get(6)?, | ||
| 411 | }), | ||
| 412 | }) | ||
| 413 | }) | ||
| 414 | .unwrap() | ||
| 415 | .collect::<Result<_, _>>() | ||
| 416 | .unwrap(); | ||
| 417 | drop(query); | ||
| 418 | pending | ||
| 419 | } else { | ||
| 420 | let mut query = db | ||
| 421 | .prepare("SELECT id,space,operation FROM edits ORDER BY id") | ||
| 422 | .unwrap(); | ||
| 423 | query | ||
| 424 | .query_map([], |r| { | ||
| 425 | Ok(PendingEdit { | ||
| 426 | id: u64::try_from(r.get::<_, i64>(0)?).unwrap(), | ||
| 427 | space: r.get::<_, String>(1)?.parse().unwrap(), | ||
| 428 | operation: serde_json::from_str(&r.get::<_, String>(2)?).unwrap(), | ||
| 429 | }) | ||
| 430 | }) | ||
| 431 | .unwrap() | ||
| 432 | .collect::<Result<_, _>>() | ||
| 433 | .unwrap() | ||
| 434 | }; | ||
| 435 | let mut states = std::collections::BTreeMap::new(); | ||
| 436 | for edit in &pending { | ||
| 437 | states.insert(edit.id, EditStatus::Pending); | ||
| 438 | } | ||
| 439 | for table in ["receipts", "attempt"] { | ||
| 440 | let mut query = db | ||
| 441 | .prepare(&format!("SELECT edit_id,revision FROM {table}")) | ||
| 442 | .unwrap(); | ||
| 443 | for row in query | ||
| 444 | .query_map([], |r| Ok((r.get::<_, i64>(0)?, r.get::<_, String>(1)?))) | ||
| 445 | .unwrap() | ||
| 446 | { | ||
| 447 | let (id, rid) = row.unwrap(); | ||
| 448 | let revision = rid.parse().unwrap(); | ||
| 449 | states.insert( | ||
| 450 | u64::try_from(id).unwrap(), | ||
| 451 | if table == "receipts" { | ||
| 452 | EditStatus::Published { revision } | ||
| 453 | } else { | ||
| 454 | EditStatus::AwaitingConfirmation { revision } | ||
| 455 | }, | ||
| 456 | ); | ||
| 457 | } | ||
| 458 | } | ||
| 459 | let mut query = db.prepare("SELECT edit_id,kind FROM conflicts").unwrap(); | ||
| 460 | for row in query | ||
| 461 | .query_map([], |r| Ok((r.get::<_, i64>(0)?, r.get::<_, u32>(1)?))) | ||
| 462 | .unwrap() | ||
| 463 | { | ||
| 464 | let (id, kind) = row.unwrap(); | ||
| 465 | assert_eq!(kind, 0); | ||
| 466 | states.insert( | ||
| 467 | u64::try_from(id).unwrap(), | ||
| 468 | EditStatus::Conflict(onestore_offline::ConflictKind::TextChanged), | ||
| 469 | ); | ||
| 470 | } | ||
| 471 | drop(query); | ||
| 472 | drop(db); | ||
| 473 | let cache = Replica::open(&output).unwrap(); | ||
| 474 | assert_eq!(cache.snapshot().unwrap(), working); | ||
| 475 | assert_eq!(cache.remote_snapshot().unwrap(), base); | ||
| 476 | assert_eq!(cache.pending().unwrap(), pending); | ||
| 477 | for (id, status) in &states { | ||
| 478 | assert_eq!(cache.status(*id).unwrap(), Some(*status)); | ||
| 479 | } | ||
| 480 | drop(cache); | ||
| 481 | let cache = Replica::open(&output).unwrap(); | ||
| 482 | assert_eq!(cache.pending().unwrap(), pending); | ||
| 483 | let store = Store::parse(&working).unwrap(); | ||
| 484 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 485 | let doc = Document::parse(&index).unwrap(); | ||
| 486 | let (sid, page) = doc.pages().unwrap()[0]; | ||
| 487 | let insertion = onestore::Insertion::outline( | ||
| 488 | page, | ||
| 489 | 144.0, | ||
| 490 | 720.0, | ||
| 491 | "After cache migration", | ||
| 492 | "Migration author", | ||
| 493 | ) | ||
| 494 | .unwrap(); | ||
| 495 | let next = cache.insert(&working, sid, &insertion).unwrap().unwrap(); | ||
| 496 | assert_eq!(next, u64::try_from(sequence + 1).unwrap()); | ||
| 497 | let updated = cache.snapshot().unwrap(); | ||
| 498 | drop(cache); | ||
| 499 | let cache = Replica::open(&output).unwrap(); | ||
| 500 | assert_eq!(cache.snapshot().unwrap(), updated); | ||
| 501 | assert_eq!( | ||
| 502 | cache.pending().unwrap().last().unwrap().operation, | ||
| 503 | Operation::Insert(insertion) | ||
| 504 | ); | ||
| 505 | println!( | ||
| 506 | "migration: {}", | ||
| 507 | serde_json::json!({"pending":pending.len(),"retained_statuses":states.len(),"next_id":next,"base_bytes":base.len(),"working_bytes":working.len()}) | ||
| 508 | ); | ||
| 509 | } | ||
| 510 | |||
| 511 | #[test] | ||
| 512 | fn twelve_local_clients_preserve_inserted_identities_and_dependent_edits() { | ||
| 513 | use onestore::Insertion; | ||
| 514 | let directory = tempfile::tempdir().unwrap(); | ||
| 515 | let path = directory.path().join("parallel.sqlite"); | ||
| 516 | let source = onestore::create_section("parallel.one", "Original", "Author").unwrap(); | ||
| 517 | let store = Store::parse(&source).unwrap(); | ||
| 518 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 519 | let doc = Document::parse(&index).unwrap(); | ||
| 520 | let (sid, page) = doc.pages().unwrap()[0]; | ||
| 521 | let cache = Replica::create(&path, &source).unwrap(); | ||
| 522 | let barrier = Barrier::new(12); | ||
| 523 | let deadline = Instant::now() + Duration::from_secs(90); | ||
| 524 | let all = std::thread::scope(|scope| { | ||
| 525 | let handles: Vec<_> = (0..12) | ||
| 526 | .map(|client| { | ||
| 527 | let (cache, barrier) = (&cache, &barrier); | ||
| 528 | scope.spawn(move || { | ||
| 529 | let outline = Insertion::outline( | ||
| 530 | page, | ||
| 531 | 72.0, | ||
| 532 | 144.0 + client as f32 * 72.0, | ||
| 533 | &format!("Client {client}"), | ||
| 534 | "Author", | ||
| 535 | ) | ||
| 536 | .unwrap(); | ||
| 537 | let mut ids = Vec::new(); | ||
| 538 | let mut objects = Vec::new(); | ||
| 539 | barrier.wait(); | ||
| 540 | for sequence in 0..4 { | ||
| 541 | let insertion = if sequence == 0 { | ||
| 542 | outline.clone() | ||
| 543 | } else { | ||
| 544 | Insertion::paragraph( | ||
| 545 | outline.object(), | ||
| 546 | None, | ||
| 547 | &format!("Paragraph {client}:{sequence}"), | ||
| 548 | "Author", | ||
| 549 | ) | ||
| 550 | .unwrap() | ||
| 551 | }; | ||
| 552 | loop { | ||
| 553 | assert!( | ||
| 554 | Instant::now() < deadline, | ||
| 555 | "Client {client} stopped at insertion {sequence}" | ||
| 556 | ); | ||
| 557 | let snapshot = cache.snapshot().unwrap(); | ||
| 558 | match cache.insert(&snapshot, sid, &insertion) { | ||
| 559 | Ok(Some(id)) => { | ||
| 560 | ids.push(id); | ||
| 561 | objects.push(insertion.text_object()); | ||
| 562 | break; | ||
| 563 | } | ||
| 564 | Err(Error::Io(e)) if e.kind() == ErrorKind::ResourceBusy => {} | ||
| 565 | other => panic!("{other:?}"), | ||
| 566 | } | ||
| 567 | } | ||
| 568 | if sequence == 0 { | ||
| 569 | continue; | ||
| 570 | } | ||
| 571 | loop { | ||
| 572 | assert!( | ||
| 573 | Instant::now() < deadline, | ||
| 574 | "Client {client} stopped at text {sequence}" | ||
| 575 | ); | ||
| 576 | let snapshot = cache.snapshot().unwrap(); | ||
| 577 | match cache.edit_text( | ||
| 578 | &snapshot, | ||
| 579 | sid, | ||
| 580 | insertion.text_object(), | ||
| 581 | 0..0, | ||
| 582 | "Edited ", | ||
| 583 | ) { | ||
| 584 | Ok(Some(id)) => { | ||
| 585 | ids.push(id); | ||
| 586 | break; | ||
| 587 | } | ||
| 588 | Err(Error::Io(e)) if e.kind() == ErrorKind::ResourceBusy => {} | ||
| 589 | other => panic!("{other:?}"), | ||
| 590 | } | ||
| 591 | } | ||
| 592 | } | ||
| 593 | (ids, objects) | ||
| 594 | }) | ||
| 595 | }) | ||
| 596 | .collect(); | ||
| 597 | handles | ||
| 598 | .into_iter() | ||
| 599 | .map(|h| h.join().unwrap()) | ||
| 600 | .collect::<Vec<_>>() | ||
| 601 | }); | ||
| 602 | let ids: BTreeSet<_> = all | ||
| 603 | .iter() | ||
| 604 | .flat_map(|(ids, _)| ids.iter().copied()) | ||
| 605 | .collect(); | ||
| 606 | assert_eq!(ids.len(), 84); | ||
| 607 | let snapshot = cache.snapshot().unwrap(); | ||
| 608 | drop(cache); | ||
| 609 | let cache = Replica::open(&path).unwrap(); | ||
| 610 | assert_eq!(cache.snapshot().unwrap(), snapshot); | ||
| 611 | assert_eq!( | ||
| 612 | cache | ||
| 613 | .pending() | ||
| 614 | .unwrap() | ||
| 615 | .iter() | ||
| 616 | .map(|e| e.id) | ||
| 617 | .collect::<BTreeSet<_>>(), | ||
| 618 | ids | ||
| 619 | ); | ||
| 620 | let store = Store::parse(&snapshot).unwrap(); | ||
| 621 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 622 | let doc = Document::parse(&index).unwrap(); | ||
| 623 | let s = &doc.spaces[&sid]; | ||
| 624 | let v = &s.revisions[&s.contexts[&ExGuid::default()]]; | ||
| 625 | for (client, (_, objects)) in all.iter().enumerate() { | ||
| 626 | for (sequence, id) in objects.iter().enumerate() { | ||
| 627 | let wanted = if sequence == 0 { | ||
| 628 | format!("Client {client}") | ||
| 629 | } else { | ||
| 630 | format!("Edited Paragraph {client}:{sequence}") | ||
| 631 | }; | ||
| 632 | assert!(matches!(&v.nodes[id].kind,Kind::RichText{text,..} if *text==wanted)); | ||
| 633 | } | ||
| 634 | } | ||
| 635 | } | ||
| 636 | |||
| 637 | #[test] | ||
| 638 | fn unrecognized_persisted_operations_are_rejected_without_dropping_fields() { | ||
| 639 | for operation in ["Text", "Insert", "Format"] { | ||
| 640 | let directory = tempfile::tempdir().unwrap(); | ||
| 641 | let path = directory.path().join("unknown.sqlite"); | ||
| 642 | let source = onestore::create_section("unknown.one", "Original", "Author").unwrap(); | ||
| 643 | let cache = Replica::create(&path, &source).unwrap(); | ||
| 644 | let (sid, oid, _) = target(&source); | ||
| 645 | if operation == "Insert" { | ||
| 646 | let store = Store::parse(&source).unwrap(); | ||
| 647 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 648 | let doc = Document::parse(&index).unwrap(); | ||
| 649 | let (_, page) = doc.pages().unwrap()[0]; | ||
| 650 | let insertion = | ||
| 651 | onestore::Insertion::outline(page, 144.0, 144.0, "Inserted", "Author").unwrap(); | ||
| 652 | cache.insert(&source, sid, &insertion).unwrap(); | ||
| 653 | } else if operation == "Text" { | ||
| 654 | cache.edit_text(&source, sid, oid, 0..0, "New ").unwrap(); | ||
| 655 | } else { | ||
| 656 | cache | ||
| 657 | .format( | ||
| 658 | &source, | ||
| 659 | sid, | ||
| 660 | oid, | ||
| 661 | 0..4, | ||
| 662 | &[onestore::TextAttribute::Bold(true)], | ||
| 663 | ) | ||
| 664 | .unwrap(); | ||
| 665 | } | ||
| 666 | drop(cache); | ||
| 667 | let db = rusqlite::Connection::open(&path).unwrap(); | ||
| 668 | let encoded: String = db | ||
| 669 | .query_row("SELECT operation FROM edits", [], |r| r.get(0)) | ||
| 670 | .unwrap(); | ||
| 671 | let mut value: serde_json::Value = serde_json::from_str(&encoded).unwrap(); | ||
| 672 | value[operation]["future_option"] = true.into(); | ||
| 673 | db.execute("UPDATE edits SET operation=?1", [value.to_string()]) | ||
| 674 | .unwrap(); | ||
| 675 | if operation != "Format" { | ||
| 676 | db.execute_batch("DROP TABLE conflicts; CREATE TABLE conflicts (edit_id INTEGER PRIMARY KEY REFERENCES edits(id) ON DELETE CASCADE, kind INTEGER NOT NULL CHECK(kind BETWEEN 0 AND 2)) STRICT; PRAGMA user_version=3;").unwrap(); | ||
| 677 | } | ||
| 678 | drop(db); | ||
| 679 | let before = fs::read(&path).unwrap(); | ||
| 680 | assert!( | ||
| 681 | matches!(Replica::open(&path),Err(Error::Io(error))if error.kind()==ErrorKind::InvalidData) | ||
| 682 | ); | ||
| 683 | assert_eq!(fs::read(&path).unwrap(), before); | ||
| 684 | } | ||
| 685 | } | ||
crates/onestore-offline/tests/sync.rs created+2618
| ... | @@ -0,0 +1,2618 @@ | ||
| 1 | use onestore::{ | ||
| 2 | CommitError, CommitIo, CommitState, ExGuid, PreparedEdit, RevisionIndex, Store, | ||
| 3 | document::{Document, Kind}, | ||
| 4 | }; | ||
| 5 | use onestore_offline::{ConflictKind, EditStatus, Error, Remote, Replica}; | ||
| 6 | use std::io; | ||
| 7 | |||
| 8 | #[derive(Clone, Copy, Default)] | ||
| 9 | enum Fault { | ||
| 10 | #[default] | ||
| 11 | None, | ||
| 12 | Before, | ||
| 13 | UnknownBefore, | ||
| 14 | UnknownAfter, | ||
| 15 | Committed, | ||
| 16 | PanicBefore, | ||
| 17 | PanicAfter, | ||
| 18 | Confirm, | ||
| 19 | ConfirmCommitted, | ||
| 20 | } | ||
| 21 | |||
| 22 | struct Server { | ||
| 23 | visible: Vec<u8>, | ||
| 24 | durable: Vec<u8>, | ||
| 25 | fault: Fault, | ||
| 26 | publications: usize, | ||
| 27 | confirmations: usize, | ||
| 28 | } | ||
| 29 | |||
| 30 | impl Server { | ||
| 31 | fn new(source: &[u8]) -> Self { | ||
| 32 | Self { | ||
| 33 | visible: source.to_vec(), | ||
| 34 | durable: source.to_vec(), | ||
| 35 | fault: Fault::None, | ||
| 36 | publications: 0, | ||
| 37 | confirmations: 0, | ||
| 38 | } | ||
| 39 | } | ||
| 40 | } | ||
| 41 | |||
| 42 | fn failure(state: CommitState) -> CommitError { | ||
| 43 | CommitError { | ||
| 44 | state, | ||
| 45 | error: io::Error::from(io::ErrorKind::ConnectionAborted), | ||
| 46 | } | ||
| 47 | } | ||
| 48 | |||
| 49 | impl CommitIo for Server { | ||
| 50 | fn read_at(&mut self, offset: u64, output: &mut [u8]) -> io::Result<usize> { | ||
| 51 | let offset = usize::try_from(offset).unwrap(); | ||
| 52 | let size = output.len().min(self.visible.len().saturating_sub(offset)); | ||
| 53 | if size > 0 { | ||
| 54 | output[..size].copy_from_slice(&self.visible[offset..offset + size]); | ||
| 55 | } | ||
| 56 | Ok(size) | ||
| 57 | } | ||
| 58 | fn write_at(&mut self, offset: u64, bytes: &[u8]) -> io::Result<usize> { | ||
| 59 | let offset = usize::try_from(offset).unwrap(); | ||
| 60 | self.visible | ||
| 61 | .resize(self.visible.len().max(offset + bytes.len()), 0); | ||
| 62 | self.visible[offset..offset + bytes.len()].copy_from_slice(bytes); | ||
| 63 | Ok(bytes.len()) | ||
| 64 | } | ||
| 65 | fn flush(&mut self) -> io::Result<()> { | ||
| 66 | self.durable.clone_from(&self.visible); | ||
| 67 | Ok(()) | ||
| 68 | } | ||
| 69 | } | ||
| 70 | |||
| 71 | impl Remote for Server { | ||
| 72 | fn read(&mut self) -> io::Result<Vec<u8>> { | ||
| 73 | Ok(self.visible.clone()) | ||
| 74 | } | ||
| 75 | fn publish(&mut self, edit: &PreparedEdit<'_>) -> Result<(), CommitError> { | ||
| 76 | self.publications += 1; | ||
| 77 | let fault = std::mem::take(&mut self.fault); | ||
| 78 | match fault { | ||
| 79 | Fault::Before => return Err(failure(CommitState::NotCommitted)), | ||
| 80 | Fault::UnknownBefore => return Err(failure(CommitState::Unknown)), | ||
| 81 | Fault::PanicBefore => panic!("Terminated before remote I/O"), | ||
| 82 | _ => {} | ||
| 83 | } | ||
| 84 | let old = self.durable.clone(); | ||
| 85 | edit.commit(self)?; | ||
| 86 | match fault { | ||
| 87 | Fault::UnknownAfter => { | ||
| 88 | self.durable = old; | ||
| 89 | Err(failure(CommitState::Unknown)) | ||
| 90 | } | ||
| 91 | Fault::Committed => Err(failure(CommitState::Committed)), | ||
| 92 | Fault::PanicAfter => panic!("Terminated after remote publication"), | ||
| 93 | _ => Ok(()), | ||
| 94 | } | ||
| 95 | } | ||
| 96 | fn confirm(&mut self, snapshot: &[u8]) -> Result<(), CommitError> { | ||
| 97 | self.confirmations += 1; | ||
| 98 | if matches!(self.fault, Fault::Confirm) { | ||
| 99 | self.fault = Fault::None; | ||
| 100 | return Err(failure(CommitState::Unknown)); | ||
| 101 | } | ||
| 102 | onestore::confirm_snapshot(self, snapshot)?; | ||
| 103 | if matches!(self.fault, Fault::ConfirmCommitted) { | ||
| 104 | self.fault = Fault::None; | ||
| 105 | return Err(failure(CommitState::Committed)); | ||
| 106 | } | ||
| 107 | Ok(()) | ||
| 108 | } | ||
| 109 | } | ||
| 110 | |||
| 111 | fn text(source: &[u8]) -> (ExGuid, ExGuid, String) { | ||
| 112 | let store = Store::parse(source).unwrap(); | ||
| 113 | assert!(store.checksum_mismatches.is_empty()); | ||
| 114 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 115 | index.validate_current().unwrap(); | ||
| 116 | let doc = Document::parse(&index).unwrap(); | ||
| 117 | doc.spaces | ||
| 118 | .iter() | ||
| 119 | .find_map(|(sid, space)| { | ||
| 120 | space.revisions[&space.contexts[&ExGuid::default()]] | ||
| 121 | .nodes | ||
| 122 | .iter() | ||
| 123 | .find_map(|(oid, node)| match &node.kind { | ||
| 124 | Kind::RichText { text, .. } => Some((*sid, *oid, text.clone())), | ||
| 125 | _ => None, | ||
| 126 | }) | ||
| 127 | }) | ||
| 128 | .unwrap() | ||
| 129 | } | ||
| 130 | |||
| 131 | #[test] | ||
| 132 | fn rebases_multiple_disjoint_remote_changes_and_persists_the_remote_receipt() { | ||
| 133 | let dir = tempfile::tempdir().unwrap(); | ||
| 134 | let path = dir.path().join("cache.sqlite"); | ||
| 135 | let source = onestore::create_section("sync.one", "ab🦀cd", "Fixture").unwrap(); | ||
| 136 | let (sid, oid, _) = text(&source); | ||
| 137 | let cache = Replica::create(&path, &source).unwrap(); | ||
| 138 | let id = cache | ||
| 139 | .edit_text(&source, sid, oid, 2..4, "🐈") | ||
| 140 | .unwrap() | ||
| 141 | .unwrap(); | ||
| 142 | let remote = onestore::replace_text(&source, sid, oid, 0..6, "Xab🦀cYd").unwrap(); | ||
| 143 | let mut server = Server::new(&remote); | ||
| 144 | let outcome = cache.sync_once(&mut server).unwrap().unwrap(); | ||
| 145 | assert_eq!(outcome.0, id); | ||
| 146 | assert!(matches!(outcome.1, EditStatus::Published { .. })); | ||
| 147 | assert_eq!(text(&server.durable).2, "Xab🐈cYd"); | ||
| 148 | assert_eq!(cache.snapshot().unwrap(), server.durable); | ||
| 149 | assert!(cache.pending().unwrap().is_empty()); | ||
| 150 | drop(cache); | ||
| 151 | let cache = Replica::open(&path).unwrap(); | ||
| 152 | assert_eq!(cache.status(id).unwrap(), Some(outcome.1)); | ||
| 153 | assert_eq!(cache.sync_once(&mut server).unwrap(), None); | ||
| 154 | assert_eq!(server.publications, 1); | ||
| 155 | assert_eq!(cache.status(id + 1).unwrap(), None); | ||
| 156 | let source = cache.snapshot().unwrap(); | ||
| 157 | let next = cache | ||
| 158 | .edit_text(&source, sid, oid, 0..0, "Later ") | ||
| 159 | .unwrap() | ||
| 160 | .unwrap(); | ||
| 161 | assert!(next > id); | ||
| 162 | } | ||
| 163 | |||
| 164 | #[test] | ||
| 165 | fn overlapping_changes_preserve_both_images_and_survive_restart() { | ||
| 166 | let dir = tempfile::tempdir().unwrap(); | ||
| 167 | let path = dir.path().join("cache.sqlite"); | ||
| 168 | let source = onestore::create_section("sync.one", "abc", "Fixture").unwrap(); | ||
| 169 | let (sid, oid, _) = text(&source); | ||
| 170 | let cache = Replica::create(&path, &source).unwrap(); | ||
| 171 | let id = cache | ||
| 172 | .edit_text(&source, sid, oid, 1..2, "L") | ||
| 173 | .unwrap() | ||
| 174 | .unwrap(); | ||
| 175 | let local = cache.snapshot().unwrap(); | ||
| 176 | let remote = onestore::replace_text(&source, sid, oid, 1..2, "R").unwrap(); | ||
| 177 | let mut server = Server::new(&remote); | ||
| 178 | assert_eq!( | ||
| 179 | cache.sync_once(&mut server).unwrap(), | ||
| 180 | Some((id, EditStatus::Conflict(ConflictKind::TextChanged))) | ||
| 181 | ); | ||
| 182 | assert_eq!(server.publications, 0); | ||
| 183 | drop(cache); | ||
| 184 | let cache = Replica::open(&path).unwrap(); | ||
| 185 | assert_eq!(cache.snapshot().unwrap(), local); | ||
| 186 | assert_eq!(cache.remote_snapshot().unwrap(), remote); | ||
| 187 | assert_eq!(cache.pending().unwrap().len(), 1); | ||
| 188 | assert_eq!( | ||
| 189 | cache.status(id).unwrap(), | ||
| 190 | Some(EditStatus::Conflict(ConflictKind::TextChanged)) | ||
| 191 | ); | ||
| 192 | } | ||
| 193 | |||
| 194 | #[test] | ||
| 195 | fn lost_replies_and_process_termination_never_blindly_replay_an_attempt() { | ||
| 196 | for fault in [ | ||
| 197 | Fault::UnknownBefore, | ||
| 198 | Fault::UnknownAfter, | ||
| 199 | Fault::PanicBefore, | ||
| 200 | Fault::PanicAfter, | ||
| 201 | ] { | ||
| 202 | let dir = tempfile::tempdir().unwrap(); | ||
| 203 | let path = dir.path().join("cache.sqlite"); | ||
| 204 | let source = onestore::create_section("sync.one", "abc", "Fixture").unwrap(); | ||
| 205 | let (sid, oid, _) = text(&source); | ||
| 206 | let cache = Replica::create(&path, &source).unwrap(); | ||
| 207 | let id = cache | ||
| 208 | .edit_text(&source, sid, oid, 0..0, "Once ") | ||
| 209 | .unwrap() | ||
| 210 | .unwrap(); | ||
| 211 | let mut server = Server::new(&source); | ||
| 212 | server.fault = fault; | ||
| 213 | let _ = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { | ||
| 214 | cache.sync_once(&mut server) | ||
| 215 | })); | ||
| 216 | let attempted = cache.status(id).unwrap().unwrap(); | ||
| 217 | assert!(matches!(attempted, EditStatus::AwaitingConfirmation { .. })); | ||
| 218 | drop(cache); | ||
| 219 | let cache = Replica::open(&path).unwrap(); | ||
| 220 | let result = cache.sync_once(&mut server).unwrap().unwrap(); | ||
| 221 | if matches!(fault, Fault::UnknownAfter | Fault::PanicAfter) { | ||
| 222 | assert!(matches!(result.1, EditStatus::Published { .. })); | ||
| 223 | assert_eq!(text(&server.durable).2, "Once abc"); | ||
| 224 | assert_eq!(server.confirmations, 1); | ||
| 225 | assert!(cache.pending().unwrap().is_empty()); | ||
| 226 | } else { | ||
| 227 | assert_eq!(result.1, attempted); | ||
| 228 | assert_eq!(cache.pending().unwrap().len(), 1); | ||
| 229 | assert_eq!(server.confirmations, 0); | ||
| 230 | assert_eq!(server.durable, source); | ||
| 231 | } | ||
| 232 | assert_eq!(server.publications, 1); | ||
| 233 | } | ||
| 234 | } | ||
| 235 | |||
| 236 | #[test] | ||
| 237 | fn failed_confirmation_does_not_promote_visible_bytes_to_a_receipt() { | ||
| 238 | let dir = tempfile::tempdir().unwrap(); | ||
| 239 | let path = dir.path().join("cache.sqlite"); | ||
| 240 | let source = onestore::create_section("sync.one", "abc", "Fixture").unwrap(); | ||
| 241 | let (sid, oid, _) = text(&source); | ||
| 242 | let cache = Replica::create(&path, &source).unwrap(); | ||
| 243 | let id = cache | ||
| 244 | .edit_text(&source, sid, oid, 0..0, "Once ") | ||
| 245 | .unwrap() | ||
| 246 | .unwrap(); | ||
| 247 | let mut server = Server::new(&source); | ||
| 248 | server.fault = Fault::UnknownAfter; | ||
| 249 | assert!(cache.sync_once(&mut server).is_err()); | ||
| 250 | server.fault = Fault::Confirm; | ||
| 251 | assert!(cache.sync_once(&mut server).is_err()); | ||
| 252 | assert_eq!(server.durable, source); | ||
| 253 | assert!(matches!( | ||
| 254 | cache.status(id).unwrap(), | ||
| 255 | Some(EditStatus::AwaitingConfirmation { .. }) | ||
| 256 | )); | ||
| 257 | assert!(matches!( | ||
| 258 | cache.sync_once(&mut server).unwrap(), | ||
| 259 | Some((_, EditStatus::Published { .. })) | ||
| 260 | )); | ||
| 261 | assert_eq!(server.publications, 1); | ||
| 262 | assert_eq!(server.confirmations, 2); | ||
| 263 | assert_eq!(server.visible, server.durable); | ||
| 264 | } | ||
| 265 | |||
| 266 | #[test] | ||
| 267 | fn confirmation_cleanup_failure_still_records_a_durable_receipt() { | ||
| 268 | let dir = tempfile::tempdir().unwrap(); | ||
| 269 | let source = onestore::create_section("sync.one", "abc", "Fixture").unwrap(); | ||
| 270 | let (sid, oid, _) = text(&source); | ||
| 271 | let cache = Replica::create(dir.path().join("cache.sqlite"), &source).unwrap(); | ||
| 272 | let id = cache | ||
| 273 | .edit_text(&source, sid, oid, 0..0, "Once ") | ||
| 274 | .unwrap() | ||
| 275 | .unwrap(); | ||
| 276 | let mut server = Server::new(&source); | ||
| 277 | server.fault = Fault::UnknownAfter; | ||
| 278 | assert!(cache.sync_once(&mut server).is_err()); | ||
| 279 | server.fault = Fault::ConfirmCommitted; | ||
| 280 | assert!( | ||
| 281 | matches!(cache.sync_once(&mut server), Err(Error::Remote(error)) if error.state == CommitState::Committed) | ||
| 282 | ); | ||
| 283 | assert!(matches!( | ||
| 284 | cache.status(id).unwrap(), | ||
| 285 | Some(EditStatus::Published { .. }) | ||
| 286 | )); | ||
| 287 | assert!(cache.pending().unwrap().is_empty()); | ||
| 288 | assert_eq!(server.visible, server.durable); | ||
| 289 | assert_eq!(server.publications, 1); | ||
| 290 | assert_eq!(server.confirmations, 1); | ||
| 291 | } | ||
| 292 | |||
| 293 | #[test] | ||
| 294 | fn proven_unpublished_attempts_retry_and_committed_cleanup_errors_keep_receipts() { | ||
| 295 | for fault in [Fault::Before, Fault::Committed] { | ||
| 296 | let dir = tempfile::tempdir().unwrap(); | ||
| 297 | let path = dir.path().join("cache.sqlite"); | ||
| 298 | let source = onestore::create_section("sync.one", "abc", "Fixture").unwrap(); | ||
| 299 | let (sid, oid, _) = text(&source); | ||
| 300 | let cache = Replica::create(&path, &source).unwrap(); | ||
| 301 | let id = cache | ||
| 302 | .edit_text(&source, sid, oid, 0..0, "Once ") | ||
| 303 | .unwrap() | ||
| 304 | .unwrap(); | ||
| 305 | let mut server = Server::new(&source); | ||
| 306 | server.fault = fault; | ||
| 307 | assert!(matches!( | ||
| 308 | cache.sync_once(&mut server), | ||
| 309 | Err(Error::Remote(_)) | ||
| 310 | )); | ||
| 311 | if matches!(fault, Fault::Before) { | ||
| 312 | assert_eq!(cache.status(id).unwrap(), Some(EditStatus::Pending)); | ||
| 313 | assert!(matches!( | ||
| 314 | cache.sync_once(&mut server).unwrap(), | ||
| 315 | Some((_, EditStatus::Published { .. })) | ||
| 316 | )); | ||
| 317 | assert_eq!(server.publications, 2); | ||
| 318 | } else { | ||
| 319 | assert!(matches!( | ||
| 320 | cache.status(id).unwrap(), | ||
| 321 | Some(EditStatus::Published { .. }) | ||
| 322 | )); | ||
| 323 | assert_eq!(cache.sync_once(&mut server).unwrap(), None); | ||
| 324 | assert_eq!(server.publications, 1); | ||
| 325 | } | ||
| 326 | assert_eq!(text(&server.durable).2, "Once abc"); | ||
| 327 | } | ||
| 328 | } | ||
| 329 | |||
| 330 | #[test] | ||
| 331 | fn database_failures_before_and_after_publication_preserve_recovery_state() { | ||
| 332 | for table in ["attempt", "receipts"] { | ||
| 333 | let dir = tempfile::tempdir().unwrap(); | ||
| 334 | let path = dir.path().join("cache.sqlite"); | ||
| 335 | let source = onestore::create_section("sync.one", "abc", "Fixture").unwrap(); | ||
| 336 | let (sid, oid, _) = text(&source); | ||
| 337 | let cache = Replica::create(&path, &source).unwrap(); | ||
| 338 | let id = cache | ||
| 339 | .edit_text(&source, sid, oid, 0..0, "Once ") | ||
| 340 | .unwrap() | ||
| 341 | .unwrap(); | ||
| 342 | drop(cache); | ||
| 343 | let db = rusqlite::Connection::open(&path).unwrap(); | ||
| 344 | db.execute_batch(&format!("CREATE TRIGGER interrupted BEFORE INSERT ON {table} BEGIN SELECT RAISE(ABORT,'Injected cache failure'); END;")).unwrap(); | ||
| 345 | drop(db); | ||
| 346 | let cache = Replica::open(&path).unwrap(); | ||
| 347 | let mut server = Server::new(&source); | ||
| 348 | assert!(matches!( | ||
| 349 | cache.sync_once(&mut server), | ||
| 350 | Err(Error::Database(_)) | ||
| 351 | )); | ||
| 352 | assert_eq!(server.publications, usize::from(table == "receipts")); | ||
| 353 | assert_eq!(cache.pending().unwrap().len(), 1); | ||
| 354 | drop(cache); | ||
| 355 | let db = rusqlite::Connection::open(&path).unwrap(); | ||
| 356 | db.execute_batch("DROP TRIGGER interrupted").unwrap(); | ||
| 357 | drop(db); | ||
| 358 | let cache = Replica::open(&path).unwrap(); | ||
| 359 | assert!(matches!( | ||
| 360 | cache.sync_once(&mut server).unwrap(), | ||
| 361 | Some((_, EditStatus::Published { .. })) | ||
| 362 | )); | ||
| 363 | assert!(matches!( | ||
| 364 | cache.status(id).unwrap(), | ||
| 365 | Some(EditStatus::Published { .. }) | ||
| 366 | )); | ||
| 367 | assert_eq!(server.publications, 1); | ||
| 368 | assert_eq!(server.confirmations, usize::from(table == "receipts")); | ||
| 369 | assert_eq!(text(&server.durable).2, "Once abc"); | ||
| 370 | } | ||
| 371 | } | ||
| 372 | |||
| 373 | #[test] | ||
| 374 | fn twelve_local_editors_progress_during_remote_reads_publication_and_confirmation() { | ||
| 375 | struct Paused { | ||
| 376 | server: Server, | ||
| 377 | phase: &'static str, | ||
| 378 | entered: std::sync::mpsc::Sender<()>, | ||
| 379 | resume: std::sync::mpsc::Receiver<()>, | ||
| 380 | } | ||
| 381 | impl Paused { | ||
| 382 | fn wait(&self, phase: &str) { | ||
| 383 | if self.phase == phase { | ||
| 384 | self.entered.send(()).unwrap(); | ||
| 385 | self.resume | ||
| 386 | .recv_timeout(std::time::Duration::from_secs(5)) | ||
| 387 | .unwrap(); | ||
| 388 | } | ||
| 389 | } | ||
| 390 | } | ||
| 391 | impl Remote for Paused { | ||
| 392 | fn read(&mut self) -> io::Result<Vec<u8>> { | ||
| 393 | self.wait("read"); | ||
| 394 | self.server.read() | ||
| 395 | } | ||
| 396 | fn publish(&mut self, edit: &PreparedEdit<'_>) -> Result<(), CommitError> { | ||
| 397 | self.wait("publish"); | ||
| 398 | self.server.publish(edit) | ||
| 399 | } | ||
| 400 | fn confirm(&mut self, source: &[u8]) -> Result<(), CommitError> { | ||
| 401 | self.wait("confirm"); | ||
| 402 | self.server.confirm(source) | ||
| 403 | } | ||
| 404 | } | ||
| 405 | for phase in ["read", "publish", "confirm"] { | ||
| 406 | let dir = tempfile::tempdir().unwrap(); | ||
| 407 | let source = onestore::create_section("sync.one", "abc", "Fixture").unwrap(); | ||
| 408 | let (sid, oid, _) = text(&source); | ||
| 409 | let cache = Replica::create(dir.path().join("cache.sqlite"), &source).unwrap(); | ||
| 410 | let first = cache | ||
| 411 | .edit_text(&source, sid, oid, 0..0, "First ") | ||
| 412 | .unwrap() | ||
| 413 | .unwrap(); | ||
| 414 | let mut server = Server::new(&source); | ||
| 415 | if phase == "confirm" { | ||
| 416 | server.fault = Fault::UnknownAfter; | ||
| 417 | assert!(cache.sync_once(&mut server).is_err()); | ||
| 418 | } | ||
| 419 | let (entered_tx, entered_rx) = std::sync::mpsc::channel(); | ||
| 420 | let (resume_tx, resume_rx) = std::sync::mpsc::channel(); | ||
| 421 | let mut paused = Paused { | ||
| 422 | server, | ||
| 423 | phase, | ||
| 424 | entered: entered_tx, | ||
| 425 | resume: resume_rx, | ||
| 426 | }; | ||
| 427 | let mut server = std::thread::scope(|scope| { | ||
| 428 | let running = scope.spawn(|| { | ||
| 429 | let result = cache.sync_once(&mut paused); | ||
| 430 | assert!( | ||
| 431 | matches!(result, Ok(Some((id, EditStatus::Published { .. }))) if id == first) | ||
| 432 | ); | ||
| 433 | paused.server | ||
| 434 | }); | ||
| 435 | entered_rx | ||
| 436 | .recv_timeout(std::time::Duration::from_secs(5)) | ||
| 437 | .unwrap(); | ||
| 438 | assert!( | ||
| 439 | matches!(cache.sync_once(&mut Server::new(&source)),Err(Error::Io(error)) if error.kind()==io::ErrorKind::WouldBlock) | ||
| 440 | ); | ||
| 441 | assert!( | ||
| 442 | matches!(cache.rebase_conflict(first, &[], &[], 0..0), Err(Error::Io(error)) if error.kind() == io::ErrorKind::WouldBlock) | ||
| 443 | ); | ||
| 444 | let started = std::time::Instant::now(); | ||
| 445 | let handles: Vec<_> = (0..12) | ||
| 446 | .map(|writer| { | ||
| 447 | let cache = &cache; | ||
| 448 | scope.spawn(move || { | ||
| 449 | loop { | ||
| 450 | let snapshot = cache.snapshot().unwrap(); | ||
| 451 | match cache.edit_text( | ||
| 452 | &snapshot, | ||
| 453 | sid, | ||
| 454 | oid, | ||
| 455 | 0..0, | ||
| 456 | &format!("[{writer}] "), | ||
| 457 | ) { | ||
| 458 | Ok(Some(id)) => break id, | ||
| 459 | Err(Error::Io(error)) | ||
| 460 | if error.kind() == io::ErrorKind::ResourceBusy => {} | ||
| 461 | other => panic!("Unexpected local outcome {other:?}"), | ||
| 462 | } | ||
| 463 | } | ||
| 464 | }) | ||
| 465 | }) | ||
| 466 | .collect(); | ||
| 467 | let ids: std::collections::BTreeSet<_> = handles | ||
| 468 | .into_iter() | ||
| 469 | .map(|handle| handle.join().unwrap()) | ||
| 470 | .collect(); | ||
| 471 | assert_eq!(ids.len(), 12); | ||
| 472 | assert!( | ||
| 473 | started.elapsed() < std::time::Duration::from_secs(2), | ||
| 474 | "Local edits waited for remote {phase}" | ||
| 475 | ); | ||
| 476 | resume_tx.send(()).unwrap(); | ||
| 477 | running.join().unwrap() | ||
| 478 | }); | ||
| 479 | assert_eq!(cache.pending().unwrap().len(), 12); | ||
| 480 | let expected = text(&cache.snapshot().unwrap()).2; | ||
| 481 | for _ in 0..12 { | ||
| 482 | assert!(matches!( | ||
| 483 | cache.sync_once(&mut server).unwrap(), | ||
| 484 | Some((_, EditStatus::Published { .. })) | ||
| 485 | )); | ||
| 486 | } | ||
| 487 | assert!(cache.pending().unwrap().is_empty()); | ||
| 488 | assert_eq!(server.publications, 13); | ||
| 489 | assert_eq!(text(&server.durable).2, expected); | ||
| 490 | assert_eq!(cache.snapshot().unwrap(), server.durable); | ||
| 491 | } | ||
| 492 | } | ||
| 493 | |||
| 494 | #[test] | ||
| 495 | fn unrelated_remote_files_never_replace_a_local_cache() { | ||
| 496 | let dir = tempfile::tempdir().unwrap(); | ||
| 497 | let source = onestore::create_section("sync.one", "abc", "Fixture").unwrap(); | ||
| 498 | let (sid, oid, _) = text(&source); | ||
| 499 | let cache = Replica::create(dir.path().join("cache.sqlite"), &source).unwrap(); | ||
| 500 | let other = onestore::create_section("other.one", "abc", "Fixture").unwrap(); | ||
| 501 | let mut server = Server::new(&other); | ||
| 502 | for pending in [false, true] { | ||
| 503 | if pending { | ||
| 504 | cache.edit_text(&source, sid, oid, 0..0, "Local ").unwrap(); | ||
| 505 | } | ||
| 506 | let before = cache.snapshot().unwrap(); | ||
| 507 | assert!( | ||
| 508 | matches!(cache.sync_once(&mut server),Err(Error::Io(error)) if error.kind()==io::ErrorKind::InvalidInput) | ||
| 509 | ); | ||
| 510 | assert_eq!(cache.snapshot().unwrap(), before); | ||
| 511 | assert_eq!(cache.remote_snapshot().unwrap(), source); | ||
| 512 | assert_eq!(server.publications, 0); | ||
| 513 | } | ||
| 514 | } | ||
| 515 | |||
| 516 | #[test] | ||
| 517 | fn version_one_cache_migration_preserves_images_intents_and_local_ids() { | ||
| 518 | let dir = tempfile::tempdir().unwrap(); | ||
| 519 | let path = dir.path().join("cache.sqlite"); | ||
| 520 | let source = onestore::create_section("sync.one", "abc", "Fixture").unwrap(); | ||
| 521 | let (sid, oid, _) = text(&source); | ||
| 522 | let cache = Replica::create(&path, &source).unwrap(); | ||
| 523 | let id = cache | ||
| 524 | .edit_text(&source, sid, oid, 0..0, "Local ") | ||
| 525 | .unwrap() | ||
| 526 | .unwrap(); | ||
| 527 | let snapshot = cache.snapshot().unwrap(); | ||
| 528 | let pending = cache.pending().unwrap(); | ||
| 529 | drop(cache); | ||
| 530 | let db = rusqlite::Connection::open(&path).unwrap(); | ||
| 531 | db.execute_batch( | ||
| 532 | "DROP TABLE attempt; DROP TABLE conflicts; DROP TABLE receipts; DROP TABLE edits; | ||
| 533 | CREATE TABLE edits ( | ||
| 534 | id INTEGER PRIMARY KEY AUTOINCREMENT CHECK(id>0), space TEXT NOT NULL, | ||
| 535 | object TEXT NOT NULL, before_text TEXT NOT NULL, | ||
| 536 | start INTEGER NOT NULL CHECK(start BETWEEN 0 AND 4294967295), | ||
| 537 | end INTEGER NOT NULL CHECK(end BETWEEN start AND 4294967295), replacement TEXT NOT NULL | ||
| 538 | ) STRICT; PRAGMA user_version=1;", | ||
| 539 | ) | ||
| 540 | .unwrap(); | ||
| 541 | db.execute("INSERT INTO edits(id,space,object,before_text,start,end,replacement) VALUES (?1,?2,?3,'abc',0,0,'Local ')",rusqlite::params![i64::try_from(id).unwrap(),sid.to_string(),oid.to_string()]).unwrap(); | ||
| 542 | drop(db); | ||
| 543 | let cache = Replica::open(&path).unwrap(); | ||
| 544 | assert_eq!(cache.snapshot().unwrap(), snapshot); | ||
| 545 | assert_eq!(cache.pending().unwrap(), pending); | ||
| 546 | assert_eq!(cache.status(id).unwrap(), Some(EditStatus::Pending)); | ||
| 547 | let mut server = Server::new(&source); | ||
| 548 | assert!(matches!( | ||
| 549 | cache.sync_once(&mut server).unwrap(), | ||
| 550 | Some((_, EditStatus::Published { .. })) | ||
| 551 | )); | ||
| 552 | drop(cache); | ||
| 553 | let db = rusqlite::Connection::open(&path).unwrap(); | ||
| 554 | assert_eq!( | ||
| 555 | db.pragma_query_value(None, "user_version", |row| row.get::<_, u32>(0)) | ||
| 556 | .unwrap(), | ||
| 557 | 4 | ||
| 558 | ); | ||
| 559 | } | ||
| 560 | |||
| 561 | #[test] | ||
| 562 | fn reviewed_conflict_rebase_preserves_twelve_dependent_edits_and_survives_reopen() { | ||
| 563 | let dir = tempfile::tempdir().unwrap(); | ||
| 564 | let path = dir.path().join("cache.sqlite"); | ||
| 565 | let source = onestore::create_section("resolve.one", "abc", "Fixture").unwrap(); | ||
| 566 | let (sid, oid, _) = text(&source); | ||
| 567 | let cache = Replica::create(&path, &source).unwrap(); | ||
| 568 | let first = cache | ||
| 569 | .edit_text(&source, sid, oid, 1..2, "L") | ||
| 570 | .unwrap() | ||
| 571 | .unwrap(); | ||
| 572 | for n in 0..12 { | ||
| 573 | cache | ||
| 574 | .edit_text( | ||
| 575 | &cache.snapshot().unwrap(), | ||
| 576 | sid, | ||
| 577 | oid, | ||
| 578 | 0..0, | ||
| 579 | &format!("[{n}] "), | ||
| 580 | ) | ||
| 581 | .unwrap(); | ||
| 582 | } | ||
| 583 | let local = cache.snapshot().unwrap(); | ||
| 584 | let pending = cache.pending().unwrap(); | ||
| 585 | let remote = onestore::replace_text(&source, sid, oid, 0..3, "aRcZ").unwrap(); | ||
| 586 | let mut server = Server::new(&remote); | ||
| 587 | assert_eq!( | ||
| 588 | cache.sync_once(&mut server).unwrap(), | ||
| 589 | Some((first, EditStatus::Conflict(ConflictKind::TextChanged))) | ||
| 590 | ); | ||
| 591 | cache.rebase_conflict(first, &local, &remote, 1..2).unwrap(); | ||
| 592 | assert_eq!(cache.snapshot().unwrap(), local); | ||
| 593 | assert_eq!(cache.remote_snapshot().unwrap(), remote); | ||
| 594 | let rebased = cache.pending().unwrap(); | ||
| 595 | assert_eq!(&rebased[1..], &pending[1..]); | ||
| 596 | assert_eq!(rebased[0].id, first); | ||
| 597 | let onestore_offline::Operation::Text(updated) = &rebased[0].operation else { | ||
| 598 | panic!() | ||
| 599 | }; | ||
| 600 | let onestore_offline::Operation::Text(previous) = &pending[0].operation else { | ||
| 601 | panic!() | ||
| 602 | }; | ||
| 603 | assert_eq!(updated.replacement, previous.replacement); | ||
| 604 | assert_eq!(updated.before, "aRcZ"); | ||
| 605 | assert_eq!(cache.status(first).unwrap(), Some(EditStatus::Pending)); | ||
| 606 | drop(cache); | ||
| 607 | let cache = Replica::open(&path).unwrap(); | ||
| 608 | assert_eq!(cache.pending().unwrap(), rebased); | ||
| 609 | assert_eq!(cache.snapshot().unwrap(), local); | ||
| 610 | for intent in &pending { | ||
| 611 | assert!( | ||
| 612 | matches!(cache.sync_once(&mut server).unwrap(), Some((id, EditStatus::Published { .. })) if id == intent.id) | ||
| 613 | ); | ||
| 614 | } | ||
| 615 | assert_eq!(server.publications, 13); | ||
| 616 | assert_eq!(text(&server.durable).2, text(&local).2 + "Z"); | ||
| 617 | assert_eq!(cache.snapshot().unwrap(), server.durable); | ||
| 618 | assert!(cache.pending().unwrap().is_empty()); | ||
| 619 | } | ||
| 620 | |||
| 621 | #[test] | ||
| 622 | fn conflict_review_rejects_stale_images_invalid_ranges_and_nonconflicting_states() { | ||
| 623 | let dir = tempfile::tempdir().unwrap(); | ||
| 624 | let source = onestore::create_section("resolve.one", "abc", "Fixture").unwrap(); | ||
| 625 | let (sid, oid, _) = text(&source); | ||
| 626 | let cache = Replica::create(dir.path().join("cache.sqlite"), &source).unwrap(); | ||
| 627 | let id = cache | ||
| 628 | .edit_text(&source, sid, oid, 1..2, "L") | ||
| 629 | .unwrap() | ||
| 630 | .unwrap(); | ||
| 631 | let local = cache.snapshot().unwrap(); | ||
| 632 | assert!( | ||
| 633 | matches!(cache.rebase_conflict(id, &local, &source, 1..2), Err(Error::Io(error)) if error.kind() == io::ErrorKind::InvalidInput) | ||
| 634 | ); | ||
| 635 | let remote = onestore::replace_text(&source, sid, oid, 0..3, "🦀Rc").unwrap(); | ||
| 636 | let mut server = Server::new(&remote); | ||
| 637 | assert!(matches!( | ||
| 638 | cache.sync_once(&mut server).unwrap(), | ||
| 639 | Some((_, EditStatus::Conflict(_))) | ||
| 640 | )); | ||
| 641 | let pending = cache.pending().unwrap(); | ||
| 642 | for range in [1..2, 100..101] { | ||
| 643 | assert!(matches!( | ||
| 644 | cache.rebase_conflict(id, &local, &remote, range), | ||
| 645 | Err(Error::Document(_)) | ||
| 646 | )); | ||
| 647 | assert_eq!(cache.pending().unwrap(), pending); | ||
| 648 | } | ||
| 649 | assert!( | ||
| 650 | matches!(cache.rebase_conflict(id + 1, &local, &remote, 2..3), Err(Error::Io(error)) if error.kind() == io::ErrorKind::InvalidInput) | ||
| 651 | ); | ||
| 652 | cache.edit_text(&local, sid, oid, 0..0, "Later ").unwrap(); | ||
| 653 | let changed = cache.snapshot().unwrap(); | ||
| 654 | assert!( | ||
| 655 | matches!(cache.rebase_conflict(id, &local, &remote, 2..3), Err(Error::Io(error)) if error.kind() == io::ErrorKind::ResourceBusy) | ||
| 656 | ); | ||
| 657 | let new_remote = onestore::replace_text(&remote, sid, oid, 2..3, "Q").unwrap(); | ||
| 658 | server.visible = new_remote.clone(); | ||
| 659 | server.durable = new_remote; | ||
| 660 | assert!(matches!( | ||
| 661 | cache.sync_once(&mut server).unwrap(), | ||
| 662 | Some((_, EditStatus::Conflict(_))) | ||
| 663 | )); | ||
| 664 | assert!( | ||
| 665 | matches!(cache.rebase_conflict(id, &changed, &remote, 2..3), Err(Error::Io(error)) if error.kind() == io::ErrorKind::ResourceBusy) | ||
| 666 | ); | ||
| 667 | assert_eq!(cache.snapshot().unwrap(), changed); | ||
| 668 | assert_eq!(cache.pending().unwrap()[0], pending[0]); | ||
| 669 | assert_eq!(server.publications, 0); | ||
| 670 | |||
| 671 | let current = cache.remote_snapshot().unwrap(); | ||
| 672 | cache.rebase_conflict(id, &changed, ¤t, 2..3).unwrap(); | ||
| 673 | server.fault = Fault::UnknownBefore; | ||
| 674 | assert!( | ||
| 675 | matches!(cache.sync_once(&mut server), Err(Error::Remote(error)) if error.state == CommitState::Unknown) | ||
| 676 | ); | ||
| 677 | let attempted = cache.status(id).unwrap(); | ||
| 678 | let pending = cache.pending().unwrap(); | ||
| 679 | assert!( | ||
| 680 | matches!(cache.rebase_conflict(id, &changed, ¤t, 2..3), Err(Error::Io(error)) if error.kind() == io::ErrorKind::InvalidInput) | ||
| 681 | ); | ||
| 682 | assert_eq!(cache.status(id).unwrap(), attempted); | ||
| 683 | assert_eq!(cache.pending().unwrap(), pending); | ||
| 684 | assert_eq!(server.publications, 1); | ||
| 685 | } | ||
| 686 | |||
| 687 | #[test] | ||
| 688 | fn failure_between_rebase_and_conflict_clear_rolls_back_the_entire_resolution() { | ||
| 689 | let dir = tempfile::tempdir().unwrap(); | ||
| 690 | let path = dir.path().join("cache.sqlite"); | ||
| 691 | let source = onestore::create_section("resolve.one", "abc", "Fixture").unwrap(); | ||
| 692 | let (sid, oid, _) = text(&source); | ||
| 693 | let cache = Replica::create(&path, &source).unwrap(); | ||
| 694 | let id = cache | ||
| 695 | .edit_text(&source, sid, oid, 1..2, "L") | ||
| 696 | .unwrap() | ||
| 697 | .unwrap(); | ||
| 698 | let remote = onestore::replace_text(&source, sid, oid, 0..3, "XaRc").unwrap(); | ||
| 699 | let mut server = Server::new(&remote); | ||
| 700 | cache.sync_once(&mut server).unwrap(); | ||
| 701 | let local = cache.snapshot().unwrap(); | ||
| 702 | let pending = cache.pending().unwrap(); | ||
| 703 | let status = cache.status(id).unwrap(); | ||
| 704 | drop(cache); | ||
| 705 | let connection = rusqlite::Connection::open(&path).unwrap(); | ||
| 706 | connection.execute_batch("CREATE TRIGGER fail_clear BEFORE DELETE ON conflicts BEGIN SELECT RAISE(ABORT, 'test conflict clear failure'); END;").unwrap(); | ||
| 707 | drop(connection); | ||
| 708 | let cache = Replica::open(&path).unwrap(); | ||
| 709 | assert!(matches!( | ||
| 710 | cache.rebase_conflict(id, &local, &remote, 2..3), | ||
| 711 | Err(Error::Database(_)) | ||
| 712 | )); | ||
| 713 | drop(cache); | ||
| 714 | let cache = Replica::open(&path).unwrap(); | ||
| 715 | assert_eq!(cache.pending().unwrap(), pending); | ||
| 716 | assert_eq!(cache.status(id).unwrap(), status); | ||
| 717 | assert_eq!(cache.snapshot().unwrap(), local); | ||
| 718 | assert_eq!(cache.remote_snapshot().unwrap(), remote); | ||
| 719 | assert_eq!(server.publications, 0); | ||
| 720 | } | ||
| 721 | |||
| 722 | #[test] | ||
| 723 | fn seeded_reviewed_ranges_preserve_unicode_and_edits_inside_the_original_replacement() { | ||
| 724 | let dir = tempfile::tempdir().unwrap(); | ||
| 725 | let original: Vec<_> = "abcdefghij🦀klmnop".chars().collect(); | ||
| 726 | let mut seed = 911_u64; | ||
| 727 | for case in 0..64 { | ||
| 728 | seed ^= seed << 13; | ||
| 729 | seed ^= seed >> 7; | ||
| 730 | seed ^= seed << 17; | ||
| 731 | let at = seed as usize % original.len(); | ||
| 732 | let prefix = "[".repeat((seed >> 8) as usize % 5); | ||
| 733 | let suffix = "]".repeat((seed >> 16) as usize % 5); | ||
| 734 | let start: u32 = original[..at].iter().map(|ch| ch.len_utf16() as u32).sum(); | ||
| 735 | let end = start + original[at].len_utf16() as u32; | ||
| 736 | let source = onestore::create_section( | ||
| 737 | "resolve.one", | ||
| 738 | &original.iter().collect::<String>(), | ||
| 739 | "Fixture", | ||
| 740 | ) | ||
| 741 | .unwrap(); | ||
| 742 | let (sid, oid, _) = text(&source); | ||
| 743 | let cache = Replica::create(dir.path().join(format!("{case}.sqlite")), &source).unwrap(); | ||
| 744 | let id = cache | ||
| 745 | .edit_text(&source, sid, oid, start..end, "λ🦊μ") | ||
| 746 | .unwrap() | ||
| 747 | .unwrap(); | ||
| 748 | let dependent = cache | ||
| 749 | .edit_text( | ||
| 750 | &cache.snapshot().unwrap(), | ||
| 751 | sid, | ||
| 752 | oid, | ||
| 753 | start + 1..start + 3, | ||
| 754 | "🐕", | ||
| 755 | ) | ||
| 756 | .unwrap() | ||
| 757 | .unwrap(); | ||
| 758 | let mut remote_text = original.clone(); | ||
| 759 | remote_text.splice(at..at + 1, "Ω🐈π".chars()); | ||
| 760 | let remote_text = prefix.clone() + &remote_text.iter().collect::<String>() + &suffix; | ||
| 761 | let remote = onestore::replace_text( | ||
| 762 | &source, | ||
| 763 | sid, | ||
| 764 | oid, | ||
| 765 | 0..original.iter().map(|ch| ch.len_utf16() as u32).sum(), | ||
| 766 | &remote_text, | ||
| 767 | ) | ||
| 768 | .unwrap(); | ||
| 769 | let mut server = Server::new(&remote); | ||
| 770 | assert!( | ||
| 771 | matches!( | ||
| 772 | cache.sync_once(&mut server).unwrap(), | ||
| 773 | Some((_, EditStatus::Conflict(_))) | ||
| 774 | ), | ||
| 775 | "case {case}, seed {seed}" | ||
| 776 | ); | ||
| 777 | let at_remote = start + prefix.len() as u32; | ||
| 778 | cache | ||
| 779 | .rebase_conflict( | ||
| 780 | id, | ||
| 781 | &cache.snapshot().unwrap(), | ||
| 782 | &remote, | ||
| 783 | at_remote..at_remote + 4, | ||
| 784 | ) | ||
| 785 | .unwrap(); | ||
| 786 | for expected in [id, dependent] { | ||
| 787 | assert!( | ||
| 788 | matches!(cache.sync_once(&mut server).unwrap(), Some((actual, EditStatus::Published { .. })) if actual == expected), | ||
| 789 | "case {case}, seed {seed}" | ||
| 790 | ); | ||
| 791 | } | ||
| 792 | let mut expected = original.clone(); | ||
| 793 | expected.splice(at..at + 1, "λ🐕μ".chars()); | ||
| 794 | let expected = prefix + &expected.iter().collect::<String>() + &suffix; | ||
| 795 | assert_eq!( | ||
| 796 | text(&server.durable).2, | ||
| 797 | expected, | ||
| 798 | "case {case}, seed {seed}" | ||
| 799 | ); | ||
| 800 | assert_eq!(server.publications, 2); | ||
| 801 | assert!(cache.pending().unwrap().is_empty()); | ||
| 802 | } | ||
| 803 | } | ||
| 804 | |||
| 805 | #[test] | ||
| 806 | fn remote_changes_after_review_cannot_be_overwritten_by_the_reviewed_placement() { | ||
| 807 | struct ChangedAfterRead { | ||
| 808 | server: Server, | ||
| 809 | change: Option<Vec<u8>>, | ||
| 810 | } | ||
| 811 | impl Remote for ChangedAfterRead { | ||
| 812 | fn read(&mut self) -> io::Result<Vec<u8>> { | ||
| 813 | let snapshot = self.server.read()?; | ||
| 814 | if let Some(changed) = self.change.take() { | ||
| 815 | self.server.visible = changed.clone(); | ||
| 816 | self.server.durable = changed; | ||
| 817 | } | ||
| 818 | Ok(snapshot) | ||
| 819 | } | ||
| 820 | fn publish(&mut self, edit: &PreparedEdit<'_>) -> Result<(), CommitError> { | ||
| 821 | self.server.publish(edit) | ||
| 822 | } | ||
| 823 | fn confirm(&mut self, snapshot: &[u8]) -> Result<(), CommitError> { | ||
| 824 | self.server.confirm(snapshot) | ||
| 825 | } | ||
| 826 | } | ||
| 827 | for after_read in [false, true] { | ||
| 828 | let dir = tempfile::tempdir().unwrap(); | ||
| 829 | let source = onestore::create_section("resolve.one", "abc", "Fixture").unwrap(); | ||
| 830 | let (sid, oid, _) = text(&source); | ||
| 831 | let cache = Replica::create(dir.path().join("cache.sqlite"), &source).unwrap(); | ||
| 832 | let id = cache | ||
| 833 | .edit_text(&source, sid, oid, 1..2, "L") | ||
| 834 | .unwrap() | ||
| 835 | .unwrap(); | ||
| 836 | let local = cache.snapshot().unwrap(); | ||
| 837 | let remote = onestore::replace_text(&source, sid, oid, 1..2, "R").unwrap(); | ||
| 838 | let mut remote = ChangedAfterRead { | ||
| 839 | server: Server::new(&remote), | ||
| 840 | change: None, | ||
| 841 | }; | ||
| 842 | assert!(matches!( | ||
| 843 | cache.sync_once(&mut remote).unwrap(), | ||
| 844 | Some((_, EditStatus::Conflict(_))) | ||
| 845 | )); | ||
| 846 | cache | ||
| 847 | .rebase_conflict(id, &local, &cache.remote_snapshot().unwrap(), 1..2) | ||
| 848 | .unwrap(); | ||
| 849 | let changed = onestore::replace_text(&remote.server.visible, sid, oid, 1..2, "Q").unwrap(); | ||
| 850 | if after_read { | ||
| 851 | remote.change = Some(changed.clone()); | ||
| 852 | } else { | ||
| 853 | remote.server.visible = changed.clone(); | ||
| 854 | remote.server.durable = changed.clone(); | ||
| 855 | } | ||
| 856 | if after_read { | ||
| 857 | assert!( | ||
| 858 | matches!(cache.sync_once(&mut remote), Err(Error::Remote(error)) if error.state == CommitState::NotCommitted) | ||
| 859 | ); | ||
| 860 | assert_eq!(cache.status(id).unwrap(), Some(EditStatus::Pending)); | ||
| 861 | } | ||
| 862 | assert_eq!( | ||
| 863 | cache.sync_once(&mut remote).unwrap(), | ||
| 864 | Some((id, EditStatus::Conflict(ConflictKind::TextChanged))) | ||
| 865 | ); | ||
| 866 | assert_eq!(remote.server.durable, changed); | ||
| 867 | assert_eq!(remote.server.visible, changed); | ||
| 868 | assert_eq!(remote.server.publications, usize::from(after_read)); | ||
| 869 | assert_eq!(cache.snapshot().unwrap(), local); | ||
| 870 | let onestore_offline::Operation::Text(edit) = &cache.pending().unwrap()[0].operation else { | ||
| 871 | panic!() | ||
| 872 | }; | ||
| 873 | assert_eq!(edit.replacement, "L"); | ||
| 874 | } | ||
| 875 | } | ||
| 876 | |||
| 877 | mod worker { | ||
| 878 | use super::*; | ||
| 879 | use std::{ | ||
| 880 | sync::{Arc, Mutex, mpsc}, | ||
| 881 | time::{Duration, Instant}, | ||
| 882 | }; | ||
| 883 | |||
| 884 | #[derive(Clone)] | ||
| 885 | struct Shared(Arc<Mutex<Server>>); | ||
| 886 | |||
| 887 | impl Remote for Shared { | ||
| 888 | fn read(&mut self) -> io::Result<Vec<u8>> { | ||
| 889 | self.0.lock().unwrap().read() | ||
| 890 | } | ||
| 891 | fn publish(&mut self, edit: &PreparedEdit<'_>) -> Result<(), CommitError> { | ||
| 892 | self.0.lock().unwrap().publish(edit) | ||
| 893 | } | ||
| 894 | fn confirm(&mut self, snapshot: &[u8]) -> Result<(), CommitError> { | ||
| 895 | self.0.lock().unwrap().confirm(snapshot) | ||
| 896 | } | ||
| 897 | } | ||
| 898 | |||
| 899 | #[test] | ||
| 900 | fn reconnects_after_connect_read_and_uncertain_publish_without_replaying() { | ||
| 901 | struct Session { | ||
| 902 | shared: Shared, | ||
| 903 | fail_read: bool, | ||
| 904 | } | ||
| 905 | impl Remote for Session { | ||
| 906 | fn read(&mut self) -> io::Result<Vec<u8>> { | ||
| 907 | if self.fail_read { | ||
| 908 | return Err(io::ErrorKind::ConnectionReset.into()); | ||
| 909 | } | ||
| 910 | self.shared.read() | ||
| 911 | } | ||
| 912 | fn publish(&mut self, edit: &PreparedEdit<'_>) -> Result<(), CommitError> { | ||
| 913 | self.shared.publish(edit) | ||
| 914 | } | ||
| 915 | fn confirm(&mut self, snapshot: &[u8]) -> Result<(), CommitError> { | ||
| 916 | self.shared.confirm(snapshot) | ||
| 917 | } | ||
| 918 | } | ||
| 919 | let dir = tempfile::tempdir().unwrap(); | ||
| 920 | let path = dir.path().join("cache.sqlite"); | ||
| 921 | let source = onestore::create_section("worker.one", "abc", "Fixture").unwrap(); | ||
| 922 | let (sid, oid, _) = text(&source); | ||
| 923 | let cache = Arc::new(Replica::create(&path, &source).unwrap()); | ||
| 924 | let id = cache | ||
| 925 | .edit_text(&source, sid, oid, 1..2, "🦀") | ||
| 926 | .unwrap() | ||
| 927 | .unwrap(); | ||
| 928 | let mut server = Server::new(&source); | ||
| 929 | server.fault = Fault::UnknownAfter; | ||
| 930 | let server = Arc::new(Mutex::new(server)); | ||
| 931 | let shared = Shared(Arc::clone(&server)); | ||
| 932 | let (connected_tx, connected_rx) = mpsc::channel(); | ||
| 933 | let (observed_tx, observed_rx) = mpsc::channel(); | ||
| 934 | let mut connections = 0; | ||
| 935 | let worker = cache | ||
| 936 | .start_sync( | ||
| 937 | Duration::from_millis(10), | ||
| 938 | move || { | ||
| 939 | connections += 1; | ||
| 940 | connected_tx.send(connections).unwrap(); | ||
| 941 | if connections <= 2 { | ||
| 942 | return Err(io::ErrorKind::ConnectionRefused.into()); | ||
| 943 | } | ||
| 944 | Ok(Session { | ||
| 945 | shared: shared.clone(), | ||
| 946 | fail_read: connections == 3, | ||
| 947 | }) | ||
| 948 | }, | ||
| 949 | move |result| { | ||
| 950 | observed_tx | ||
| 951 | .send(result.as_ref().copied().map_err(|error| error.to_string())) | ||
| 952 | .unwrap(); | ||
| 953 | }, | ||
| 954 | ) | ||
| 955 | .unwrap(); | ||
| 956 | let mut errors = 0; | ||
| 957 | let published = loop { | ||
| 958 | match observed_rx.recv_timeout(Duration::from_secs(5)).unwrap() { | ||
| 959 | Err(_) => errors += 1, | ||
| 960 | Ok(Some((actual, status @ EditStatus::Published { .. }))) => { | ||
| 961 | assert_eq!(actual, id); | ||
| 962 | break status; | ||
| 963 | } | ||
| 964 | other => panic!("Unexpected result: {other:?}"), | ||
| 965 | } | ||
| 966 | }; | ||
| 967 | worker.stop().unwrap(); | ||
| 968 | assert_eq!(errors, 4); | ||
| 969 | assert_eq!(connected_rx.try_iter().collect::<Vec<_>>(), [1, 2, 3, 4, 5]); | ||
| 970 | let server = server.lock().unwrap(); | ||
| 971 | assert_eq!(server.publications, 1); | ||
| 972 | assert_eq!(server.confirmations, 1); | ||
| 973 | assert_eq!(text(&server.durable).2, "a🦀c"); | ||
| 974 | drop(cache); | ||
| 975 | let cache = Replica::open(&path).unwrap(); | ||
| 976 | assert_eq!(cache.status(id).unwrap(), Some(published)); | ||
| 977 | assert_eq!(cache.snapshot().unwrap(), server.durable); | ||
| 978 | } | ||
| 979 | |||
| 980 | #[test] | ||
| 981 | fn local_edits_wake_an_idle_worker_and_coalesced_notifications_drain_twelve_writers() { | ||
| 982 | let dir = tempfile::tempdir().unwrap(); | ||
| 983 | let source = onestore::create_section("worker.one", "abc", "Fixture").unwrap(); | ||
| 984 | let (sid, oid, _) = text(&source); | ||
| 985 | let cache = Arc::new(Replica::create(dir.path().join("cache.sqlite"), &source).unwrap()); | ||
| 986 | let server = Arc::new(Mutex::new(Server::new(&source))); | ||
| 987 | let shared = Shared(Arc::clone(&server)); | ||
| 988 | let (observed_tx, observed_rx) = mpsc::channel(); | ||
| 989 | let (resume_tx, resume_rx) = mpsc::channel(); | ||
| 990 | let mut first = true; | ||
| 991 | let worker = cache | ||
| 992 | .start_sync( | ||
| 993 | Duration::from_secs(3600), | ||
| 994 | move || Ok(shared.clone()), | ||
| 995 | move |result| { | ||
| 996 | observed_tx.send(*result.as_ref().unwrap()).unwrap(); | ||
| 997 | if first { | ||
| 998 | first = false; | ||
| 999 | resume_rx.recv_timeout(Duration::from_secs(5)).unwrap(); | ||
| 1000 | } | ||
| 1001 | }, | ||
| 1002 | ) | ||
| 1003 | .unwrap(); | ||
| 1004 | assert_eq!( | ||
| 1005 | observed_rx.recv_timeout(Duration::from_secs(5)).unwrap(), | ||
| 1006 | None | ||
| 1007 | ); | ||
| 1008 | let started = Instant::now(); | ||
| 1009 | let edits = std::thread::scope(|scope| { | ||
| 1010 | (0..12) | ||
| 1011 | .map(|writer| { | ||
| 1012 | let cache = &cache; | ||
| 1013 | scope.spawn(move || { | ||
| 1014 | let replacement = format!("[{writer}] "); | ||
| 1015 | loop { | ||
| 1016 | let snapshot = cache.snapshot().unwrap(); | ||
| 1017 | match cache.edit_text(&snapshot, sid, oid, 0..0, &replacement) { | ||
| 1018 | Ok(Some(id)) => break (id, replacement), | ||
| 1019 | Err(Error::Io(error)) | ||
| 1020 | if error.kind() == io::ErrorKind::ResourceBusy => {} | ||
| 1021 | other => panic!("Unexpected local outcome: {other:?}"), | ||
| 1022 | } | ||
| 1023 | } | ||
| 1024 | }) | ||
| 1025 | }) | ||
| 1026 | .collect::<Vec<_>>() | ||
| 1027 | .into_iter() | ||
| 1028 | .map(|join| join.join().unwrap()) | ||
| 1029 | .collect::<std::collections::BTreeMap<_, _>>() | ||
| 1030 | }); | ||
| 1031 | resume_tx.send(()).unwrap(); | ||
| 1032 | let mut published = std::collections::BTreeSet::new(); | ||
| 1033 | while published.len() < 12 { | ||
| 1034 | if let Some((id, status)) = observed_rx.recv_timeout(Duration::from_secs(5)).unwrap() { | ||
| 1035 | assert!(matches!(status, EditStatus::Published { .. }), "{status:?}"); | ||
| 1036 | assert!(published.insert(id)); | ||
| 1037 | } | ||
| 1038 | } | ||
| 1039 | assert!( | ||
| 1040 | started.elapsed() < Duration::from_secs(5), | ||
| 1041 | "Edits waited for the hourly poll" | ||
| 1042 | ); | ||
| 1043 | worker.stop().unwrap(); | ||
| 1044 | assert_eq!(published, edits.keys().copied().collect()); | ||
| 1045 | let expected = edits.values().rev().cloned().collect::<String>() + "abc"; | ||
| 1046 | let server = server.lock().unwrap(); | ||
| 1047 | assert_eq!(text(&server.durable).2, expected); | ||
| 1048 | assert_eq!(server.publications, 12); | ||
| 1049 | assert_eq!(cache.snapshot().unwrap(), server.durable); | ||
| 1050 | assert!(cache.pending().unwrap().is_empty()); | ||
| 1051 | } | ||
| 1052 | |||
| 1053 | #[test] | ||
| 1054 | fn dropping_during_publication_is_nonblocking_and_retains_ownership_until_recovery_is_recorded() | ||
| 1055 | { | ||
| 1056 | struct Paused { | ||
| 1057 | shared: Shared, | ||
| 1058 | entered: mpsc::Sender<()>, | ||
| 1059 | resume: mpsc::Receiver<()>, | ||
| 1060 | } | ||
| 1061 | impl Remote for Paused { | ||
| 1062 | fn read(&mut self) -> io::Result<Vec<u8>> { | ||
| 1063 | self.shared.read() | ||
| 1064 | } | ||
| 1065 | fn publish(&mut self, edit: &PreparedEdit<'_>) -> Result<(), CommitError> { | ||
| 1066 | self.entered.send(()).unwrap(); | ||
| 1067 | self.resume.recv_timeout(Duration::from_secs(5)).unwrap(); | ||
| 1068 | self.shared.publish(edit) | ||
| 1069 | } | ||
| 1070 | fn confirm(&mut self, snapshot: &[u8]) -> Result<(), CommitError> { | ||
| 1071 | self.shared.confirm(snapshot) | ||
| 1072 | } | ||
| 1073 | } | ||
| 1074 | let dir = tempfile::tempdir().unwrap(); | ||
| 1075 | let path = dir.path().join("cache.sqlite"); | ||
| 1076 | let source = onestore::create_section("worker.one", "abc", "Fixture").unwrap(); | ||
| 1077 | let (sid, oid, _) = text(&source); | ||
| 1078 | let cache = Arc::new(Replica::create(&path, &source).unwrap()); | ||
| 1079 | let id = cache | ||
| 1080 | .edit_text(&source, sid, oid, 0..0, "L ") | ||
| 1081 | .unwrap() | ||
| 1082 | .unwrap(); | ||
| 1083 | let mut server = Server::new(&source); | ||
| 1084 | server.fault = Fault::UnknownAfter; | ||
| 1085 | let server = Arc::new(Mutex::new(server)); | ||
| 1086 | let shared = Shared(Arc::clone(&server)); | ||
| 1087 | let (entered_tx, entered_rx) = mpsc::channel(); | ||
| 1088 | let (resume_tx, resume_rx) = mpsc::channel(); | ||
| 1089 | let mut session = Some(Paused { | ||
| 1090 | shared, | ||
| 1091 | entered: entered_tx, | ||
| 1092 | resume: resume_rx, | ||
| 1093 | }); | ||
| 1094 | let worker = cache | ||
| 1095 | .start_sync( | ||
| 1096 | Duration::from_secs(3600), | ||
| 1097 | move || Ok(session.take().unwrap()), | ||
| 1098 | |_| {}, | ||
| 1099 | ) | ||
| 1100 | .unwrap(); | ||
| 1101 | entered_rx.recv_timeout(Duration::from_secs(5)).unwrap(); | ||
| 1102 | let stopped = Instant::now(); | ||
| 1103 | drop(worker); | ||
| 1104 | assert!(stopped.elapsed() < Duration::from_millis(500)); | ||
| 1105 | let shared = Shared(Arc::clone(&server)); | ||
| 1106 | let start = cache.start_sync( | ||
| 1107 | Duration::from_secs(3600), | ||
| 1108 | move || Ok(shared.clone()), | ||
| 1109 | |_| {}, | ||
| 1110 | ); | ||
| 1111 | assert!(matches!(start, Err(error) if error.kind() == io::ErrorKind::WouldBlock)); | ||
| 1112 | let weak = Arc::downgrade(&cache); | ||
| 1113 | drop(cache); | ||
| 1114 | resume_tx.send(()).unwrap(); | ||
| 1115 | while weak.upgrade().is_some() { | ||
| 1116 | assert!(stopped.elapsed() < Duration::from_secs(5)); | ||
| 1117 | std::thread::sleep(Duration::from_millis(1)); | ||
| 1118 | } | ||
| 1119 | let cache = Arc::new(Replica::open(&path).unwrap()); | ||
| 1120 | assert!(matches!( | ||
| 1121 | cache.status(id).unwrap(), | ||
| 1122 | Some(EditStatus::AwaitingConfirmation { .. }) | ||
| 1123 | )); | ||
| 1124 | assert_eq!(server.lock().unwrap().publications, 1); | ||
| 1125 | let shared = Shared(Arc::clone(&server)); | ||
| 1126 | let (tx, rx) = mpsc::channel(); | ||
| 1127 | let worker = cache | ||
| 1128 | .start_sync( | ||
| 1129 | Duration::from_secs(3600), | ||
| 1130 | move || Ok(shared.clone()), | ||
| 1131 | move |result| { | ||
| 1132 | tx.send(*result.as_ref().unwrap()).unwrap(); | ||
| 1133 | }, | ||
| 1134 | ) | ||
| 1135 | .unwrap(); | ||
| 1136 | assert!( | ||
| 1137 | matches!(rx.recv_timeout(Duration::from_secs(5)).unwrap(), Some((actual, EditStatus::Published { .. })) if actual == id) | ||
| 1138 | ); | ||
| 1139 | worker.stop().unwrap(); | ||
| 1140 | let server = server.lock().unwrap(); | ||
| 1141 | assert_eq!(server.publications, 1); | ||
| 1142 | assert_eq!(server.confirmations, 1); | ||
| 1143 | assert_eq!(text(&server.durable).2, "L abc"); | ||
| 1144 | } | ||
| 1145 | |||
| 1146 | #[test] | ||
| 1147 | fn cache_failures_stop_retries_and_return_the_error_without_remote_publication() { | ||
| 1148 | let dir = tempfile::tempdir().unwrap(); | ||
| 1149 | let path = dir.path().join("cache.sqlite"); | ||
| 1150 | let source = onestore::create_section("worker.one", "abc", "Fixture").unwrap(); | ||
| 1151 | let (sid, oid, _) = text(&source); | ||
| 1152 | let cache = Replica::create(&path, &source).unwrap(); | ||
| 1153 | let id = cache | ||
| 1154 | .edit_text(&source, sid, oid, 0..0, "L ") | ||
| 1155 | .unwrap() | ||
| 1156 | .unwrap(); | ||
| 1157 | drop(cache); | ||
| 1158 | let connection = rusqlite::Connection::open(&path).unwrap(); | ||
| 1159 | connection.execute_batch("CREATE TRIGGER fail_attempt BEFORE INSERT ON attempt BEGIN SELECT RAISE(ABORT, 'test cache write failure'); END;").unwrap(); | ||
| 1160 | drop(connection); | ||
| 1161 | let cache = Arc::new(Replica::open(&path).unwrap()); | ||
| 1162 | let server = Arc::new(Mutex::new(Server::new(&source))); | ||
| 1163 | let shared = Shared(Arc::clone(&server)); | ||
| 1164 | let (tx, rx) = mpsc::channel(); | ||
| 1165 | let worker = cache | ||
| 1166 | .start_sync( | ||
| 1167 | Duration::from_millis(1), | ||
| 1168 | move || Ok(shared.clone()), | ||
| 1169 | move |result| { | ||
| 1170 | tx.send(matches!(result, Err(Error::Database(_)))).unwrap(); | ||
| 1171 | }, | ||
| 1172 | ) | ||
| 1173 | .unwrap(); | ||
| 1174 | assert!(rx.recv_timeout(Duration::from_secs(5)).unwrap()); | ||
| 1175 | assert!(matches!(worker.stop(), Err(Error::Database(_)))); | ||
| 1176 | assert!(rx.try_iter().next().is_none()); | ||
| 1177 | assert_eq!(server.lock().unwrap().publications, 0); | ||
| 1178 | assert_eq!(cache.status(id).unwrap(), Some(EditStatus::Pending)); | ||
| 1179 | assert_eq!(text(&cache.snapshot().unwrap()).2, "L abc"); | ||
| 1180 | } | ||
| 1181 | |||
| 1182 | #[test] | ||
| 1183 | fn polling_preserves_conflicts_and_absent_uncertain_revisions_without_replay() { | ||
| 1184 | for uncertain in [false, true] { | ||
| 1185 | let dir = tempfile::tempdir().unwrap(); | ||
| 1186 | let source = onestore::create_section("worker.one", "abc", "Fixture").unwrap(); | ||
| 1187 | let (sid, oid, _) = text(&source); | ||
| 1188 | let cache = | ||
| 1189 | Arc::new(Replica::create(dir.path().join("cache.sqlite"), &source).unwrap()); | ||
| 1190 | let id = cache | ||
| 1191 | .edit_text(&source, sid, oid, 1..2, "L") | ||
| 1192 | .unwrap() | ||
| 1193 | .unwrap(); | ||
| 1194 | let local = cache.snapshot().unwrap(); | ||
| 1195 | let mut server = if uncertain { | ||
| 1196 | Server::new(&source) | ||
| 1197 | } else { | ||
| 1198 | Server::new(&onestore::replace_text(&source, sid, oid, 1..2, "R").unwrap()) | ||
| 1199 | }; | ||
| 1200 | if uncertain { | ||
| 1201 | server.fault = Fault::UnknownBefore; | ||
| 1202 | } | ||
| 1203 | let server = Arc::new(Mutex::new(server)); | ||
| 1204 | let shared = Shared(Arc::clone(&server)); | ||
| 1205 | let (tx, rx) = mpsc::channel(); | ||
| 1206 | let worker = cache | ||
| 1207 | .start_sync( | ||
| 1208 | Duration::from_millis(10), | ||
| 1209 | move || Ok(shared.clone()), | ||
| 1210 | move |result| { | ||
| 1211 | tx.send(result.as_ref().copied().map_err(|_| ())).unwrap(); | ||
| 1212 | }, | ||
| 1213 | ) | ||
| 1214 | .unwrap(); | ||
| 1215 | if uncertain { | ||
| 1216 | assert!(rx.recv_timeout(Duration::from_secs(5)).unwrap().is_err()); | ||
| 1217 | } | ||
| 1218 | let mut previous = None; | ||
| 1219 | let started = Instant::now(); | ||
| 1220 | for _ in 0..5 { | ||
| 1221 | let (actual, status) = rx | ||
| 1222 | .recv_timeout(Duration::from_secs(5)) | ||
| 1223 | .unwrap() | ||
| 1224 | .unwrap() | ||
| 1225 | .unwrap(); | ||
| 1226 | assert_eq!(actual, id); | ||
| 1227 | if uncertain { | ||
| 1228 | assert!(matches!(status, EditStatus::AwaitingConfirmation { .. })); | ||
| 1229 | } else { | ||
| 1230 | assert_eq!(status, EditStatus::Conflict(ConflictKind::TextChanged)); | ||
| 1231 | } | ||
| 1232 | if let Some(previous) = previous { | ||
| 1233 | assert_eq!(previous, status); | ||
| 1234 | } | ||
| 1235 | previous = Some(status); | ||
| 1236 | } | ||
| 1237 | assert!( | ||
| 1238 | started.elapsed() >= Duration::from_millis(30), | ||
| 1239 | "Worker spun instead of waiting between retries" | ||
| 1240 | ); | ||
| 1241 | worker.stop().unwrap(); | ||
| 1242 | assert_eq!(cache.snapshot().unwrap(), local); | ||
| 1243 | assert_eq!(cache.pending().unwrap().len(), 1); | ||
| 1244 | let server = server.lock().unwrap(); | ||
| 1245 | assert_eq!(server.publications, usize::from(uncertain)); | ||
| 1246 | assert_eq!(server.confirmations, 0); | ||
| 1247 | } | ||
| 1248 | } | ||
| 1249 | |||
| 1250 | #[test] | ||
| 1251 | fn reachability_notification_retries_without_waiting_for_the_poll() { | ||
| 1252 | let dir = tempfile::tempdir().unwrap(); | ||
| 1253 | let source = onestore::create_section("worker.one", "abc", "Fixture").unwrap(); | ||
| 1254 | let cache = Arc::new(Replica::create(dir.path().join("cache.sqlite"), &source).unwrap()); | ||
| 1255 | let (tx, rx) = mpsc::channel(); | ||
| 1256 | let worker = cache | ||
| 1257 | .start_sync( | ||
| 1258 | Duration::from_secs(3600), | ||
| 1259 | || -> io::Result<Shared> { Err(io::ErrorKind::NotConnected.into()) }, | ||
| 1260 | move |result| { | ||
| 1261 | tx.send(matches!(result, Err(Error::RemoteIo(_)))).unwrap(); | ||
| 1262 | }, | ||
| 1263 | ) | ||
| 1264 | .unwrap(); | ||
| 1265 | assert!(rx.recv_timeout(Duration::from_secs(5)).unwrap()); | ||
| 1266 | worker.wake(); | ||
| 1267 | assert!(rx.recv_timeout(Duration::from_secs(5)).unwrap()); | ||
| 1268 | worker.stop().unwrap(); | ||
| 1269 | assert!(rx.try_iter().next().is_none()); | ||
| 1270 | } | ||
| 1271 | |||
| 1272 | #[test] | ||
| 1273 | fn cancellation_during_connect_does_not_read_or_report_a_false_refresh() { | ||
| 1274 | let dir = tempfile::tempdir().unwrap(); | ||
| 1275 | let source = onestore::create_section("worker.one", "abc", "Fixture").unwrap(); | ||
| 1276 | let cache = Arc::new(Replica::create(dir.path().join("cache.sqlite"), &source).unwrap()); | ||
| 1277 | let (entered_tx, entered_rx) = mpsc::channel(); | ||
| 1278 | let (resume_tx, resume_rx) = mpsc::channel(); | ||
| 1279 | let (tx, rx) = mpsc::channel(); | ||
| 1280 | // An invalid image makes any unexpected read observable as an error callback. | ||
| 1281 | let shared = Shared(Arc::new(Mutex::new(Server::new(&[])))); | ||
| 1282 | let worker = cache | ||
| 1283 | .start_sync( | ||
| 1284 | Duration::from_secs(3600), | ||
| 1285 | move || { | ||
| 1286 | entered_tx.send(()).unwrap(); | ||
| 1287 | resume_rx.recv_timeout(Duration::from_secs(5)).unwrap(); | ||
| 1288 | Ok(shared.clone()) | ||
| 1289 | }, | ||
| 1290 | move |_| { | ||
| 1291 | tx.send(()).unwrap(); | ||
| 1292 | }, | ||
| 1293 | ) | ||
| 1294 | .unwrap(); | ||
| 1295 | entered_rx.recv_timeout(Duration::from_secs(5)).unwrap(); | ||
| 1296 | drop(worker); | ||
| 1297 | let weak = Arc::downgrade(&cache); | ||
| 1298 | drop(cache); | ||
| 1299 | resume_tx.send(()).unwrap(); | ||
| 1300 | let started = Instant::now(); | ||
| 1301 | while weak.upgrade().is_some() { | ||
| 1302 | assert!(started.elapsed() < Duration::from_secs(5)); | ||
| 1303 | std::thread::sleep(Duration::from_millis(1)); | ||
| 1304 | } | ||
| 1305 | assert_eq!( | ||
| 1306 | rx.recv_timeout(Duration::from_secs(5)), | ||
| 1307 | Err(mpsc::RecvTimeoutError::Disconnected) | ||
| 1308 | ); | ||
| 1309 | } | ||
| 1310 | |||
| 1311 | #[test] | ||
| 1312 | fn invalid_intervals_and_callback_panics_leave_worker_ownership_recoverable() { | ||
| 1313 | let dir = tempfile::tempdir().unwrap(); | ||
| 1314 | let source = onestore::create_section("worker.one", "abc", "Fixture").unwrap(); | ||
| 1315 | let cache = Arc::new(Replica::create(dir.path().join("cache.sqlite"), &source).unwrap()); | ||
| 1316 | for interval in [Duration::ZERO, Duration::MAX] { | ||
| 1317 | assert!( | ||
| 1318 | matches!(cache.start_sync(interval, || -> io::Result<Shared> { panic!("Unexpected connection") }, |_| {}), Err(error) if error.kind() == io::ErrorKind::InvalidInput) | ||
| 1319 | ); | ||
| 1320 | } | ||
| 1321 | let shared = Shared(Arc::new(Mutex::new(Server::new(&source)))); | ||
| 1322 | let first = shared.clone(); | ||
| 1323 | let (tx, rx) = mpsc::channel(); | ||
| 1324 | let worker = cache | ||
| 1325 | .start_sync( | ||
| 1326 | Duration::from_secs(3600), | ||
| 1327 | move || Ok(first.clone()), | ||
| 1328 | move |_| { | ||
| 1329 | tx.send(()).unwrap(); | ||
| 1330 | panic!("Test observer panic"); | ||
| 1331 | }, | ||
| 1332 | ) | ||
| 1333 | .unwrap(); | ||
| 1334 | rx.recv_timeout(Duration::from_secs(5)).unwrap(); | ||
| 1335 | assert!(matches!(worker.stop(), Err(Error::Io(_)))); | ||
| 1336 | let (tx, rx) = mpsc::channel(); | ||
| 1337 | let worker = cache | ||
| 1338 | .start_sync( | ||
| 1339 | Duration::from_secs(3600), | ||
| 1340 | move || Ok(shared.clone()), | ||
| 1341 | move |result| { | ||
| 1342 | tx.send(*result.as_ref().unwrap()).unwrap(); | ||
| 1343 | }, | ||
| 1344 | ) | ||
| 1345 | .unwrap(); | ||
| 1346 | assert_eq!(rx.recv_timeout(Duration::from_secs(5)).unwrap(), None); | ||
| 1347 | worker.stop().unwrap(); | ||
| 1348 | assert_eq!(cache.snapshot().unwrap(), source); | ||
| 1349 | } | ||
| 1350 | |||
| 1351 | #[test] | ||
| 1352 | fn reviewed_conflict_wakes_the_worker_and_publishes_the_original_intent_once() { | ||
| 1353 | let dir = tempfile::tempdir().unwrap(); | ||
| 1354 | let source = onestore::create_section("resolve.one", "abc", "Fixture").unwrap(); | ||
| 1355 | let (sid, oid, _) = text(&source); | ||
| 1356 | let cache = Arc::new(Replica::create(dir.path().join("cache.sqlite"), &source).unwrap()); | ||
| 1357 | let id = cache | ||
| 1358 | .edit_text(&source, sid, oid, 1..2, "L") | ||
| 1359 | .unwrap() | ||
| 1360 | .unwrap(); | ||
| 1361 | let remote = onestore::replace_text(&source, sid, oid, 1..2, "R").unwrap(); | ||
| 1362 | let server = Arc::new(Mutex::new(Server::new(&remote))); | ||
| 1363 | let shared = Shared(Arc::clone(&server)); | ||
| 1364 | let (tx, rx) = mpsc::channel(); | ||
| 1365 | let (resume_tx, resume_rx) = mpsc::channel(); | ||
| 1366 | let mut first = true; | ||
| 1367 | let worker = cache | ||
| 1368 | .start_sync( | ||
| 1369 | Duration::from_secs(3600), | ||
| 1370 | move || Ok(shared.clone()), | ||
| 1371 | move |result| { | ||
| 1372 | tx.send(*result.as_ref().unwrap()).unwrap(); | ||
| 1373 | if first { | ||
| 1374 | first = false; | ||
| 1375 | resume_rx.recv_timeout(Duration::from_secs(5)).unwrap(); | ||
| 1376 | } | ||
| 1377 | }, | ||
| 1378 | ) | ||
| 1379 | .unwrap(); | ||
| 1380 | assert_eq!( | ||
| 1381 | rx.recv_timeout(Duration::from_secs(5)).unwrap(), | ||
| 1382 | Some((id, EditStatus::Conflict(ConflictKind::TextChanged))) | ||
| 1383 | ); | ||
| 1384 | cache | ||
| 1385 | .rebase_conflict( | ||
| 1386 | id, | ||
| 1387 | &cache.snapshot().unwrap(), | ||
| 1388 | &cache.remote_snapshot().unwrap(), | ||
| 1389 | 1..2, | ||
| 1390 | ) | ||
| 1391 | .unwrap(); | ||
| 1392 | assert_eq!(cache.status(id).unwrap(), Some(EditStatus::Pending)); | ||
| 1393 | resume_tx.send(()).unwrap(); | ||
| 1394 | assert!( | ||
| 1395 | matches!(rx.recv_timeout(Duration::from_secs(5)).unwrap(), Some((actual, EditStatus::Published { .. })) if actual == id) | ||
| 1396 | ); | ||
| 1397 | worker.stop().unwrap(); | ||
| 1398 | assert!(cache.pending().unwrap().is_empty()); | ||
| 1399 | let server = server.lock().unwrap(); | ||
| 1400 | assert_eq!(server.publications, 1); | ||
| 1401 | assert_eq!(text(&server.durable).2, "aLc"); | ||
| 1402 | assert_eq!(cache.snapshot().unwrap(), server.durable); | ||
| 1403 | } | ||
| 1404 | |||
| 1405 | #[test] | ||
| 1406 | fn ordinary_read_and_unpublished_write_contention_reuse_the_connection() { | ||
| 1407 | struct Busy { | ||
| 1408 | server: Server, | ||
| 1409 | reads: usize, | ||
| 1410 | writes: usize, | ||
| 1411 | } | ||
| 1412 | impl Remote for Busy { | ||
| 1413 | fn read(&mut self) -> io::Result<Vec<u8>> { | ||
| 1414 | self.reads += 1; | ||
| 1415 | match self.reads { | ||
| 1416 | 1 => Err(io::ErrorKind::WouldBlock.into()), | ||
| 1417 | 2 => Err(io::ErrorKind::ResourceBusy.into()), | ||
| 1418 | _ => self.server.read(), | ||
| 1419 | } | ||
| 1420 | } | ||
| 1421 | fn publish(&mut self, edit: &PreparedEdit<'_>) -> Result<(), CommitError> { | ||
| 1422 | self.writes += 1; | ||
| 1423 | match self.writes { | ||
| 1424 | 1 | 2 => Err(CommitError { | ||
| 1425 | state: CommitState::NotCommitted, | ||
| 1426 | error: if self.writes == 1 { | ||
| 1427 | io::ErrorKind::WouldBlock | ||
| 1428 | } else { | ||
| 1429 | io::ErrorKind::ResourceBusy | ||
| 1430 | } | ||
| 1431 | .into(), | ||
| 1432 | }), | ||
| 1433 | _ => self.server.publish(edit), | ||
| 1434 | } | ||
| 1435 | } | ||
| 1436 | fn confirm(&mut self, source: &[u8]) -> Result<(), CommitError> { | ||
| 1437 | self.server.confirm(source) | ||
| 1438 | } | ||
| 1439 | } | ||
| 1440 | let dir = tempfile::tempdir().unwrap(); | ||
| 1441 | let source = onestore::create_section("worker.one", "abc", "Fixture").unwrap(); | ||
| 1442 | let (sid, oid, _) = text(&source); | ||
| 1443 | let cache = Arc::new(Replica::create(dir.path().join("cache.sqlite"), &source).unwrap()); | ||
| 1444 | let id = cache | ||
| 1445 | .edit_text(&source, sid, oid, 0..0, "L ") | ||
| 1446 | .unwrap() | ||
| 1447 | .unwrap(); | ||
| 1448 | let mut remote = Some(Busy { | ||
| 1449 | server: Server::new(&source), | ||
| 1450 | reads: 0, | ||
| 1451 | writes: 0, | ||
| 1452 | }); | ||
| 1453 | let (tx, rx) = mpsc::channel(); | ||
| 1454 | let worker = cache | ||
| 1455 | .start_sync( | ||
| 1456 | Duration::from_millis(1), | ||
| 1457 | move || Ok(remote.take().expect("Contention caused a reconnect")), | ||
| 1458 | move |result| { | ||
| 1459 | tx.send(result.as_ref().copied().map_err(|error| error.to_string())) | ||
| 1460 | .unwrap(); | ||
| 1461 | }, | ||
| 1462 | ) | ||
| 1463 | .unwrap(); | ||
| 1464 | for _ in 0..4 { | ||
| 1465 | assert!(rx.recv_timeout(Duration::from_secs(5)).unwrap().is_err()); | ||
| 1466 | } | ||
| 1467 | assert!( | ||
| 1468 | matches!(rx.recv_timeout(Duration::from_secs(5)).unwrap().unwrap(), Some((actual, EditStatus::Published { .. })) if actual == id) | ||
| 1469 | ); | ||
| 1470 | worker.stop().unwrap(); | ||
| 1471 | assert_eq!(text(&cache.snapshot().unwrap()).2, "L abc"); | ||
| 1472 | assert!(cache.pending().unwrap().is_empty()); | ||
| 1473 | } | ||
| 1474 | #[test] | ||
| 1475 | fn publication_backoff_drains_local_wakes_without_waiting_for_the_idle_poll() { | ||
| 1476 | struct BusyOnce(Server); | ||
| 1477 | impl Remote for BusyOnce { | ||
| 1478 | fn read(&mut self) -> io::Result<Vec<u8>> { | ||
| 1479 | self.0.read() | ||
| 1480 | } | ||
| 1481 | fn publish(&mut self, edit: &PreparedEdit<'_>) -> Result<(), CommitError> { | ||
| 1482 | let server = &mut self.0; | ||
| 1483 | if server.publications == 0 { | ||
| 1484 | server.publications += 1; | ||
| 1485 | return Err(CommitError { | ||
| 1486 | state: CommitState::NotCommitted, | ||
| 1487 | error: io::ErrorKind::ResourceBusy.into(), | ||
| 1488 | }); | ||
| 1489 | } | ||
| 1490 | server.publish(edit) | ||
| 1491 | } | ||
| 1492 | fn confirm(&mut self, source: &[u8]) -> Result<(), CommitError> { | ||
| 1493 | self.0.confirm(source) | ||
| 1494 | } | ||
| 1495 | } | ||
| 1496 | let dir = tempfile::tempdir().unwrap(); | ||
| 1497 | let source = onestore::create_section("worker.one", "abc", "Fixture").unwrap(); | ||
| 1498 | let (sid, oid, _) = text(&source); | ||
| 1499 | let cache = Arc::new(Replica::create(dir.path().join("cache.sqlite"), &source).unwrap()); | ||
| 1500 | let first = cache | ||
| 1501 | .edit_text(&source, sid, oid, 0..0, "L ") | ||
| 1502 | .unwrap() | ||
| 1503 | .unwrap(); | ||
| 1504 | let mut remote = Some(BusyOnce(Server::new(&source))); | ||
| 1505 | let observed = Arc::clone(&cache); | ||
| 1506 | let (tx, rx) = mpsc::channel(); | ||
| 1507 | let mut failed = false; | ||
| 1508 | let worker = cache | ||
| 1509 | .start_sync( | ||
| 1510 | Duration::from_secs(3600), | ||
| 1511 | move || Ok(remote.take().expect("Contention must retain the session")), | ||
| 1512 | move |result| match result { | ||
| 1513 | Err(Error::Remote(error)) if error.state == CommitState::NotCommitted => { | ||
| 1514 | assert!(!failed); | ||
| 1515 | failed = true; | ||
| 1516 | let source = observed.snapshot().unwrap(); | ||
| 1517 | let second = observed | ||
| 1518 | .edit_text(&source, sid, oid, 0..0, "Q ") | ||
| 1519 | .unwrap() | ||
| 1520 | .unwrap(); | ||
| 1521 | tx.send((second, None)).unwrap(); | ||
| 1522 | } | ||
| 1523 | Ok(Some((id, status))) => tx.send((*id, Some(*status))).unwrap(), | ||
| 1524 | Ok(None) => {} | ||
| 1525 | other => panic!("Unexpected worker result: {other:?}"), | ||
| 1526 | }, | ||
| 1527 | ) | ||
| 1528 | .unwrap(); | ||
| 1529 | let (second, status) = rx.recv_timeout(Duration::from_secs(5)).unwrap(); | ||
| 1530 | assert_eq!(status, None); | ||
| 1531 | for expected in [first, second] { | ||
| 1532 | let (id, status) = rx.recv_timeout(Duration::from_secs(5)).unwrap(); | ||
| 1533 | assert_eq!(id, expected); | ||
| 1534 | assert!(matches!(status, Some(EditStatus::Published { .. }))); | ||
| 1535 | } | ||
| 1536 | worker.stop().unwrap(); | ||
| 1537 | assert!(cache.pending().unwrap().is_empty()); | ||
| 1538 | assert_eq!(text(&cache.snapshot().unwrap()).2, "Q L abc"); | ||
| 1539 | } | ||
| 1540 | } | ||
| 1541 | |||
| 1542 | #[test] | ||
| 1543 | fn offline_insertions_survive_reopen_rebase_and_dependent_text_edits() { | ||
| 1544 | use onestore::Insertion; | ||
| 1545 | let dir = tempfile::tempdir().unwrap(); | ||
| 1546 | let path = dir.path().join("insert.sqlite"); | ||
| 1547 | let source = onestore::create_section("insert.one", "Original", "Author").unwrap(); | ||
| 1548 | let (sid, original, _) = text(&source); | ||
| 1549 | let store = Store::parse(&source).unwrap(); | ||
| 1550 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 1551 | let doc = Document::parse(&index).unwrap(); | ||
| 1552 | let (_, page) = doc.pages().unwrap()[0]; | ||
| 1553 | let cache = Replica::create(&path, &source).unwrap(); | ||
| 1554 | let outline = | ||
| 1555 | Insertion::outline(page, 72.0, 144.0, "Offline outline", "Offline author").unwrap(); | ||
| 1556 | let first = cache.insert(&source, sid, &outline).unwrap().unwrap(); | ||
| 1557 | let snapshot = cache.snapshot().unwrap(); | ||
| 1558 | let paragraph = Insertion::paragraph( | ||
| 1559 | outline.object(), | ||
| 1560 | None, | ||
| 1561 | "Offline paragraph 🦀", | ||
| 1562 | "Offline author", | ||
| 1563 | ) | ||
| 1564 | .unwrap(); | ||
| 1565 | let second = cache.insert(&snapshot, sid, ¶graph).unwrap().unwrap(); | ||
| 1566 | let third = cache | ||
| 1567 | .edit_text( | ||
| 1568 | &cache.snapshot().unwrap(), | ||
| 1569 | sid, | ||
| 1570 | paragraph.text_object(), | ||
| 1571 | 0..0, | ||
| 1572 | "Edited ", | ||
| 1573 | ) | ||
| 1574 | .unwrap() | ||
| 1575 | .unwrap(); | ||
| 1576 | let local = cache.snapshot().unwrap(); | ||
| 1577 | let pending = cache.pending().unwrap(); | ||
| 1578 | drop(cache); | ||
| 1579 | let cache = Replica::open(&path).unwrap(); | ||
| 1580 | assert_eq!(cache.snapshot().unwrap(), local); | ||
| 1581 | assert_eq!(cache.pending().unwrap(), pending); | ||
| 1582 | let remote = onestore::replace_text(&source, sid, original, 0..0, "Remote ").unwrap(); | ||
| 1583 | let mut server = Server::new(&remote); | ||
| 1584 | for id in [first, second, third] { | ||
| 1585 | assert!( | ||
| 1586 | matches!(cache.sync_once(&mut server).unwrap(),Some((observed,EditStatus::Published{..})) if observed==id) | ||
| 1587 | ); | ||
| 1588 | } | ||
| 1589 | assert_eq!(server.publications, 3); | ||
| 1590 | assert!(cache.pending().unwrap().is_empty()); | ||
| 1591 | assert_eq!(cache.snapshot().unwrap(), server.durable); | ||
| 1592 | let store = Store::parse(&server.durable).unwrap(); | ||
| 1593 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 1594 | let doc = Document::parse(&index).unwrap(); | ||
| 1595 | let s = &doc.spaces[&sid]; | ||
| 1596 | let v = &s.revisions[&s.contexts[&ExGuid::default()]]; | ||
| 1597 | for (id, wanted) in [ | ||
| 1598 | (original, "Remote Original"), | ||
| 1599 | (outline.text_object(), "Offline outline"), | ||
| 1600 | (paragraph.text_object(), "Edited Offline paragraph 🦀"), | ||
| 1601 | ] { | ||
| 1602 | assert!(matches!(&v.nodes[&id].kind,Kind::RichText{text,..} if text==wanted)); | ||
| 1603 | } | ||
| 1604 | assert_eq!( | ||
| 1605 | v.nodes[&outline.object()].children.last(), | ||
| 1606 | Some(¶graph.object()) | ||
| 1607 | ); | ||
| 1608 | } | ||
| 1609 | |||
| 1610 | #[test] | ||
| 1611 | fn uncertain_insertions_reconcile_the_original_revision_without_duplicate_objects() { | ||
| 1612 | use onestore::Insertion; | ||
| 1613 | for fault in [ | ||
| 1614 | Fault::Before, | ||
| 1615 | Fault::UnknownBefore, | ||
| 1616 | Fault::UnknownAfter, | ||
| 1617 | Fault::PanicBefore, | ||
| 1618 | Fault::PanicAfter, | ||
| 1619 | Fault::Committed, | ||
| 1620 | ] { | ||
| 1621 | let dir = tempfile::tempdir().unwrap(); | ||
| 1622 | let path = dir.path().join("uncertain-insert.sqlite"); | ||
| 1623 | let source = onestore::create_section("insert.one", "Original", "Author").unwrap(); | ||
| 1624 | let store = Store::parse(&source).unwrap(); | ||
| 1625 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 1626 | let doc = Document::parse(&index).unwrap(); | ||
| 1627 | let (sid, page) = doc.pages().unwrap()[0]; | ||
| 1628 | let cache = Replica::create(&path, &source).unwrap(); | ||
| 1629 | let insertion = | ||
| 1630 | Insertion::outline(page, 144.0, 144.0, "Uncertain insertion", "Author").unwrap(); | ||
| 1631 | let id = cache.insert(&source, sid, &insertion).unwrap().unwrap(); | ||
| 1632 | let local = cache.snapshot().unwrap(); | ||
| 1633 | let mut server = Server::new(&source); | ||
| 1634 | server.fault = fault; | ||
| 1635 | let _ = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { | ||
| 1636 | cache.sync_once(&mut server) | ||
| 1637 | })); | ||
| 1638 | drop(cache); | ||
| 1639 | let cache = Replica::open(&path).unwrap(); | ||
| 1640 | assert_eq!(server.publications, 1); | ||
| 1641 | if matches!(fault, Fault::UnknownBefore | Fault::PanicBefore) { | ||
| 1642 | let status = cache.status(id).unwrap(); | ||
| 1643 | assert!(matches!( | ||
| 1644 | status, | ||
| 1645 | Some(EditStatus::AwaitingConfirmation { .. }) | ||
| 1646 | )); | ||
| 1647 | for _ in 0..5 { | ||
| 1648 | assert_eq!( | ||
| 1649 | cache.sync_once(&mut server).unwrap(), | ||
| 1650 | status.map(|state| (id, state)) | ||
| 1651 | ); | ||
| 1652 | } | ||
| 1653 | assert_eq!(server.publications, 1); | ||
| 1654 | assert_eq!(server.durable, source); | ||
| 1655 | assert_eq!(cache.snapshot().unwrap(), local); | ||
| 1656 | } else { | ||
| 1657 | if !matches!(fault, Fault::Committed) { | ||
| 1658 | assert!(matches!( | ||
| 1659 | cache.sync_once(&mut server).unwrap(), | ||
| 1660 | Some((_, EditStatus::Published { .. })) | ||
| 1661 | )); | ||
| 1662 | } | ||
| 1663 | assert_eq!( | ||
| 1664 | server.publications, | ||
| 1665 | if matches!(fault, Fault::Before) { 2 } else { 1 } | ||
| 1666 | ); | ||
| 1667 | assert_eq!( | ||
| 1668 | server.confirmations, | ||
| 1669 | usize::from(matches!(fault, Fault::UnknownAfter | Fault::PanicAfter)) | ||
| 1670 | ); | ||
| 1671 | let store = Store::parse(&server.durable).unwrap(); | ||
| 1672 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 1673 | let doc = Document::parse(&index).unwrap(); | ||
| 1674 | let s = &doc.spaces[&sid]; | ||
| 1675 | let v = &s.revisions[&s.contexts[&ExGuid::default()]]; | ||
| 1676 | assert_eq!(v.nodes.values().filter(|node|matches!(&node.kind,Kind::RichText{text,..} if text=="Uncertain insertion")).count(),1); | ||
| 1677 | assert!(v.nodes.contains_key(&insertion.object())); | ||
| 1678 | assert!(cache.pending().unwrap().is_empty()); | ||
| 1679 | } | ||
| 1680 | } | ||
| 1681 | } | ||
| 1682 | |||
| 1683 | #[test] | ||
| 1684 | fn offline_formatting_rebases_text_and_merges_independent_attributes() { | ||
| 1685 | use onestore::TextAttribute as A; | ||
| 1686 | let directory = tempfile::tempdir().unwrap(); | ||
| 1687 | let path = directory.path().join("format.sqlite"); | ||
| 1688 | let source = onestore::create_section("format.one", "ab🦀cd", "Author").unwrap(); | ||
| 1689 | let (sid, id, _) = text(&source); | ||
| 1690 | let cache = Replica::create(&path, &source).unwrap(); | ||
| 1691 | let local = cache | ||
| 1692 | .format(&source, sid, id, 2..4, &[A::Bold(true)]) | ||
| 1693 | .unwrap() | ||
| 1694 | .unwrap(); | ||
| 1695 | let pending = cache.pending().unwrap(); | ||
| 1696 | drop(cache); | ||
| 1697 | let cache = Replica::open(&path).unwrap(); | ||
| 1698 | assert_eq!(cache.pending().unwrap(), pending); | ||
| 1699 | let moved = PreparedEdit::text(&source, sid, id, 0..0, "Prefix ").unwrap(); | ||
| 1700 | let styled = | ||
| 1701 | PreparedEdit::format(moved.as_bytes(), sid, id, 9..11, &[A::Italic(true)]).unwrap(); | ||
| 1702 | let mut server = Server::new(styled.as_bytes()); | ||
| 1703 | assert!( | ||
| 1704 | matches!(cache.sync_once(&mut server).unwrap(),Some((observed,EditStatus::Published{..}))if observed==local) | ||
| 1705 | ); | ||
| 1706 | let store = Store::parse(&server.durable).unwrap(); | ||
| 1707 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 1708 | let doc = Document::parse(&index).unwrap(); | ||
| 1709 | let space = &doc.spaces[&sid]; | ||
| 1710 | let view = &space.revisions[&space.contexts[&ExGuid::default()]]; | ||
| 1711 | let runs = view.text_runs(id).unwrap(); | ||
| 1712 | let selected = runs.iter().find(|r| r.text == "🦀").unwrap(); | ||
| 1713 | assert_eq!(selected.format.bold, Some(true)); | ||
| 1714 | assert_eq!(selected.format.italic, Some(true)); | ||
| 1715 | assert_eq!( | ||
| 1716 | runs.iter().map(|r| r.text).collect::<String>(), | ||
| 1717 | "Prefix ab🦀cd" | ||
| 1718 | ); | ||
| 1719 | } | ||
| 1720 | |||
| 1721 | #[test] | ||
| 1722 | fn competing_font_changes_remain_preserved_conflicts() { | ||
| 1723 | use onestore::TextAttribute as A; | ||
| 1724 | let directory = tempfile::tempdir().unwrap(); | ||
| 1725 | let path = directory.path().join("conflict.sqlite"); | ||
| 1726 | let source = onestore::create_section("format.one", "abcdef", "Author").unwrap(); | ||
| 1727 | let (sid, id, _) = text(&source); | ||
| 1728 | let cache = Replica::create(&path, &source).unwrap(); | ||
| 1729 | let local_id = cache | ||
| 1730 | .format(&source, sid, id, 1..5, &[A::FontSize(14.0)]) | ||
| 1731 | .unwrap() | ||
| 1732 | .unwrap(); | ||
| 1733 | let local = cache.snapshot().unwrap(); | ||
| 1734 | let pending = cache.pending().unwrap(); | ||
| 1735 | let remote = PreparedEdit::format(&source, sid, id, 2..4, &[A::FontSize(18.0)]).unwrap(); | ||
| 1736 | let mut server = Server::new(remote.as_bytes()); | ||
| 1737 | for _ in 0..3 { | ||
| 1738 | assert_eq!( | ||
| 1739 | cache.sync_once(&mut server).unwrap(), | ||
| 1740 | Some(( | ||
| 1741 | local_id, | ||
| 1742 | EditStatus::Conflict(ConflictKind::FormattingChanged) | ||
| 1743 | )) | ||
| 1744 | ); | ||
| 1745 | } | ||
| 1746 | assert_eq!(server.publications, 0); | ||
| 1747 | assert_eq!(server.confirmations, 0); | ||
| 1748 | assert_eq!(cache.snapshot().unwrap(), local); | ||
| 1749 | assert_eq!(cache.pending().unwrap(), pending); | ||
| 1750 | drop(cache); | ||
| 1751 | let cache = Replica::open(&path).unwrap(); | ||
| 1752 | assert_eq!( | ||
| 1753 | cache.status(local_id).unwrap(), | ||
| 1754 | Some(EditStatus::Conflict(ConflictKind::FormattingChanged)) | ||
| 1755 | ); | ||
| 1756 | assert_eq!(cache.snapshot().unwrap(), local); | ||
| 1757 | } | ||
| 1758 | |||
| 1759 | #[test] | ||
| 1760 | fn independently_satisfied_formatting_requires_confirmation_before_a_receipt() { | ||
| 1761 | use onestore::TextAttribute as A; | ||
| 1762 | let directory = tempfile::tempdir().unwrap(); | ||
| 1763 | let path = directory.path().join("satisfied.sqlite"); | ||
| 1764 | let source = onestore::create_section("format.one", "abcdef", "Author").unwrap(); | ||
| 1765 | let (sid, id, _) = text(&source); | ||
| 1766 | let cache = Replica::create(&path, &source).unwrap(); | ||
| 1767 | let local_id = cache | ||
| 1768 | .format(&source, sid, id, 1..5, &[A::Bold(true)]) | ||
| 1769 | .unwrap() | ||
| 1770 | .unwrap(); | ||
| 1771 | let remote = PreparedEdit::format(&source, sid, id, 1..5, &[A::Bold(true)]).unwrap(); | ||
| 1772 | let mut server = Server::new(remote.as_bytes()); | ||
| 1773 | server.durable = source.clone(); | ||
| 1774 | server.fault = Fault::Confirm; | ||
| 1775 | assert!( | ||
| 1776 | matches!(cache.sync_once(&mut server),Err(Error::Remote(e))if e.state==CommitState::Unknown) | ||
| 1777 | ); | ||
| 1778 | assert_eq!(cache.status(local_id).unwrap(), Some(EditStatus::Pending)); | ||
| 1779 | assert_eq!(server.publications, 0); | ||
| 1780 | assert_eq!(server.durable, source); | ||
| 1781 | drop(cache); | ||
| 1782 | let cache = Replica::open(&path).unwrap(); | ||
| 1783 | assert!( | ||
| 1784 | matches!(cache.sync_once(&mut server).unwrap(),Some((id,EditStatus::Published{..}))if id==local_id) | ||
| 1785 | ); | ||
| 1786 | assert_eq!(server.publications, 0); | ||
| 1787 | assert_eq!(server.confirmations, 2); | ||
| 1788 | assert_ne!(server.durable, source); | ||
| 1789 | assert!(cache.pending().unwrap().is_empty()); | ||
| 1790 | } | ||
| 1791 | |||
| 1792 | #[test] | ||
| 1793 | fn retired_format_attempts_require_the_complete_durable_effect_without_replay() { | ||
| 1794 | use onestore::TextAttribute as A; | ||
| 1795 | for (complete, fault) in [ | ||
| 1796 | (true, Fault::None), | ||
| 1797 | (true, Fault::Confirm), | ||
| 1798 | (true, Fault::ConfirmCommitted), | ||
| 1799 | (false, Fault::None), | ||
| 1800 | ] { | ||
| 1801 | let directory = tempfile::tempdir().unwrap(); | ||
| 1802 | let path = directory.path().join("retired-format.sqlite"); | ||
| 1803 | let source = onestore::create_section("format.one", "abcdef", "Author").unwrap(); | ||
| 1804 | let (sid, object, _) = text(&source); | ||
| 1805 | let cache = Replica::create(&path, &source).unwrap(); | ||
| 1806 | let id = cache | ||
| 1807 | .format( | ||
| 1808 | &source, | ||
| 1809 | sid, | ||
| 1810 | object, | ||
| 1811 | 1..5, | ||
| 1812 | &[A::Bold(true), A::FontSize(18.0)], | ||
| 1813 | ) | ||
| 1814 | .unwrap() | ||
| 1815 | .unwrap(); | ||
| 1816 | let local = cache.snapshot().unwrap(); | ||
| 1817 | let mut server = Server::new(&source); | ||
| 1818 | server.fault = Fault::UnknownAfter; | ||
| 1819 | assert!(cache.sync_once(&mut server).is_err()); | ||
| 1820 | let attempted = cache.status(id).unwrap(); | ||
| 1821 | let Some(EditStatus::AwaitingConfirmation { revision: retired }) = attempted else { | ||
| 1822 | panic!() | ||
| 1823 | }; | ||
| 1824 | drop(cache); | ||
| 1825 | let prefix = PreparedEdit::text(&source, sid, object, 0..0, "prefix ").unwrap(); | ||
| 1826 | let mut attributes = vec![A::Bold(true), A::Italic(true)]; | ||
| 1827 | if complete { | ||
| 1828 | attributes.push(A::FontSize(18.0)); | ||
| 1829 | } | ||
| 1830 | server.visible = PreparedEdit::format(prefix.as_bytes(), sid, object, 8..12, &attributes) | ||
| 1831 | .unwrap() | ||
| 1832 | .as_bytes() | ||
| 1833 | .to_vec(); | ||
| 1834 | let store = Store::parse(&server.visible).unwrap(); | ||
| 1835 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 1836 | let current = index.spaces[&sid].labels[&(ExGuid::default(), 1)]; | ||
| 1837 | assert!(!index.spaces[&sid].revisions.contains_key(&retired)); | ||
| 1838 | let cache = Replica::open(&path).unwrap(); | ||
| 1839 | server.fault = fault; | ||
| 1840 | let result = cache.sync_once(&mut server); | ||
| 1841 | if !complete { | ||
| 1842 | assert_eq!(result.unwrap(), attempted.map(|s| (id, s))); | ||
| 1843 | assert_eq!(server.confirmations, 0); | ||
| 1844 | assert_eq!(cache.snapshot().unwrap(), local); | ||
| 1845 | assert_eq!(server.durable, source); | ||
| 1846 | } else { | ||
| 1847 | if matches!(fault, Fault::Confirm) { | ||
| 1848 | assert!(matches!(result, Err(Error::Remote(e)) if e.state == CommitState::Unknown)); | ||
| 1849 | assert_eq!(cache.status(id).unwrap(), attempted); | ||
| 1850 | assert_eq!(cache.snapshot().unwrap(), local); | ||
| 1851 | assert_eq!(server.durable, source); | ||
| 1852 | let complete = server.visible.clone(); | ||
| 1853 | server.visible = | ||
| 1854 | PreparedEdit::format(&complete, sid, object, 8..12, &[A::Bold(false)]) | ||
| 1855 | .unwrap() | ||
| 1856 | .as_bytes() | ||
| 1857 | .to_vec(); | ||
| 1858 | assert_eq!( | ||
| 1859 | cache.sync_once(&mut server).unwrap(), | ||
| 1860 | attempted.map(|s| (id, s)) | ||
| 1861 | ); | ||
| 1862 | assert_eq!(server.confirmations, 1); | ||
| 1863 | assert_eq!(cache.snapshot().unwrap(), local); | ||
| 1864 | server.visible = complete; | ||
| 1865 | cache.sync_once(&mut server).unwrap(); | ||
| 1866 | } else if matches!(fault, Fault::ConfirmCommitted) { | ||
| 1867 | assert!( | ||
| 1868 | matches!(result, Err(Error::Remote(e)) if e.state == CommitState::Committed) | ||
| 1869 | ); | ||
| 1870 | } else { | ||
| 1871 | assert_eq!( | ||
| 1872 | result.unwrap(), | ||
| 1873 | Some((id, EditStatus::Published { revision: current })) | ||
| 1874 | ); | ||
| 1875 | } | ||
| 1876 | assert_ne!(current, retired); | ||
| 1877 | assert_eq!( | ||
| 1878 | cache.status(id).unwrap(), | ||
| 1879 | Some(EditStatus::Published { revision: current }) | ||
| 1880 | ); | ||
| 1881 | assert!(cache.pending().unwrap().is_empty()); | ||
| 1882 | assert_eq!(text(&server.durable).2, "prefix abcdef"); | ||
| 1883 | assert_ne!(server.durable, source); | ||
| 1884 | drop(cache); | ||
| 1885 | let cache = Replica::open(&path).unwrap(); | ||
| 1886 | assert_eq!( | ||
| 1887 | cache.status(id).unwrap(), | ||
| 1888 | Some(EditStatus::Published { revision: current }) | ||
| 1889 | ); | ||
| 1890 | } | ||
| 1891 | assert_eq!(server.publications, 1); | ||
| 1892 | } | ||
| 1893 | } | ||
| 1894 | |||
| 1895 | #[test] | ||
| 1896 | fn uncertain_formatting_keeps_the_original_attempt_and_never_replays() { | ||
| 1897 | use onestore::TextAttribute as A; | ||
| 1898 | for fault in [Fault::UnknownBefore, Fault::UnknownAfter] { | ||
| 1899 | let directory = tempfile::tempdir().unwrap(); | ||
| 1900 | let path = directory.path().join("unknown-format.sqlite"); | ||
| 1901 | let source = onestore::create_section("format.one", "abcdef", "Author").unwrap(); | ||
| 1902 | let (sid, id, _) = text(&source); | ||
| 1903 | let cache = Replica::create(&path, &source).unwrap(); | ||
| 1904 | let local_id = cache | ||
| 1905 | .format(&source, sid, id, 1..5, &[A::Bold(true)]) | ||
| 1906 | .unwrap() | ||
| 1907 | .unwrap(); | ||
| 1908 | let mut server = Server::new(&source); | ||
| 1909 | server.fault = fault; | ||
| 1910 | assert!( | ||
| 1911 | matches!(cache.sync_once(&mut server),Err(Error::Remote(e))if e.state==CommitState::Unknown) | ||
| 1912 | ); | ||
| 1913 | let status = cache.status(local_id).unwrap(); | ||
| 1914 | drop(cache); | ||
| 1915 | let cache = Replica::open(&path).unwrap(); | ||
| 1916 | if matches!(fault, Fault::UnknownBefore) { | ||
| 1917 | for _ in 0..4 { | ||
| 1918 | assert_eq!( | ||
| 1919 | cache.sync_once(&mut server).unwrap(), | ||
| 1920 | status.map(|s| (local_id, s)) | ||
| 1921 | ); | ||
| 1922 | } | ||
| 1923 | assert_eq!(server.confirmations, 0); | ||
| 1924 | } else { | ||
| 1925 | assert!(matches!( | ||
| 1926 | cache.sync_once(&mut server).unwrap(), | ||
| 1927 | Some((_, EditStatus::Published { .. })) | ||
| 1928 | )); | ||
| 1929 | assert_eq!(server.confirmations, 1); | ||
| 1930 | } | ||
| 1931 | assert_eq!(server.publications, 1); | ||
| 1932 | } | ||
| 1933 | } | ||
| 1934 | |||
| 1935 | #[test] | ||
| 1936 | fn reviewed_format_conflicts_preserve_dependent_edits_and_recheck_later_remote_changes() { | ||
| 1937 | use onestore::TextAttribute as A; | ||
| 1938 | for changed_again in [false, true] { | ||
| 1939 | let directory = tempfile::tempdir().unwrap(); | ||
| 1940 | let path = directory.path().join("review-format.sqlite"); | ||
| 1941 | let source = onestore::create_section("format.one", "abcdef", "Author").unwrap(); | ||
| 1942 | let (sid, id, _) = text(&source); | ||
| 1943 | let cache = Replica::create(&path, &source).unwrap(); | ||
| 1944 | let first = cache | ||
| 1945 | .format(&source, sid, id, 1..5, &[A::FontSize(14.0)]) | ||
| 1946 | .unwrap() | ||
| 1947 | .unwrap(); | ||
| 1948 | let second = cache | ||
| 1949 | .edit_text(&cache.snapshot().unwrap(), sid, id, 2..2, "X") | ||
| 1950 | .unwrap() | ||
| 1951 | .unwrap(); | ||
| 1952 | let local = cache.snapshot().unwrap(); | ||
| 1953 | let pending = cache.pending().unwrap(); | ||
| 1954 | let remote = PreparedEdit::format(&source, sid, id, 1..5, &[A::FontSize(18.0)]).unwrap(); | ||
| 1955 | let mut server = Server::new(remote.as_bytes()); | ||
| 1956 | assert_eq!( | ||
| 1957 | cache.sync_once(&mut server).unwrap(), | ||
| 1958 | Some((first, EditStatus::Conflict(ConflictKind::FormattingChanged))) | ||
| 1959 | ); | ||
| 1960 | assert!( | ||
| 1961 | matches!(cache.rebase_conflict(first,&source,remote.as_bytes(),1..5),Err(Error::Io(e))if e.kind()==io::ErrorKind::ResourceBusy) | ||
| 1962 | ); | ||
| 1963 | cache | ||
| 1964 | .rebase_conflict(first, &local, remote.as_bytes(), 1..5) | ||
| 1965 | .unwrap(); | ||
| 1966 | assert_eq!(cache.snapshot().unwrap(), local); | ||
| 1967 | assert_eq!(cache.pending().unwrap()[1], pending[1]); | ||
| 1968 | drop(cache); | ||
| 1969 | let cache = Replica::open(&path).unwrap(); | ||
| 1970 | if changed_again { | ||
| 1971 | let newer = | ||
| 1972 | PreparedEdit::format(&server.visible, sid, id, 1..5, &[A::FontSize(20.0)]).unwrap(); | ||
| 1973 | server = Server::new(newer.as_bytes()); | ||
| 1974 | assert_eq!( | ||
| 1975 | cache.sync_once(&mut server).unwrap(), | ||
| 1976 | Some((first, EditStatus::Conflict(ConflictKind::FormattingChanged))) | ||
| 1977 | ); | ||
| 1978 | assert_eq!(server.publications, 0); | ||
| 1979 | assert_eq!(cache.snapshot().unwrap(), local); | ||
| 1980 | } else { | ||
| 1981 | for expected in [first, second] { | ||
| 1982 | assert!( | ||
| 1983 | matches!(cache.sync_once(&mut server).unwrap(),Some((id,EditStatus::Published{..}))if id==expected) | ||
| 1984 | ); | ||
| 1985 | } | ||
| 1986 | assert_eq!(text(&server.durable).2, "abXcdef"); | ||
| 1987 | assert!(cache.pending().unwrap().is_empty()); | ||
| 1988 | } | ||
| 1989 | } | ||
| 1990 | } | ||
| 1991 | |||
| 1992 | #[test] | ||
| 1993 | fn superscript_reconciliation_checks_the_implicit_subscript_change() { | ||
| 1994 | use onestore::TextAttribute as A; | ||
| 1995 | let directory = tempfile::tempdir().unwrap(); | ||
| 1996 | let source = onestore::create_section("script.one", "abc", "Author").unwrap(); | ||
| 1997 | let (sid, id, _) = text(&source); | ||
| 1998 | let cache = Replica::create(directory.path().join("script.sqlite"), &source).unwrap(); | ||
| 1999 | let local = cache | ||
| 2000 | .format(&source, sid, id, 0..3, &[A::Superscript(true)]) | ||
| 2001 | .unwrap() | ||
| 2002 | .unwrap(); | ||
| 2003 | let remote = PreparedEdit::format(&source, sid, id, 0..3, &[A::Subscript(true)]).unwrap(); | ||
| 2004 | let mut server = Server::new(remote.as_bytes()); | ||
| 2005 | assert_eq!( | ||
| 2006 | cache.sync_once(&mut server).unwrap(), | ||
| 2007 | Some((local, EditStatus::Conflict(ConflictKind::FormattingChanged))) | ||
| 2008 | ); | ||
| 2009 | assert_eq!(server.publications, 0); | ||
| 2010 | } | ||
| 2011 | |||
| 2012 | #[test] | ||
| 2013 | fn seeded_formatting_reconciliation_matches_a_character_model() { | ||
| 2014 | use onestore::TextAttribute as A; | ||
| 2015 | #[derive(Clone, Debug, PartialEq)] | ||
| 2016 | struct Style { | ||
| 2017 | size: f32, | ||
| 2018 | color: u32, | ||
| 2019 | bold: bool, | ||
| 2020 | italic: bool, | ||
| 2021 | } | ||
| 2022 | let mut conflicts = 0; | ||
| 2023 | let mut published = 0; | ||
| 2024 | let mut satisfied_parts = 0; | ||
| 2025 | for seed in 1_u64..=128 { | ||
| 2026 | let mut random = seed; | ||
| 2027 | let mut next = || { | ||
| 2028 | random = random | ||
| 2029 | .wrapping_mul(6364136223846793005) | ||
| 2030 | .wrapping_add(1442695040888963407); | ||
| 2031 | random >> 32 | ||
| 2032 | }; | ||
| 2033 | let directory = tempfile::tempdir().unwrap(); | ||
| 2034 | let path = directory.path().join("model.sqlite"); | ||
| 2035 | let mut source = | ||
| 2036 | onestore::create_section("model.one", "abcdefghijklmnopqrstuvwxyz012345", "Author") | ||
| 2037 | .unwrap(); | ||
| 2038 | let (sid, object, original_text) = text(&source); | ||
| 2039 | let mut baseline = vec![ | ||
| 2040 | Style { | ||
| 2041 | size: 11.0, | ||
| 2042 | color: 0xff000000, | ||
| 2043 | bold: false, | ||
| 2044 | italic: false | ||
| 2045 | }; | ||
| 2046 | 32 | ||
| 2047 | ]; | ||
| 2048 | for _ in 0..6 { | ||
| 2049 | let start = next() as usize % 32; | ||
| 2050 | let end = start + 1 + next() as usize % (32 - start); | ||
| 2051 | let size = [12.0, 14.0, 18.0][next() as usize % 3]; | ||
| 2052 | let color: u32 = [0xabcdef, 0x987654, 0xff000000][next() as usize % 3]; | ||
| 2053 | let bold = next() % 2 == 0; | ||
| 2054 | source = PreparedEdit::format( | ||
| 2055 | &source, | ||
| 2056 | sid, | ||
| 2057 | object, | ||
| 2058 | start as u32..end as u32, | ||
| 2059 | &[ | ||
| 2060 | A::FontSize(size), | ||
| 2061 | A::Color((color != 0xff000000).then(|| { | ||
| 2062 | let b = color.to_le_bytes(); | ||
| 2063 | [b[0], b[1], b[2]] | ||
| 2064 | })), | ||
| 2065 | A::Bold(bold), | ||
| 2066 | ], | ||
| 2067 | ) | ||
| 2068 | .unwrap() | ||
| 2069 | .as_bytes() | ||
| 2070 | .to_vec(); | ||
| 2071 | for style in &mut baseline[start..end] { | ||
| 2072 | style.size = size; | ||
| 2073 | style.color = color; | ||
| 2074 | style.bold = bold; | ||
| 2075 | } | ||
| 2076 | } | ||
| 2077 | let start = next() as usize % 24; | ||
| 2078 | let end = start + 2 + next() as usize % (31 - start); | ||
| 2079 | let attribute = match seed % 3 { | ||
| 2080 | 0 => A::FontSize(21.0), | ||
| 2081 | 1 => A::Color(Some([0x12, 0x34, 0x56])), | ||
| 2082 | _ => A::Bold(!baseline[start].bold), | ||
| 2083 | }; | ||
| 2084 | let cache = Replica::create(&path, &source).unwrap(); | ||
| 2085 | let id = cache | ||
| 2086 | .format( | ||
| 2087 | &source, | ||
| 2088 | sid, | ||
| 2089 | object, | ||
| 2090 | start as u32..end as u32, | ||
| 2091 | std::slice::from_ref(&attribute), | ||
| 2092 | ) | ||
| 2093 | .unwrap() | ||
| 2094 | .unwrap(); | ||
| 2095 | let local = cache.snapshot().unwrap(); | ||
| 2096 | let mut remote = source.clone(); | ||
| 2097 | let mut expected = baseline.clone(); | ||
| 2098 | for step in 0..8 { | ||
| 2099 | let left = next() as usize % 32; | ||
| 2100 | let right = left + 1 + next() as usize % (32 - left); | ||
| 2101 | let update = if step % 3 == 0 { | ||
| 2102 | attribute.clone() | ||
| 2103 | } else { | ||
| 2104 | match next() % 4 { | ||
| 2105 | 0 => A::FontSize([11.0, 14.0, 18.0, 21.0][next() as usize % 4]), | ||
| 2106 | 1 => A::Color(Some([0x99, 0x88, 0x77])), | ||
| 2107 | 2 => A::Bold(next() % 2 == 0), | ||
| 2108 | _ => A::Italic(true), | ||
| 2109 | } | ||
| 2110 | }; | ||
| 2111 | remote = PreparedEdit::format( | ||
| 2112 | &remote, | ||
| 2113 | sid, | ||
| 2114 | object, | ||
| 2115 | left as u32..right as u32, | ||
| 2116 | std::slice::from_ref(&update), | ||
| 2117 | ) | ||
| 2118 | .unwrap() | ||
| 2119 | .as_bytes() | ||
| 2120 | .to_vec(); | ||
| 2121 | for style in &mut expected[left..right] { | ||
| 2122 | match update { | ||
| 2123 | A::FontSize(value) => style.size = value, | ||
| 2124 | A::Color(Some([r, g, b])) => style.color = u32::from_le_bytes([r, g, b, 0]), | ||
| 2125 | A::Bold(value) => style.bold = value, | ||
| 2126 | A::Italic(value) => style.italic = value, | ||
| 2127 | _ => unreachable!(), | ||
| 2128 | } | ||
| 2129 | } | ||
| 2130 | } | ||
| 2131 | let conflict = (start..end).any(|i| match attribute { | ||
| 2132 | A::FontSize(value) => expected[i].size != baseline[i].size && expected[i].size != value, | ||
| 2133 | A::Color(_) => expected[i].color != baseline[i].color && expected[i].color != 0x563412, | ||
| 2134 | A::Bold(value) => expected[i].bold != baseline[i].bold && expected[i].bold != value, | ||
| 2135 | _ => unreachable!(), | ||
| 2136 | }); | ||
| 2137 | drop(cache); | ||
| 2138 | let cache = Replica::open(&path).unwrap(); | ||
| 2139 | let mut server = Server::new(&remote); | ||
| 2140 | let result = cache.sync_once(&mut server).unwrap().unwrap(); | ||
| 2141 | assert_eq!(result.0, id, "seed {seed}"); | ||
| 2142 | if conflict { | ||
| 2143 | conflicts += 1; | ||
| 2144 | assert_eq!( | ||
| 2145 | result.1, | ||
| 2146 | EditStatus::Conflict(ConflictKind::FormattingChanged), | ||
| 2147 | "seed {seed}" | ||
| 2148 | ); | ||
| 2149 | assert_eq!(server.publications, 0, "seed {seed}"); | ||
| 2150 | assert_eq!(cache.snapshot().unwrap(), local, "seed {seed}"); | ||
| 2151 | continue; | ||
| 2152 | } | ||
| 2153 | published += 1; | ||
| 2154 | assert!( | ||
| 2155 | matches!(result.1, EditStatus::Published { .. }), | ||
| 2156 | "seed {seed}: {result:?}" | ||
| 2157 | ); | ||
| 2158 | for style in &mut expected[start..end] { | ||
| 2159 | match attribute { | ||
| 2160 | A::FontSize(value) => { | ||
| 2161 | satisfied_parts += usize::from(style.size == value); | ||
| 2162 | style.size = value; | ||
| 2163 | } | ||
| 2164 | A::Color(_) => { | ||
| 2165 | satisfied_parts += usize::from(style.color == 0x563412); | ||
| 2166 | style.color = 0x563412; | ||
| 2167 | } | ||
| 2168 | A::Bold(value) => { | ||
| 2169 | satisfied_parts += usize::from(style.bold == value); | ||
| 2170 | style.bold = value; | ||
| 2171 | } | ||
| 2172 | _ => unreachable!(), | ||
| 2173 | } | ||
| 2174 | } | ||
| 2175 | assert_eq!(text(&server.durable).2, original_text, "seed {seed}"); | ||
| 2176 | let store = Store::parse(&server.durable).unwrap(); | ||
| 2177 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 2178 | let doc = Document::parse(&index).unwrap(); | ||
| 2179 | let space = &doc.spaces[&sid]; | ||
| 2180 | let revision = &space.revisions[&space.contexts[&ExGuid::default()]]; | ||
| 2181 | let actual: Vec<_> = revision | ||
| 2182 | .text_runs(object) | ||
| 2183 | .unwrap() | ||
| 2184 | .iter() | ||
| 2185 | .flat_map(|run| { | ||
| 2186 | std::iter::repeat_n( | ||
| 2187 | Style { | ||
| 2188 | size: run.format.font_size.unwrap(), | ||
| 2189 | color: run.format.color.unwrap_or(0xff000000), | ||
| 2190 | bold: run.format.bold.unwrap_or(false), | ||
| 2191 | italic: run.format.italic.unwrap_or(false), | ||
| 2192 | }, | ||
| 2193 | run.text.chars().count(), | ||
| 2194 | ) | ||
| 2195 | }) | ||
| 2196 | .collect(); | ||
| 2197 | assert_eq!(actual, expected, "seed {seed}"); | ||
| 2198 | drop(cache); | ||
| 2199 | let cache = Replica::open(&path).unwrap(); | ||
| 2200 | assert_eq!(cache.status(id).unwrap(), Some(result.1)); | ||
| 2201 | let snapshot = cache.snapshot().unwrap(); | ||
| 2202 | assert_eq!(snapshot.len(), server.durable.len(), "seed {seed}"); | ||
| 2203 | assert!( | ||
| 2204 | snapshot | ||
| 2205 | .iter() | ||
| 2206 | .zip(&server.durable) | ||
| 2207 | .enumerate() | ||
| 2208 | .all(|(offset, (a, b))| a == b || (212..252).contains(&offset)), | ||
| 2209 | "seed {seed}: only confirmation version metadata may change" | ||
| 2210 | ); | ||
| 2211 | assert_eq!(cache.sync_once(&mut server).unwrap(), None); | ||
| 2212 | assert!( | ||
| 2213 | cache.snapshot().unwrap() == server.durable, | ||
| 2214 | "seed {seed}: refreshed snapshot" | ||
| 2215 | ); | ||
| 2216 | } | ||
| 2217 | assert!( | ||
| 2218 | conflicts >= 16 && published >= 32 && satisfied_parts >= 128, | ||
| 2219 | "conflicts={conflicts}, published={published}, already desired characters={satisfied_parts}" | ||
| 2220 | ); | ||
| 2221 | println!( | ||
| 2222 | "128 seeds: {conflicts} conflicts, {published} publications, {satisfied_parts} already desired characters" | ||
| 2223 | ); | ||
| 2224 | } | ||
| 2225 | |||
| 2226 | #[test] | ||
| 2227 | fn inserted_empty_text_retains_formatting_and_dependent_text_across_sync() { | ||
| 2228 | use onestore::{Insertion, TextAttribute as A}; | ||
| 2229 | let directory = tempfile::tempdir().unwrap(); | ||
| 2230 | let path = directory.path().join("empty.sqlite"); | ||
| 2231 | let source = onestore::create_section("empty.one", "Original", "Author").unwrap(); | ||
| 2232 | let store = Store::parse(&source).unwrap(); | ||
| 2233 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 2234 | let doc = Document::parse(&index).unwrap(); | ||
| 2235 | let (sid, page) = doc.pages().unwrap()[0]; | ||
| 2236 | let insertion = Insertion::outline(page, 144.0, 144.0, "", "Author").unwrap(); | ||
| 2237 | let cache = Replica::create(&path, &source).unwrap(); | ||
| 2238 | let first = cache.insert(&source, sid, &insertion).unwrap().unwrap(); | ||
| 2239 | let second = cache | ||
| 2240 | .format( | ||
| 2241 | &cache.snapshot().unwrap(), | ||
| 2242 | sid, | ||
| 2243 | insertion.text_object(), | ||
| 2244 | 0..0, | ||
| 2245 | &[A::Bold(true), A::FontSize(18.0)], | ||
| 2246 | ) | ||
| 2247 | .unwrap() | ||
| 2248 | .unwrap(); | ||
| 2249 | let third = cache | ||
| 2250 | .edit_text( | ||
| 2251 | &cache.snapshot().unwrap(), | ||
| 2252 | sid, | ||
| 2253 | insertion.text_object(), | ||
| 2254 | 0..0, | ||
| 2255 | "Typed 🦀", | ||
| 2256 | ) | ||
| 2257 | .unwrap() | ||
| 2258 | .unwrap(); | ||
| 2259 | let pending = cache.pending().unwrap(); | ||
| 2260 | drop(cache); | ||
| 2261 | let cache = Replica::open(&path).unwrap(); | ||
| 2262 | assert_eq!(cache.pending().unwrap(), pending); | ||
| 2263 | let mut server = Server::new(&source); | ||
| 2264 | for id in [first, second, third] { | ||
| 2265 | let result = cache.sync_once(&mut server).unwrap(); | ||
| 2266 | assert!( | ||
| 2267 | matches!(result,Some((actual,EditStatus::Published{..}))if actual==id), | ||
| 2268 | "{result:?}" | ||
| 2269 | ); | ||
| 2270 | } | ||
| 2271 | let store = Store::parse(&server.durable).unwrap(); | ||
| 2272 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 2273 | let doc = Document::parse(&index).unwrap(); | ||
| 2274 | let space = &doc.spaces[&sid]; | ||
| 2275 | let revision = &space.revisions[&space.contexts[&ExGuid::default()]]; | ||
| 2276 | let runs = revision.text_runs(insertion.text_object()).unwrap(); | ||
| 2277 | assert_eq!(runs.iter().map(|r| r.text).collect::<String>(), "Typed 🦀"); | ||
| 2278 | assert!( | ||
| 2279 | runs.iter() | ||
| 2280 | .all(|r| r.format.bold == Some(true) && r.format.font_size == Some(18.0)) | ||
| 2281 | ); | ||
| 2282 | assert_eq!(cache.snapshot().unwrap(), server.durable); | ||
| 2283 | } | ||
| 2284 | |||
| 2285 | #[test] | ||
| 2286 | fn every_visual_attribute_rebases_with_an_independent_remote_attribute() { | ||
| 2287 | use onestore::TextAttribute as A; | ||
| 2288 | use serde_json::json; | ||
| 2289 | let mut cases = Vec::new(); | ||
| 2290 | for value in [false, true] { | ||
| 2291 | cases.extend([ | ||
| 2292 | (A::Bold(!value), A::Bold(value), "bold", json!(value)), | ||
| 2293 | (A::Italic(!value), A::Italic(value), "italic", json!(value)), | ||
| 2294 | ( | ||
| 2295 | A::Underline(!value), | ||
| 2296 | A::Underline(value), | ||
| 2297 | "underline", | ||
| 2298 | json!(value), | ||
| 2299 | ), | ||
| 2300 | (A::Strike(!value), A::Strike(value), "strike", json!(value)), | ||
| 2301 | ( | ||
| 2302 | A::Superscript(!value), | ||
| 2303 | A::Superscript(value), | ||
| 2304 | "superscript", | ||
| 2305 | json!(value), | ||
| 2306 | ), | ||
| 2307 | ( | ||
| 2308 | A::Subscript(!value), | ||
| 2309 | A::Subscript(value), | ||
| 2310 | "subscript", | ||
| 2311 | json!(value), | ||
| 2312 | ), | ||
| 2313 | ]); | ||
| 2314 | } | ||
| 2315 | cases.extend([ | ||
| 2316 | ( | ||
| 2317 | A::Font("Georgia".into()), | ||
| 2318 | A::Font("Arial".into()), | ||
| 2319 | "font", | ||
| 2320 | json!("Arial"), | ||
| 2321 | ), | ||
| 2322 | ( | ||
| 2323 | A::FontSize(11.0), | ||
| 2324 | A::FontSize(18.0), | ||
| 2325 | "font_size", | ||
| 2326 | json!(18.0), | ||
| 2327 | ), | ||
| 2328 | ( | ||
| 2329 | A::Color(None), | ||
| 2330 | A::Color(Some([1, 2, 3])), | ||
| 2331 | "color", | ||
| 2332 | json!(0x030201), | ||
| 2333 | ), | ||
| 2334 | ( | ||
| 2335 | A::Color(Some([1, 2, 3])), | ||
| 2336 | A::Color(None), | ||
| 2337 | "color", | ||
| 2338 | json!(0xff000000_u32), | ||
| 2339 | ), | ||
| 2340 | ( | ||
| 2341 | A::Highlight(None), | ||
| 2342 | A::Highlight(Some([4, 5, 6])), | ||
| 2343 | "highlight", | ||
| 2344 | json!(0x060504), | ||
| 2345 | ), | ||
| 2346 | ( | ||
| 2347 | A::Highlight(Some([4, 5, 6])), | ||
| 2348 | A::Highlight(None), | ||
| 2349 | "highlight", | ||
| 2350 | json!(0xff000000_u32), | ||
| 2351 | ), | ||
| 2352 | ]); | ||
| 2353 | for (baseline, desired, field, value) in cases { | ||
| 2354 | let directory = tempfile::tempdir().unwrap(); | ||
| 2355 | let path = directory.path().join("attribute.sqlite"); | ||
| 2356 | let source = onestore::create_section("attribute.one", "abc", "Author").unwrap(); | ||
| 2357 | let (sid, object, _) = text(&source); | ||
| 2358 | let source = PreparedEdit::format(&source, sid, object, 0..3, &[baseline]) | ||
| 2359 | .unwrap() | ||
| 2360 | .as_bytes() | ||
| 2361 | .to_vec(); | ||
| 2362 | let cache = Replica::create(&path, &source).unwrap(); | ||
| 2363 | let id = cache | ||
| 2364 | .format(&source, sid, object, 0..3, &[desired]) | ||
| 2365 | .unwrap() | ||
| 2366 | .unwrap(); | ||
| 2367 | let moved = PreparedEdit::text(&source, sid, object, 0..0, "Z").unwrap(); | ||
| 2368 | let (other, other_field, other_value) = if field == "font_size" { | ||
| 2369 | (A::Italic(true), "italic", json!(true)) | ||
| 2370 | } else { | ||
| 2371 | (A::FontSize(22.0), "font_size", json!(22.0)) | ||
| 2372 | }; | ||
| 2373 | let remote = PreparedEdit::format(moved.as_bytes(), sid, object, 1..4, &[other]).unwrap(); | ||
| 2374 | let mut server = Server::new(remote.as_bytes()); | ||
| 2375 | drop(cache); | ||
| 2376 | let cache = Replica::open(&path).unwrap(); | ||
| 2377 | let result = cache.sync_once(&mut server).unwrap(); | ||
| 2378 | assert!( | ||
| 2379 | matches!(result,Some((observed,EditStatus::Published{..}))if observed==id), | ||
| 2380 | "{field}={value}: {result:?}" | ||
| 2381 | ); | ||
| 2382 | assert_eq!(text(&server.durable).2, "Zabc"); | ||
| 2383 | let store = Store::parse(&server.durable).unwrap(); | ||
| 2384 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 2385 | let doc = Document::parse(&index).unwrap(); | ||
| 2386 | let space = &doc.spaces[&sid]; | ||
| 2387 | let revision = &space.revisions[&space.contexts[&ExGuid::default()]]; | ||
| 2388 | let mut position = 0; | ||
| 2389 | for run in revision.text_runs(object).unwrap() { | ||
| 2390 | let format = serde_json::to_value(run.format).unwrap(); | ||
| 2391 | for _ in run.text.chars() { | ||
| 2392 | if position > 0 { | ||
| 2393 | assert_eq!(format[field], value, "{field}, character {position}"); | ||
| 2394 | assert_eq!( | ||
| 2395 | format[other_field], other_value, | ||
| 2396 | "{field}, character {position}" | ||
| 2397 | ); | ||
| 2398 | } | ||
| 2399 | position += 1; | ||
| 2400 | } | ||
| 2401 | } | ||
| 2402 | assert_eq!(position, 4); | ||
| 2403 | } | ||
| 2404 | } | ||
| 2405 | |||
| 2406 | #[test] | ||
| 2407 | fn reviewed_insertion_placements_preserve_identity_and_dependent_operations() { | ||
| 2408 | use onestore::{Insertion, TextAttribute as A}; | ||
| 2409 | use onestore_offline::Operation; | ||
| 2410 | for outline_case in [false, true] { | ||
| 2411 | let directory = tempfile::tempdir().unwrap(); | ||
| 2412 | let path = directory.path().join("placement.sqlite"); | ||
| 2413 | let base = onestore::create_section("placement.one", "Original", "Author").unwrap(); | ||
| 2414 | let (sid, _, _) = text(&base); | ||
| 2415 | let store = Store::parse(&base).unwrap(); | ||
| 2416 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 2417 | let doc = Document::parse(&index).unwrap(); | ||
| 2418 | let (_, page) = doc.pages().unwrap()[0]; | ||
| 2419 | let space = &doc.spaces[&sid]; | ||
| 2420 | let view = &space.revisions[&space.contexts[&ExGuid::default()]]; | ||
| 2421 | let parent = *view.nodes[&page] | ||
| 2422 | .children | ||
| 2423 | .iter() | ||
| 2424 | .find(|id| matches!(view.nodes[id].kind, Kind::Outline { .. })) | ||
| 2425 | .unwrap(); | ||
| 2426 | let anchor = Insertion::paragraph(parent, None, "Temporary anchor", "Author").unwrap(); | ||
| 2427 | let source = PreparedEdit::insert(&base, sid, &anchor) | ||
| 2428 | .unwrap() | ||
| 2429 | .as_bytes() | ||
| 2430 | .to_vec(); | ||
| 2431 | let insertion = if outline_case { | ||
| 2432 | Insertion::outline(page, 144.0, 144.0, "Offline", "Author").unwrap() | ||
| 2433 | } else { | ||
| 2434 | Insertion::paragraph(parent, Some(anchor.object()), "Offline", "Author").unwrap() | ||
| 2435 | }; | ||
| 2436 | let cache = Replica::create(&path, &source).unwrap(); | ||
| 2437 | let first = cache.insert(&source, sid, &insertion).unwrap().unwrap(); | ||
| 2438 | let second = cache | ||
| 2439 | .format( | ||
| 2440 | &cache.snapshot().unwrap(), | ||
| 2441 | sid, | ||
| 2442 | insertion.text_object(), | ||
| 2443 | 0..7, | ||
| 2444 | &[A::Bold(true)], | ||
| 2445 | ) | ||
| 2446 | .unwrap() | ||
| 2447 | .unwrap(); | ||
| 2448 | let third = cache | ||
| 2449 | .edit_text( | ||
| 2450 | &cache.snapshot().unwrap(), | ||
| 2451 | sid, | ||
| 2452 | insertion.text_object(), | ||
| 2453 | 7..7, | ||
| 2454 | " 🦀", | ||
| 2455 | ) | ||
| 2456 | .unwrap() | ||
| 2457 | .unwrap(); | ||
| 2458 | let local = cache.snapshot().unwrap(); | ||
| 2459 | let pending = cache.pending().unwrap(); | ||
| 2460 | let mut server = Server::new(&base); | ||
| 2461 | if outline_case { | ||
| 2462 | drop(cache); | ||
| 2463 | let db = rusqlite::Connection::open(&path).unwrap(); | ||
| 2464 | db.execute( | ||
| 2465 | "INSERT INTO conflicts VALUES (?1,2)", | ||
| 2466 | [i64::try_from(first).unwrap()], | ||
| 2467 | ) | ||
| 2468 | .unwrap(); | ||
| 2469 | db.execute("UPDATE replica SET base=?1", [&base]).unwrap(); | ||
| 2470 | drop(db); | ||
| 2471 | } else { | ||
| 2472 | assert_eq!( | ||
| 2473 | cache.sync_once(&mut server).unwrap(), | ||
| 2474 | Some((first, EditStatus::Conflict(ConflictKind::UnsupportedEdit))) | ||
| 2475 | ); | ||
| 2476 | drop(cache); | ||
| 2477 | } | ||
| 2478 | let cache = Replica::open(&path).unwrap(); | ||
| 2479 | if outline_case { | ||
| 2480 | assert!( | ||
| 2481 | cache | ||
| 2482 | .rebase_paragraph_conflict(first, &local, &base, parent, None) | ||
| 2483 | .is_err() | ||
| 2484 | ); | ||
| 2485 | assert!( | ||
| 2486 | cache | ||
| 2487 | .rebase_outline_conflict(first, &local, &base, page, f32::NAN, 288.0) | ||
| 2488 | .is_err() | ||
| 2489 | ); | ||
| 2490 | assert!( | ||
| 2491 | matches!(cache.rebase_outline_conflict(first,&source,&base,page,288.0,360.0),Err(Error::Io(e))if e.kind()==io::ErrorKind::ResourceBusy) | ||
| 2492 | ); | ||
| 2493 | cache | ||
| 2494 | .rebase_outline_conflict(first, &local, &base, page, 288.0, 360.0) | ||
| 2495 | .unwrap(); | ||
| 2496 | } else { | ||
| 2497 | assert!( | ||
| 2498 | cache | ||
| 2499 | .rebase_outline_conflict(first, &local, &base, page, 288.0, 360.0) | ||
| 2500 | .is_err() | ||
| 2501 | ); | ||
| 2502 | assert!( | ||
| 2503 | cache | ||
| 2504 | .rebase_paragraph_conflict(first, &local, &base, parent, Some(anchor.object())) | ||
| 2505 | .is_err() | ||
| 2506 | ); | ||
| 2507 | assert!( | ||
| 2508 | matches!(cache.rebase_paragraph_conflict(first,&source,&base,parent,None),Err(Error::Io(e))if e.kind()==io::ErrorKind::ResourceBusy) | ||
| 2509 | ); | ||
| 2510 | cache | ||
| 2511 | .rebase_paragraph_conflict(first, &local, &base, parent, None) | ||
| 2512 | .unwrap(); | ||
| 2513 | } | ||
| 2514 | assert_eq!(cache.snapshot().unwrap(), local); | ||
| 2515 | assert_eq!(cache.pending().unwrap()[1..], pending[1..]); | ||
| 2516 | let Operation::Insert(rebased) = cache.pending().unwrap().remove(0).operation else { | ||
| 2517 | panic!() | ||
| 2518 | }; | ||
| 2519 | assert_eq!(rebased.object(), insertion.object()); | ||
| 2520 | assert_eq!(rebased.text_object(), insertion.text_object()); | ||
| 2521 | assert_eq!(cache.status(first).unwrap(), Some(EditStatus::Pending)); | ||
| 2522 | drop(cache); | ||
| 2523 | let cache = Replica::open(&path).unwrap(); | ||
| 2524 | for id in [first, second, third] { | ||
| 2525 | let result = cache.sync_once(&mut server).unwrap(); | ||
| 2526 | assert!( | ||
| 2527 | matches!(result,Some((actual,EditStatus::Published{..}))if actual==id), | ||
| 2528 | "{result:?}" | ||
| 2529 | ); | ||
| 2530 | } | ||
| 2531 | let store = Store::parse(&server.durable).unwrap(); | ||
| 2532 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 2533 | let doc = Document::parse(&index).unwrap(); | ||
| 2534 | let space = &doc.spaces[&sid]; | ||
| 2535 | let view = &space.revisions[&space.contexts[&ExGuid::default()]]; | ||
| 2536 | let runs = view.text_runs(insertion.text_object()).unwrap(); | ||
| 2537 | assert_eq!( | ||
| 2538 | runs.iter().map(|r| r.text).collect::<String>(), | ||
| 2539 | "Offline 🦀" | ||
| 2540 | ); | ||
| 2541 | assert!(runs.iter().all(|r| r.format.bold == Some(true))); | ||
| 2542 | if outline_case { | ||
| 2543 | assert_eq!( | ||
| 2544 | ( | ||
| 2545 | view.nodes[&insertion.object()].layout.x, | ||
| 2546 | view.nodes[&insertion.object()].layout.y | ||
| 2547 | ), | ||
| 2548 | (Some(288.0), Some(360.0)) | ||
| 2549 | ); | ||
| 2550 | } else { | ||
| 2551 | assert_eq!( | ||
| 2552 | view.nodes[&parent].children.last(), | ||
| 2553 | Some(&insertion.object()) | ||
| 2554 | ); | ||
| 2555 | } | ||
| 2556 | assert!(!view.nodes.contains_key(&anchor.object())); | ||
| 2557 | assert_eq!(server.publications, 3); | ||
| 2558 | assert!(cache.pending().unwrap().is_empty()); | ||
| 2559 | assert_eq!(cache.snapshot().unwrap(), server.durable); | ||
| 2560 | } | ||
| 2561 | } | ||
| 2562 | |||
| 2563 | #[test] | ||
| 2564 | fn placement_reviews_cannot_replace_pending_or_uncertain_attempts() { | ||
| 2565 | use onestore::Insertion; | ||
| 2566 | for outline_case in [false, true] { | ||
| 2567 | let directory = tempfile::tempdir().unwrap(); | ||
| 2568 | let path = directory.path().join("attempt-placement.sqlite"); | ||
| 2569 | let source = onestore::create_section("placement.one", "Original", "Author").unwrap(); | ||
| 2570 | let store = Store::parse(&source).unwrap(); | ||
| 2571 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 2572 | let doc = Document::parse(&index).unwrap(); | ||
| 2573 | let (sid, page) = doc.pages().unwrap()[0]; | ||
| 2574 | let space = &doc.spaces[&sid]; | ||
| 2575 | let view = &space.revisions[&space.contexts[&ExGuid::default()]]; | ||
| 2576 | let parent = *view.nodes[&page] | ||
| 2577 | .children | ||
| 2578 | .iter() | ||
| 2579 | .find(|id| matches!(view.nodes[id].kind, Kind::Outline { .. })) | ||
| 2580 | .unwrap(); | ||
| 2581 | let insertion = if outline_case { | ||
| 2582 | Insertion::outline(page, 144.0, 144.0, "Offline", "Author").unwrap() | ||
| 2583 | } else { | ||
| 2584 | Insertion::paragraph(parent, None, "Offline", "Author").unwrap() | ||
| 2585 | }; | ||
| 2586 | let cache = Replica::create(&path, &source).unwrap(); | ||
| 2587 | let id = cache.insert(&source, sid, &insertion).unwrap().unwrap(); | ||
| 2588 | let local = cache.snapshot().unwrap(); | ||
| 2589 | let pending = cache.pending().unwrap(); | ||
| 2590 | let mut server = Server::new(&source); | ||
| 2591 | for attempted in [false, true] { | ||
| 2592 | if attempted { | ||
| 2593 | server.fault = Fault::UnknownBefore; | ||
| 2594 | assert!( | ||
| 2595 | matches!(cache.sync_once(&mut server),Err(Error::Remote(e))if e.state==CommitState::Unknown) | ||
| 2596 | ); | ||
| 2597 | } | ||
| 2598 | let state = cache.status(id).unwrap(); | ||
| 2599 | let result = if outline_case { | ||
| 2600 | cache.rebase_outline_conflict(id, &local, &source, page, 288.0, 360.0) | ||
| 2601 | } else { | ||
| 2602 | cache.rebase_paragraph_conflict(id, &local, &source, parent, None) | ||
| 2603 | }; | ||
| 2604 | assert!(matches!(result,Err(Error::Io(e))if e.kind()==io::ErrorKind::InvalidInput)); | ||
| 2605 | assert_eq!(cache.pending().unwrap(), pending); | ||
| 2606 | assert_eq!(cache.status(id).unwrap(), state); | ||
| 2607 | assert_eq!(cache.snapshot().unwrap(), local); | ||
| 2608 | } | ||
| 2609 | drop(cache); | ||
| 2610 | let cache = Replica::open(&path).unwrap(); | ||
| 2611 | assert_eq!(cache.pending().unwrap(), pending); | ||
| 2612 | assert!(matches!( | ||
| 2613 | cache.sync_once(&mut server).unwrap(), | ||
| 2614 | Some((_, EditStatus::AwaitingConfirmation { .. })) | ||
| 2615 | )); | ||
| 2616 | assert_eq!(server.publications, 1); | ||
| 2617 | } | ||
| 2618 | } | ||
crates/onestore-smb/Cargo.toml created+13
| ... | @@ -0,0 +1,13 @@ | ||
| 1 | [package] | ||
| 2 | name = "onestore-smb" | ||
| 3 | version = "0.1.0" | ||
| 4 | edition = "2024" | ||
| 5 | publish = false | ||
| 6 | |||
| 7 | [dependencies] | ||
| 8 | onestore = { path = "../onestore" } | ||
| 9 | smb2 = "=0.21.0" | ||
| 10 | tokio = { version = "1", features = ["rt-multi-thread", "time"] } | ||
| 11 | |||
| 12 | [dev-dependencies] | ||
| 13 | serde_json = "1" | ||
crates/onestore-smb/README.md created+43
| ... | @@ -0,0 +1,43 @@ | ||
| 1 | # onestore-smb | ||
| 2 | |||
| 3 | Optional blocking SMB access for OneNote sections and table-of-contents files. | ||
| 4 | The core `onestore` crate remains independent of network runtimes. This is an | ||
| 5 | experimental Rust API with native interoperability evidence in the repository's | ||
| 6 | [Milestone 9](../../evidence/MILESTONE9.md). | ||
| 7 | |||
| 8 | ```no_run | ||
| 9 | use onestore_smb::{Client, Credentials}; | ||
| 10 | use std::time::Duration; | ||
| 11 | |||
| 12 | let client = Client::connect( | ||
| 13 | "server:445", | ||
| 14 | "notes", | ||
| 15 | Credentials { username: "user", password: "password", domain: "" }, | ||
| 16 | Duration::from_secs(5), | ||
| 17 | )?; | ||
| 18 | let snapshot = client.read("Personal/Video.one", 64 * 1024 * 1024)?; | ||
| 19 | let store = onestore::Store::parse(&snapshot)?; | ||
| 20 | let revisions = onestore::RevisionIndex::parse(&store)?; | ||
| 21 | let document = onestore::document::Document::parse(&revisions)?; | ||
| 22 | # Ok::<(), Box<dyn std::error::Error>>(()) | ||
| 23 | ``` | ||
| 24 | |||
| 25 | Paths are relative to the share. The read limit bounds the complete physical | ||
| 26 | snapshot. Call from a background thread outside a Tokio runtime. Use identities | ||
| 27 | from the document and the same snapshot with `Client::commit_text` or | ||
| 28 | `Client::commit_property_bytes`; their errors retain `onestore::CommitState`. | ||
| 29 | `PreparedEdit::{text,insert,format}` separate preparation from I/O: inspect the immutable image | ||
| 30 | and persist the intended revision identity before `Client::commit_prepared`. | ||
| 31 | `Client::confirm_snapshot` compares and flushes an observed image, then refreshes | ||
| 32 | its header version metadata without adding a revision. The caller must first | ||
| 33 | establish which intents that image contains and reread before another commit. | ||
| 34 | |||
| 35 | Readers use shared native guards while writers publish under native write-open | ||
| 36 | and byte-lock exclusion. Maintenance is excluded during each operation; pathname | ||
| 37 | identity is checked after acquiring the guards. Connection loss retires the | ||
| 38 | client. Reconnect for subsequent operations, and reconcile an `Unknown` edit | ||
| 39 | before retrying it. The transport does not automatically replay requests. | ||
| 40 | |||
| 41 | Device and simulator builds link for iOS. Native acceptance uses disposable | ||
| 42 | OneNote 2010 clients and Samba; it does not establish on-device execution or | ||
| 43 | physical power-loss durability. | ||
crates/onestore-smb/examples/smb_concurrent_client.rs created+22
| ... | @@ -0,0 +1,22 @@ | ||
| 1 | #[path = "../../onestore/examples/support/concurrent.rs"] | ||
| 2 | mod concurrent; | ||
| 3 | |||
| 4 | use onestore_smb::{Client, Credentials}; | ||
| 5 | use std::{env, time::Duration}; | ||
| 6 | |||
| 7 | fn main() -> Result<(), Box<dyn std::error::Error>> { | ||
| 8 | let client = Client::connect( | ||
| 9 | &env::var("ONESTORE_SMB_LAB")?, | ||
| 10 | &env::var("ONESTORE_SMB_SHARE")?, | ||
| 11 | Credentials::default(), | ||
| 12 | Duration::from_secs(10), | ||
| 13 | )?; | ||
| 14 | let args: Vec<_> = env::args().skip(1).collect(); | ||
| 15 | concurrent::run( | ||
| 16 | &args, | ||
| 17 | |path| client.read(path, 256 * 1024 * 1024), | ||
| 18 | |path, source, space, object, range, replacement| { | ||
| 19 | client.commit_text(path, source, space, object, range, replacement) | ||
| 20 | }, | ||
| 21 | ) | ||
| 22 | } | ||
crates/onestore-smb/examples/smb_reconnect_client.rs created+267
| ... | @@ -0,0 +1,267 @@ | ||
| 1 | #[path = "../../onestore/examples/support/concurrent.rs"] | ||
| 2 | mod concurrent; | ||
| 3 | |||
| 4 | use onestore::{ | ||
| 5 | CommitError, CommitState, ExGuid, RevisionIndex, Store, | ||
| 6 | document::{Document, Kind}, | ||
| 7 | }; | ||
| 8 | use onestore_smb::{Client, Credentials}; | ||
| 9 | use serde_json::json; | ||
| 10 | use std::{ | ||
| 11 | cell::RefCell, | ||
| 12 | env, io, thread, | ||
| 13 | time::{Duration, Instant, SystemTime, UNIX_EPOCH}, | ||
| 14 | }; | ||
| 15 | |||
| 16 | fn paragraph( | ||
| 17 | bytes: &[u8], | ||
| 18 | space: ExGuid, | ||
| 19 | object: ExGuid, | ||
| 20 | ) -> Result<String, Box<dyn std::error::Error>> { | ||
| 21 | let store = Store::parse(bytes)?; | ||
| 22 | let index = RevisionIndex::parse(&store)?; | ||
| 23 | index.validate_current()?; | ||
| 24 | let document = Document::parse(&index)?; | ||
| 25 | let space = &document.spaces[&space]; | ||
| 26 | let revision = &space.revisions[&space.contexts[&ExGuid::default()]]; | ||
| 27 | let Kind::RichText { text, .. } = &revision.nodes[&object].kind else { | ||
| 28 | return Err("The append target is no longer text.".into()); | ||
| 29 | }; | ||
| 30 | Ok(text.clone()) | ||
| 31 | } | ||
| 32 | |||
| 33 | // Only the owned append workload guarantees unique tokens that no writer removes. | ||
| 34 | fn retained(before: &str, token: &str, current: &str, state: CommitState) -> io::Result<bool> { | ||
| 35 | let count = current.matches(token).count(); | ||
| 36 | if count == 0 && state != CommitState::Committed && current.starts_with(before) { | ||
| 37 | return Ok(false); | ||
| 38 | } | ||
| 39 | if count == 1 | ||
| 40 | && state != CommitState::NotCommitted | ||
| 41 | && current.starts_with(&format!("{before}{token}")) | ||
| 42 | { | ||
| 43 | return Ok(true); | ||
| 44 | } | ||
| 45 | Err(io::Error::other( | ||
| 46 | "The append history contradicts the commit outcome.", | ||
| 47 | )) | ||
| 48 | } | ||
| 49 | |||
| 50 | fn main() -> Result<(), Box<dyn std::error::Error>> { | ||
| 51 | let args: Vec<_> = env::args().skip(1).collect(); | ||
| 52 | if args | ||
| 53 | .first() | ||
| 54 | .is_none_or(|mode| !["read", "write"].contains(&mode.as_str())) | ||
| 55 | { | ||
| 56 | return Err("Reconnect testing requires the append-only workload.".into()); | ||
| 57 | } | ||
| 58 | let address = env::var("ONESTORE_SMB_LAB")?; | ||
| 59 | let share = env::var("ONESTORE_SMB_SHARE")?; | ||
| 60 | let client = RefCell::new(Some(Client::connect( | ||
| 61 | &address, | ||
| 62 | &share, | ||
| 63 | Credentials::default(), | ||
| 64 | Duration::from_secs(5), | ||
| 65 | )?)); | ||
| 66 | let read = |path: &str| { | ||
| 67 | let mut session = client.borrow_mut(); | ||
| 68 | if session.is_none() { | ||
| 69 | match Client::connect( | ||
| 70 | &address, | ||
| 71 | &share, | ||
| 72 | Credentials::default(), | ||
| 73 | Duration::from_secs(5), | ||
| 74 | ) { | ||
| 75 | Ok(fresh) => { | ||
| 76 | *session = Some(fresh); | ||
| 77 | println!( | ||
| 78 | "{}", | ||
| 79 | json!({"event":"transport_connected", "at_us":SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_micros()}) | ||
| 80 | ); | ||
| 81 | } | ||
| 82 | Err(error) => { | ||
| 83 | println!( | ||
| 84 | "{}", | ||
| 85 | json!({"event":"transport_connect_error","error":error.to_string()}) | ||
| 86 | ); | ||
| 87 | return Err(io::ErrorKind::WouldBlock.into()); | ||
| 88 | } | ||
| 89 | } | ||
| 90 | } | ||
| 91 | let started = SystemTime::now() | ||
| 92 | .duration_since(UNIX_EPOCH) | ||
| 93 | .unwrap() | ||
| 94 | .as_micros(); | ||
| 95 | match session.as_ref().unwrap().read(path, 256 * 1024 * 1024) { | ||
| 96 | Ok(bytes) => { | ||
| 97 | if args.get(2).is_some_and(|actor| actor == "r0") | ||
| 98 | && let Some(folder) = env::var_os("ONESTORE_OFFLINE_FORMAT_REPLY_DIR") | ||
| 99 | { | ||
| 100 | let folder = std::path::PathBuf::from(folder); | ||
| 101 | let captured = folder.join("offline-retired.one"); | ||
| 102 | if !captured.exists() | ||
| 103 | && let Ok(marker) = std::fs::read(folder.join("offline-paused-w0.isolate")) | ||
| 104 | && let Ok(watched) = serde_json::from_slice::<serde_json::Value>(&marker) | ||
| 105 | && watched["after_us"] | ||
| 106 | .as_u64() | ||
| 107 | .is_some_and(|after| started > u128::from(after)) | ||
| 108 | { | ||
| 109 | let sid: ExGuid = watched["space"] | ||
| 110 | .as_str() | ||
| 111 | .ok_or_else(|| io::Error::other("Missing watched space"))? | ||
| 112 | .parse() | ||
| 113 | .map_err(io::Error::other)?; | ||
| 114 | let rid: ExGuid = watched["revision"] | ||
| 115 | .as_str() | ||
| 116 | .ok_or_else(|| io::Error::other("Missing watched revision"))? | ||
| 117 | .parse() | ||
| 118 | .map_err(io::Error::other)?; | ||
| 119 | let store = Store::parse(&bytes).map_err(io::Error::other)?; | ||
| 120 | let index = RevisionIndex::parse(&store).map_err(io::Error::other)?; | ||
| 121 | if index | ||
| 122 | .spaces | ||
| 123 | .get(&sid) | ||
| 124 | .is_some_and(|space| !space.revisions.contains_key(&rid)) | ||
| 125 | { | ||
| 126 | index.validate_current().map_err(io::Error::other)?; | ||
| 127 | std::fs::write(captured.with_extension("tmp"), &bytes)?; | ||
| 128 | std::fs::rename(captured.with_extension("tmp"), captured)?; | ||
| 129 | println!( | ||
| 130 | "{}", | ||
| 131 | json!({"event":"revision_retired", "space":sid.to_string(), "revision":rid.to_string(), "started_us":started, "finished_us":SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_micros()}) | ||
| 132 | ); | ||
| 133 | } | ||
| 134 | } | ||
| 135 | } | ||
| 136 | Ok(bytes) | ||
| 137 | } | ||
| 138 | Err(error) | ||
| 139 | if matches!( | ||
| 140 | error.kind(), | ||
| 141 | io::ErrorKind::Other | io::ErrorKind::TimedOut | io::ErrorKind::NotConnected | ||
| 142 | ) => | ||
| 143 | { | ||
| 144 | println!( | ||
| 145 | "{}", | ||
| 146 | json!({"event":"transport_read_error","error":error.to_string()}) | ||
| 147 | ); | ||
| 148 | *session = None; | ||
| 149 | Err(io::ErrorKind::WouldBlock.into()) | ||
| 150 | } | ||
| 151 | result => result, | ||
| 152 | } | ||
| 153 | }; | ||
| 154 | concurrent::run( | ||
| 155 | &args, | ||
| 156 | read, | ||
| 157 | |path, source, space, object, range, replacement| { | ||
| 158 | let outcome = client.borrow().as_ref().unwrap().commit_text( | ||
| 159 | path, | ||
| 160 | source, | ||
| 161 | space, | ||
| 162 | object, | ||
| 163 | range, | ||
| 164 | replacement, | ||
| 165 | ); | ||
| 166 | let Err(error) = outcome else { | ||
| 167 | return Ok(()); | ||
| 168 | }; | ||
| 169 | if error.state == CommitState::NotCommitted | ||
| 170 | && matches!( | ||
| 171 | error.error.kind(), | ||
| 172 | io::ErrorKind::WouldBlock | ||
| 173 | | io::ErrorKind::ResourceBusy | ||
| 174 | | io::ErrorKind::PermissionDenied | ||
| 175 | | io::ErrorKind::NotFound | ||
| 176 | ) | ||
| 177 | { | ||
| 178 | return Err(error); | ||
| 179 | } | ||
| 180 | println!( | ||
| 181 | "{}", | ||
| 182 | json!({"event":"transport_commit_error","state":format!("{:?}",error.state),"token":replacement,"error":error.error.to_string()}) | ||
| 183 | ); | ||
| 184 | *client.borrow_mut() = None; | ||
| 185 | let deadline = Instant::now() + Duration::from_secs(60); | ||
| 186 | loop { | ||
| 187 | if Instant::now() >= deadline { | ||
| 188 | return Err(error); | ||
| 189 | } | ||
| 190 | let current = match read(path) { | ||
| 191 | Ok(bytes) => bytes, | ||
| 192 | Err(retry) if retry.kind() == io::ErrorKind::WouldBlock => { | ||
| 193 | thread::sleep(Duration::from_millis(100)); | ||
| 194 | continue; | ||
| 195 | } | ||
| 196 | Err(_) => return Err(error), | ||
| 197 | }; | ||
| 198 | let published = paragraph(source, space, object) | ||
| 199 | .and_then(|before| { | ||
| 200 | Ok(retained( | ||
| 201 | &before, | ||
| 202 | replacement, | ||
| 203 | ¶graph(¤t, space, object)?, | ||
| 204 | error.state, | ||
| 205 | )?) | ||
| 206 | }) | ||
| 207 | .map_err(|failure| CommitError { | ||
| 208 | state: error.state, | ||
| 209 | error: io::Error::other(failure.to_string()), | ||
| 210 | })?; | ||
| 211 | if published { | ||
| 212 | let confirmation = client.borrow().as_ref().unwrap().commit_text( | ||
| 213 | path, | ||
| 214 | ¤t, | ||
| 215 | space, | ||
| 216 | object, | ||
| 217 | 0..0, | ||
| 218 | "", | ||
| 219 | ); | ||
| 220 | if let Err(failure) = confirmation | ||
| 221 | && failure.state != CommitState::Committed | ||
| 222 | { | ||
| 223 | println!( | ||
| 224 | "{}", | ||
| 225 | json!({"event":"transport_confirmation_error","state":format!("{:?}", failure.state),"error":failure.error.to_string()}) | ||
| 226 | ); | ||
| 227 | *client.borrow_mut() = None; | ||
| 228 | thread::sleep(Duration::from_millis(100)); | ||
| 229 | continue; | ||
| 230 | } | ||
| 231 | println!( | ||
| 232 | "{}", | ||
| 233 | json!({"event":"transport_reconciled","token":replacement,"published":true,"flush_confirmed":true}) | ||
| 234 | ); | ||
| 235 | return Ok(()); | ||
| 236 | } | ||
| 237 | println!( | ||
| 238 | "{}", | ||
| 239 | json!({"event":"transport_reconciled","token":replacement,"published":false}) | ||
| 240 | ); | ||
| 241 | return Err(CommitError { | ||
| 242 | state: CommitState::NotCommitted, | ||
| 243 | error: io::ErrorKind::ResourceBusy.into(), | ||
| 244 | }); | ||
| 245 | } | ||
| 246 | }, | ||
| 247 | ) | ||
| 248 | } | ||
| 249 | |||
| 250 | #[test] | ||
| 251 | fn uncertain_append_requires_one_retained_token_and_its_predecessor() { | ||
| 252 | for state in [CommitState::Unknown, CommitState::Committed] { | ||
| 253 | assert!(retained("before", " [w0:0]", "before [w0:0] [w1:0]", state).unwrap()); | ||
| 254 | } | ||
| 255 | for state in [CommitState::Unknown, CommitState::NotCommitted] { | ||
| 256 | assert!(!retained("before", " [w0:0]", "before [w1:0]", state).unwrap()); | ||
| 257 | } | ||
| 258 | for (current, state) in [ | ||
| 259 | ("before [w0:0] [w0:0]", CommitState::Unknown), | ||
| 260 | ("changed [w0:0]", CommitState::Unknown), | ||
| 261 | ("befor", CommitState::Unknown), | ||
| 262 | ("before", CommitState::Committed), | ||
| 263 | ("before [w0:0]", CommitState::NotCommitted), | ||
| 264 | ] { | ||
| 265 | assert!(retained("before", " [w0:0]", current, state).is_err()); | ||
| 266 | } | ||
| 267 | } | ||
crates/onestore-smb/src/lib.rs created+466
| ... | @@ -0,0 +1,466 @@ | ||
| 1 | #![forbid(unsafe_code)] | ||
| 2 | #![doc = include_str!("../README.md")] | ||
| 3 | |||
| 4 | use onestore::{CommitError, CommitIo, CommitState, ExGuid}; | ||
| 5 | use smb2::{ | ||
| 6 | Session, Tree, | ||
| 7 | client::connection::{Connection, NegotiatedParams}, | ||
| 8 | msg::{ | ||
| 9 | close::{CloseRequest, CloseResponse}, | ||
| 10 | create::{ | ||
| 11 | CreateDisposition, CreateRequest, CreateResponse, ImpersonationLevel, ShareAccess, | ||
| 12 | }, | ||
| 13 | flush::{FlushRequest, FlushResponse}, | ||
| 14 | lock::{LockElement, LockRequest, LockResponse}, | ||
| 15 | query_info::{InfoType, QueryInfoRequest, QueryInfoResponse}, | ||
| 16 | read::{ReadRequest, ReadResponse}, | ||
| 17 | write::{WriteRequest, WriteResponse}, | ||
| 18 | }, | ||
| 19 | pack::{Pack, ReadCursor, Unpack}, | ||
| 20 | types::{ | ||
| 21 | Command, CreditCharge, Dialect, FileId, OplockLevel, | ||
| 22 | flags::{Capabilities, FileAccessMask}, | ||
| 23 | }, | ||
| 24 | }; | ||
| 25 | use std::{io, ops::Range, sync::Mutex, time::Duration}; | ||
| 26 | use tokio::runtime::{Handle, Runtime}; | ||
| 27 | |||
| 28 | #[derive(Default)] | ||
| 29 | pub struct Credentials<'a> { | ||
| 30 | pub username: &'a str, | ||
| 31 | pub password: &'a str, | ||
| 32 | pub domain: &'a str, | ||
| 33 | } | ||
| 34 | |||
| 35 | /// Blocking connection with no automatic request replay or cached file contents. | ||
| 36 | /// Call from a background thread outside a Tokio runtime. | ||
| 37 | /// Paths are relative to the share; both `/` and `\` are separators. | ||
| 38 | pub struct Client { | ||
| 39 | connection: Mutex<Option<Connection>>, | ||
| 40 | tree: Tree, | ||
| 41 | timeout: Duration, | ||
| 42 | runtime: Mutex<Option<Runtime>>, | ||
| 43 | } | ||
| 44 | |||
| 45 | impl Client { | ||
| 46 | pub fn connect( | ||
| 47 | address: &str, | ||
| 48 | share: &str, | ||
| 49 | credentials: Credentials<'_>, | ||
| 50 | timeout: Duration, | ||
| 51 | ) -> io::Result<Self> { | ||
| 52 | if Handle::try_current().is_ok() || timeout.is_zero() { | ||
| 53 | return Err(io::ErrorKind::InvalidInput.into()); | ||
| 54 | } | ||
| 55 | let runtime = tokio::runtime::Builder::new_multi_thread() | ||
| 56 | .worker_threads(1) | ||
| 57 | .enable_all() | ||
| 58 | .build()?; | ||
| 59 | let (connection, tree) = runtime | ||
| 60 | .block_on(async { | ||
| 61 | tokio::time::timeout(timeout, async { | ||
| 62 | let mut connection = Connection::connect(address, timeout).await?; | ||
| 63 | connection.set_compression_requested(false); | ||
| 64 | connection.negotiate().await?; | ||
| 65 | Session::setup( | ||
| 66 | &mut connection, | ||
| 67 | credentials.username, | ||
| 68 | credentials.password, | ||
| 69 | credentials.domain, | ||
| 70 | ) | ||
| 71 | .await?; | ||
| 72 | let tree = Tree::connect(&mut connection, share).await?; | ||
| 73 | Ok::<_, smb2::Error>((connection, tree)) | ||
| 74 | }) | ||
| 75 | .await | ||
| 76 | }) | ||
| 77 | .map_err(|_| io::Error::from(io::ErrorKind::TimedOut))? | ||
| 78 | .map_err(io::Error::other)?; | ||
| 79 | Ok(Self { | ||
| 80 | connection: Mutex::new(Some(connection)), | ||
| 81 | tree, | ||
| 82 | timeout, | ||
| 83 | runtime: Mutex::new(Some(runtime)), | ||
| 84 | }) | ||
| 85 | } | ||
| 86 | |||
| 87 | fn retire(&self) { | ||
| 88 | if let Some(connection) = self | ||
| 89 | .connection | ||
| 90 | .lock() | ||
| 91 | .unwrap_or_else(|error| error.into_inner()) | ||
| 92 | .take() | ||
| 93 | { | ||
| 94 | connection.mark_dead(); | ||
| 95 | } | ||
| 96 | if let Some(runtime) = self | ||
| 97 | .runtime | ||
| 98 | .lock() | ||
| 99 | .unwrap_or_else(|error| error.into_inner()) | ||
| 100 | .take() | ||
| 101 | { | ||
| 102 | runtime.shutdown_background(); | ||
| 103 | } | ||
| 104 | } | ||
| 105 | |||
| 106 | fn request<T: Unpack>(&self, command: Command, body: impl Pack) -> io::Result<T> { | ||
| 107 | self.request_with(command, |_| (body, CreditCharge(1))) | ||
| 108 | } | ||
| 109 | |||
| 110 | fn request_with<T: Unpack, B: Pack>( | ||
| 111 | &self, | ||
| 112 | command: Command, | ||
| 113 | prepare: impl FnOnce(&Connection) -> (B, CreditCharge), | ||
| 114 | ) -> io::Result<T> { | ||
| 115 | if Handle::try_current().is_ok() { | ||
| 116 | return Err(io::ErrorKind::InvalidInput.into()); | ||
| 117 | } | ||
| 118 | let connection = self | ||
| 119 | .connection | ||
| 120 | .lock() | ||
| 121 | .map_err(|_| io::ErrorKind::Other)? | ||
| 122 | .clone() | ||
| 123 | .ok_or(io::ErrorKind::NotConnected)?; | ||
| 124 | let frame = { | ||
| 125 | let runtime = self.runtime.lock().map_err(|_| io::ErrorKind::Other)?; | ||
| 126 | let (body, charge) = prepare(&connection); | ||
| 127 | runtime | ||
| 128 | .as_ref() | ||
| 129 | .ok_or(io::ErrorKind::NotConnected)? | ||
| 130 | .block_on(async { | ||
| 131 | tokio::time::timeout( | ||
| 132 | self.timeout, | ||
| 133 | connection.execute_with_credits( | ||
| 134 | command, | ||
| 135 | &body, | ||
| 136 | Some(self.tree.tree_id), | ||
| 137 | charge, | ||
| 138 | ), | ||
| 139 | ) | ||
| 140 | .await | ||
| 141 | }) | ||
| 142 | }; | ||
| 143 | let frame = frame | ||
| 144 | .map_err(|_| io::Error::from(io::ErrorKind::TimedOut)) | ||
| 145 | .and_then(|result| result.map_err(io::Error::other)) | ||
| 146 | .inspect_err(|_| self.retire())?; | ||
| 147 | if frame.header.command != command { | ||
| 148 | self.retire(); | ||
| 149 | return Err(io::ErrorKind::InvalidData.into()); | ||
| 150 | } | ||
| 151 | if frame.header.status.0 != 0 { | ||
| 152 | let kind = match frame.header.status.0 { | ||
| 153 | 0xc0000043 | 0xc0000054 | 0xc0000055 => io::ErrorKind::WouldBlock, | ||
| 154 | 0xc0000011 => io::ErrorKind::UnexpectedEof, | ||
| 155 | 0xc0000034 | 0xc000003a => io::ErrorKind::NotFound, | ||
| 156 | _ => io::ErrorKind::Other, | ||
| 157 | }; | ||
| 158 | return Err(io::Error::new( | ||
| 159 | kind, | ||
| 160 | smb2::Error::Protocol { | ||
| 161 | status: frame.header.status, | ||
| 162 | command, | ||
| 163 | }, | ||
| 164 | )); | ||
| 165 | } | ||
| 166 | T::unpack(&mut ReadCursor::new(&frame.body)).map_err(|error| { | ||
| 167 | self.retire(); | ||
| 168 | io::Error::new(io::ErrorKind::InvalidData, error) | ||
| 169 | }) | ||
| 170 | } | ||
| 171 | |||
| 172 | fn open(&self, path: &str, write: bool) -> io::Result<File<'_>> { | ||
| 173 | if path.is_empty() || path.contains('\0') || path.encode_utf16().count() > 32767 { | ||
| 174 | return Err(io::ErrorKind::InvalidInput.into()); | ||
| 175 | } | ||
| 176 | let response: CreateResponse = self.request( | ||
| 177 | Command::Create, | ||
| 178 | CreateRequest { | ||
| 179 | requested_oplock_level: OplockLevel::None, | ||
| 180 | impersonation_level: ImpersonationLevel::Impersonation, | ||
| 181 | desired_access: FileAccessMask::new(if write { 0xc0000000 } else { 0x80000000 }), | ||
| 182 | file_attributes: 0, | ||
| 183 | share_access: ShareAccess(if write { 5 } else { 7 }), | ||
| 184 | create_disposition: CreateDisposition::FileOpen, | ||
| 185 | create_options: 0x42, | ||
| 186 | name: smb2::encode_path(&path.replace('\\', "/")), | ||
| 187 | create_contexts: Vec::new(), | ||
| 188 | }, | ||
| 189 | )?; | ||
| 190 | Ok(File { | ||
| 191 | client: self, | ||
| 192 | id: Some(response.file_id), | ||
| 193 | }) | ||
| 194 | } | ||
| 195 | |||
| 196 | /// Reads one bounded, consistent snapshot; contention returns WouldBlock. | ||
| 197 | pub fn read(&self, path: &str, limit: usize) -> io::Result<Vec<u8>> { | ||
| 198 | let mut file = self.open(path, false)?.coordinate(path, false)?; | ||
| 199 | let result = onestore::read_snapshot(|offset, output| file.read_at(offset, output), limit) | ||
| 200 | .and_then(|snapshot| snapshot.ok_or_else(|| io::ErrorKind::WouldBlock.into())); | ||
| 201 | let closed = file.close(); | ||
| 202 | let snapshot = result?; | ||
| 203 | closed?; | ||
| 204 | Ok(snapshot) | ||
| 205 | } | ||
| 206 | |||
| 207 | pub fn commit_text( | ||
| 208 | &self, | ||
| 209 | path: &str, | ||
| 210 | source: &[u8], | ||
| 211 | space: ExGuid, | ||
| 212 | object: ExGuid, | ||
| 213 | range: Range<u32>, | ||
| 214 | replacement: &str, | ||
| 215 | ) -> Result<(), CommitError> { | ||
| 216 | self.commit(path, |file| { | ||
| 217 | onestore::commit_text(file, source, space, object, range, replacement) | ||
| 218 | }) | ||
| 219 | } | ||
| 220 | |||
| 221 | pub fn commit_property_bytes( | ||
| 222 | &self, | ||
| 223 | path: &str, | ||
| 224 | source: &[u8], | ||
| 225 | space: ExGuid, | ||
| 226 | object: ExGuid, | ||
| 227 | property: u32, | ||
| 228 | value: &[u8], | ||
| 229 | ) -> Result<(), CommitError> { | ||
| 230 | self.commit(path, |file| { | ||
| 231 | onestore::commit_property_bytes(file, source, space, object, property, value) | ||
| 232 | }) | ||
| 233 | } | ||
| 234 | |||
| 235 | /// Publishes a prepared edit using the same native writer coordination as text commits. | ||
| 236 | pub fn commit_prepared( | ||
| 237 | &self, | ||
| 238 | path: &str, | ||
| 239 | edit: &onestore::PreparedEdit<'_>, | ||
| 240 | ) -> Result<(), CommitError> { | ||
| 241 | self.commit(path, |file| edit.commit(file)) | ||
| 242 | } | ||
| 243 | |||
| 244 | /// Confirms an observed snapshot's durability under native writer coordination. | ||
| 245 | pub fn confirm_snapshot(&self, path: &str, source: &[u8]) -> Result<(), CommitError> { | ||
| 246 | self.commit(path, |file| onestore::confirm_snapshot(file, source)) | ||
| 247 | } | ||
| 248 | |||
| 249 | fn commit( | ||
| 250 | &self, | ||
| 251 | path: &str, | ||
| 252 | operation: impl FnOnce(&mut File<'_>) -> Result<(), CommitError>, | ||
| 253 | ) -> Result<(), CommitError> { | ||
| 254 | let mut file = self | ||
| 255 | .open(path, true) | ||
| 256 | .and_then(|file| file.coordinate(path, true)) | ||
| 257 | .map_err(|error| CommitError { | ||
| 258 | state: CommitState::NotCommitted, | ||
| 259 | error, | ||
| 260 | })?; | ||
| 261 | let result = operation(&mut file); | ||
| 262 | let closed = file.close(); | ||
| 263 | result?; | ||
| 264 | closed.map_err(|error| CommitError { | ||
| 265 | state: CommitState::Committed, | ||
| 266 | error, | ||
| 267 | }) | ||
| 268 | } | ||
| 269 | } | ||
| 270 | |||
| 271 | impl Drop for Client { | ||
| 272 | fn drop(&mut self) { | ||
| 273 | self.retire(); | ||
| 274 | } | ||
| 275 | } | ||
| 276 | |||
| 277 | struct File<'a> { | ||
| 278 | client: &'a Client, | ||
| 279 | id: Option<FileId>, | ||
| 280 | } | ||
| 281 | impl File<'_> { | ||
| 282 | fn coordinate(self, path: &str, write: bool) -> io::Result<Self> { | ||
| 283 | self.lock(0xfffffffb, 0x11)?; | ||
| 284 | if write { | ||
| 285 | self.lock(0xfffffffd, 0x12)?; | ||
| 286 | } | ||
| 287 | let current = self.client.open(path, false)?; | ||
| 288 | let same = self.identity()? == current.identity()?; | ||
| 289 | current.close()?; | ||
| 290 | if !same { | ||
| 291 | return Err(io::ErrorKind::ResourceBusy.into()); | ||
| 292 | } | ||
| 293 | Ok(self) | ||
| 294 | } | ||
| 295 | |||
| 296 | fn lock(&self, offset: u64, flags: u32) -> io::Result<()> { | ||
| 297 | let _: LockResponse = self.client.request( | ||
| 298 | Command::Lock, | ||
| 299 | LockRequest { | ||
| 300 | file_id: self.id.unwrap(), | ||
| 301 | lock_sequence: 0, | ||
| 302 | locks: vec![LockElement { | ||
| 303 | offset, | ||
| 304 | length: 1, | ||
| 305 | flags, | ||
| 306 | }], | ||
| 307 | }, | ||
| 308 | )?; | ||
| 309 | Ok(()) | ||
| 310 | } | ||
| 311 | |||
| 312 | fn identity(&self) -> io::Result<(u64, u32)> { | ||
| 313 | let index: QueryInfoResponse = self.client.request( | ||
| 314 | Command::QueryInfo, | ||
| 315 | QueryInfoRequest { | ||
| 316 | info_type: InfoType::File, | ||
| 317 | file_info_class: 6, | ||
| 318 | output_buffer_length: 8, | ||
| 319 | additional_information: 0, | ||
| 320 | flags: 0, | ||
| 321 | file_id: self.id.unwrap(), | ||
| 322 | input_buffer: Vec::new(), | ||
| 323 | }, | ||
| 324 | )?; | ||
| 325 | let index = u64::from_le_bytes( | ||
| 326 | index | ||
| 327 | .output_buffer | ||
| 328 | .try_into() | ||
| 329 | .map_err(|_| io::ErrorKind::InvalidData)?, | ||
| 330 | ); | ||
| 331 | if index == 0 { | ||
| 332 | return Err(io::ErrorKind::Unsupported.into()); | ||
| 333 | } | ||
| 334 | let volume: QueryInfoResponse = self.client.request( | ||
| 335 | Command::QueryInfo, | ||
| 336 | QueryInfoRequest { | ||
| 337 | info_type: InfoType::Filesystem, | ||
| 338 | file_info_class: 1, | ||
| 339 | output_buffer_length: 1024, | ||
| 340 | additional_information: 0, | ||
| 341 | flags: 0, | ||
| 342 | file_id: self.id.unwrap(), | ||
| 343 | input_buffer: Vec::new(), | ||
| 344 | }, | ||
| 345 | )?; | ||
| 346 | let serial = volume | ||
| 347 | .output_buffer | ||
| 348 | .get(8..12) | ||
| 349 | .ok_or(io::ErrorKind::InvalidData)?; | ||
| 350 | Ok((index, u32::from_le_bytes(serial.try_into().unwrap()))) | ||
| 351 | } | ||
| 352 | |||
| 353 | fn close(mut self) -> io::Result<()> { | ||
| 354 | self.release() | ||
| 355 | } | ||
| 356 | |||
| 357 | fn release(&mut self) -> io::Result<()> { | ||
| 358 | if let Some(file_id) = self.id.take() { | ||
| 359 | let result: io::Result<CloseResponse> = self | ||
| 360 | .client | ||
| 361 | .request(Command::Close, CloseRequest { file_id, flags: 0 }); | ||
| 362 | if result.is_err() { | ||
| 363 | self.client.retire(); | ||
| 364 | } | ||
| 365 | result?; | ||
| 366 | } | ||
| 367 | Ok(()) | ||
| 368 | } | ||
| 369 | } | ||
| 370 | impl Drop for File<'_> { | ||
| 371 | fn drop(&mut self) { | ||
| 372 | let _ = self.release(); | ||
| 373 | } | ||
| 374 | } | ||
| 375 | fn read_size(params: &NegotiatedParams, credits: u16, requested: usize) -> usize { | ||
| 376 | let budget = if params.dialect != Dialect::Smb2_0_2 | ||
| 377 | && params.capabilities.contains(Capabilities::LARGE_MTU) | ||
| 378 | { | ||
| 379 | usize::from(credits.max(1)) * 65536 | ||
| 380 | } else { | ||
| 381 | 65536 | ||
| 382 | }; | ||
| 383 | requested | ||
| 384 | .min(params.max_read_size as usize) | ||
| 385 | .min(budget) | ||
| 386 | .min(1024 * 1024) | ||
| 387 | } | ||
| 388 | |||
| 389 | impl CommitIo for File<'_> { | ||
| 390 | fn read_at(&mut self, offset: u64, output: &mut [u8]) -> io::Result<usize> { | ||
| 391 | if output.is_empty() { | ||
| 392 | return Ok(0); | ||
| 393 | } | ||
| 394 | let mut size = 0; | ||
| 395 | let response: io::Result<ReadResponse> = | ||
| 396 | self.client.request_with(Command::Read, |connection| { | ||
| 397 | size = read_size( | ||
| 398 | &connection | ||
| 399 | .params() | ||
| 400 | .expect("connected SMB session is negotiated"), | ||
| 401 | connection.credits(), | ||
| 402 | output.len(), | ||
| 403 | ); | ||
| 404 | ( | ||
| 405 | ReadRequest { | ||
| 406 | file_id: self.id.unwrap(), | ||
| 407 | offset, | ||
| 408 | length: size as u32, | ||
| 409 | minimum_count: 1, | ||
| 410 | flags: 0, | ||
| 411 | padding: 0, | ||
| 412 | channel: 0, | ||
| 413 | remaining_bytes: 0, | ||
| 414 | read_channel_info: Vec::new(), | ||
| 415 | }, | ||
| 416 | CreditCharge(size.div_ceil(65536) as u16), | ||
| 417 | ) | ||
| 418 | }); | ||
| 419 | let response = match response { | ||
| 420 | Err(error) if error.kind() == io::ErrorKind::UnexpectedEof => return Ok(0), | ||
| 421 | result => result?, | ||
| 422 | }; | ||
| 423 | if response.data.len() > size { | ||
| 424 | self.client.retire(); | ||
| 425 | return Err(io::ErrorKind::InvalidData.into()); | ||
| 426 | } | ||
| 427 | output[..response.data.len()].copy_from_slice(&response.data); | ||
| 428 | Ok(response.data.len()) | ||
| 429 | } | ||
| 430 | fn write_at(&mut self, offset: u64, bytes: &[u8]) -> io::Result<usize> { | ||
| 431 | if bytes.is_empty() { | ||
| 432 | return Ok(0); | ||
| 433 | } | ||
| 434 | let size = bytes.len().min(65536); | ||
| 435 | let response: WriteResponse = self.client.request( | ||
| 436 | Command::Write, | ||
| 437 | WriteRequest { | ||
| 438 | file_id: self.id.unwrap(), | ||
| 439 | offset, | ||
| 440 | data: bytes[..size].to_vec(), | ||
| 441 | data_offset: 112, | ||
| 442 | flags: 1, | ||
| 443 | channel: 0, | ||
| 444 | remaining_bytes: 0, | ||
| 445 | write_channel_info_offset: 0, | ||
| 446 | write_channel_info_length: 0, | ||
| 447 | }, | ||
| 448 | )?; | ||
| 449 | if response.count as usize > size { | ||
| 450 | return Err(io::ErrorKind::InvalidData.into()); | ||
| 451 | } | ||
| 452 | Ok(response.count as usize) | ||
| 453 | } | ||
| 454 | fn flush(&mut self) -> io::Result<()> { | ||
| 455 | let _: FlushResponse = self.client.request( | ||
| 456 | Command::Flush, | ||
| 457 | FlushRequest { | ||
| 458 | file_id: self.id.unwrap(), | ||
| 459 | }, | ||
| 460 | )?; | ||
| 461 | Ok(()) | ||
| 462 | } | ||
| 463 | } | ||
| 464 | |||
| 465 | #[cfg(test)] | ||
| 466 | mod tests; | ||
crates/onestore-smb/src/tests.rs created+340
| ... | @@ -0,0 +1,340 @@ | ||
| 1 | use super::*; | ||
| 2 | use onestore::{ | ||
| 3 | RevisionIndex, Store, | ||
| 4 | document::{Document, Kind}, | ||
| 5 | }; | ||
| 6 | use std::{ | ||
| 7 | fs, | ||
| 8 | time::{Instant, SystemTime, UNIX_EPOCH}, | ||
| 9 | }; | ||
| 10 | |||
| 11 | mod faults; | ||
| 12 | |||
| 13 | fn client() -> Client { | ||
| 14 | Client::connect( | ||
| 15 | &std::env::var("ONESTORE_SMB_LAB").unwrap(), | ||
| 16 | "agent", | ||
| 17 | Credentials::default(), | ||
| 18 | Duration::from_secs(5), | ||
| 19 | ) | ||
| 20 | .unwrap() | ||
| 21 | } | ||
| 22 | fn create(client: &Client, path: &str, bytes: &[u8]) { | ||
| 23 | let response: CreateResponse = client | ||
| 24 | .request( | ||
| 25 | Command::Create, | ||
| 26 | CreateRequest { | ||
| 27 | requested_oplock_level: OplockLevel::None, | ||
| 28 | impersonation_level: ImpersonationLevel::Impersonation, | ||
| 29 | desired_access: FileAccessMask::new(0xc0000000), | ||
| 30 | file_attributes: 0, | ||
| 31 | share_access: ShareAccess(7), | ||
| 32 | create_disposition: CreateDisposition::FileCreate, | ||
| 33 | create_options: 0x42, | ||
| 34 | name: path.to_owned(), | ||
| 35 | create_contexts: Vec::new(), | ||
| 36 | }, | ||
| 37 | ) | ||
| 38 | .unwrap(); | ||
| 39 | let mut file = File { | ||
| 40 | client, | ||
| 41 | id: Some(response.file_id), | ||
| 42 | }; | ||
| 43 | let mut offset = 0; | ||
| 44 | while offset < bytes.len() { | ||
| 45 | offset += file.write_at(offset as u64, &bytes[offset..]).unwrap(); | ||
| 46 | } | ||
| 47 | file.flush().unwrap(); | ||
| 48 | file.close().unwrap(); | ||
| 49 | } | ||
| 50 | fn text(bytes: &[u8]) -> (ExGuid, ExGuid, String) { | ||
| 51 | let store = Store::parse(bytes).unwrap(); | ||
| 52 | assert!(store.checksum_mismatches.is_empty()); | ||
| 53 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 54 | index.validate_current().unwrap(); | ||
| 55 | let doc = Document::parse(&index).unwrap(); | ||
| 56 | doc.spaces | ||
| 57 | .iter() | ||
| 58 | .find_map(|(sid, space)| { | ||
| 59 | let revision = &space.revisions[&space.contexts[&ExGuid::default()]]; | ||
| 60 | revision | ||
| 61 | .nodes | ||
| 62 | .iter() | ||
| 63 | .find_map(|(oid, node)| match &node.kind { | ||
| 64 | Kind::RichText { text, .. } => Some((*sid, *oid, text.clone())), | ||
| 65 | _ => None, | ||
| 66 | }) | ||
| 67 | }) | ||
| 68 | .unwrap() | ||
| 69 | } | ||
| 70 | #[test] | ||
| 71 | #[ignore = "requires ONESTORE_SMB_LAB pointing to disposable Samba"] | ||
| 72 | fn live_coordination() { | ||
| 73 | let writer = client(); | ||
| 74 | let path = format!( | ||
| 75 | "adapter-{}.one", | ||
| 76 | SystemTime::now() | ||
| 77 | .duration_since(UNIX_EPOCH) | ||
| 78 | .unwrap() | ||
| 79 | .as_nanos() | ||
| 80 | ); | ||
| 81 | let source = onestore::create_section(&path, "Before café 🦀", "Fixture").unwrap(); | ||
| 82 | create(&writer, &path, &source); | ||
| 83 | assert_eq!(writer.read(&path, 1 << 20).unwrap(), source); | ||
| 84 | let readers: Vec<_> = (0..12).map(|_| client()).collect(); | ||
| 85 | let mut held: Vec<_> = readers | ||
| 86 | .iter() | ||
| 87 | .map(|client| { | ||
| 88 | client | ||
| 89 | .open(&path, false) | ||
| 90 | .unwrap() | ||
| 91 | .coordinate(&path, false) | ||
| 92 | .unwrap() | ||
| 93 | }) | ||
| 94 | .collect(); | ||
| 95 | let (sid, oid, before) = text(&source); | ||
| 96 | let replacement = "After café 🦀"; | ||
| 97 | writer | ||
| 98 | .commit_text( | ||
| 99 | &path, | ||
| 100 | &source, | ||
| 101 | sid, | ||
| 102 | oid, | ||
| 103 | 0..before.encode_utf16().count() as u32, | ||
| 104 | replacement, | ||
| 105 | ) | ||
| 106 | .unwrap(); | ||
| 107 | let after = writer.read(&path, 1 << 20).unwrap(); | ||
| 108 | assert_eq!(text(&after).2, replacement); | ||
| 109 | for file in &mut held { | ||
| 110 | let snapshot = onestore::read_snapshot(|offset, out| file.read_at(offset, out), 1 << 20) | ||
| 111 | .unwrap() | ||
| 112 | .unwrap(); | ||
| 113 | assert_eq!(snapshot, after); | ||
| 114 | } | ||
| 115 | let error = writer | ||
| 116 | .commit_text(&path, &source, sid, oid, 0..1, "X") | ||
| 117 | .unwrap_err(); | ||
| 118 | assert_eq!(error.state, CommitState::NotCommitted); | ||
| 119 | assert_eq!(error.error.kind(), io::ErrorKind::ResourceBusy); | ||
| 120 | assert_eq!(writer.read(&path, 1 << 20).unwrap(), after); | ||
| 121 | drop(held); | ||
| 122 | |||
| 123 | let stale = writer.open(&path, true).unwrap(); | ||
| 124 | let maintenance = client(); | ||
| 125 | let guard = maintenance.open(&path, false).unwrap(); | ||
| 126 | let _: LockResponse = maintenance | ||
| 127 | .request( | ||
| 128 | Command::Lock, | ||
| 129 | LockRequest { | ||
| 130 | file_id: guard.id.unwrap(), | ||
| 131 | lock_sequence: 0, | ||
| 132 | locks: vec![ | ||
| 133 | LockElement { | ||
| 134 | offset: 0xfffffffc, | ||
| 135 | length: 1, | ||
| 136 | flags: 0x12, | ||
| 137 | }, | ||
| 138 | LockElement { | ||
| 139 | offset: 0xffffeffc, | ||
| 140 | length: 4096, | ||
| 141 | flags: 0x12, | ||
| 142 | }, | ||
| 143 | ], | ||
| 144 | }, | ||
| 145 | ) | ||
| 146 | .unwrap(); | ||
| 147 | let replacement_path = format!("{path}.replacement"); | ||
| 148 | create(&maintenance, &replacement_path, &source); | ||
| 149 | let mut connection = maintenance | ||
| 150 | .connection | ||
| 151 | .lock() | ||
| 152 | .unwrap() | ||
| 153 | .as_ref() | ||
| 154 | .unwrap() | ||
| 155 | .clone(); | ||
| 156 | maintenance | ||
| 157 | .runtime | ||
| 158 | .lock() | ||
| 159 | .unwrap() | ||
| 160 | .as_ref() | ||
| 161 | .unwrap() | ||
| 162 | .block_on( | ||
| 163 | maintenance | ||
| 164 | .tree | ||
| 165 | .rename(&mut connection, &path, &format!("{path}.old")), | ||
| 166 | ) | ||
| 167 | .unwrap(); | ||
| 168 | maintenance | ||
| 169 | .runtime | ||
| 170 | .lock() | ||
| 171 | .unwrap() | ||
| 172 | .as_ref() | ||
| 173 | .unwrap() | ||
| 174 | .block_on( | ||
| 175 | maintenance | ||
| 176 | .tree | ||
| 177 | .rename(&mut connection, &replacement_path, &path), | ||
| 178 | ) | ||
| 179 | .unwrap(); | ||
| 180 | drop(connection); | ||
| 181 | guard.close().unwrap(); | ||
| 182 | let error = stale.coordinate(&path, true).err().unwrap(); | ||
| 183 | assert_eq!(error.kind(), io::ErrorKind::ResourceBusy); | ||
| 184 | assert_eq!(writer.read(&path, 1 << 20).unwrap(), source); | ||
| 185 | |||
| 186 | let retiring = client(); | ||
| 187 | let file = retiring | ||
| 188 | .open(&path, true) | ||
| 189 | .unwrap() | ||
| 190 | .coordinate(&path, true) | ||
| 191 | .unwrap(); | ||
| 192 | retiring.retire(); | ||
| 193 | assert_eq!( | ||
| 194 | retiring.read(&path, 1 << 20).unwrap_err().kind(), | ||
| 195 | io::ErrorKind::NotConnected | ||
| 196 | ); | ||
| 197 | drop(file); | ||
| 198 | let deadline = Instant::now() + Duration::from_secs(5); | ||
| 199 | loop { | ||
| 200 | match writer | ||
| 201 | .open(&path, true) | ||
| 202 | .and_then(|file| file.coordinate(&path, true)) | ||
| 203 | { | ||
| 204 | Ok(file) => { | ||
| 205 | file.close().unwrap(); | ||
| 206 | break; | ||
| 207 | } | ||
| 208 | Err(error) | ||
| 209 | if error.kind() == io::ErrorKind::WouldBlock && Instant::now() < deadline => | ||
| 210 | { | ||
| 211 | std::thread::sleep(Duration::from_millis(10)) | ||
| 212 | } | ||
| 213 | Err(error) => panic!("retired connection retained locks: {error}"), | ||
| 214 | } | ||
| 215 | } | ||
| 216 | |||
| 217 | let mut unfinished = writer | ||
| 218 | .open(&path, true) | ||
| 219 | .unwrap() | ||
| 220 | .coordinate(&path, true) | ||
| 221 | .unwrap(); | ||
| 222 | assert_eq!( | ||
| 223 | unfinished | ||
| 224 | .write_at(source.len() as u64, b"unpublished") | ||
| 225 | .unwrap(), | ||
| 226 | 11 | ||
| 227 | ); | ||
| 228 | unfinished.flush().unwrap(); | ||
| 229 | unfinished.close().unwrap(); | ||
| 230 | let snapshot = writer.read(&path, 1 << 20).unwrap(); | ||
| 231 | assert_eq!(snapshot.len(), source.len() + 11); | ||
| 232 | assert_eq!(text(&snapshot).2, before); | ||
| 233 | writer | ||
| 234 | .commit_text( | ||
| 235 | &path, | ||
| 236 | &snapshot, | ||
| 237 | sid, | ||
| 238 | oid, | ||
| 239 | 0..before.encode_utf16().count() as u32, | ||
| 240 | "Recovered café 🦀", | ||
| 241 | ) | ||
| 242 | .unwrap(); | ||
| 243 | let recovered = writer.read(&path, 1 << 20).unwrap(); | ||
| 244 | assert_eq!(text(&recovered).2, "Recovered café 🦀"); | ||
| 245 | let spare = client(); | ||
| 246 | tokio::runtime::Builder::new_current_thread() | ||
| 247 | .build() | ||
| 248 | .unwrap() | ||
| 249 | .block_on(async { | ||
| 250 | drop(spare); | ||
| 251 | }); | ||
| 252 | let output = std::env::var("ONESTORE_SMB_EVIDENCE").unwrap(); | ||
| 253 | fs::write(std::path::Path::new(&output).join("source.one"), &source).unwrap(); | ||
| 254 | fs::write(std::path::Path::new(&output).join("committed.one"), &after).unwrap(); | ||
| 255 | fs::write( | ||
| 256 | std::path::Path::new(&output).join("recovered.one"), | ||
| 257 | &recovered, | ||
| 258 | ) | ||
| 259 | .unwrap(); | ||
| 260 | println!( | ||
| 261 | "{}", | ||
| 262 | serde_json::json!({"path":path,"readers":12,"committed_while_readers_held":true,"fresh_reads":12,"stale_snapshot_rejected":true,"replaced_handle_rejected":true,"retirement_releases_locks":true,"unpublished_tail_recovered":true,"drop_inside_runtime":true}) | ||
| 263 | ); | ||
| 264 | } | ||
| 265 | |||
| 266 | #[test] | ||
| 267 | #[ignore = "requires an owned Samba fixture and maintenance controller"] | ||
| 268 | fn live_reader_hold() { | ||
| 269 | let client = client(); | ||
| 270 | let path = std::env::var("ONESTORE_SMB_PATH").unwrap(); | ||
| 271 | let output = std::path::PathBuf::from(std::env::var("ONESTORE_SMB_HOLD").unwrap()); | ||
| 272 | assert!(!output.join("ready").exists() && !output.join("release").exists()); | ||
| 273 | let mut file = client | ||
| 274 | .open(&path, false) | ||
| 275 | .unwrap() | ||
| 276 | .coordinate(&path, false) | ||
| 277 | .unwrap(); | ||
| 278 | fs::write(output.join("ready"), b"held").unwrap(); | ||
| 279 | let deadline = Instant::now() + Duration::from_secs(300); | ||
| 280 | let mut accepted = 0; | ||
| 281 | let mut retries = 0; | ||
| 282 | while !output.join("release").exists() { | ||
| 283 | assert!( | ||
| 284 | Instant::now() < deadline, | ||
| 285 | "maintenance controller timed out" | ||
| 286 | ); | ||
| 287 | match onestore::read_snapshot(|offset, out| file.read_at(offset, out), 256 << 20).unwrap() { | ||
| 288 | Some(_) => accepted += 1, | ||
| 289 | None => retries += 1, | ||
| 290 | } | ||
| 291 | std::thread::sleep(Duration::from_millis(5)); | ||
| 292 | } | ||
| 293 | file.close().unwrap(); | ||
| 294 | assert!(accepted > 0); | ||
| 295 | fs::write( | ||
| 296 | output.join("released.json"), | ||
| 297 | serde_json::to_vec(&serde_json::json!({"accepted": accepted, "retries": retries})).unwrap(), | ||
| 298 | ) | ||
| 299 | .unwrap(); | ||
| 300 | } | ||
| 301 | |||
| 302 | #[test] | ||
| 303 | fn read_limits_respect_negotiation_and_available_credits() { | ||
| 304 | for dialect in Dialect::ALL { | ||
| 305 | for large_mtu in [false, true] { | ||
| 306 | for max_read_size in [65536, 65537, 131072, 1048576, u32::MAX] { | ||
| 307 | let params = NegotiatedParams { | ||
| 308 | dialect: *dialect, | ||
| 309 | max_read_size, | ||
| 310 | max_write_size: 65536, | ||
| 311 | max_transact_size: 65536, | ||
| 312 | server_guid: Default::default(), | ||
| 313 | signing_required: false, | ||
| 314 | capabilities: Capabilities(if large_mtu { | ||
| 315 | Capabilities::LARGE_MTU | ||
| 316 | } else { | ||
| 317 | 0 | ||
| 318 | }), | ||
| 319 | gmac_negotiated: false, | ||
| 320 | cipher: None, | ||
| 321 | compression_supported: false, | ||
| 322 | }; | ||
| 323 | for credits in [0, 1, 2, 3, 15, 16, 17, u16::MAX] { | ||
| 324 | for requested in [1, 1024, 65535, 65536, 65537, 131072, 1048576, usize::MAX] { | ||
| 325 | let size = read_size(¶ms, credits, requested); | ||
| 326 | assert!(size > 0 && size <= requested && size <= max_read_size as usize); | ||
| 327 | assert!(size <= 1048576); | ||
| 328 | assert!(size.div_ceil(65536) <= usize::from(credits.max(1))); | ||
| 329 | if *dialect == Dialect::Smb2_0_2 || !large_mtu { | ||
| 330 | assert!(size <= 65536); | ||
| 331 | } else if credits >= 16 && max_read_size >= 1048576 && requested >= 1048576 | ||
| 332 | { | ||
| 333 | assert_eq!(size, 1048576); | ||
| 334 | } | ||
| 335 | } | ||
| 336 | } | ||
| 337 | } | ||
| 338 | } | ||
| 339 | } | ||
| 340 | } | ||
crates/onestore-smb/src/tests/faults.rs created+444
| ... | @@ -0,0 +1,444 @@ | ||
| 1 | use super::*; | ||
| 2 | use serde_json::{Value, json}; | ||
| 3 | use std::{collections::BTreeMap, path::Path, process::Child}; | ||
| 4 | |||
| 5 | #[derive(Clone)] | ||
| 6 | struct Snapshot { | ||
| 7 | content: BTreeMap<ExGuid, String>, | ||
| 8 | modified: BTreeMap<(ExGuid, ExGuid), u32>, | ||
| 9 | } | ||
| 10 | |||
| 11 | fn snapshot(bytes: &[u8]) -> Snapshot { | ||
| 12 | let store = Store::parse(bytes).unwrap(); | ||
| 13 | assert!(store.checksum_mismatches.is_empty()); | ||
| 14 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 15 | index.validate_current().unwrap(); | ||
| 16 | Document::parse(&index).unwrap(); | ||
| 17 | let mut content = BTreeMap::new(); | ||
| 18 | let mut modified = BTreeMap::new(); | ||
| 19 | for (sid, space) in &index.spaces { | ||
| 20 | let revision = index | ||
| 21 | .resolve(*sid, space.labels[&(ExGuid::default(), 1)]) | ||
| 22 | .unwrap(); | ||
| 23 | let mut objects = BTreeMap::new(); | ||
| 24 | for (oid, object) in &revision.objects { | ||
| 25 | if let Some(onestore::FileDataReference::Internal(guid)) = | ||
| 26 | object.file_reference().unwrap() | ||
| 27 | { | ||
| 28 | store.file_data(guid).unwrap(); | ||
| 29 | } | ||
| 30 | let data = match object.data { | ||
| 31 | onestore::ObjectData::Properties(bytes) => { | ||
| 32 | let mut properties = onestore::PropertySets::parse(bytes).unwrap(); | ||
| 33 | for property in &mut properties.sets[0] { | ||
| 34 | if property.id == 0x14001d7a { | ||
| 35 | let onestore::Value::Bytes(value) = property.value else { | ||
| 36 | panic!() | ||
| 37 | }; | ||
| 38 | assert!( | ||
| 39 | modified | ||
| 40 | .insert( | ||
| 41 | (*sid, *oid), | ||
| 42 | u32::from_le_bytes(value.try_into().unwrap()) | ||
| 43 | ) | ||
| 44 | .is_none() | ||
| 45 | ); | ||
| 46 | property.value = onestore::Value::Bytes(&[0; 4]); | ||
| 47 | } | ||
| 48 | } | ||
| 49 | format!("{properties:?}") | ||
| 50 | } | ||
| 51 | other => format!("{other:?}"), | ||
| 52 | }; | ||
| 53 | objects.insert( | ||
| 54 | *oid, | ||
| 55 | ( | ||
| 56 | object.jcid, | ||
| 57 | object.reference_count, | ||
| 58 | data, | ||
| 59 | format!("{:?}", object.references().unwrap()), | ||
| 60 | ), | ||
| 61 | ); | ||
| 62 | } | ||
| 63 | content.insert(*sid, format!("{:?} {objects:?}", revision.roots)); | ||
| 64 | } | ||
| 65 | Snapshot { content, modified } | ||
| 66 | } | ||
| 67 | |||
| 68 | fn stamp() -> u32 { | ||
| 69 | (SystemTime::now() | ||
| 70 | .duration_since(UNIX_EPOCH) | ||
| 71 | .unwrap() | ||
| 72 | .as_secs() | ||
| 73 | - 315532800) | ||
| 74 | .try_into() | ||
| 75 | .unwrap() | ||
| 76 | } | ||
| 77 | |||
| 78 | fn published( | ||
| 79 | actual: &Snapshot, | ||
| 80 | old: &Snapshot, | ||
| 81 | new: &Snapshot, | ||
| 82 | time: std::ops::RangeInclusive<u32>, | ||
| 83 | ) -> bool { | ||
| 84 | if actual.content == old.content { | ||
| 85 | assert_eq!( | ||
| 86 | actual.modified, old.modified, | ||
| 87 | "Unpublished modification time changed" | ||
| 88 | ); | ||
| 89 | return false; | ||
| 90 | } | ||
| 91 | assert_eq!( | ||
| 92 | actual.content, new.content, | ||
| 93 | "Partial or unexpected publication" | ||
| 94 | ); | ||
| 95 | assert!(actual.modified.keys().eq(new.modified.keys())); | ||
| 96 | for (key, value) in &actual.modified { | ||
| 97 | if old.modified.get(key) != new.modified.get(key) { | ||
| 98 | assert!( | ||
| 99 | time.contains(value), | ||
| 100 | "Modification time outside the commit interval" | ||
| 101 | ); | ||
| 102 | } else { | ||
| 103 | assert_eq!( | ||
| 104 | *value, new.modified[key], | ||
| 105 | "Unrelated modification time changed" | ||
| 106 | ); | ||
| 107 | } | ||
| 108 | } | ||
| 109 | true | ||
| 110 | } | ||
| 111 | |||
| 112 | #[test] | ||
| 113 | fn publication_oracle_bounds_changed_timestamps_and_preserves_others() { | ||
| 114 | let id = ExGuid::default(); | ||
| 115 | let other = ExGuid { n: 1, ..id }; | ||
| 116 | let old = Snapshot { | ||
| 117 | content: BTreeMap::from([(id, "old".into())]), | ||
| 118 | modified: BTreeMap::from([((id, id), 10), ((id, other), 7)]), | ||
| 119 | }; | ||
| 120 | let new = Snapshot { | ||
| 121 | content: BTreeMap::from([(id, "new".into())]), | ||
| 122 | modified: BTreeMap::from([((id, id), 20), ((id, other), 7)]), | ||
| 123 | }; | ||
| 124 | let mut actual = new.clone(); | ||
| 125 | actual.modified.insert((id, id), 30); | ||
| 126 | assert!(published(&actual, &old, &new, 25..=35)); | ||
| 127 | assert!(!published(&old, &old, &new, 25..=35)); | ||
| 128 | for (key, value) in [((id, id), 24), ((id, id), 36), ((id, other), 30)] { | ||
| 129 | let mut changed = actual.clone(); | ||
| 130 | changed.modified.insert(key, value); | ||
| 131 | assert!(std::panic::catch_unwind(|| published(&changed, &old, &new, 25..=35)).is_err()); | ||
| 132 | } | ||
| 133 | actual.content = old.content.clone(); | ||
| 134 | assert!(std::panic::catch_unwind(|| published(&actual, &old, &new, 25..=35)).is_err()); | ||
| 135 | } | ||
| 136 | |||
| 137 | struct Proxy(Child); | ||
| 138 | impl Drop for Proxy { | ||
| 139 | fn drop(&mut self) { | ||
| 140 | let _ = self.0.kill(); | ||
| 141 | let _ = self.0.wait(); | ||
| 142 | } | ||
| 143 | } | ||
| 144 | |||
| 145 | fn records(output: &Path) -> Vec<Value> { | ||
| 146 | let data = fs::read_to_string(output.join("proxy.jsonl")).unwrap(); | ||
| 147 | data.rsplit_once('\n') | ||
| 148 | .map_or("", |(complete, _)| complete) | ||
| 149 | .lines() | ||
| 150 | .map(|line| serde_json::from_str(line).unwrap()) | ||
| 151 | .collect() | ||
| 152 | } | ||
| 153 | |||
| 154 | fn configure(output: &Path, state: Value) -> usize { | ||
| 155 | fs::write(output.join("control.tmp"), state.to_string()).unwrap(); | ||
| 156 | fs::rename(output.join("control.tmp"), output.join("control.json")).unwrap(); | ||
| 157 | let deadline = Instant::now() + Duration::from_secs(5); | ||
| 158 | loop { | ||
| 159 | let events = records(output); | ||
| 160 | if let Some(index) = events | ||
| 161 | .iter() | ||
| 162 | .rposition(|event| event.get("control") == Some(&state)) | ||
| 163 | { | ||
| 164 | return index + 1; | ||
| 165 | } | ||
| 166 | assert!( | ||
| 167 | Instant::now() < deadline, | ||
| 168 | "proxy did not acknowledge its control state" | ||
| 169 | ); | ||
| 170 | std::thread::sleep(Duration::from_millis(5)); | ||
| 171 | } | ||
| 172 | } | ||
| 173 | |||
| 174 | #[test] | ||
| 175 | #[ignore = "requires an owned Samba share and a new ONESTORE_SMB_EVIDENCE directory"] | ||
| 176 | fn live_message_loss() { | ||
| 177 | let output = std::path::PathBuf::from(std::env::var("ONESTORE_SMB_EVIDENCE").unwrap()); | ||
| 178 | fs::create_dir(&output).unwrap(); | ||
| 179 | fs::create_dir(output.join("interrupted")).unwrap(); | ||
| 180 | fs::create_dir(output.join("recovered")).unwrap(); | ||
| 181 | fs::create_dir(output.join("source")).unwrap(); | ||
| 182 | let address = std::env::var("ONESTORE_SMB_LAB").unwrap(); | ||
| 183 | let (host, port) = address.rsplit_once(':').unwrap(); | ||
| 184 | let mut proxy = Proxy( | ||
| 185 | std::process::Command::new("python3") | ||
| 186 | .arg(Path::new(env!("CARGO_MANIFEST_DIR")).join("../../tools/smb-proxy.py")) | ||
| 187 | .arg(output.join("control.json")) | ||
| 188 | .args(["--port", "0", "--server", host, "--server-port", port]) | ||
| 189 | .stdout(fs::File::create(output.join("proxy.jsonl")).unwrap()) | ||
| 190 | .stderr(fs::File::create(output.join("proxy.stderr")).unwrap()) | ||
| 191 | .spawn() | ||
| 192 | .unwrap(), | ||
| 193 | ); | ||
| 194 | let deadline = Instant::now() + Duration::from_secs(5); | ||
| 195 | let port = loop { | ||
| 196 | if let Some(port) = records(&output) | ||
| 197 | .iter() | ||
| 198 | .find_map(|event| event["listening"].as_u64()) | ||
| 199 | { | ||
| 200 | break port; | ||
| 201 | } | ||
| 202 | assert!(proxy.0.try_wait().unwrap().is_none()); | ||
| 203 | assert!(Instant::now() < deadline, "proxy did not start"); | ||
| 204 | std::thread::sleep(Duration::from_millis(5)); | ||
| 205 | }; | ||
| 206 | let proxied = format!("127.0.0.1:{port}"); | ||
| 207 | let observer = client(); | ||
| 208 | let prefix = SystemTime::now() | ||
| 209 | .duration_since(UNIX_EPOCH) | ||
| 210 | .unwrap() | ||
| 211 | .as_nanos(); | ||
| 212 | let fixtures = [ | ||
| 213 | ( | ||
| 214 | "chunked", | ||
| 215 | onestore::create_section("fault.one", "Before café 🦀", "Fault test").unwrap(), | ||
| 216 | ), | ||
| 217 | ( | ||
| 218 | "carry", | ||
| 219 | fs::read( | ||
| 220 | Path::new(env!("CARGO_MANIFEST_DIR")) | ||
| 221 | .join("../../corpus/append/round-01/tx-255/notebook/synthetic.one"), | ||
| 222 | ) | ||
| 223 | .unwrap(), | ||
| 224 | ), | ||
| 225 | ]; | ||
| 226 | let mut results = Vec::new(); | ||
| 227 | for (fixture, source) in fixtures { | ||
| 228 | fs::write( | ||
| 229 | output.join("source").join(format!("{fixture}.one")), | ||
| 230 | &source, | ||
| 231 | ) | ||
| 232 | .unwrap(); | ||
| 233 | let (sid, oid, before) = text(&source); | ||
| 234 | let after = if fixture == "chunked" { | ||
| 235 | "After café 🦀 ".repeat(6000) | ||
| 236 | } else { | ||
| 237 | "After café 🦀".to_owned() | ||
| 238 | }; | ||
| 239 | let suffix = " [reconnected]"; | ||
| 240 | fs::write( | ||
| 241 | output.join(format!("{fixture}-intent.json")), | ||
| 242 | serde_json::to_vec(&json!({"before":before,"after":after,"suffix":suffix})).unwrap(), | ||
| 243 | ) | ||
| 244 | .unwrap(); | ||
| 245 | let old = snapshot(&source); | ||
| 246 | let deadline = Instant::now() + Duration::from_secs(2); | ||
| 247 | while old.modified.values().any(|value| *value >= stamp()) { | ||
| 248 | assert!( | ||
| 249 | Instant::now() < deadline, | ||
| 250 | "source timestamps did not precede the test" | ||
| 251 | ); | ||
| 252 | std::thread::sleep(Duration::from_millis(5)); | ||
| 253 | } | ||
| 254 | let expected = onestore::replace_text( | ||
| 255 | &source, | ||
| 256 | sid, | ||
| 257 | oid, | ||
| 258 | 0..before.encode_utf16().count() as u32, | ||
| 259 | &after, | ||
| 260 | ) | ||
| 261 | .unwrap(); | ||
| 262 | let new = snapshot(&expected); | ||
| 263 | let baseline_path = format!("fault-{prefix}-{fixture}-baseline.one"); | ||
| 264 | create(&observer, &baseline_path, &source); | ||
| 265 | configure(&output, json!({"phase":format!("{fixture}-setup")})); | ||
| 266 | let baseline = Client::connect( | ||
| 267 | &proxied, | ||
| 268 | "agent", | ||
| 269 | Credentials::default(), | ||
| 270 | Duration::from_secs(5), | ||
| 271 | ) | ||
| 272 | .unwrap(); | ||
| 273 | let start = configure(&output, json!({"phase":format!("{fixture}-baseline")})); | ||
| 274 | let began = stamp(); | ||
| 275 | baseline | ||
| 276 | .commit_text( | ||
| 277 | &baseline_path, | ||
| 278 | &source, | ||
| 279 | sid, | ||
| 280 | oid, | ||
| 281 | 0..before.encode_utf16().count() as u32, | ||
| 282 | &after, | ||
| 283 | ) | ||
| 284 | .unwrap(); | ||
| 285 | let events = records(&output); | ||
| 286 | let mut occurrences = BTreeMap::new(); | ||
| 287 | let mut cuts = Vec::new(); | ||
| 288 | for event in &events[start..] { | ||
| 289 | let Some(direction) = event["direction"].as_str() else { | ||
| 290 | continue; | ||
| 291 | }; | ||
| 292 | if event["status"] == "0x103" { | ||
| 293 | continue; | ||
| 294 | } | ||
| 295 | let command = event["command"].as_u64().unwrap(); | ||
| 296 | let status = event["status"].as_str().map(str::to_owned); | ||
| 297 | let count = occurrences | ||
| 298 | .entry((direction.to_owned(), command, status.clone())) | ||
| 299 | .or_insert(0); | ||
| 300 | *count += 1; | ||
| 301 | cuts.push((direction.to_owned(), command, status, *count)); | ||
| 302 | } | ||
| 303 | assert!( | ||
| 304 | cuts.iter() | ||
| 305 | .any(|(direction, command, _, count)| direction == "response" | ||
| 306 | && *command == 7 | ||
| 307 | && *count == 3) | ||
| 308 | ); | ||
| 309 | assert!(published( | ||
| 310 | &snapshot(&observer.read(&baseline_path, 1 << 20).unwrap()), | ||
| 311 | &old, | ||
| 312 | &new, | ||
| 313 | began..=stamp() | ||
| 314 | )); | ||
| 315 | drop(baseline); | ||
| 316 | for (case, (direction, command, status, occurrence)) in cuts.iter().enumerate() { | ||
| 317 | let name = format!("{fixture}-{case:03}"); | ||
| 318 | let path = format!("fault-{prefix}-{name}.one"); | ||
| 319 | create(&observer, &path, &source); | ||
| 320 | configure(&output, json!({"phase":format!("{name}-setup")})); | ||
| 321 | let interrupted = Client::connect( | ||
| 322 | &proxied, | ||
| 323 | "agent", | ||
| 324 | Credentials::default(), | ||
| 325 | Duration::from_secs(5), | ||
| 326 | ) | ||
| 327 | .unwrap(); | ||
| 328 | let start = configure( | ||
| 329 | &output, | ||
| 330 | json!({"phase":name, "direction":direction, "cut":command, "status":status, "occurrence":occurrence}), | ||
| 331 | ); | ||
| 332 | let began = stamp(); | ||
| 333 | let error = interrupted | ||
| 334 | .commit_text( | ||
| 335 | &path, | ||
| 336 | &source, | ||
| 337 | sid, | ||
| 338 | oid, | ||
| 339 | 0..before.encode_utf16().count() as u32, | ||
| 340 | &after, | ||
| 341 | ) | ||
| 342 | .unwrap_err(); | ||
| 343 | assert_eq!( | ||
| 344 | interrupted.read(&path, 1 << 20).unwrap_err().kind(), | ||
| 345 | io::ErrorKind::NotConnected | ||
| 346 | ); | ||
| 347 | let events = records(&output); | ||
| 348 | assert_eq!( | ||
| 349 | events[start..] | ||
| 350 | .iter() | ||
| 351 | .filter(|event| event.get("cut").is_some()) | ||
| 352 | .count(), | ||
| 353 | 1 | ||
| 354 | ); | ||
| 355 | assert!( | ||
| 356 | !events | ||
| 357 | .iter() | ||
| 358 | .any(|event| event.get("trace_error").is_some()) | ||
| 359 | ); | ||
| 360 | let fresh = client(); | ||
| 361 | let deadline = Instant::now() + Duration::from_secs(5); | ||
| 362 | loop { | ||
| 363 | match fresh | ||
| 364 | .open(&path, true) | ||
| 365 | .and_then(|file| file.coordinate(&path, true)) | ||
| 366 | { | ||
| 367 | Ok(file) => { | ||
| 368 | file.close().unwrap(); | ||
| 369 | break; | ||
| 370 | } | ||
| 371 | Err(error) | ||
| 372 | if error.kind() == io::ErrorKind::WouldBlock | ||
| 373 | && Instant::now() < deadline => | ||
| 374 | { | ||
| 375 | std::thread::sleep(Duration::from_millis(5)) | ||
| 376 | } | ||
| 377 | Err(error) => panic!("{name}: retired session retained exclusion: {error}"), | ||
| 378 | } | ||
| 379 | } | ||
| 380 | let saved = fresh.read(&path, 1 << 20).unwrap(); | ||
| 381 | fs::write( | ||
| 382 | output.join("interrupted").join(format!("{name}.one")), | ||
| 383 | &saved, | ||
| 384 | ) | ||
| 385 | .unwrap(); | ||
| 386 | let visible = published(&snapshot(&saved), &old, &new, began..=stamp()); | ||
| 387 | match error.state { | ||
| 388 | CommitState::NotCommitted => assert!(!visible, "{name}"), | ||
| 389 | CommitState::Committed => assert!(visible, "{name}"), | ||
| 390 | CommitState::Unknown => {} | ||
| 391 | } | ||
| 392 | if !visible { | ||
| 393 | fresh | ||
| 394 | .commit_text( | ||
| 395 | &path, | ||
| 396 | &saved, | ||
| 397 | sid, | ||
| 398 | oid, | ||
| 399 | 0..before.encode_utf16().count() as u32, | ||
| 400 | &after, | ||
| 401 | ) | ||
| 402 | .unwrap(); | ||
| 403 | } | ||
| 404 | let saved = fresh.read(&path, 1 << 20).unwrap(); | ||
| 405 | assert!( | ||
| 406 | published(&snapshot(&saved), &old, &new, began..=stamp()), | ||
| 407 | "{name}: replay did not publish" | ||
| 408 | ); | ||
| 409 | let end = after.encode_utf16().count() as u32; | ||
| 410 | fresh | ||
| 411 | .commit_text(&path, &saved, sid, oid, end..end, suffix) | ||
| 412 | .unwrap(); | ||
| 413 | let recovered = fresh.read(&path, 1 << 20).unwrap(); | ||
| 414 | assert_eq!(text(&recovered).2, format!("{after}{suffix}")); | ||
| 415 | fs::write( | ||
| 416 | output.join("recovered").join(format!("{name}.one")), | ||
| 417 | recovered, | ||
| 418 | ) | ||
| 419 | .unwrap(); | ||
| 420 | results.push(json!({"case":name,"path":path,"direction":direction,"command":command,"status":status,"occurrence":occurrence, | ||
| 421 | "state":format!("{:?}",error.state),"visible":if visible {"after"} else {"before"}, | ||
| 422 | "retired_session_rejected":true,"exclusion_released":true,"fresh_commit_succeeded":true})); | ||
| 423 | fs::write( | ||
| 424 | output.join("results.json"), | ||
| 425 | serde_json::to_vec_pretty(&results).unwrap(), | ||
| 426 | ) | ||
| 427 | .unwrap(); | ||
| 428 | } | ||
| 429 | } | ||
| 430 | for state in ["NotCommitted", "Unknown", "Committed"] { | ||
| 431 | assert!(results.iter().any(|result| result["state"] == state)); | ||
| 432 | } | ||
| 433 | assert!( | ||
| 434 | results | ||
| 435 | .iter() | ||
| 436 | .any(|result| result["state"] == "Unknown" && result["visible"] == "before") | ||
| 437 | ); | ||
| 438 | assert!( | ||
| 439 | results | ||
| 440 | .iter() | ||
| 441 | .any(|result| result["state"] == "Unknown" && result["visible"] == "after") | ||
| 442 | ); | ||
| 443 | println!("{} message-loss cases passed", results.len()); | ||
| 444 | } | ||
crates/onestore/README.md created+242
| ... | @@ -0,0 +1,242 @@ | ||
| 1 | # onestore | ||
| 2 | |||
| 3 | An experimental native Rust library for OneNote revision stores (`.one` and | ||
| 4 | `.onetoc2`). It reads committed object graphs, creates a small notebook without | ||
| 5 | a template, appends property, text, paragraph, outline and formatting edits with a | ||
| 6 | recoverable commit protocol, and interprets MS-ONE document structure, formatting, media, and historical pages. | ||
| 7 | The storage gates are recorded in [PROGRESS.md](../../evidence/PROGRESS.md); document-model | ||
| 8 | verification is recorded in [M6-ACCEPTANCE.md](../../evidence/M6-ACCEPTANCE.md). Concurrent-editing | ||
| 9 | verification is recorded in [MILESTONE7.md](../../evidence/MILESTONE7.md). Crash recovery and the | ||
| 10 | read/write HTML diagnostic editor are recorded in [MILESTONE8.md](../../evidence/MILESTONE8.md). | ||
| 11 | Embedded SMB coordination, durable offline editing and document-growth acceptance | ||
| 12 | are recorded in [MILESTONE9.md](../../evidence/MILESTONE9.md#document-writer-and-offline-acceptance). | ||
| 13 | |||
| 14 | Use disposable copies for notebook editing. Header version notification now | ||
| 15 | follows durable transaction publication, fixing a native cached-reader race. | ||
| 16 | The [lost-reply acceptance](../../evidence/MILESTONE9.md#lost-reply-acceptance-with-version-notification-published-last) | ||
| 17 | records the failure, reduced regression model, twelve-client repeat and cold | ||
| 18 | OneNote verification of all 3200 editing intents. | ||
| 19 | |||
| 20 | ## Workspace | ||
| 21 | |||
| 22 | `crates/onestore` contains the library, examples and integration tests. New Rust | ||
| 23 | prototypes belong in sibling directories under `crates/` and depend on | ||
| 24 | `onestore = { path = "../onestore" }`. The root manifest discovers these crates. | ||
| 25 | The consumer boundary and API tradeoffs are recorded in [API-AUDIT.md](../../evidence/API-AUDIT.md). | ||
| 26 | `crates/onestore-diagnostic` backs the [HTML diagnostic editor](../../evidence/DIAGNOSTIC.md). | ||
| 27 | [`onestore-smb`](../onestore-smb/README.md) provides optional embedded network | ||
| 28 | access; [`onestore-offline`](../onestore-offline/README.md) provides local | ||
| 29 | SQLite persistence and reconnect reconciliation for text, insertion and formatting. | ||
| 30 | Shared native fixtures, specifications, evidence and Python/VM tools stay at the | ||
| 31 | repository root; `fuzz/` remains an independent cargo-fuzz workspace. | ||
| 32 | |||
| 33 | Run Cargo commands from the root. Select `-p onestore` when working only on the | ||
| 34 | library, or `--workspace` for checks across all crates. Example binary paths | ||
| 35 | remain `target/debug/examples/…` for the native verification tools. The collaboration | ||
| 36 | harness also accepts `--client-profile release`. | ||
| 37 | |||
| 38 | ## Supported surface | ||
| 39 | |||
| 40 | | API | Contract | | ||
| 41 | | --- | --- | | ||
| 42 | | `Store`, `RevisionIndex`, `ResolvedRevision` | Parse storage, resolve revisions and reference graphs, expose roots and objects | | ||
| 43 | | `PropertySets`, `Object::references` | Decode properties and ID streams while retaining raw values | | ||
| 44 | | `Object::file_reference`, `Store::file_data` | Identify internal/external payloads and read internal payload bytes | | ||
| 45 | | `document::Document`, `Revision::text_runs` | Interpret document objects and inherited text formatting while retaining unknown properties and revision identities | | ||
| 46 | | `create_section` | Create one page containing one plain-text paragraph and an author, including Unicode | | ||
| 47 | | `create_table_of_contents` | Create ordered section entries from filenames and file identities | | ||
| 48 | | `replace_property_bytes` | Append one scalar-property revision; preserve prior revisions and unrelated property values and references | | ||
| 49 | | `replace_text`, `commit_text`, `commit_file_text` | Replace a UTF-16 range within one ordinary text run; publish text, run boundaries and modification time together | | ||
| 50 | | `Insertion`, `PreparedEdit::insert` | Insert paragraphs into editable containers or positioned outlines into a page, retaining intent identities across rebases | | ||
| 51 | | `TextAttribute`, `PreparedEdit::format` | Change character formatting over a UTF-16 range while sharing immutable styles; preserve unselected runs | | ||
| 52 | | `PreparedEdit::commit`, `PreparedEdit::commit_file` | Publish the exact prepared image under caller-held exclusion or the conservative filesystem adapter | | ||
| 53 | | `read_file` | Read a snapshot under whole-file exclusion | | ||
| 54 | | `read_snapshot` | Read a validated snapshot through fresh positioned I/O while the caller excludes maintenance | | ||
| 55 | | `commit_file_property` | Lock, compare the source snapshot, append and flush, then publish the revision | | ||
| 56 | | `CommitIo`, `commit_property_bytes` | Supply another storage backend with equivalent exclusion and ordered durability | | ||
| 57 | |||
| 58 | Scalar edits accept encoded values and require the caller to maintain MS-ONE | ||
| 59 | semantics. Text edits maintain run boundaries, inherit the insertion run's formatting, | ||
| 60 | and promote legacy text to Unicode when needed. Explicit and body-derived | ||
| 61 | navigation titles update in the same transaction; unsupported fields, | ||
| 62 | protected objects and split surrogate pairs are | ||
| 63 | rejected before writing. Appended snapshots cap revision dependency depth at 512 | ||
| 64 | while retaining historical revisions. TOC snapshots can remap encoded CompactIDs | ||
| 65 | without changing their resolved references. Password-protected | ||
| 66 | sections retain their encrypted structure and payloads; the library does not | ||
| 67 | derive password keys or decrypt their pages. | ||
| 68 | Insertions update child references, reference counts, modification times and automatic | ||
| 69 | titles atomically. Paragraphs can be nested or inserted into table cells; outline | ||
| 70 | coordinates use points. Retain the `Insertion` value for rebasing: creating another | ||
| 71 | value creates different object identities. Duplicate insertion identities require | ||
| 72 | reconciliation. Formatting accepts explicit attributes, preserves inherited values, | ||
| 73 | and gives retired immutable styles zero current references while retaining history. | ||
| 74 | Generated fields, protected targets and unsupported run-data boundary changes are | ||
| 75 | rejected before publication. Local caches expose text, insertion and formatting edits; | ||
| 76 | the [document-writer acceptance](../../evidence/MILESTONE9.md#document-writer-and-offline-acceptance) | ||
| 77 | includes twelve mixed native/Rust clients, outages, lost replies and native revision retirement. | ||
| 78 | |||
| 79 | External `.onebin` references identify payloads for the caller to obtain. Cloud | ||
| 80 | FSSHTTP synchronization and a C ABI are outside the implemented surface. | ||
| 81 | |||
| 82 | ## Try it | ||
| 83 | |||
| 84 | Requires Rust 1.97 or later for the verified build. Examples create new destinations | ||
| 85 | and refuse to overwrite them. The Python tools require Python 3.10 or later and Pillow. | ||
| 86 | |||
| 87 | ```sh | ||
| 88 | cargo run --example create_notebook -- /tmp/one-demo 'Hello from Rust.' 'Example Author' | ||
| 89 | cargo run --example inventory -- /tmp/one-demo/synthetic.one | ||
| 90 | cargo run --example inspect -- /tmp/one-demo/synthetic.one | ||
| 91 | cargo run --example document -- /tmp/one-demo/synthetic.one /tmp/one-model | ||
| 92 | python3 tools/notebook_report.py /tmp/one-demo /tmp/one-report --timezone America/Los_Angeles | ||
| 93 | ``` | ||
| 94 | |||
| 95 | A seeded text edit on a disposable copy records its page, UTF-16 range, | ||
| 96 | replacement and outcome as JSON: | ||
| 97 | |||
| 98 | ```sh | ||
| 99 | cargo run --example random_edit -- /tmp/one-demo/synthetic.one /tmp/edited.one 42 | ||
| 100 | ``` | ||
| 101 | |||
| 102 | The report contains readable pages, document JSON, assets, source identities and | ||
| 103 | coordinates. It preserves paragraph nesting, lists, tables, links and tags. | ||
| 104 | Historical contexts, recycle-bin pages and default templates are represented | ||
| 105 | separately. Native ink and structured equations retain their source data and | ||
| 106 | appear explicitly as opaque content. The report is a reading view; its native | ||
| 107 | PDF references supply the original canvas layout. | ||
| 108 | |||
| 109 | Read and validate a snapshot before interpreting its graph: | ||
| 110 | |||
| 111 | ```rust,no_run | ||
| 112 | use onestore::{read_file, RevisionIndex, Store}; | ||
| 113 | |||
| 114 | fn main() -> Result<(), Box<dyn std::error::Error>> { | ||
| 115 | let bytes = read_file("notebook/synthetic.one")?; | ||
| 116 | let store = Store::parse(&bytes)?; | ||
| 117 | if !store.checksum_mismatches.is_empty() { | ||
| 118 | return Err("Transaction checksum damage".into()); | ||
| 119 | } | ||
| 120 | let index = RevisionIndex::parse(&store)?; | ||
| 121 | index.validate_current()?; | ||
| 122 | Ok(()) | ||
| 123 | } | ||
| 124 | ``` | ||
| 125 | |||
| 126 | `Store::parse` exposes checksum mismatches for diagnostic readers; the writer | ||
| 127 | rejects them. The resolved graph borrows the snapshot. Select the object-space ID, | ||
| 128 | object ID, and property from that graph, then pass those IDs, the same snapshot, | ||
| 129 | and the replacement's encoded bytes to `commit_file_property`. The | ||
| 130 | `edit_property` example demonstrates selection by JCID/property/expected bytes, | ||
| 131 | with optional explicit IDs when conflict copies contain identical text. | ||
| 132 | |||
| 133 | ## Commit behavior | ||
| 134 | |||
| 135 | ```text | ||
| 136 | exclusive lock → exact snapshot comparison | ||
| 137 | → append data → flush | ||
| 138 | → prepare header metadata → flush | ||
| 139 | → publish transaction counter → flush | ||
| 140 | → finish counter rollover → flush | ||
| 141 | → notify cached readers → flush → unlock | ||
| 142 | ``` | ||
| 143 | |||
| 144 | Stale snapshots fail before writing. Live readers must use equivalent exclusion. | ||
| 145 | Native conflict creation can still expose cross-space references before their | ||
| 146 | targets are saved; such snapshots must be rejected and reread while synchronization | ||
| 147 | proceeds. `read_file` and `commit_file_property` serialize within the process because | ||
| 148 | macOS SMB locks can be reentrant. The lock is nonblocking across processes; | ||
| 149 | contention requires a fresh read and a later retry. | ||
| 150 | |||
| 151 | | Error state | Meaning and caller action | | ||
| 152 | | --- | --- | | ||
| 153 | | `NotCommitted` | This edit was not published. Preparation bytes may exist. Reread before retrying. | | ||
| 154 | | `Unknown` | Publication may have persisted despite the error. Reread and resolve the intended edit before retrying. | | ||
| 155 | | `Committed` | Publication was durably acknowledged; counter cleanup or lock release failed. Reopen the committed result instead of replaying the edit. | | ||
| 156 | |||
| 157 | At counter rollover, empty transactions make intermediate published counts valid. | ||
| 158 | The highest changed byte is flushed before lower bytes are cleaned up. The | ||
| 159 | 255→256 and 65535→65536 boundaries and interrupted cleanup states have independent | ||
| 160 | native acceptance captures. A no-op still flushes; a failed flush has an unknown | ||
| 161 | durability outcome. | ||
| 162 | |||
| 163 | The filesystem adapter uses whole-file locking. On macOS it acquires the lock | ||
| 164 | as part of opening the file (`O_EXLOCK | O_NONBLOCK`): separate open and `flock` | ||
| 165 | calls allowed overlapping exclusive holders and stranded server locks under | ||
| 166 | multi-process SMB contention. `tools/smb_lock_race.py` reproduces that failure | ||
| 167 | without notebook parsing or writing. On the tested macOS SMB mount, | ||
| 168 | POSIX byte-range locks returned `ENOTSUP`; whole-file locks excluded native | ||
| 169 | OneNote's lock ranges. This adapter serializes readers and writers during each | ||
| 170 | operation; it does not reproduce native reader/writer concurrency. The | ||
| 171 | [locking audit](../../evidence/LOCKING.md) records native coordination bytes, write-open share | ||
| 172 | modes, and a working macOS SMB-specific byte-range lock probe. `sync_all` falls | ||
| 173 | back to `fsync` on macOS only when `F_FULLFSYNC` is unsupported. Successful SMB FLUSH replies were observed on the | ||
| 174 | wire. Correctness requires the backend to honor exclusion and ordered flushes. | ||
| 175 | The evidence covers transport failures, not physical server power loss or every | ||
| 176 | filesystem's lock implementation. | ||
| 177 | |||
| 178 | For shared network notebooks, use the optional | ||
| 179 | [`onestore-smb`](../onestore-smb/README.md) crate. It uses native share modes, | ||
| 180 | shared reader guards, writer exclusion and fresh pathname identity checks without | ||
| 181 | an OS-mounted share. Its [coordination acceptance](../../evidence/MILESTONE9.md) covers native | ||
| 182 | maintenance, mixed readers/writers, reconnects and uncertain publication. The | ||
| 183 | filesystem adapter retains its conservative locking; mounted-path freshness | ||
| 184 | across native replacement is not established by that exclusion. | ||
| 185 | |||
| 186 | ## Verification | ||
| 187 | |||
| 188 | ```sh | ||
| 189 | cargo test --all-targets | ||
| 190 | cargo clippy --all-targets -- -D warnings | ||
| 191 | python3 tools/verify-corpus.py | ||
| 192 | python3 tools/verify-reader.py # requires Pillow and the local private corpus | ||
| 193 | python3 tools/verify-writer.py | ||
| 194 | python3 tools/verify-collaboration.py | ||
| 195 | python3 tools/verify-document.py /path/to/copied/notebook /path/to/native/read | ||
| 196 | ``` | ||
| 197 | |||
| 198 | The frozen private corpus is excluded from version control. Its verifier checks 26 pages, | ||
| 199 | 581 text objects, hyperlink targets, exact image bytes, and native image conversions. | ||
| 200 | Synthetic corpus manifests bind binary fixtures to independent native XML and | ||
| 201 | attachment captures. `verify-corpus.py` also requires that private corpus. | ||
| 202 | |||
| 203 | Storage tests exercise malformed references, deep properties, historical revision | ||
| 204 | preservation, short I/O, stale snapshots, counter tears, and every injected I/O | ||
| 205 | failure point at ordinary and rollover commits. The crash model persists arbitrary | ||
| 206 | subsets of unflushed bytes and is shared with the stateful commit fuzzer. | ||
| 207 | |||
| 208 | ```sh | ||
| 209 | cargo +nightly fuzz run revisions -- -max_total_time=120 -max_len=262144 -rss_limit_mb=2048 | ||
| 210 | cargo +nightly fuzz run commit -- -max_total_time=300 -max_len=4096 -rss_limit_mb=2048 | ||
| 211 | ``` | ||
| 212 | |||
| 213 | Fuzz targets cover storage, properties, revisions, scalar edits, creation, and | ||
| 214 | multi-edit interrupted commits. Seed the revision target with native `.one` files | ||
| 215 | using symlinks under `fuzz/corpus/revisions`; empty seed directories mostly exercise | ||
| 216 | header rejection. Bounded run counts and native findings live in `PROGRESS.md`. | ||
| 217 | |||
| 218 | The document fuzzer mutates native property streams, repairs their checksums, and | ||
| 219 | traverses every retained revision and resolved text run. Its public seeds live in | ||
| 220 | the source target; private seeds are supplied only at runtime. Native edit-history | ||
| 221 | tests compare independently generated operations, OneNote XML and the Rust model; | ||
| 222 | failed histories can be replayed and shrunk in fresh disposable clones. The | ||
| 223 | document feature matrix is in [FEATURES.md](../../evidence/FEATURES.md), and the milestone's | ||
| 224 | acceptance contract is in [MILESTONE6.md](../../evidence/MILESTONE6.md). | ||
| 225 | |||
| 226 | The stage-5 gate uses OneNote 2010 build 14.0.7015.1000 on Windows 7, a macOS SMB | ||
| 227 | mount, and Samba on zenith. [The collaboration corpus](../../corpus/collaboration/round-01) | ||
| 228 | captures native lock contention, different-paragraph merging, same-paragraph | ||
| 229 | conflicts, offline editing/reconnection, and lost successful FLUSH replies at | ||
| 230 | preparation, publication, and counter cleanup. Each final notebook was reopened | ||
| 231 | from a fresh native cache. Competing text survives as native conflict pages; | ||
| 232 | OneNote's COM hierarchy omits those pages, so the offline case also includes a | ||
| 233 | native UI capture. Full-page COM updates produced an extra conflict copy during | ||
| 234 | the disjoint case and two recorded geometry changes; the verifier checks those | ||
| 235 | exact changes as well as retained image, ink, table, and attachment content. | ||
| 236 | |||
| 237 | `tools/native/profile.ps1` parks/restores the personal native profile and cache; | ||
| 238 | `cold.ps1`, `read.ps1`, and `collaborate.ps1` operate on disposable test roots. | ||
| 239 | `tools/zenith-locks` observes server locks. `tools/smb-proxy.py` traces and interrupts | ||
| 240 | a dedicated loopback test session; its control JSON selects the successful response | ||
| 241 | and occurrence to withhold. The captured trace and result files are the regression | ||
| 242 | oracle; replaying the native experiments requires the supplied Windows/share setup. | ||
crates/onestore/examples/concurrent_client.rs+10-193
| ... | @@ -1,18 +1,9 @@ | ... | @@ -1,18 +1,9 @@ |
| 1 | #[path = "support/concurrent.rs"] | ||
| 2 | mod concurrent; | ||
| 1 | #[path = "../src/flush.rs"] | 3 | #[path = "../src/flush.rs"] |
| 2 | mod flush; | 4 | mod flush; |
| 3 | 5 | ||
| 4 | use onestore::{ | 6 | use std::{env, fs, io::Write}; |
| 5 | CommitState, ExGuid, RevisionIndex, Store, | ||
| 6 | document::{Document, Kind}, | ||
| 7 | }; | ||
| 8 | use serde_json::json; | ||
| 9 | use std::{ | ||
| 10 | env, fs, | ||
| 11 | io::{self, Write}, | ||
| 12 | path::Path, | ||
| 13 | thread, | ||
| 14 | time::{Duration, Instant, SystemTime, UNIX_EPOCH}, | ||
| 15 | }; | ||
| 16 | 7 | ||
| 17 | fn main() -> Result<(), Box<dyn std::error::Error>> { | 8 | fn main() -> Result<(), Box<dyn std::error::Error>> { |
| 18 | let args: Vec<_> = env::args().skip(1).collect(); | 9 | let args: Vec<_> = env::args().skip(1).collect(); |
| ... | @@ -27,185 +18,11 @@ fn main() -> Result<(), Box<dyn std::error::Error>> { | ... | @@ -27,185 +18,11 @@ fn main() -> Result<(), Box<dyn std::error::Error>> { |
| 27 | flush::flush(&file)?; | 18 | flush::flush(&file)?; |
| 28 | return Ok(()); | 19 | return Ok(()); |
| 29 | } | 20 | } |
| 30 | if args.len() != 7 || !["read", "write", "edit"].contains(&args[0].as_str()) { | 21 | concurrent::run( |
| 31 | return Err("Usage: concurrent_client init FILE | read|write|edit FILE ACTOR OPERATIONS START_FILE STOP_FILE SEED".into()); | 22 | &args, |
| 32 | } | 23 | |path| onestore::read_file(path), |
| 33 | let mut random: u64 = args[6].parse()?; | 24 | |path, source, space, object, range, replacement| { |
| 34 | let operations: usize = args[3].parse()?; | 25 | onestore::commit_file_text(path, source, space, object, range, replacement) |
| 35 | if operations == 0 { | 26 | }, |
| 36 | return Err("Choose at least one operation.".into()); | 27 | ) |
| 37 | } | ||
| 38 | let deadline = Instant::now() + Duration::from_secs(600); | ||
| 39 | let mut output = io::stdout().lock(); | ||
| 40 | let mut log = |event: serde_json::Value| -> io::Result<()> { | ||
| 41 | writeln!(output, "{event}")?; | ||
| 42 | output.flush() | ||
| 43 | }; | ||
| 44 | log(json!({"event": "ready", "pid": std::process::id(), "actor": args[2]}))?; | ||
| 45 | while !Path::new(&args[4]).exists() { | ||
| 46 | if Instant::now() > deadline { | ||
| 47 | return Err("Start barrier timed out.".into()); | ||
| 48 | } | ||
| 49 | thread::sleep(Duration::from_millis(5)); | ||
| 50 | } | ||
| 51 | let mut completed = 0; | ||
| 52 | let mut attempts = 0; | ||
| 53 | while completed < operations || (args[0] == "read" && !Path::new(&args[5]).exists()) { | ||
| 54 | if Instant::now() > deadline { | ||
| 55 | return Err("Concurrent client timed out.".into()); | ||
| 56 | } | ||
| 57 | attempts += 1; | ||
| 58 | random = random | ||
| 59 | .wrapping_mul(6364136223846793005) | ||
| 60 | .wrapping_add(1442695040888963407); | ||
| 61 | thread::sleep(Duration::from_millis((random >> 32) % 7)); | ||
| 62 | let started = SystemTime::now().duration_since(UNIX_EPOCH)?.as_micros(); | ||
| 63 | let source = match onestore::read_file(&args[1]) { | ||
| 64 | Ok(source) => source, | ||
| 65 | Err(error) | ||
| 66 | if [ | ||
| 67 | io::ErrorKind::WouldBlock, | ||
| 68 | io::ErrorKind::PermissionDenied, | ||
| 69 | io::ErrorKind::NotFound, | ||
| 70 | ] | ||
| 71 | .contains(&error.kind()) => | ||
| 72 | { | ||
| 73 | log( | ||
| 74 | json!({"event": "read_busy", "attempt": attempts, "kind": format!("{:?}", error.kind())}), | ||
| 75 | )?; | ||
| 76 | thread::sleep(Duration::from_millis(100)); | ||
| 77 | continue; | ||
| 78 | } | ||
| 79 | Err(error) => { | ||
| 80 | log( | ||
| 81 | json!({"event": "read_error", "attempt": attempts, "kind": format!("{:?}", error.kind())}), | ||
| 82 | )?; | ||
| 83 | return Err(error.into()); | ||
| 84 | } | ||
| 85 | }; | ||
| 86 | let preserve = |error: onestore::Error| { | ||
| 87 | let path = Path::new(&args[4]) | ||
| 88 | .parent() | ||
| 89 | .unwrap() | ||
| 90 | .join(format!("invalid-{}-{attempts}.one", std::process::id())); | ||
| 91 | if let Err(failure) = fs::write(&path, &source) { | ||
| 92 | eprintln!( | ||
| 93 | "Could not save invalid snapshot {}: {failure}", | ||
| 94 | path.display() | ||
| 95 | ); | ||
| 96 | } | ||
| 97 | error | ||
| 98 | }; | ||
| 99 | let store = Store::parse(&source).map_err(preserve)?; | ||
| 100 | if !store.checksum_mismatches.is_empty() { | ||
| 101 | return Err(preserve(onestore::Error { | ||
| 102 | offset: store.checksum_mismatches[0], | ||
| 103 | message: "A reader observed transaction checksum damage.", | ||
| 104 | }) | ||
| 105 | .into()); | ||
| 106 | } | ||
| 107 | let index = RevisionIndex::parse(&store).map_err(preserve)?; | ||
| 108 | index.validate_current().map_err(preserve)?; | ||
| 109 | let document = Document::parse(&index).map_err(preserve)?; | ||
| 110 | let mut targets = Vec::new(); | ||
| 111 | for (sid, page) in document.pages().map_err(preserve)? { | ||
| 112 | let space = &document.spaces[&sid]; | ||
| 113 | let revision = &space.revisions[&space.contexts[&ExGuid::default()]]; | ||
| 114 | let mut pending = vec![page]; | ||
| 115 | let mut seen = std::collections::BTreeSet::new(); | ||
| 116 | while let Some(oid) = pending.pop() { | ||
| 117 | if !seen.insert(oid) { | ||
| 118 | continue; | ||
| 119 | } | ||
| 120 | let node = &revision.nodes[&oid]; | ||
| 121 | pending.extend( | ||
| 122 | node.children | ||
| 123 | .iter() | ||
| 124 | .chain(&node.content) | ||
| 125 | .chain(&node.structure) | ||
| 126 | .copied(), | ||
| 127 | ); | ||
| 128 | if let Kind::RichText { text, .. } = &node.kind | ||
| 129 | && text.starts_with("Concurrent edits:") | ||
| 130 | { | ||
| 131 | revision.text_runs(oid).map_err(preserve)?; | ||
| 132 | targets.push((sid, oid, text)); | ||
| 133 | } | ||
| 134 | } | ||
| 135 | } | ||
| 136 | let [(sid, oid, text)] = targets.as_slice() else { | ||
| 137 | return Err(preserve(onestore::Error { | ||
| 138 | offset: 0, | ||
| 139 | message: "Expected one concurrent-edit paragraph.", | ||
| 140 | }) | ||
| 141 | .into()); | ||
| 142 | }; | ||
| 143 | let read_finished = SystemTime::now().duration_since(UNIX_EPOCH)?.as_micros(); | ||
| 144 | log( | ||
| 145 | json!({"event": "read", "attempt": attempts, "started_us": started, "finished_us": read_finished, | ||
| 146 | "transaction": store.header.transaction_count, "text": text}), | ||
| 147 | )?; | ||
| 148 | if args[0] == "read" { | ||
| 149 | completed += 1; | ||
| 150 | continue; | ||
| 151 | } | ||
| 152 | let token = format!(" [{}:{}]", args[2], completed); | ||
| 153 | let offset = u32::try_from(text.encode_utf16().count())?; | ||
| 154 | let mut range = offset..offset; | ||
| 155 | let mut replacement = token.clone(); | ||
| 156 | if args[0] == "edit" { | ||
| 157 | let prefix = "Concurrent edits:"; | ||
| 158 | let mut boundaries = vec![u32::try_from(prefix.encode_utf16().count())?]; | ||
| 159 | for character in text[prefix.len()..].chars() { | ||
| 160 | boundaries.push(boundaries.last().unwrap() + character.len_utf16() as u32); | ||
| 161 | } | ||
| 162 | let first = ((random >> 16) % boundaries.len() as u64) as usize; | ||
| 163 | let second = ((random >> 40) % boundaries.len() as u64) as usize; | ||
| 164 | range = boundaries[first.min(second)]..boundaries[first.max(second)]; | ||
| 165 | replacement = format!(" café 🦀{token}"); | ||
| 166 | } | ||
| 167 | log( | ||
| 168 | json!({"event": "intent", "attempt": attempts, "operation": completed, | ||
| 169 | "source_transaction": store.header.transaction_count, "before": text, | ||
| 170 | "range": [range.start, range.end], "replacement": replacement, "token": token}), | ||
| 171 | )?; | ||
| 172 | thread::sleep(Duration::from_millis((random >> 48) % 13)); | ||
| 173 | let commit_started = SystemTime::now().duration_since(UNIX_EPOCH)?.as_micros(); | ||
| 174 | let result = onestore::commit_file_text(&args[1], &source, *sid, *oid, range, &replacement); | ||
| 175 | let finished = SystemTime::now().duration_since(UNIX_EPOCH)?.as_micros(); | ||
| 176 | match result { | ||
| 177 | Ok(()) => { | ||
| 178 | log( | ||
| 179 | json!({"event": "commit", "attempt": attempts, "operation": completed, "token": token, | ||
| 180 | "started_us": commit_started, "finished_us": finished, "source_transaction": store.header.transaction_count}), | ||
| 181 | )?; | ||
| 182 | completed += 1; | ||
| 183 | } | ||
| 184 | Err(error) | ||
| 185 | if error.state == CommitState::NotCommitted | ||
| 186 | && [ | ||
| 187 | io::ErrorKind::WouldBlock, | ||
| 188 | io::ErrorKind::ResourceBusy, | ||
| 189 | io::ErrorKind::PermissionDenied, | ||
| 190 | io::ErrorKind::NotFound, | ||
| 191 | ] | ||
| 192 | .contains(&error.error.kind()) => | ||
| 193 | { | ||
| 194 | log( | ||
| 195 | json!({"event": "retry", "attempt": attempts, "started_us": commit_started, | ||
| 196 | "finished_us": finished, "kind": format!("{:?}", error.error.kind())}), | ||
| 197 | )?; | ||
| 198 | } | ||
| 199 | Err(error) => { | ||
| 200 | log( | ||
| 201 | json!({"event": "commit_error", "attempt": attempts, "operation": completed, | ||
| 202 | "token": token, "state": format!("{:?}", error.state), "kind": format!("{:?}", error.error.kind()), | ||
| 203 | "started_us": commit_started, "finished_us": finished}), | ||
| 204 | )?; | ||
| 205 | return Err(error.into()); | ||
| 206 | } | ||
| 207 | } | ||
| 208 | } | ||
| 209 | log(json!({"event": "done", "completed": completed, "attempts": attempts}))?; | ||
| 210 | Ok(()) | ||
| 211 | } | 28 | } |
crates/onestore/examples/document.rs+19-7
| ... | @@ -2,7 +2,22 @@ use onestore::{ | ... | @@ -2,7 +2,22 @@ use onestore::{ |
| 2 | FileDataReference, RevisionIndex, Store, | 2 | FileDataReference, RevisionIndex, Store, |
| 3 | document::{Document, Kind}, | 3 | document::{Document, Kind}, |
| 4 | }; | 4 | }; |
| 5 | use std::{collections::BTreeSet, env, fs, io, path::PathBuf}; | 5 | use std::{ |
| 6 | collections::BTreeSet, | ||
| 7 | env, fs, | ||
| 8 | io::{self, BufWriter, Write}, | ||
| 9 | path::{Path, PathBuf}, | ||
| 10 | }; | ||
| 11 | |||
| 12 | fn write_json( | ||
| 13 | path: impl AsRef<Path>, | ||
| 14 | value: &impl serde::Serialize, | ||
| 15 | ) -> Result<(), Box<dyn std::error::Error>> { | ||
| 16 | let mut output = BufWriter::new(fs::File::create(path)?); | ||
| 17 | serde_json::to_writer(&mut output, value)?; | ||
| 18 | output.flush()?; | ||
| 19 | Ok(()) | ||
| 20 | } | ||
| 6 | 21 | ||
| 7 | fn main() -> Result<(), Box<dyn std::error::Error>> { | 22 | fn main() -> Result<(), Box<dyn std::error::Error>> { |
| 8 | let mut args = env::args_os().skip(1); | 23 | let mut args = env::args_os().skip(1); |
| ... | @@ -38,7 +53,7 @@ fn main() -> Result<(), Box<dyn std::error::Error>> { | ... | @@ -38,7 +53,7 @@ fn main() -> Result<(), Box<dyn std::error::Error>> { |
| 38 | assets.push(serde_json::json!({"reference": FileDataReference::Internal(*guid), "path": path})); | 53 | assets.push(serde_json::json!({"reference": FileDataReference::Internal(*guid), "path": path})); |
| 39 | } | 54 | } |
| 40 | } | 55 | } |
| 41 | serde_json::to_writer(fs::File::create(destination.join("assets.json"))?, &assets)?; | 56 | write_json(destination.join("assets.json"), &assets)?; |
| 42 | let mut text = std::collections::BTreeMap::new(); | 57 | let mut text = std::collections::BTreeMap::new(); |
| 43 | for (sid, space) in &document.spaces { | 58 | for (sid, space) in &document.spaces { |
| 44 | let mut revisions = std::collections::BTreeMap::new(); | 59 | let mut revisions = std::collections::BTreeMap::new(); |
| ... | @@ -53,11 +68,8 @@ fn main() -> Result<(), Box<dyn std::error::Error>> { | ... | @@ -53,11 +68,8 @@ fn main() -> Result<(), Box<dyn std::error::Error>> { |
| 53 | } | 68 | } |
| 54 | text.insert(*sid, revisions); | 69 | text.insert(*sid, revisions); |
| 55 | } | 70 | } |
| 56 | serde_json::to_writer(fs::File::create(destination.join("text.json"))?, &text)?; | 71 | write_json(destination.join("text.json"), &text)?; |
| 57 | serde_json::to_writer( | 72 | write_json(destination.join("document.json"), &document)?; |
| 58 | fs::File::create(destination.join("document.json"))?, | ||
| 59 | &document, | ||
| 60 | )?; | ||
| 61 | } else { | 73 | } else { |
| 62 | serde_json::to_writer(io::stdout().lock(), &document)?; | 74 | serde_json::to_writer(io::stdout().lock(), &document)?; |
| 63 | } | 75 | } |
crates/onestore/examples/insert.rs created+52
| ... | @@ -0,0 +1,52 @@ | ||
| 1 | #[path = "../src/flush.rs"] | ||
| 2 | mod flush; | ||
| 3 | |||
| 4 | use onestore::{Insertion, PreparedEdit}; | ||
| 5 | use std::{fs, io::Write}; | ||
| 6 | |||
| 7 | fn main() -> Result<(), Box<dyn std::error::Error>> { | ||
| 8 | let args: Vec<_> = std::env::args().skip(1).collect(); | ||
| 9 | let usage = "Usage: insert paragraph INPUT OUTPUT|--in-place SPACE PARENT BEFORE|- TEXT AUTHOR\n insert outline INPUT OUTPUT|--in-place SPACE PAGE X Y TEXT AUTHOR"; | ||
| 10 | let insertion = match args.first().map(String::as_str) { | ||
| 11 | Some("paragraph") if args.len() == 8 => Insertion::paragraph( | ||
| 12 | args[4].parse()?, | ||
| 13 | if args[5] == "-" { | ||
| 14 | None | ||
| 15 | } else { | ||
| 16 | Some(args[5].parse()?) | ||
| 17 | }, | ||
| 18 | &args[6], | ||
| 19 | &args[7], | ||
| 20 | )?, | ||
| 21 | Some("outline") if args.len() == 9 => Insertion::outline( | ||
| 22 | args[4].parse()?, | ||
| 23 | args[5].parse()?, | ||
| 24 | args[6].parse()?, | ||
| 25 | &args[7], | ||
| 26 | &args[8], | ||
| 27 | )?, | ||
| 28 | _ => return Err(usage.into()), | ||
| 29 | }; | ||
| 30 | let source = onestore::read_file(&args[1])?; | ||
| 31 | let prepared = PreparedEdit::insert(&source, args[3].parse()?, &insertion)?; | ||
| 32 | let mut record = serde_json::json!({"intent": insertion, "object": insertion.object(), "text_object": insertion.text_object()}); | ||
| 33 | if args[2] == "--in-place" { | ||
| 34 | if let Err(error) = prepared.commit_file(&args[1]) { | ||
| 35 | record["state"] = format!("{:?}", error.state).into(); | ||
| 36 | record["error"] = error.error.to_string().into(); | ||
| 37 | println!("{record}"); | ||
| 38 | std::process::exit(2); | ||
| 39 | } | ||
| 40 | record["state"] = "Committed".into(); | ||
| 41 | } else { | ||
| 42 | let mut file = fs::OpenOptions::new() | ||
| 43 | .write(true) | ||
| 44 | .create_new(true) | ||
| 45 | .open(&args[2])?; | ||
| 46 | file.write_all(prepared.as_bytes())?; | ||
| 47 | flush::flush(&file)?; | ||
| 48 | record["state"] = "Created".into(); | ||
| 49 | } | ||
| 50 | println!("{record}"); | ||
| 51 | Ok(()) | ||
| 52 | } | ||
crates/onestore/examples/maintenance_fixture.rs created+63
| ... | @@ -0,0 +1,63 @@ | ||
| 1 | use onestore::{ | ||
| 2 | ExGuid, RevisionIndex, Store, | ||
| 3 | document::{Document, Kind}, | ||
| 4 | }; | ||
| 5 | use std::{fs, io, path::PathBuf}; | ||
| 6 | |||
| 7 | fn main() -> Result<(), Box<dyn std::error::Error>> { | ||
| 8 | let mut args = std::env::args_os().skip(1); | ||
| 9 | let output = PathBuf::from( | ||
| 10 | args.next() | ||
| 11 | .ok_or(io::Error::from(io::ErrorKind::InvalidInput))?, | ||
| 12 | ); | ||
| 13 | let current = args | ||
| 14 | .next() | ||
| 15 | .map(|value| value.into_string()) | ||
| 16 | .transpose() | ||
| 17 | .map_err(|_| io::Error::from(io::ErrorKind::InvalidInput))?; | ||
| 18 | if args.next().is_some() { | ||
| 19 | return Err(io::Error::from(io::ErrorKind::InvalidInput).into()); | ||
| 20 | } | ||
| 21 | fs::create_dir(&output)?; | ||
| 22 | let mut bytes = onestore::create_section("synthetic.one", "Maintenance baseline.", "Fixture")?; | ||
| 23 | for revision in 0..25 { | ||
| 24 | let store = Store::parse(&bytes)?; | ||
| 25 | let index = RevisionIndex::parse(&store)?; | ||
| 26 | let document = Document::parse(&index)?; | ||
| 27 | let (sid, oid, end) = document | ||
| 28 | .spaces | ||
| 29 | .iter() | ||
| 30 | .find_map(|(sid, space)| { | ||
| 31 | space.revisions[&space.contexts[&ExGuid::default()]] | ||
| 32 | .nodes | ||
| 33 | .iter() | ||
| 34 | .find_map(|(oid, node)| { | ||
| 35 | if let Kind::RichText { text, .. } = &node.kind { | ||
| 36 | return Some((*sid, *oid, text.encode_utf16().count() as u32)); | ||
| 37 | } | ||
| 38 | None | ||
| 39 | }) | ||
| 40 | }) | ||
| 41 | .ok_or(io::Error::from(io::ErrorKind::InvalidData))?; | ||
| 42 | let text = if revision == 24 { | ||
| 43 | current | ||
| 44 | .clone() | ||
| 45 | .unwrap_or_else(|| "Maintenance current.".to_owned()) | ||
| 46 | } else { | ||
| 47 | format!("Revision {revision}: {}", "x".repeat(65536)) | ||
| 48 | }; | ||
| 49 | bytes = onestore::replace_text(&bytes, sid, oid, 0..end, &text)?; | ||
| 50 | } | ||
| 51 | let store = Store::parse(&bytes)?; | ||
| 52 | let toc = onestore::create_table_of_contents( | ||
| 53 | "Open Notebook.onetoc2", | ||
| 54 | &[("synthetic.one", store.header.file_id)], | ||
| 55 | )?; | ||
| 56 | fs::write(output.join("synthetic.one"), &bytes)?; | ||
| 57 | fs::write(output.join("Open Notebook.onetoc2"), toc)?; | ||
| 58 | println!( | ||
| 59 | "{}", | ||
| 60 | serde_json::json!({"bytes":bytes.len(),"transactions":store.header.transaction_count}) | ||
| 61 | ); | ||
| 62 | Ok(()) | ||
| 63 | } | ||
crates/onestore/examples/power_loss.rs+17-64
| ... | @@ -1,74 +1,27 @@ | ... | @@ -1,74 +1,27 @@ |
| 1 | #[path = "../tests/support/current.rs"] | ||
| 2 | mod current; | ||
| 3 | use current::current; | ||
| 4 | |||
| 5 | #[path = "../tests/support/trace.rs"] | ||
| 6 | mod trace; | ||
| 7 | use trace::{Event, Trace}; | ||
| 8 | |||
| 1 | #[path = "../tests/support/checkpoint.rs"] | 9 | #[path = "../tests/support/checkpoint.rs"] |
| 2 | mod checkpoint; | 10 | mod checkpoint; |
| 3 | 11 | ||
| 4 | use onestore::{ | 12 | use onestore::{ |
| 5 | CommitIo, ExGuid, RevisionIndex, Store, | 13 | ExGuid, RevisionIndex, Store, |
| 6 | document::{Document, Kind}, | 14 | document::{Document, Kind}, |
| 7 | }; | 15 | }; |
| 8 | use std::{collections::BTreeMap, fs, io, path::PathBuf}; | 16 | use std::{collections::BTreeMap, fs, path::PathBuf}; |
| 9 | |||
| 10 | enum Event { | ||
| 11 | Write(usize, Vec<u8>), | ||
| 12 | Flush, | ||
| 13 | } | ||
| 14 | |||
| 15 | struct Trace { | ||
| 16 | bytes: Vec<u8>, | ||
| 17 | events: Vec<Event>, | ||
| 18 | } | ||
| 19 | |||
| 20 | impl CommitIo for Trace { | ||
| 21 | fn read_at(&mut self, offset: u64, output: &mut [u8]) -> io::Result<usize> { | ||
| 22 | let offset = offset as usize; | ||
| 23 | let count = output.len().min(self.bytes.len().saturating_sub(offset)); | ||
| 24 | output[..count].copy_from_slice(&self.bytes[offset..offset + count]); | ||
| 25 | Ok(count) | ||
| 26 | } | ||
| 27 | fn write_at(&mut self, offset: u64, bytes: &[u8]) -> io::Result<usize> { | ||
| 28 | let offset = offset as usize; | ||
| 29 | let count = bytes.len().min(4096); | ||
| 30 | self.bytes.resize(self.bytes.len().max(offset + count), 0); | ||
| 31 | self.bytes[offset..offset + count].copy_from_slice(&bytes[..count]); | ||
| 32 | self.events | ||
| 33 | .push(Event::Write(offset, bytes[..count].to_vec())); | ||
| 34 | Ok(count) | ||
| 35 | } | ||
| 36 | fn flush(&mut self) -> io::Result<()> { | ||
| 37 | self.events.push(Event::Flush); | ||
| 38 | Ok(()) | ||
| 39 | } | ||
| 40 | } | ||
| 41 | |||
| 42 | fn current(bytes: &[u8]) -> BTreeMap<ExGuid, String> { | ||
| 43 | let store = Store::parse(bytes).unwrap(); | ||
| 44 | assert!(store.checksum_mismatches.is_empty()); | ||
| 45 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 46 | index.validate_current().unwrap(); | ||
| 47 | Document::parse(&index).unwrap(); | ||
| 48 | index | ||
| 49 | .spaces | ||
| 50 | .iter() | ||
| 51 | .map(|(sid, space)| { | ||
| 52 | let rid = space.labels[&(ExGuid::default(), 1)]; | ||
| 53 | let revision = index.resolve(*sid, rid).unwrap(); | ||
| 54 | for object in revision.objects.values() { | ||
| 55 | if let Some(onestore::FileDataReference::Internal(guid)) = | ||
| 56 | object.file_reference().unwrap() | ||
| 57 | { | ||
| 58 | store.file_data(guid).unwrap(); | ||
| 59 | } | ||
| 60 | } | ||
| 61 | (*sid, format!("{revision:?}")) | ||
| 62 | }) | ||
| 63 | .collect() | ||
| 64 | } | ||
| 65 | 17 | ||
| 66 | fn main() -> Result<(), Box<dyn std::error::Error>> { | 18 | fn main() -> Result<(), Box<dyn std::error::Error>> { |
| 67 | let destination = PathBuf::from( | 19 | let mut args = std::env::args_os().skip(1); |
| 68 | std::env::args_os() | 20 | let destination = PathBuf::from(args.next().ok_or("Provide a new evidence directory")?); |
| 69 | .nth(1) | 21 | let option = args.next(); |
| 70 | .ok_or("Provide a new evidence directory")?, | 22 | if option.as_deref().is_some_and(|flag| flag != "--toc") || args.next().is_some() { |
| 71 | ); | 23 | return Err("Usage: power_loss NEW_DIRECTORY [--toc]".into()); |
| 24 | } | ||
| 72 | fs::create_dir(&destination)?; | 25 | fs::create_dir(&destination)?; |
| 73 | let fixtures = [ | 26 | let fixtures = [ |
| 74 | ( | 27 | ( |
| ... | @@ -107,7 +60,7 @@ fn main() -> Result<(), Box<dyn std::error::Error>> { | ... | @@ -107,7 +60,7 @@ fn main() -> Result<(), Box<dyn std::error::Error>> { |
| 107 | let mut records = Vec::new(); | 60 | let mut records = Vec::new(); |
| 108 | let mut saved = BTreeMap::new(); | 61 | let mut saved = BTreeMap::new(); |
| 109 | for (name, path) in fixtures { | 62 | for (name, path) in fixtures { |
| 110 | if std::env::args().nth(2).as_deref() == Some("--toc") && !name.starts_with("toc") { | 63 | if option.is_some() && !name.starts_with("toc") { |
| 111 | continue; | 64 | continue; |
| 112 | } | 65 | } |
| 113 | println!("Checking {name}"); | 66 | println!("Checking {name}"); |
crates/onestore/examples/support/concurrent.rs created+258
| ... | @@ -0,0 +1,258 @@ | ||
| 1 | use onestore::{ | ||
| 2 | CommitState, ExGuid, RevisionIndex, Store, | ||
| 3 | document::{Document, Kind}, | ||
| 4 | }; | ||
| 5 | use serde_json::json; | ||
| 6 | use std::{ | ||
| 7 | fs, | ||
| 8 | io::{self, Write}, | ||
| 9 | path::Path, | ||
| 10 | thread, | ||
| 11 | time::{Duration, Instant, SystemTime, UNIX_EPOCH}, | ||
| 12 | }; | ||
| 13 | |||
| 14 | pub fn document_view(bytes: &[u8]) -> Result<serde_json::Value, onestore::Error> { | ||
| 15 | let store = Store::parse(bytes)?; | ||
| 16 | let index = RevisionIndex::parse(&store)?; | ||
| 17 | index.validate_current()?; | ||
| 18 | let document = Document::parse(&index)?; | ||
| 19 | let mut texts = serde_json::Map::new(); | ||
| 20 | for (sid, _) in document.pages()? { | ||
| 21 | let space = &document.spaces[&sid]; | ||
| 22 | let revision = &space.revisions[&space.contexts[&ExGuid::default()]]; | ||
| 23 | for (id, node) in &revision.nodes { | ||
| 24 | if let Kind::RichText { text, .. } = &node.kind | ||
| 25 | && text.starts_with("Document w") | ||
| 26 | { | ||
| 27 | let runs=revision.text_runs(*id)?.into_iter().map(|run|json!({"text":run.text,"bold":run.format.bold.unwrap_or(false),"size":run.format.font_size,"color":run.format.color.unwrap_or(0xff000000)})).collect::<Vec<_>>(); | ||
| 28 | texts.insert(id.to_string(), json!({"text":text,"runs":runs})); | ||
| 29 | } | ||
| 30 | } | ||
| 31 | } | ||
| 32 | Ok(texts.into()) | ||
| 33 | } | ||
| 34 | |||
| 35 | pub fn run( | ||
| 36 | args: &[String], | ||
| 37 | mut read: impl FnMut(&str) -> io::Result<Vec<u8>>, | ||
| 38 | mut commit: impl FnMut( | ||
| 39 | &str, | ||
| 40 | &[u8], | ||
| 41 | ExGuid, | ||
| 42 | ExGuid, | ||
| 43 | std::ops::Range<u32>, | ||
| 44 | &str, | ||
| 45 | ) -> Result<(), onestore::CommitError>, | ||
| 46 | ) -> Result<(), Box<dyn std::error::Error>> { | ||
| 47 | if args.len() != 7 || !["read", "write", "edit"].contains(&args[0].as_str()) { | ||
| 48 | return Err( | ||
| 49 | "Expected read|write|edit FILE ACTOR OPERATIONS START_FILE STOP_FILE SEED.".into(), | ||
| 50 | ); | ||
| 51 | } | ||
| 52 | let mut random: u64 = args[6].parse()?; | ||
| 53 | let operations: usize = args[3].parse()?; | ||
| 54 | if operations == 0 { | ||
| 55 | return Err("Choose at least one operation.".into()); | ||
| 56 | } | ||
| 57 | let timeout = match std::env::var("ONESTORE_CLIENT_TIMEOUT_MS") { | ||
| 58 | Ok(value) => value.parse::<u64>()?, | ||
| 59 | Err(std::env::VarError::NotPresent) => 600_000, | ||
| 60 | Err(error) => return Err(error.into()), | ||
| 61 | }; | ||
| 62 | if timeout == 0 { | ||
| 63 | return Err("Choose a positive client timeout.".into()); | ||
| 64 | } | ||
| 65 | let deadline = Instant::now() | ||
| 66 | .checked_add(Duration::from_millis(timeout)) | ||
| 67 | .ok_or("Client timeout exceeds the clock range.")?; | ||
| 68 | let mut output = io::stdout().lock(); | ||
| 69 | let mut log = |event: serde_json::Value| -> io::Result<()> { | ||
| 70 | writeln!(output, "{event}")?; | ||
| 71 | output.flush() | ||
| 72 | }; | ||
| 73 | log(json!({"event": "ready", "pid": std::process::id(), "actor": args[2]}))?; | ||
| 74 | while !Path::new(&args[4]).exists() { | ||
| 75 | if Instant::now() > deadline { | ||
| 76 | return Err("Start barrier timed out.".into()); | ||
| 77 | } | ||
| 78 | thread::sleep(Duration::from_millis(5)); | ||
| 79 | } | ||
| 80 | let documents = std::env::var_os("ONESTORE_OFFLINE_DOCUMENTS").is_some(); | ||
| 81 | let maintenance = std::env::var_os("ONESTORE_MAINTENANCE_DIR").map(std::path::PathBuf::from); | ||
| 82 | let mut completed = 0; | ||
| 83 | let mut attempts = 0; | ||
| 84 | while completed < operations || (args[0] == "read" && !Path::new(&args[5]).exists()) { | ||
| 85 | if Instant::now() > deadline { | ||
| 86 | return Err("Concurrent client timed out.".into()); | ||
| 87 | } | ||
| 88 | if let Some(control) = &maintenance | ||
| 89 | && !control.join("resume").exists() | ||
| 90 | && ((args[0] != "read" && completed == operations / 2) | ||
| 91 | || (args[0] == "read" && control.join("pause").exists())) | ||
| 92 | { | ||
| 93 | fs::write(control.join(format!("paused-{}", args[2])), b"paused")?; | ||
| 94 | log(json!({"event": "paused", "completed": completed}))?; | ||
| 95 | while !control.join("resume").exists() { | ||
| 96 | if Instant::now() > deadline { | ||
| 97 | return Err("Maintenance pause timed out.".into()); | ||
| 98 | } | ||
| 99 | thread::sleep(Duration::from_millis(10)); | ||
| 100 | } | ||
| 101 | log(json!({"event": "resumed", "completed": completed}))?; | ||
| 102 | } | ||
| 103 | attempts += 1; | ||
| 104 | random = random | ||
| 105 | .wrapping_mul(6364136223846793005) | ||
| 106 | .wrapping_add(1442695040888963407); | ||
| 107 | thread::sleep(Duration::from_millis((random >> 32) % 7)); | ||
| 108 | let started = SystemTime::now().duration_since(UNIX_EPOCH)?.as_micros(); | ||
| 109 | let source = match read(&args[1]) { | ||
| 110 | Ok(source) => source, | ||
| 111 | Err(error) | ||
| 112 | if [ | ||
| 113 | io::ErrorKind::WouldBlock, | ||
| 114 | io::ErrorKind::ResourceBusy, | ||
| 115 | io::ErrorKind::PermissionDenied, | ||
| 116 | io::ErrorKind::NotFound, | ||
| 117 | ] | ||
| 118 | .contains(&error.kind()) => | ||
| 119 | { | ||
| 120 | log( | ||
| 121 | json!({"event": "read_busy", "attempt": attempts, "kind": format!("{:?}", error.kind())}), | ||
| 122 | )?; | ||
| 123 | thread::sleep(Duration::from_millis(100)); | ||
| 124 | continue; | ||
| 125 | } | ||
| 126 | Err(error) => { | ||
| 127 | log( | ||
| 128 | json!({"event": "read_error", "attempt": attempts, "kind": format!("{:?}", error.kind())}), | ||
| 129 | )?; | ||
| 130 | return Err(error.into()); | ||
| 131 | } | ||
| 132 | }; | ||
| 133 | let preserve = |error: onestore::Error| { | ||
| 134 | let path = Path::new(&args[4]) | ||
| 135 | .parent() | ||
| 136 | .unwrap() | ||
| 137 | .join(format!("invalid-{}-{attempts}.one", std::process::id())); | ||
| 138 | if let Err(failure) = fs::write(&path, &source) { | ||
| 139 | eprintln!( | ||
| 140 | "Could not save invalid snapshot {}: {failure}", | ||
| 141 | path.display() | ||
| 142 | ); | ||
| 143 | } | ||
| 144 | error | ||
| 145 | }; | ||
| 146 | let store = Store::parse(&source).map_err(preserve)?; | ||
| 147 | if !store.checksum_mismatches.is_empty() { | ||
| 148 | return Err(preserve(onestore::Error { | ||
| 149 | offset: store.checksum_mismatches[0], | ||
| 150 | message: "A reader observed transaction checksum damage.", | ||
| 151 | }) | ||
| 152 | .into()); | ||
| 153 | } | ||
| 154 | let index = RevisionIndex::parse(&store).map_err(preserve)?; | ||
| 155 | index.validate_current().map_err(preserve)?; | ||
| 156 | let document = Document::parse(&index).map_err(preserve)?; | ||
| 157 | let mut targets = Vec::new(); | ||
| 158 | for (sid, page) in document.pages().map_err(preserve)? { | ||
| 159 | let space = &document.spaces[&sid]; | ||
| 160 | let revision = &space.revisions[&space.contexts[&ExGuid::default()]]; | ||
| 161 | let mut pending = vec![page]; | ||
| 162 | let mut seen = std::collections::BTreeSet::new(); | ||
| 163 | while let Some(oid) = pending.pop() { | ||
| 164 | if !seen.insert(oid) { | ||
| 165 | continue; | ||
| 166 | } | ||
| 167 | let node = &revision.nodes[&oid]; | ||
| 168 | pending.extend( | ||
| 169 | node.children | ||
| 170 | .iter() | ||
| 171 | .chain(&node.content) | ||
| 172 | .chain(&node.structure) | ||
| 173 | .copied(), | ||
| 174 | ); | ||
| 175 | if let Kind::RichText { text, .. } = &node.kind | ||
| 176 | && text.starts_with("Concurrent edits:") | ||
| 177 | { | ||
| 178 | revision.text_runs(oid).map_err(preserve)?; | ||
| 179 | targets.push((sid, oid, text)); | ||
| 180 | } | ||
| 181 | } | ||
| 182 | } | ||
| 183 | let [(sid, oid, text)] = targets.as_slice() else { | ||
| 184 | return Err(preserve(onestore::Error { | ||
| 185 | offset: 0, | ||
| 186 | message: "Expected one concurrent-edit paragraph.", | ||
| 187 | }) | ||
| 188 | .into()); | ||
| 189 | }; | ||
| 190 | let read_finished = SystemTime::now().duration_since(UNIX_EPOCH)?.as_micros(); | ||
| 191 | log( | ||
| 192 | json!({"event": "read", "attempt": attempts, "started_us": started, "finished_us": read_finished, | ||
| 193 | "transaction": store.header.transaction_count, "text": text, "documents":if documents {Some(document_view(&source).map_err(preserve)?)}else{None}}), | ||
| 194 | )?; | ||
| 195 | if args[0] == "read" { | ||
| 196 | completed += 1; | ||
| 197 | continue; | ||
| 198 | } | ||
| 199 | let token = format!(" [{}:{}]", args[2], completed); | ||
| 200 | let offset = u32::try_from(text.encode_utf16().count())?; | ||
| 201 | let mut range = offset..offset; | ||
| 202 | let mut replacement = token.clone(); | ||
| 203 | if args[0] == "edit" { | ||
| 204 | let prefix = "Concurrent edits:"; | ||
| 205 | let mut boundaries = vec![u32::try_from(prefix.encode_utf16().count())?]; | ||
| 206 | for character in text[prefix.len()..].chars() { | ||
| 207 | boundaries.push(boundaries.last().unwrap() + character.len_utf16() as u32); | ||
| 208 | } | ||
| 209 | let first = ((random >> 16) % boundaries.len() as u64) as usize; | ||
| 210 | let second = ((random >> 40) % boundaries.len() as u64) as usize; | ||
| 211 | range = boundaries[first.min(second)]..boundaries[first.max(second)]; | ||
| 212 | replacement = format!(" café 🦀{token}"); | ||
| 213 | } | ||
| 214 | log( | ||
| 215 | json!({"event": "intent", "attempt": attempts, "operation": completed, | ||
| 216 | "source_transaction": store.header.transaction_count, "before": text, | ||
| 217 | "range": [range.start, range.end], "replacement": replacement, "token": token}), | ||
| 218 | )?; | ||
| 219 | thread::sleep(Duration::from_millis((random >> 48) % 13)); | ||
| 220 | let commit_started = SystemTime::now().duration_since(UNIX_EPOCH)?.as_micros(); | ||
| 221 | let result = commit(&args[1], &source, *sid, *oid, range, &replacement); | ||
| 222 | let finished = SystemTime::now().duration_since(UNIX_EPOCH)?.as_micros(); | ||
| 223 | match result { | ||
| 224 | Ok(()) => { | ||
| 225 | log( | ||
| 226 | json!({"event": "commit", "attempt": attempts, "operation": completed, "token": token, | ||
| 227 | "started_us": commit_started, "finished_us": finished, "source_transaction": store.header.transaction_count}), | ||
| 228 | )?; | ||
| 229 | completed += 1; | ||
| 230 | } | ||
| 231 | Err(error) | ||
| 232 | if error.state == CommitState::NotCommitted | ||
| 233 | && [ | ||
| 234 | io::ErrorKind::WouldBlock, | ||
| 235 | io::ErrorKind::ResourceBusy, | ||
| 236 | io::ErrorKind::PermissionDenied, | ||
| 237 | io::ErrorKind::NotFound, | ||
| 238 | ] | ||
| 239 | .contains(&error.error.kind()) => | ||
| 240 | { | ||
| 241 | log( | ||
| 242 | json!({"event": "retry", "attempt": attempts, "started_us": commit_started, | ||
| 243 | "finished_us": finished, "kind": format!("{:?}", error.error.kind())}), | ||
| 244 | )?; | ||
| 245 | } | ||
| 246 | Err(error) => { | ||
| 247 | log( | ||
| 248 | json!({"event": "commit_error", "attempt": attempts, "operation": completed, | ||
| 249 | "token": token, "state": format!("{:?}", error.state), "kind": format!("{:?}", error.error.kind()), | ||
| 250 | "started_us": commit_started, "finished_us": finished}), | ||
| 251 | )?; | ||
| 252 | return Err(error.into()); | ||
| 253 | } | ||
| 254 | } | ||
| 255 | } | ||
| 256 | log(json!({"event": "done", "completed": completed, "attempts": attempts}))?; | ||
| 257 | Ok(()) | ||
| 258 | } | ||
crates/onestore/src/commit.rs+147-35
| ... | @@ -153,14 +153,113 @@ pub fn commit_text( | ... | @@ -153,14 +153,113 @@ pub fn commit_text( |
| 153 | range: std::ops::Range<u32>, | 153 | range: std::ops::Range<u32>, |
| 154 | replacement: &str, | 154 | replacement: &str, |
| 155 | ) -> Result<(), CommitError> { | 155 | ) -> Result<(), CommitError> { |
| 156 | let written = | 156 | PreparedEdit::text(source, space, object, range, replacement) |
| 157 | crate::replace_text(source, space, object, range, replacement).map_err(|error| { | 157 | .map_err(|error| CommitError { |
| 158 | CommitError { | 158 | state: CommitState::NotCommitted, |
| 159 | state: CommitState::NotCommitted, | 159 | error: io::Error::new(ErrorKind::InvalidData, error), |
| 160 | error: io::Error::new(ErrorKind::InvalidData, error), | 160 | })? |
| 161 | } | 161 | .commit(io) |
| 162 | } | ||
| 163 | |||
| 164 | /// An immutable writer-generated transition tied to its original snapshot. | ||
| 165 | /// Persist intended revision identities from `as_bytes` before publishing an offline edit. | ||
| 166 | /// Missing identities after native maintenance do not prove an edit was never published. | ||
| 167 | pub struct PreparedEdit<'a> { | ||
| 168 | source: &'a [u8], | ||
| 169 | written: Vec<u8>, | ||
| 170 | } | ||
| 171 | |||
| 172 | impl<'a> PreparedEdit<'a> { | ||
| 173 | /// Prepares an insertion and its dependent metadata in one revision, without I/O. | ||
| 174 | pub fn insert( | ||
| 175 | source: &'a [u8], | ||
| 176 | space: ExGuid, | ||
| 177 | insertion: &crate::Insertion, | ||
| 178 | ) -> Result<Self, crate::Error> { | ||
| 179 | Ok(Self { | ||
| 180 | source, | ||
| 181 | written: insertion.apply(source, space)?, | ||
| 182 | }) | ||
| 183 | } | ||
| 184 | |||
| 185 | /// Validates and prepares a text edit without I/O, with `replace_text` semantics. | ||
| 186 | pub fn text( | ||
| 187 | source: &'a [u8], | ||
| 188 | space: ExGuid, | ||
| 189 | object: ExGuid, | ||
| 190 | range: std::ops::Range<u32>, | ||
| 191 | replacement: &str, | ||
| 192 | ) -> Result<Self, crate::Error> { | ||
| 193 | Ok(Self { | ||
| 194 | source, | ||
| 195 | written: crate::replace_text(source, space, object, range, replacement)?, | ||
| 196 | }) | ||
| 197 | } | ||
| 198 | |||
| 199 | /// Changes character formatting over a UTF-16 range, preserving unselected runs and styles. | ||
| 200 | /// A zero-length range sets the insertion style only when the paragraph is empty. | ||
| 201 | /// Fields, associated run objects and boundaries splitting preserved run data are rejected. | ||
| 202 | pub fn format( | ||
| 203 | source: &'a [u8], | ||
| 204 | space: ExGuid, | ||
| 205 | object: ExGuid, | ||
| 206 | range: std::ops::Range<u32>, | ||
| 207 | attributes: &[crate::TextAttribute], | ||
| 208 | ) -> Result<Self, crate::Error> { | ||
| 209 | Ok(Self { | ||
| 210 | source, | ||
| 211 | written: crate::formatting::format_text(source, space, object, range, attributes)?, | ||
| 212 | }) | ||
| 213 | } | ||
| 214 | |||
| 215 | /// The exact complete image this edit will publish; identities do not regenerate on commit. | ||
| 216 | /// Do not overwrite a live notebook with this image; use `commit` under exclusion. | ||
| 217 | pub fn as_bytes(&self) -> &[u8] { | ||
| 218 | &self.written | ||
| 219 | } | ||
| 220 | |||
| 221 | /// Publishes these prepared bytes after comparing the entire original snapshot. | ||
| 222 | /// The caller must retain OneNote-compatible exclusion through the returned outcome. | ||
| 223 | pub fn commit(&self, io: &mut impl CommitIo) -> Result<(), CommitError> { | ||
| 224 | commit_bytes(io, self.source, &self.written) | ||
| 225 | } | ||
| 226 | |||
| 227 | /// Publishes these exact bytes through the conservative whole-file filesystem adapter. | ||
| 228 | /// A changed source returns ResourceBusy; an uncertain outcome must be reconciled before replay. | ||
| 229 | #[cfg(any(unix, windows))] | ||
| 230 | pub fn commit_file(&self, path: impl AsRef<Path>) -> Result<(), CommitError> { | ||
| 231 | let mut io = FileIo::open(path, true).map_err(|error| CommitError { | ||
| 232 | state: CommitState::NotCommitted, | ||
| 233 | error, | ||
| 162 | })?; | 234 | })?; |
| 163 | commit_bytes(io, source, &written) | 235 | let result = self.commit(&mut io); |
| 236 | io.finish(result) | ||
| 237 | } | ||
| 238 | } | ||
| 239 | |||
| 240 | /// Compares and flushes a snapshot, then refreshes its header version metadata. | ||
| 241 | /// No revision is added; reread before using the snapshot for another physical commit. | ||
| 242 | /// The caller must hold OneNote-compatible exclusion and independently establish which | ||
| 243 | /// intents the snapshot contains. A successful read alone is not a durable acknowledgement. | ||
| 244 | pub fn confirm_snapshot(io: &mut impl CommitIo, source: &[u8]) -> Result<(), CommitError> { | ||
| 245 | let mut state = CommitState::NotCommitted; | ||
| 246 | let result = (|| -> io::Result<()> { | ||
| 247 | let header = crate::Header::parse(source).map_err(io::Error::other)?; | ||
| 248 | let generation = header | ||
| 249 | .generation | ||
| 250 | .checked_add(1) | ||
| 251 | .ok_or(ErrorKind::InvalidData)?; | ||
| 252 | let mut version = [0; 40]; | ||
| 253 | version[..16].copy_from_slice(&crate::write::fresh_guid().map_err(io::Error::other)?); | ||
| 254 | version[16..24].copy_from_slice(&generation.to_le_bytes()); | ||
| 255 | version[24..].copy_from_slice(&crate::write::fresh_guid().map_err(io::Error::other)?); | ||
| 256 | compare_snapshot(io, source)?; | ||
| 257 | state = CommitState::Unknown; | ||
| 258 | io.flush()?; | ||
| 259 | write_all(io, 212, &version)?; | ||
| 260 | io.flush() | ||
| 261 | })(); | ||
| 262 | result.map_err(|error| CommitError { state, error }) | ||
| 164 | } | 263 | } |
| 165 | 264 | ||
| 166 | /// The caller must hold OneNote-compatible exclusion for the entire operation. | 265 | /// The caller must hold OneNote-compatible exclusion for the entire operation. |
| ... | @@ -221,7 +320,7 @@ fn write_all(io: &mut impl CommitIo, mut offset: usize, mut bytes: &[u8]) -> io: | ... | @@ -221,7 +320,7 @@ fn write_all(io: &mut impl CommitIo, mut offset: usize, mut bytes: &[u8]) -> io: |
| 221 | } | 320 | } |
| 222 | 321 | ||
| 223 | /// Publishes a scalar revision after checking the locked file against its snapshot. | 322 | /// Publishes a scalar revision after checking the locked file against its snapshot. |
| 224 | /// Unknown outcomes require rereading; Committed errors affect counter cleanup or lock release. | 323 | /// Unknown outcomes require rereading; Committed errors affect lock release. |
| 225 | pub fn commit_property_bytes( | 324 | pub fn commit_property_bytes( |
| 226 | io: &mut impl CommitIo, | 325 | io: &mut impl CommitIo, |
| 227 | source: &[u8], | 326 | source: &[u8], |
| ... | @@ -240,6 +339,40 @@ pub fn commit_property_bytes( | ... | @@ -240,6 +339,40 @@ pub fn commit_property_bytes( |
| 240 | commit_bytes(io, source, &written) | 339 | commit_bytes(io, source, &written) |
| 241 | } | 340 | } |
| 242 | 341 | ||
| 342 | fn compare_snapshot(io: &mut impl CommitIo, source: &[u8]) -> io::Result<()> { | ||
| 343 | let capacity = source.len().clamp(1, 1024 * 1024); | ||
| 344 | let mut buffer = Vec::new(); | ||
| 345 | buffer | ||
| 346 | .try_reserve_exact(capacity) | ||
| 347 | .map_err(io::Error::other)?; | ||
| 348 | buffer.resize(capacity, 0); | ||
| 349 | let mut offset = 0; | ||
| 350 | while offset < source.len() { | ||
| 351 | let size = buffer.len().min(source.len() - offset); | ||
| 352 | let count = match io.read_at(offset as u64, &mut buffer[..size]) { | ||
| 353 | Err(error) if error.kind() == ErrorKind::Interrupted => continue, | ||
| 354 | result => result?, | ||
| 355 | }; | ||
| 356 | if count == 0 { | ||
| 357 | return Err(io::Error::from(ErrorKind::UnexpectedEof)); | ||
| 358 | } | ||
| 359 | if count > size || buffer[..count] != source[offset..offset + count] { | ||
| 360 | return Err(io::Error::new( | ||
| 361 | ErrorKind::ResourceBusy, | ||
| 362 | "The locked file differs from the edit snapshot", | ||
| 363 | )); | ||
| 364 | } | ||
| 365 | offset += count; | ||
| 366 | } | ||
| 367 | if io.read_at(source.len() as u64, &mut buffer[..1])? != 0 { | ||
| 368 | return Err(io::Error::new( | ||
| 369 | ErrorKind::ResourceBusy, | ||
| 370 | "The locked file grew after the edit snapshot", | ||
| 371 | )); | ||
| 372 | } | ||
| 373 | Ok(()) | ||
| 374 | } | ||
| 375 | |||
| 243 | pub(crate) fn commit_bytes( | 376 | pub(crate) fn commit_bytes( |
| 244 | io: &mut impl CommitIo, | 377 | io: &mut impl CommitIo, |
| 245 | source: &[u8], | 378 | source: &[u8], |
| ... | @@ -247,31 +380,7 @@ pub(crate) fn commit_bytes( | ... | @@ -247,31 +380,7 @@ pub(crate) fn commit_bytes( |
| 247 | ) -> Result<(), CommitError> { | 380 | ) -> Result<(), CommitError> { |
| 248 | let mut state = CommitState::NotCommitted; | 381 | let mut state = CommitState::NotCommitted; |
| 249 | let result = (|| -> io::Result<()> { | 382 | let result = (|| -> io::Result<()> { |
| 250 | let mut buffer = [0; 65536]; | 383 | compare_snapshot(io, source)?; |
| 251 | let mut offset = 0; | ||
| 252 | while offset < source.len() { | ||
| 253 | let size = buffer.len().min(source.len() - offset); | ||
| 254 | let count = match io.read_at(offset as u64, &mut buffer[..size]) { | ||
| 255 | Err(error) if error.kind() == ErrorKind::Interrupted => continue, | ||
| 256 | result => result?, | ||
| 257 | }; | ||
| 258 | if count == 0 { | ||
| 259 | return Err(io::Error::from(ErrorKind::UnexpectedEof)); | ||
| 260 | } | ||
| 261 | if count > size || buffer[..count] != source[offset..offset + count] { | ||
| 262 | return Err(io::Error::new( | ||
| 263 | ErrorKind::ResourceBusy, | ||
| 264 | "The locked file differs from the edit snapshot", | ||
| 265 | )); | ||
| 266 | } | ||
| 267 | offset += count; | ||
| 268 | } | ||
| 269 | if io.read_at(source.len() as u64, &mut buffer[..1])? != 0 { | ||
| 270 | return Err(io::Error::new( | ||
| 271 | ErrorKind::ResourceBusy, | ||
| 272 | "The locked file grew after the edit snapshot", | ||
| 273 | )); | ||
| 274 | } | ||
| 275 | if written == source { | 384 | if written == source { |
| 276 | state = CommitState::Unknown; | 385 | state = CommitState::Unknown; |
| 277 | io.flush()?; | 386 | io.flush()?; |
| ... | @@ -291,17 +400,20 @@ pub(crate) fn commit_bytes( | ... | @@ -291,17 +400,20 @@ pub(crate) fn commit_bytes( |
| 291 | write_all(io, start, &written[start..offset])?; | 400 | write_all(io, start, &written[start..offset])?; |
| 292 | } | 401 | } |
| 293 | io.flush()?; | 402 | io.flush()?; |
| 294 | write_all(io, 100, &written[100..1024])?; | 403 | write_all(io, 100, &written[100..212])?; |
| 404 | write_all(io, 252, &written[252..1024])?; | ||
| 295 | io.flush()?; | 405 | io.flush()?; |
| 296 | let highest = (96..100).rfind(|at| source[*at] != written[*at]).unwrap(); | 406 | let highest = (96..100).rfind(|at| source[*at] != written[*at]).unwrap(); |
| 297 | state = CommitState::Unknown; | 407 | state = CommitState::Unknown; |
| 298 | write_all(io, highest, &written[highest..highest + 1])?; | 408 | write_all(io, highest, &written[highest..highest + 1])?; |
| 299 | io.flush()?; | 409 | io.flush()?; |
| 300 | state = CommitState::Committed; | ||
| 301 | if highest > 96 { | 410 | if highest > 96 { |
| 302 | write_all(io, 96, &written[96..highest])?; | 411 | write_all(io, 96, &written[96..highest])?; |
| 303 | io.flush()?; | 412 | io.flush()?; |
| 304 | } | 413 | } |
| 414 | // Native readers cache the version GUID without rechecking the transaction count. | ||
| 415 | write_all(io, 212, &written[212..252])?; | ||
| 416 | io.flush()?; | ||
| 305 | Ok(()) | 417 | Ok(()) |
| 306 | })(); | 418 | })(); |
| 307 | result.map_err(|error| CommitError { state, error }) | 419 | result.map_err(|error| CommitError { state, error }) |
crates/onestore/src/create.rs+11-7
| ... | @@ -8,7 +8,7 @@ use std::time::{SystemTime, UNIX_EPOCH}; | ... | @@ -8,7 +8,7 @@ use std::time::{SystemTime, UNIX_EPOCH}; |
| 8 | 8 | ||
| 9 | type Result<T> = std::result::Result<T, Error>; | 9 | type Result<T> = std::result::Result<T, Error>; |
| 10 | 10 | ||
| 11 | fn string(value: &str) -> Vec<u8> { | 11 | pub(crate) fn string(value: &str) -> Vec<u8> { |
| 12 | value | 12 | value |
| 13 | .encode_utf16() | 13 | .encode_utf16() |
| 14 | .chain([0]) | 14 | .chain([0]) |
| ... | @@ -16,7 +16,15 @@ fn string(value: &str) -> Vec<u8> { | ... | @@ -16,7 +16,15 @@ fn string(value: &str) -> Vec<u8> { |
| 16 | .collect() | 16 | .collect() |
| 17 | } | 17 | } |
| 18 | 18 | ||
| 19 | fn properties(values: &[(u32, Vec<u8>)]) -> Result<Vec<u8>> { | 19 | pub(crate) fn default_text_style() -> Vec<(u32, Vec<u8>)> { |
| 20 | vec![ | ||
| 21 | (0x14001c3b, 0x409_u32.to_le_bytes().to_vec()), | ||
| 22 | (0x1c001c0a, string("Calibri")), | ||
| 23 | (0x10001c0b, 22_u16.to_le_bytes().to_vec()), | ||
| 24 | ] | ||
| 25 | } | ||
| 26 | |||
| 27 | pub(crate) fn properties(values: &[(u32, Vec<u8>)]) -> Result<Vec<u8>> { | ||
| 20 | let mut streams: [Vec<u8>; 3] = std::array::from_fn(|_| Vec::new()); | 28 | let mut streams: [Vec<u8>; 3] = std::array::from_fn(|_| Vec::new()); |
| 21 | let mut fields = Vec::new(); | 29 | let mut fields = Vec::new(); |
| 22 | for (id, value) in values { | 30 | for (id, value) in values { |
| ... | @@ -239,11 +247,7 @@ pub fn create_section(file_name: &str, text: &str, author: &str) -> Result<Vec<u | ... | @@ -239,11 +247,7 @@ pub fn create_section(file_name: &str, text: &str, author: &str) -> Result<Vec<u |
| 239 | NewObject { | 247 | NewObject { |
| 240 | id: 27, | 248 | id: 27, |
| 241 | jcid: 0x12004d, | 249 | jcid: 0x12004d, |
| 242 | properties: vec![ | 250 | properties: default_text_style(), |
| 243 | (0x14001c3b, id(0x409)), | ||
| 244 | (0x1c001c0a, string("Calibri")), | ||
| 245 | (0x10001c0b, 22_u16.to_le_bytes().to_vec()), | ||
| 246 | ], | ||
| 247 | }, | 251 | }, |
| 248 | ], | 252 | ], |
| 249 | }, | 253 | }, |
crates/onestore/src/document.rs+1-1
| ... | @@ -747,7 +747,7 @@ impl<'a> Document<'a> { | ... | @@ -747,7 +747,7 @@ impl<'a> Document<'a> { |
| 747 | } | 747 | } |
| 748 | 748 | ||
| 749 | impl<'a> Element<'a> { | 749 | impl<'a> Element<'a> { |
| 750 | fn parse(object: &Object<'a>, store: &Store<'a>) -> Result<Self> { | 750 | pub(crate) fn parse(object: &Object<'a>, store: &Store<'a>) -> Result<Self> { |
| 751 | let empty = |kind| Self { | 751 | let empty = |kind| Self { |
| 752 | jcid: object.jcid, | 752 | jcid: object.jcid, |
| 753 | children: vec![], | 753 | children: vec![], |
crates/onestore/src/edit.rs+145-93
| ... | @@ -58,38 +58,7 @@ pub fn replace_text( | ... | @@ -58,38 +58,7 @@ pub fn replace_text( |
| 58 | .into_iter() | 58 | .into_iter() |
| 59 | .filter_map(|(sid, page)| (sid == space).then_some(page)) | 59 | .filter_map(|(sid, page)| (sid == space).then_some(page)) |
| 60 | .collect(); | 60 | .collect(); |
| 61 | let mut pending: Vec<_> = pages.iter().map(|page| (*page, false)).collect(); | 61 | let parents = editable_parents(revision, &pages, object)?; |
| 62 | let mut seen = std::collections::BTreeSet::new(); | ||
| 63 | let mut parents = std::collections::BTreeMap::<_, Vec<_>>::new(); | ||
| 64 | let mut editable = false; | ||
| 65 | while let Some((id, read_only)) = pending.pop() { | ||
| 66 | if !seen.insert((id, read_only)) { | ||
| 67 | continue; | ||
| 68 | } | ||
| 69 | let element = revision | ||
| 70 | .nodes | ||
| 71 | .get(&id) | ||
| 72 | .ok_or_else(|| invalid("Page content is unavailable"))?; | ||
| 73 | let read_only = read_only || element.extra[0].iter().any(|field| field.id == 0x88001cde); | ||
| 74 | if id == object { | ||
| 75 | if read_only { | ||
| 76 | return Err(invalid("This page or its content is read-only")); | ||
| 77 | } | ||
| 78 | editable = true; | ||
| 79 | } | ||
| 80 | for child in element | ||
| 81 | .children | ||
| 82 | .iter() | ||
| 83 | .chain(&element.content) | ||
| 84 | .chain(&element.structure) | ||
| 85 | { | ||
| 86 | parents.entry(*child).or_default().push(id); | ||
| 87 | pending.push((*child, read_only)); | ||
| 88 | } | ||
| 89 | } | ||
| 90 | if !editable { | ||
| 91 | return Err(invalid("Select text on an active editable page")); | ||
| 92 | } | ||
| 93 | let node = revision | 62 | let node = revision |
| 94 | .nodes | 63 | .nodes |
| 95 | .get(&object) | 64 | .get(&object) |
| ... | @@ -219,7 +188,7 @@ pub fn replace_text( | ... | @@ -219,7 +188,7 @@ pub fn replace_text( |
| 219 | let mut edits = vec![crate::write::ObjectEdit { | 188 | let mut edits = vec![crate::write::ObjectEdit { |
| 220 | object, | 189 | object, |
| 221 | updates: &updates, | 190 | updates: &updates, |
| 222 | insert, | 191 | inserts: insert.as_slice(), |
| 223 | }]; | 192 | }]; |
| 224 | // Native conflict merges can discard descendant edits when ancestor timestamps stay stale. | 193 | // Native conflict merges can discard descendant edits when ancestor timestamps stay stale. |
| 225 | let modified_update = [(0x14001d7a, modified.as_slice())]; | 194 | let modified_update = [(0x14001d7a, modified.as_slice())]; |
| ... | @@ -233,18 +202,143 @@ pub fn replace_text( | ... | @@ -233,18 +202,143 @@ pub fn replace_text( |
| 233 | edits.push(crate::write::ObjectEdit { | 202 | edits.push(crate::write::ObjectEdit { |
| 234 | object: id, | 203 | object: id, |
| 235 | updates: &modified_update, | 204 | updates: &modified_update, |
| 236 | insert: None, | 205 | inserts: &[], |
| 237 | }); | 206 | }); |
| 238 | } | 207 | } |
| 239 | pending.extend(parents.get(&id).into_iter().flatten().copied()); | 208 | pending.extend(parents.get(&id).into_iter().flatten().copied()); |
| 240 | } | 209 | } |
| 210 | let Some((page, automatic, title_text)) = | ||
| 211 | page_title(revision, &pages, Some((object, &changed)))? | ||
| 212 | else { | ||
| 213 | return crate::write::replace_objects(source, space, &edits); | ||
| 214 | }; | ||
| 215 | let Kind::Page { | ||
| 216 | alternate_title, .. | ||
| 217 | } = &revision.nodes[&page].kind | ||
| 218 | else { | ||
| 219 | unreachable!() | ||
| 220 | }; | ||
| 221 | let metadata = revision | ||
| 222 | .roots | ||
| 223 | .get(&2) | ||
| 224 | .ok_or_else(|| invalid("Page title metadata is unavailable"))?; | ||
| 225 | let Kind::Metadata { title, .. } = &revision.nodes[metadata].kind else { | ||
| 226 | return Err(invalid("Page title metadata is unavailable")); | ||
| 227 | }; | ||
| 228 | let cached: Vec<_> = title_text | ||
| 229 | .encode_utf16() | ||
| 230 | .chain([0]) | ||
| 231 | .flat_map(u16::to_le_bytes) | ||
| 232 | .collect(); | ||
| 233 | let metadata_update = [(0x1c001cf3, cached.as_slice())]; | ||
| 234 | edits.push(crate::write::ObjectEdit { | ||
| 235 | object: *metadata, | ||
| 236 | updates: if title.is_some() { | ||
| 237 | &metadata_update | ||
| 238 | } else { | ||
| 239 | &[] | ||
| 240 | }, | ||
| 241 | inserts: if title.is_none() { | ||
| 242 | &metadata_update | ||
| 243 | } else { | ||
| 244 | &[] | ||
| 245 | }, | ||
| 246 | }); | ||
| 247 | let mut alternate_update = vec![( | ||
| 248 | 0x1c001d3c, | ||
| 249 | if automatic { | ||
| 250 | cached.as_slice() | ||
| 251 | } else { | ||
| 252 | &[0u8, 0][..] | ||
| 253 | }, | ||
| 254 | )]; | ||
| 255 | if revision.nodes[&page].modified.is_some() { | ||
| 256 | alternate_update.push(modified_update[0]); | ||
| 257 | } | ||
| 258 | edits.retain(|edit| edit.object != page); | ||
| 259 | edits.push(crate::write::ObjectEdit { | ||
| 260 | object: page, | ||
| 261 | updates: if alternate_title.is_some() { | ||
| 262 | &alternate_update | ||
| 263 | } else { | ||
| 264 | &alternate_update[1..] | ||
| 265 | }, | ||
| 266 | inserts: if alternate_title.is_none() { | ||
| 267 | &alternate_update[..1] | ||
| 268 | } else { | ||
| 269 | &[] | ||
| 270 | }, | ||
| 271 | }); | ||
| 272 | crate::write::replace_objects(source, space, &edits) | ||
| 273 | } | ||
| 274 | |||
| 275 | pub(crate) fn editable_parents( | ||
| 276 | revision: &crate::document::Revision<'_>, | ||
| 277 | pages: &[ExGuid], | ||
| 278 | object: ExGuid, | ||
| 279 | ) -> Result<std::collections::BTreeMap<ExGuid, Vec<ExGuid>>, Error> { | ||
| 280 | let invalid = |message| Error { offset: 0, message }; | ||
| 281 | let mut pending: Vec<_> = pages.iter().map(|page| (*page, false)).collect(); | ||
| 282 | let mut seen = std::collections::BTreeSet::new(); | ||
| 283 | let mut parents = std::collections::BTreeMap::<_, Vec<_>>::new(); | ||
| 284 | let mut editable = false; | ||
| 285 | while let Some((id, read_only)) = pending.pop() { | ||
| 286 | if !seen.insert((id, read_only)) { | ||
| 287 | continue; | ||
| 288 | } | ||
| 289 | let element = revision | ||
| 290 | .nodes | ||
| 291 | .get(&id) | ||
| 292 | .ok_or_else(|| invalid("Page content is unavailable"))?; | ||
| 293 | let read_only = read_only || element.extra[0].iter().any(|field| field.id == 0x88001cde); | ||
| 294 | if id == object { | ||
| 295 | if read_only { | ||
| 296 | return Err(invalid("This page or its content is read-only")); | ||
| 297 | } | ||
| 298 | editable = true; | ||
| 299 | } | ||
| 300 | for child in element | ||
| 301 | .children | ||
| 302 | .iter() | ||
| 303 | .chain(&element.content) | ||
| 304 | .chain(&element.structure) | ||
| 305 | { | ||
| 306 | parents.entry(*child).or_default().push(id); | ||
| 307 | pending.push((*child, read_only)); | ||
| 308 | } | ||
| 309 | } | ||
| 310 | if !editable { | ||
| 311 | return Err(invalid("Select content on an active editable page")); | ||
| 312 | } | ||
| 313 | Ok(parents) | ||
| 314 | } | ||
| 315 | |||
| 316 | pub(crate) fn page_title( | ||
| 317 | revision: &crate::document::Revision<'_>, | ||
| 318 | pages: &[ExGuid], | ||
| 319 | text_update: Option<(ExGuid, &str)>, | ||
| 320 | ) -> Result<Option<(ExGuid, bool, String)>, Error> { | ||
| 321 | let invalid = |message| Error { offset: 0, message }; | ||
| 322 | let mut pending = pages.to_vec(); | ||
| 323 | let mut seen = std::collections::BTreeSet::new(); | ||
| 324 | while let Some(id) = pending.pop() { | ||
| 325 | if !seen.insert(id) { | ||
| 326 | continue; | ||
| 327 | } | ||
| 328 | let node = &revision.nodes[&id]; | ||
| 329 | pending.extend( | ||
| 330 | node.children | ||
| 331 | .iter() | ||
| 332 | .chain(&node.content) | ||
| 333 | .chain(&node.structure) | ||
| 334 | .copied(), | ||
| 335 | ); | ||
| 336 | } | ||
| 241 | let titles: Vec<_> = revision | 337 | let titles: Vec<_> = revision |
| 242 | .nodes | 338 | .nodes |
| 243 | .iter() | 339 | .iter() |
| 244 | .filter_map(|(id, node)| { | 340 | .filter_map(|(id, node)| { |
| 245 | if !(seen.contains(&(*id, false)) || seen.contains(&(*id, true))) | 341 | if !seen.contains(id) || !node.extra[0].iter().any(|field| field.id == 0x88001cb4) { |
| 246 | || !node.extra[0].iter().any(|field| field.id == 0x88001cb4) | ||
| 247 | { | ||
| 248 | return None; | 342 | return None; |
| 249 | } | 343 | } |
| 250 | match &node.kind { | 344 | match &node.kind { |
| ... | @@ -260,8 +354,8 @@ pub fn replace_text( | ... | @@ -260,8 +354,8 @@ pub fn replace_text( |
| 260 | let title_text = match titles.as_slice() { | 354 | let title_text = match titles.as_slice() { |
| 261 | [] => "", | 355 | [] => "", |
| 262 | [(id, text)] => { | 356 | [(id, text)] => { |
| 263 | if *id == object { | 357 | if let Some((_, changed)) = text_update.filter(|(object, _)| object == id) { |
| 264 | changed.as_str() | 358 | changed |
| 265 | } else { | 359 | } else { |
| 266 | text | 360 | text |
| 267 | } | 361 | } |
| ... | @@ -269,18 +363,13 @@ pub fn replace_text( | ... | @@ -269,18 +363,13 @@ pub fn replace_text( |
| 269 | _ => return Err(invalid("Title editing requires a single title text object")), | 363 | _ => return Err(invalid("Title editing requires a single title text object")), |
| 270 | }; | 364 | }; |
| 271 | let automatic = title_line(title_text).is_empty(); | 365 | let automatic = title_line(title_text).is_empty(); |
| 272 | if !automatic && titles[0].0 != object { | 366 | if !automatic && text_update.is_none_or(|(id, _)| titles[0].0 != id) { |
| 273 | return crate::write::replace_objects(source, space, &edits); | 367 | return Ok(None); |
| 274 | } | 368 | } |
| 275 | let [page] = pages.as_slice() else { | 369 | let [page] = pages else { |
| 276 | return Err(invalid("Title editing requires a single active page")); | 370 | return Err(invalid("Title editing requires a single active page")); |
| 277 | }; | 371 | }; |
| 278 | let Kind::Page { | 372 | let Kind::Page { rtl, .. } = &revision.nodes[page].kind else { |
| 279 | alternate_title, | ||
| 280 | rtl, | ||
| 281 | .. | ||
| 282 | } = &revision.nodes[page].kind | ||
| 283 | else { | ||
| 284 | unreachable!() | 373 | unreachable!() |
| 285 | }; | 374 | }; |
| 286 | let mut title_text = title_line(title_text); | 375 | let mut title_text = title_line(title_text); |
| ... | @@ -312,7 +401,13 @@ pub fn replace_text( | ... | @@ -312,7 +401,13 @@ pub fn replace_text( |
| 312 | .. | 401 | .. |
| 313 | } = &node.kind | 402 | } = &node.kind |
| 314 | { | 403 | { |
| 315 | title_text = automatic_title(if id == object { &changed } else { text }); | 404 | title_text = automatic_title( |
| 405 | if let Some((_, changed)) = text_update.filter(|(object, _)| *object == id) { | ||
| 406 | changed | ||
| 407 | } else { | ||
| 408 | text | ||
| 409 | }, | ||
| 410 | ); | ||
| 316 | if !title_text.is_empty() { | 411 | if !title_text.is_empty() { |
| 317 | break; | 412 | break; |
| 318 | } | 413 | } |
| ... | @@ -326,48 +421,5 @@ pub fn replace_text( | ... | @@ -326,48 +421,5 @@ pub fn replace_text( |
| 326 | pending.extend(node.structure.iter().rev().copied()); | 421 | pending.extend(node.structure.iter().rev().copied()); |
| 327 | } | 422 | } |
| 328 | } | 423 | } |
| 329 | let metadata = revision | 424 | Ok(Some((*page, automatic, title_text.to_owned()))) |
| 330 | .roots | ||
| 331 | .get(&2) | ||
| 332 | .ok_or_else(|| invalid("Page title metadata is unavailable"))?; | ||
| 333 | let Kind::Metadata { title, .. } = &revision.nodes[metadata].kind else { | ||
| 334 | return Err(invalid("Page title metadata is unavailable")); | ||
| 335 | }; | ||
| 336 | let cached: Vec<_> = title_text | ||
| 337 | .encode_utf16() | ||
| 338 | .chain([0]) | ||
| 339 | .flat_map(u16::to_le_bytes) | ||
| 340 | .collect(); | ||
| 341 | let metadata_update = [(0x1c001cf3, cached.as_slice())]; | ||
| 342 | edits.push(crate::write::ObjectEdit { | ||
| 343 | object: *metadata, | ||
| 344 | updates: if title.is_some() { | ||
| 345 | &metadata_update | ||
| 346 | } else { | ||
| 347 | &[] | ||
| 348 | }, | ||
| 349 | insert: title.is_none().then_some(metadata_update[0]), | ||
| 350 | }); | ||
| 351 | let mut alternate_update = vec![( | ||
| 352 | 0x1c001d3c, | ||
| 353 | if automatic { | ||
| 354 | cached.as_slice() | ||
| 355 | } else { | ||
| 356 | &[0u8, 0][..] | ||
| 357 | }, | ||
| 358 | )]; | ||
| 359 | if revision.nodes[page].modified.is_some() { | ||
| 360 | alternate_update.push(modified_update[0]); | ||
| 361 | } | ||
| 362 | edits.retain(|edit| edit.object != *page); | ||
| 363 | edits.push(crate::write::ObjectEdit { | ||
| 364 | object: *page, | ||
| 365 | updates: if alternate_title.is_some() { | ||
| 366 | &alternate_update | ||
| 367 | } else { | ||
| 368 | &alternate_update[1..] | ||
| 369 | }, | ||
| 370 | insert: alternate_title.is_none().then_some(alternate_update[0]), | ||
| 371 | }); | ||
| 372 | crate::write::replace_objects(source, space, &edits) | ||
| 373 | } | 425 | } |
crates/onestore/src/formatting.rs created+288
| ... | @@ -0,0 +1,288 @@ | ||
| 1 | use crate::{ | ||
| 2 | Error, ExGuid, PropertySets, RevisionIndex, Store, | ||
| 3 | create::{current_timestamps, properties, string}, | ||
| 4 | document::{Document, Kind}, | ||
| 5 | edit::editable_parents, | ||
| 6 | write::{PropertyObject, fresh_guid, write_revision}, | ||
| 7 | }; | ||
| 8 | use serde::{Deserialize, Serialize}; | ||
| 9 | use std::{ | ||
| 10 | collections::{BTreeMap, BTreeSet}, | ||
| 11 | ops::Range, | ||
| 12 | sync::Arc, | ||
| 13 | }; | ||
| 14 | |||
| 15 | fn invalid(message: &'static str) -> Error { | ||
| 16 | Error { offset: 0, message } | ||
| 17 | } | ||
| 18 | |||
| 19 | /// An explicit character-format change; omitted attributes retain their current values. | ||
| 20 | #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] | ||
| 21 | pub enum TextAttribute { | ||
| 22 | Bold(bool), | ||
| 23 | Italic(bool), | ||
| 24 | Underline(bool), | ||
| 25 | Strike(bool), | ||
| 26 | /// Enabling superscript clears subscript. | ||
| 27 | Superscript(bool), | ||
| 28 | /// Enabling subscript clears superscript. | ||
| 29 | Subscript(bool), | ||
| 30 | Font(String), | ||
| 31 | /// Points, from 6 through 130 in half-point increments. | ||
| 32 | /// OneNote 2010 clamps larger sizes despite the specification allowing 144. | ||
| 33 | FontSize(f32), | ||
| 34 | /// RGB, or None for automatic text color. | ||
| 35 | Color(Option<[u8; 3]>), | ||
| 36 | /// RGB, or None to clear highlighting. | ||
| 37 | Highlight(Option<[u8; 3]>), | ||
| 38 | } | ||
| 39 | |||
| 40 | impl TextAttribute { | ||
| 41 | fn property(&self) -> Result<(u32, Vec<u8>), Error> { | ||
| 42 | let boolean = |id, value: bool| (id | (u32::from(value) << 31), Vec::new()); | ||
| 43 | Ok(match self { | ||
| 44 | Self::Bold(value) => boolean(0x08001c04, *value), | ||
| 45 | Self::Italic(value) => boolean(0x08001c05, *value), | ||
| 46 | Self::Underline(value) => boolean(0x08001c06, *value), | ||
| 47 | Self::Strike(value) => boolean(0x08001c07, *value), | ||
| 48 | Self::Superscript(value) => boolean(0x08001c08, *value), | ||
| 49 | Self::Subscript(value) => boolean(0x08001c09, *value), | ||
| 50 | Self::Font(font) => { | ||
| 51 | if font.is_empty() || font.contains('\0') { | ||
| 52 | return Err(invalid("Font names must be nonempty and contain no NUL")); | ||
| 53 | } | ||
| 54 | (0x1c001c0a, string(font)) | ||
| 55 | } | ||
| 56 | Self::FontSize(points) => { | ||
| 57 | if !points.is_finite() | ||
| 58 | || !(6.0..=130.0).contains(points) | ||
| 59 | || (points * 2.0).fract() != 0.0 | ||
| 60 | { | ||
| 61 | return Err(invalid( | ||
| 62 | "Font size must be 6 to 130 points in half-point increments", | ||
| 63 | )); | ||
| 64 | } | ||
| 65 | (0x10001c0b, ((*points * 2.0) as u16).to_le_bytes().to_vec()) | ||
| 66 | } | ||
| 67 | Self::Color(color) | Self::Highlight(color) => ( | ||
| 68 | if matches!(self, Self::Color(_)) { | ||
| 69 | 0x14001c0c | ||
| 70 | } else { | ||
| 71 | 0x14001c0d | ||
| 72 | }, | ||
| 73 | color | ||
| 74 | .map_or(0xff000000, |[r, g, b]| u32::from_le_bytes([r, g, b, 0])) | ||
| 75 | .to_le_bytes() | ||
| 76 | .to_vec(), | ||
| 77 | ), | ||
| 78 | }) | ||
| 79 | } | ||
| 80 | } | ||
| 81 | |||
| 82 | pub(crate) fn format_text( | ||
| 83 | source: &[u8], | ||
| 84 | space: ExGuid, | ||
| 85 | object: ExGuid, | ||
| 86 | range: Range<u32>, | ||
| 87 | attributes: &[TextAttribute], | ||
| 88 | ) -> Result<Vec<u8>, Error> { | ||
| 89 | if attributes.is_empty() || range.start > range.end { | ||
| 90 | return Err(invalid( | ||
| 91 | "Select a text range and at least one formatting attribute", | ||
| 92 | )); | ||
| 93 | } | ||
| 94 | let mut values = Vec::new(); | ||
| 95 | let mut seen = BTreeSet::new(); | ||
| 96 | for attribute in attributes { | ||
| 97 | let (id, value) = attribute.property()?; | ||
| 98 | if !seen.insert(id & 0x7fffffff) { | ||
| 99 | return Err(invalid("Specify each formatting attribute once")); | ||
| 100 | } | ||
| 101 | values.push((id, value)); | ||
| 102 | } | ||
| 103 | if attributes.contains(&TextAttribute::Superscript(true)) | ||
| 104 | && attributes.contains(&TextAttribute::Subscript(true)) | ||
| 105 | { | ||
| 106 | return Err(invalid("Text cannot be both superscript and subscript")); | ||
| 107 | } | ||
| 108 | // Setting either script position clears its mutually exclusive counterpart. | ||
| 109 | for (set, opposite) in [(0x88001c08, 0x08001c09), (0x88001c09, 0x08001c08)] { | ||
| 110 | if values.iter().any(|(id, _)| *id == set) && !seen.contains(&opposite) { | ||
| 111 | values.push((opposite, Vec::new())); | ||
| 112 | } | ||
| 113 | } | ||
| 114 | let store = Store::parse(source)?; | ||
| 115 | let index = RevisionIndex::parse(&store)?; | ||
| 116 | index.validate_current()?; | ||
| 117 | let document = Document::parse(&index)?; | ||
| 118 | let semantic = document | ||
| 119 | .spaces | ||
| 120 | .get(&space) | ||
| 121 | .ok_or_else(|| invalid("The active page is unavailable"))?; | ||
| 122 | let rid = semantic.contexts[&ExGuid::default()]; | ||
| 123 | let view = &semantic.revisions[&rid]; | ||
| 124 | let pages: Vec<_> = document | ||
| 125 | .pages()? | ||
| 126 | .into_iter() | ||
| 127 | .filter_map(|(sid, page)| (sid == space).then_some(page)) | ||
| 128 | .collect(); | ||
| 129 | let parents = editable_parents(view, &pages, object)?; | ||
| 130 | let node = &view.nodes[&object]; | ||
| 131 | let Kind::RichText { | ||
| 132 | text, | ||
| 133 | runs, | ||
| 134 | boilerplate, | ||
| 135 | .. | ||
| 136 | } = &node.kind | ||
| 137 | else { | ||
| 138 | return Err(invalid("Select a rich-text object")); | ||
| 139 | }; | ||
| 140 | if *boilerplate { | ||
| 141 | return Err(invalid( | ||
| 142 | "Generated title fields cannot be formatted as ordinary text", | ||
| 143 | )); | ||
| 144 | } | ||
| 145 | let (mut valid_start, mut valid_end) = (range.start == 0, range.end == 0); | ||
| 146 | let mut offset = 0; | ||
| 147 | for character in text.chars() { | ||
| 148 | offset += character.len_utf16() as u32; | ||
| 149 | valid_start |= offset == range.start; | ||
| 150 | valid_end |= offset == range.end; | ||
| 151 | } | ||
| 152 | if !valid_start || !valid_end { | ||
| 153 | return Err(invalid( | ||
| 154 | "The format range splits a surrogate pair or exceeds the text", | ||
| 155 | )); | ||
| 156 | } | ||
| 157 | if range.is_empty() && !text.is_empty() { | ||
| 158 | return Err(invalid( | ||
| 159 | "Select characters, or an empty paragraph's insertion style", | ||
| 160 | )); | ||
| 161 | } | ||
| 162 | let resolved = view.text_runs(object)?; | ||
| 163 | let mut segments = Vec::new(); | ||
| 164 | for (i, run) in runs.iter().enumerate() { | ||
| 165 | let selected = if text.is_empty() { | ||
| 166 | true | ||
| 167 | } else { | ||
| 168 | run.start < range.end && range.start < run.end | ||
| 169 | }; | ||
| 170 | if selected { | ||
| 171 | let format = &resolved[i].format; | ||
| 172 | if [ | ||
| 173 | format.hidden, | ||
| 174 | format.hyperlink, | ||
| 175 | format.math, | ||
| 176 | format.embedded_object, | ||
| 177 | ] | ||
| 178 | .contains(&Some(true)) | ||
| 179 | || resolved[i].text.contains(['\u{fffc}', '\u{fddf}']) | ||
| 180 | { | ||
| 181 | return Err(invalid( | ||
| 182 | "This format range contains a field or embedded data", | ||
| 183 | )); | ||
| 184 | } | ||
| 185 | if run.start < range.start { | ||
| 186 | segments.push((i, range.start, false)); | ||
| 187 | } | ||
| 188 | segments.push((i, run.end.min(range.end), true)); | ||
| 189 | if range.end < run.end { | ||
| 190 | segments.push((i, run.end, false)); | ||
| 191 | } | ||
| 192 | } else { | ||
| 193 | segments.push((i, run.end, false)); | ||
| 194 | } | ||
| 195 | } | ||
| 196 | let modified = current_timestamps()?.0.to_le_bytes(); | ||
| 197 | write_revision(source, space, |raw| { | ||
| 198 | let mut target = PropertyObject::from_object(&raw.objects[&object])?; | ||
| 199 | let fields = PropertySets::parse(&target.bytes)?; | ||
| 200 | if fields.sets[0].iter().any(|p| p.id == 0x24003458) { | ||
| 201 | return Err(invalid("This text object contains associated run objects")); | ||
| 202 | } | ||
| 203 | if segments.len() != runs.len() && fields.sets[0].iter().any(|p| p.id == 0x40003499) { | ||
| 204 | return Err(invalid( | ||
| 205 | "Formatting boundaries cannot split preserved run data", | ||
| 206 | )); | ||
| 207 | } | ||
| 208 | let mut styles = BTreeMap::new(); | ||
| 209 | let mut changed = BTreeMap::new(); | ||
| 210 | let mut references = Vec::new(); | ||
| 211 | let mut ends = Vec::new(); | ||
| 212 | let mut updated = false; | ||
| 213 | let changes: Vec<_> = values | ||
| 214 | .iter() | ||
| 215 | .map(|(id, value)| (*id, value.as_slice())) | ||
| 216 | .collect(); | ||
| 217 | for &(i, end, selected) in &segments { | ||
| 218 | let previous = runs[i].format; | ||
| 219 | let key = (previous, selected); | ||
| 220 | let id = if let Some(id) = styles.get(&key) { | ||
| 221 | *id | ||
| 222 | } else if let Some(id) = previous.filter(|_| !selected) { | ||
| 223 | id | ||
| 224 | } else { | ||
| 225 | let mut style = match previous { | ||
| 226 | Some(id) => PropertyObject::from_object(&raw.objects[&id])?, | ||
| 227 | None => PropertyObject { | ||
| 228 | jcid: 0x12004d, | ||
| 229 | bytes: properties(&[])?, | ||
| 230 | global_ids: Arc::new(BTreeMap::new()), | ||
| 231 | }, | ||
| 232 | }; | ||
| 233 | if selected { | ||
| 234 | style.set(&changes)?; | ||
| 235 | } | ||
| 236 | if let Some(id) = previous.filter(|id| { | ||
| 237 | raw.objects[id].data == crate::ObjectData::Properties(&style.bytes) | ||
| 238 | }) { | ||
| 239 | styles.insert(key, id); | ||
| 240 | id | ||
| 241 | } else { | ||
| 242 | if !PropertySets::parse(&style.bytes)?.sets[0] | ||
| 243 | .iter() | ||
| 244 | .any(|p| p.id == 0x14001c3b) | ||
| 245 | { | ||
| 246 | style.set(&[( | ||
| 247 | 0x14001c3b, | ||
| 248 | &resolved[i].format.language.unwrap_or(0x409).to_le_bytes(), | ||
| 249 | )])?; | ||
| 250 | } | ||
| 251 | let id = ExGuid { | ||
| 252 | guid: fresh_guid()?, | ||
| 253 | n: 1, | ||
| 254 | }; | ||
| 255 | style.reference(id)?; | ||
| 256 | changed.insert(id, style); | ||
| 257 | styles.insert(key, id); | ||
| 258 | id | ||
| 259 | } | ||
| 260 | }; | ||
| 261 | updated |= selected && previous != Some(id); | ||
| 262 | references.extend_from_slice(&target.reference(id)?); | ||
| 263 | ends.extend_from_slice(&end.to_le_bytes()); | ||
| 264 | } | ||
| 265 | if !updated { | ||
| 266 | return Ok(BTreeMap::new()); | ||
| 267 | } | ||
| 268 | ends.truncate(ends.len() - 4); | ||
| 269 | target.set(&[ | ||
| 270 | (0x24001e13, &references), | ||
| 271 | (0x1c001e12, &ends), | ||
| 272 | (0x14001d7a, &modified), | ||
| 273 | ])?; | ||
| 274 | changed.insert(object, target); | ||
| 275 | let mut pending = parents.get(&object).cloned().unwrap_or_default(); | ||
| 276 | let mut ancestors = BTreeSet::new(); | ||
| 277 | while let Some(id) = pending.pop() { | ||
| 278 | if !ancestors.insert(id) { | ||
| 279 | continue; | ||
| 280 | } | ||
| 281 | let mut ancestor = PropertyObject::from_object(&raw.objects[&id])?; | ||
| 282 | ancestor.set(&[(0x14001d7a, &modified)])?; | ||
| 283 | changed.insert(id, ancestor); | ||
| 284 | pending.extend(parents.get(&id).into_iter().flatten().copied()); | ||
| 285 | } | ||
| 286 | Ok(changed) | ||
| 287 | }) | ||
| 288 | } | ||
crates/onestore/src/insertion.rs created+343
| ... | @@ -0,0 +1,343 @@ | ||
| 1 | use crate::{ | ||
| 2 | Error, ExGuid, Object, ObjectData, RevisionIndex, Store, | ||
| 3 | create::{current_timestamps, default_text_style, properties, string}, | ||
| 4 | document::{Document, Element, Kind}, | ||
| 5 | edit::{editable_parents, page_title}, | ||
| 6 | write::{PropertyObject, fresh_guid, write_revision}, | ||
| 7 | }; | ||
| 8 | use serde::{Deserialize, Serialize}; | ||
| 9 | use std::{ | ||
| 10 | collections::{BTreeMap, BTreeSet}, | ||
| 11 | sync::Arc, | ||
| 12 | }; | ||
| 13 | |||
| 14 | fn invalid(message: &'static str) -> Error { | ||
| 15 | Error { offset: 0, message } | ||
| 16 | } | ||
| 17 | |||
| 18 | #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] | ||
| 19 | #[serde(deny_unknown_fields)] | ||
| 20 | enum Placement { | ||
| 21 | Paragraph { before: Option<ExGuid> }, | ||
| 22 | Outline { x: f32, y: f32 }, | ||
| 23 | } | ||
| 24 | |||
| 25 | /// A paragraph or outline insertion with stable object identities and creation time. | ||
| 26 | /// Retain this intent across rebases; constructing another intent allocates different identities. | ||
| 27 | #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] | ||
| 28 | #[serde(deny_unknown_fields)] | ||
| 29 | pub struct Insertion { | ||
| 30 | guid: [u8; 16], | ||
| 31 | parent: ExGuid, | ||
| 32 | placement: Placement, | ||
| 33 | text: String, | ||
| 34 | author: String, | ||
| 35 | created: u32, | ||
| 36 | } | ||
| 37 | |||
| 38 | impl Insertion { | ||
| 39 | /// Inserts before a direct child, or appends when `before` is None. | ||
| 40 | /// The parent must be an editable outline, paragraph, outline group or table cell. | ||
| 41 | /// Carriage returns represent soft line breaks; line feeds and embedded-field markers are rejected. | ||
| 42 | pub fn paragraph( | ||
| 43 | parent: ExGuid, | ||
| 44 | before: Option<ExGuid>, | ||
| 45 | text: &str, | ||
| 46 | author: &str, | ||
| 47 | ) -> Result<Self, Error> { | ||
| 48 | Self::new(parent, Placement::Paragraph { before }, text, author) | ||
| 49 | } | ||
| 50 | |||
| 51 | /// Adds an outline to an editable page at coordinates measured in points. | ||
| 52 | pub fn outline(page: ExGuid, x: f32, y: f32, text: &str, author: &str) -> Result<Self, Error> { | ||
| 53 | Self::new(page, Placement::Outline { x, y }, text, author) | ||
| 54 | } | ||
| 55 | |||
| 56 | /// Changes a paragraph intent's placement while retaining its identities, text and author. | ||
| 57 | pub fn reposition_paragraph( | ||
| 58 | &self, | ||
| 59 | parent: ExGuid, | ||
| 60 | before: Option<ExGuid>, | ||
| 61 | ) -> Result<Self, Error> { | ||
| 62 | if !matches!(self.placement, Placement::Paragraph { .. }) { | ||
| 63 | return Err(invalid("An outline intent cannot become a paragraph")); | ||
| 64 | } | ||
| 65 | let mut intent = self.clone(); | ||
| 66 | intent.parent = parent; | ||
| 67 | intent.placement = Placement::Paragraph { before }; | ||
| 68 | intent.validate()?; | ||
| 69 | Ok(intent) | ||
| 70 | } | ||
| 71 | |||
| 72 | /// Changes an outline intent's placement while retaining its identities, text and author. | ||
| 73 | pub fn reposition_outline(&self, page: ExGuid, x: f32, y: f32) -> Result<Self, Error> { | ||
| 74 | if !matches!(self.placement, Placement::Outline { .. }) { | ||
| 75 | return Err(invalid("A paragraph intent cannot become an outline")); | ||
| 76 | } | ||
| 77 | let mut intent = self.clone(); | ||
| 78 | intent.parent = page; | ||
| 79 | intent.placement = Placement::Outline { x, y }; | ||
| 80 | intent.validate()?; | ||
| 81 | Ok(intent) | ||
| 82 | } | ||
| 83 | |||
| 84 | fn new(parent: ExGuid, placement: Placement, text: &str, author: &str) -> Result<Self, Error> { | ||
| 85 | let intent = Self { | ||
| 86 | guid: fresh_guid()?, | ||
| 87 | parent, | ||
| 88 | placement, | ||
| 89 | text: text.to_owned(), | ||
| 90 | author: author.to_owned(), | ||
| 91 | created: current_timestamps()?.0, | ||
| 92 | }; | ||
| 93 | intent.validate()?; | ||
| 94 | Ok(intent) | ||
| 95 | } | ||
| 96 | |||
| 97 | /// Identity of the new paragraph, or the new outline for an outline insertion. | ||
| 98 | pub fn object(&self) -> ExGuid { | ||
| 99 | ExGuid { | ||
| 100 | guid: self.guid, | ||
| 101 | n: 1, | ||
| 102 | } | ||
| 103 | } | ||
| 104 | |||
| 105 | /// Identity of the insertion's ordinary rich-text object. | ||
| 106 | pub fn text_object(&self) -> ExGuid { | ||
| 107 | ExGuid { | ||
| 108 | guid: self.guid, | ||
| 109 | n: 2, | ||
| 110 | } | ||
| 111 | } | ||
| 112 | |||
| 113 | fn validate(&self) -> Result<(), Error> { | ||
| 114 | if self.guid == [0; 16] | ||
| 115 | || self.parent.guid == [0; 16] | ||
| 116 | || self.text.contains(['\0', '\n', '\u{fffc}', '\u{fddf}']) | ||
| 117 | || self.author.contains('\0') | ||
| 118 | { | ||
| 119 | return Err(invalid( | ||
| 120 | "Use an editable parent, ordinary paragraph text and a valid author", | ||
| 121 | )); | ||
| 122 | } | ||
| 123 | if let Placement::Outline { x, y } = self.placement | ||
| 124 | && (!x.is_finite() || !y.is_finite()) | ||
| 125 | { | ||
| 126 | return Err(invalid("Outline coordinates must be finite")); | ||
| 127 | } | ||
| 128 | Ok(()) | ||
| 129 | } | ||
| 130 | |||
| 131 | pub(crate) fn apply(&self, source: &[u8], space: ExGuid) -> Result<Vec<u8>, Error> { | ||
| 132 | self.validate()?; | ||
| 133 | let store = Store::parse(source)?; | ||
| 134 | let index = RevisionIndex::parse(&store)?; | ||
| 135 | index.validate_current()?; | ||
| 136 | let mut document = Document::parse(&index)?; | ||
| 137 | let pages: Vec<_> = document | ||
| 138 | .pages()? | ||
| 139 | .into_iter() | ||
| 140 | .filter_map(|(sid, id)| (sid == space).then_some(id)) | ||
| 141 | .collect(); | ||
| 142 | let [page] = pages.as_slice() else { | ||
| 143 | return Err(invalid("Insertion requires a single active page")); | ||
| 144 | }; | ||
| 145 | let semantic_space = document | ||
| 146 | .spaces | ||
| 147 | .remove(&space) | ||
| 148 | .ok_or_else(|| invalid("The active page is unavailable"))?; | ||
| 149 | let rid = semantic_space.contexts[&ExGuid::default()]; | ||
| 150 | let mut view = semantic_space | ||
| 151 | .revisions | ||
| 152 | .into_iter() | ||
| 153 | .find_map(|(id, revision)| (id == rid).then_some(revision)) | ||
| 154 | .unwrap(); | ||
| 155 | let parents = editable_parents(&view, &pages, self.parent)?; | ||
| 156 | let parent = &view.nodes[&self.parent]; | ||
| 157 | let position = match self.placement { | ||
| 158 | Placement::Paragraph { before } => { | ||
| 159 | if !matches!( | ||
| 160 | parent.kind, | ||
| 161 | Kind::Outline { .. } | ||
| 162 | | Kind::Paragraph { .. } | ||
| 163 | | Kind::OutlineGroup | ||
| 164 | | Kind::Cell { .. } | ||
| 165 | ) { | ||
| 166 | return Err(invalid( | ||
| 167 | "Select an outline, paragraph, outline group or table cell", | ||
| 168 | )); | ||
| 169 | } | ||
| 170 | if let Some(id) = before { | ||
| 171 | parent | ||
| 172 | .children | ||
| 173 | .iter() | ||
| 174 | .position(|child| *child == id) | ||
| 175 | .ok_or_else(|| { | ||
| 176 | invalid("The insertion anchor is no longer a direct child") | ||
| 177 | })? | ||
| 178 | } else { | ||
| 179 | parent.children.len() | ||
| 180 | } | ||
| 181 | } | ||
| 182 | Placement::Outline { .. } => { | ||
| 183 | if self.parent != *page || !matches!(parent.kind, Kind::Page { .. }) { | ||
| 184 | return Err(invalid("Select an active page for the new outline")); | ||
| 185 | } | ||
| 186 | parent.children.len() | ||
| 187 | } | ||
| 188 | }; | ||
| 189 | let mut ancestors = BTreeSet::new(); | ||
| 190 | let mut pending = vec![self.parent]; | ||
| 191 | while let Some(id) = pending.pop() { | ||
| 192 | if !ancestors.insert(id) { | ||
| 193 | continue; | ||
| 194 | } | ||
| 195 | if matches!(view.nodes[&id].kind, Kind::Title) { | ||
| 196 | return Err(invalid( | ||
| 197 | "Title containers do not accept ordinary paragraphs", | ||
| 198 | )); | ||
| 199 | } | ||
| 200 | pending.extend(parents.get(&id).into_iter().flatten().copied()); | ||
| 201 | } | ||
| 202 | let modified = current_timestamps()?.0.to_le_bytes(); | ||
| 203 | let paragraph_n = if matches!(self.placement, Placement::Outline { .. }) { | ||
| 204 | 3 | ||
| 205 | } else { | ||
| 206 | 1 | ||
| 207 | }; | ||
| 208 | let table = Arc::new(BTreeMap::from([(0, self.guid)])); | ||
| 209 | let reference = |n: u32| n.to_le_bytes().to_vec(); | ||
| 210 | let mut new = BTreeMap::new(); | ||
| 211 | for (n, jcid, values) in [ | ||
| 212 | ( | ||
| 213 | paragraph_n, | ||
| 214 | 0x6000d, | ||
| 215 | vec![ | ||
| 216 | (0x14001d7a, modified.to_vec()), | ||
| 217 | (0x14001d09, self.created.to_le_bytes().to_vec()), | ||
| 218 | (0x0c001c03, vec![1]), | ||
| 219 | (0x24001c1f, reference(2)), | ||
| 220 | (0x20001d78, reference(4)), | ||
| 221 | (0x20001d79, reference(4)), | ||
| 222 | ], | ||
| 223 | ), | ||
| 224 | ( | ||
| 225 | 2, | ||
| 226 | 0x6000e, | ||
| 227 | vec![ | ||
| 228 | (0x14001d7a, modified.to_vec()), | ||
| 229 | (0x1c001c22, string(&self.text)), | ||
| 230 | (0x24001e13, reference(5)), | ||
| 231 | (0x10001cfe, 0x409_u16.to_le_bytes().to_vec()), | ||
| 232 | ], | ||
| 233 | ), | ||
| 234 | (4, 0x120001, vec![(0x1c001d75, string(&self.author))]), | ||
| 235 | (5, 0x12004d, default_text_style()), | ||
| 236 | ] { | ||
| 237 | new.insert( | ||
| 238 | ExGuid { guid: self.guid, n }, | ||
| 239 | PropertyObject { | ||
| 240 | jcid, | ||
| 241 | bytes: properties(&values)?, | ||
| 242 | global_ids: Arc::clone(&table), | ||
| 243 | }, | ||
| 244 | ); | ||
| 245 | } | ||
| 246 | if let Placement::Outline { x, y } = self.placement { | ||
| 247 | new.insert( | ||
| 248 | self.object(), | ||
| 249 | PropertyObject { | ||
| 250 | jcid: 0x6000c, | ||
| 251 | global_ids: table, | ||
| 252 | bytes: properties(&[ | ||
| 253 | (0x14001d7a, modified.to_vec()), | ||
| 254 | (0x24001c20, reference(3)), | ||
| 255 | (0x0c001c03, vec![1]), | ||
| 256 | (0x1c001c12, vec![1, 0, 0, 0, 0, 0, 0, 0]), | ||
| 257 | (0x14001c14, (x / 36.0).to_le_bytes().to_vec()), | ||
| 258 | (0x14001c15, (y / 36.0).to_le_bytes().to_vec()), | ||
| 259 | (0x14001c1b, 13_f32.to_le_bytes().to_vec()), | ||
| 260 | (0x14001c1c, 0.6_f32.to_le_bytes().to_vec()), | ||
| 261 | ])?, | ||
| 262 | }, | ||
| 263 | ); | ||
| 264 | } | ||
| 265 | let raw = index.resolve(space, rid)?; | ||
| 266 | if new.keys().any(|id| raw.objects.contains_key(id)) { | ||
| 267 | return Err(invalid( | ||
| 268 | "An insertion identity is already present; reconcile the existing edit", | ||
| 269 | )); | ||
| 270 | } | ||
| 271 | // Drop the semantic view before moving the property bytes it borrows. | ||
| 272 | let title = { | ||
| 273 | view.nodes | ||
| 274 | .get_mut(&self.parent) | ||
| 275 | .unwrap() | ||
| 276 | .children | ||
| 277 | .insert(position, self.object()); | ||
| 278 | for (id, object) in &new { | ||
| 279 | view.nodes.insert( | ||
| 280 | *id, | ||
| 281 | Element::parse( | ||
| 282 | &Object { | ||
| 283 | jcid: object.jcid, | ||
| 284 | reference_count: 0, | ||
| 285 | data: ObjectData::Properties(&object.bytes), | ||
| 286 | global_ids: Arc::clone(&object.global_ids), | ||
| 287 | }, | ||
| 288 | &store, | ||
| 289 | )?, | ||
| 290 | ); | ||
| 291 | } | ||
| 292 | let title = page_title(&view, &pages, None)?; | ||
| 293 | drop(view); | ||
| 294 | title | ||
| 295 | }; | ||
| 296 | write_revision(source, space, |raw| { | ||
| 297 | let mut changed = new; | ||
| 298 | for id in &ancestors { | ||
| 299 | let mut object = PropertyObject::from_object(&raw.objects[id])?; | ||
| 300 | object.set(&[(0x14001d7a, &modified)])?; | ||
| 301 | changed.insert(*id, object); | ||
| 302 | } | ||
| 303 | let parent = changed.get_mut(&self.parent).unwrap(); | ||
| 304 | let properties = crate::PropertySets::parse(&parent.bytes)?; | ||
| 305 | let existing = properties.sets[0].iter().find(|p| p.id == 0x24001c20); | ||
| 306 | let mut ids = match existing.map(|p| &p.value) { | ||
| 307 | Some(crate::Value::References { compact_ids, .. }) => compact_ids.to_vec(), | ||
| 308 | None => Vec::new(), | ||
| 309 | _ => return Err(invalid("The parent has an invalid child list")), | ||
| 310 | }; | ||
| 311 | let child = parent.reference(self.object())?; | ||
| 312 | if position > ids.len() / 4 { | ||
| 313 | return Err(invalid("The parent has an invalid child list")); | ||
| 314 | } | ||
| 315 | ids.splice(position * 4..position * 4, child); | ||
| 316 | parent.set(&[(0x24001c20, &ids)])?; | ||
| 317 | if matches!(self.placement, Placement::Paragraph { .. }) { | ||
| 318 | let properties = crate::PropertySets::parse(&parent.bytes)?; | ||
| 319 | if !properties.sets[0].iter().any(|p| p.id == 0x0c001c03) { | ||
| 320 | parent.set(&[(0x0c001c03, &[1])])?; | ||
| 321 | } | ||
| 322 | } | ||
| 323 | if let Some((page, automatic, title)) = title { | ||
| 324 | let metadata = raw | ||
| 325 | .roots | ||
| 326 | .get(&2) | ||
| 327 | .ok_or_else(|| invalid("Page title metadata is unavailable"))?; | ||
| 328 | if raw.objects[metadata].jcid != 0x20030 { | ||
| 329 | return Err(invalid("Page title metadata is unavailable")); | ||
| 330 | } | ||
| 331 | let title = string(&title); | ||
| 332 | let mut metadata_object = PropertyObject::from_object(&raw.objects[metadata])?; | ||
| 333 | metadata_object.set(&[(0x1c001cf3, &title)])?; | ||
| 334 | changed.insert(*metadata, metadata_object); | ||
| 335 | changed | ||
| 336 | .get_mut(&page) | ||
| 337 | .unwrap() | ||
| 338 | .set(&[(0x1c001d3c, if automatic { &title } else { &[0, 0] })])?; | ||
| 339 | } | ||
| 340 | Ok(changed) | ||
| 341 | }) | ||
| 342 | } | ||
| 343 | } | ||
crates/onestore/src/lib.rs+11-2
| ... | @@ -1,5 +1,5 @@ | ... | @@ -1,5 +1,5 @@ |
| 1 | #![forbid(unsafe_code)] | 1 | #![forbid(unsafe_code)] |
| 2 | #![doc = include_str!("../../../README.md")] | 2 | #![doc = include_str!("../README.md")] |
| 3 | 3 | ||
| 4 | mod bytes; | 4 | mod bytes; |
| 5 | mod commit; | 5 | mod commit; |
| ... | @@ -8,20 +8,29 @@ pub mod document; | ... | @@ -8,20 +8,29 @@ pub mod document; |
| 8 | mod edit; | 8 | mod edit; |
| 9 | mod files; | 9 | mod files; |
| 10 | mod flush; | 10 | mod flush; |
| 11 | mod formatting; | ||
| 12 | mod insertion; | ||
| 11 | mod objects; | 13 | mod objects; |
| 12 | mod properties; | 14 | mod properties; |
| 13 | mod revisions; | 15 | mod revisions; |
| 16 | mod snapshot; | ||
| 14 | mod store; | 17 | mod store; |
| 15 | mod write; | 18 | mod write; |
| 16 | 19 | ||
| 17 | pub use commit::{CommitError, CommitIo, CommitState, commit_property_bytes, commit_text}; | 20 | pub use commit::{ |
| 21 | CommitError, CommitIo, CommitState, PreparedEdit, commit_property_bytes, commit_text, | ||
| 22 | confirm_snapshot, | ||
| 23 | }; | ||
| 18 | #[cfg(any(unix, windows))] | 24 | #[cfg(any(unix, windows))] |
| 19 | pub use commit::{commit_file_property, commit_file_text, read_file}; | 25 | pub use commit::{commit_file_property, commit_file_text, read_file}; |
| 20 | pub use create::{create_section, create_table_of_contents}; | 26 | pub use create::{create_section, create_table_of_contents}; |
| 21 | pub use edit::replace_text; | 27 | pub use edit::replace_text; |
| 22 | pub use files::FileDataReference; | 28 | pub use files::FileDataReference; |
| 29 | pub use formatting::TextAttribute; | ||
| 30 | pub use insertion::Insertion; | ||
| 23 | pub use objects::{Object, ObjectData, ObjectReferences, ResolvedRevision}; | 31 | pub use objects::{Object, ObjectData, ObjectReferences, ResolvedRevision}; |
| 24 | pub use properties::{IdStream, Property, PropertySets, Value}; | 32 | pub use properties::{IdStream, Property, PropertySets, Value}; |
| 25 | pub use revisions::{ExGuid, ObjectSpace, Revision, RevisionIndex}; | 33 | pub use revisions::{ExGuid, ObjectSpace, Revision, RevisionIndex}; |
| 34 | pub use snapshot::read_snapshot; | ||
| 26 | pub use store::{Chunk, Error, FileType, Header, Node, NodeList, Reference, Store}; | 35 | pub use store::{Chunk, Error, FileType, Header, Node, NodeList, Reference, Store}; |
| 27 | pub use write::replace_property_bytes; | 36 | pub use write::replace_property_bytes; |
crates/onestore/src/objects.rs+14-9
| ... | @@ -81,6 +81,19 @@ impl Object<'_> { | ... | @@ -81,6 +81,19 @@ impl Object<'_> { |
| 81 | 81 | ||
| 82 | impl ResolvedRevision<'_> { | 82 | impl ResolvedRevision<'_> { |
| 83 | pub fn reachable(&self) -> Result<BTreeSet<ExGuid>> { | 83 | pub fn reachable(&self) -> Result<BTreeSet<ExGuid>> { |
| 84 | let incoming = self.reference_counts()?; | ||
| 85 | for (id, count) in &incoming { | ||
| 86 | if self.objects[id].reference_count != *count { | ||
| 87 | return Err(Error { | ||
| 88 | offset: 0, | ||
| 89 | message: "Stored object reference count disagrees with the reachable graph", | ||
| 90 | }); | ||
| 91 | } | ||
| 92 | } | ||
| 93 | Ok(incoming.into_keys().collect()) | ||
| 94 | } | ||
| 95 | |||
| 96 | pub(crate) fn reference_counts(&self) -> Result<BTreeMap<ExGuid, u32>> { | ||
| 84 | let mut pending: Vec<_> = self.roots.values().copied().collect(); | 97 | let mut pending: Vec<_> = self.roots.values().copied().collect(); |
| 85 | let mut incoming = BTreeMap::<ExGuid, u32>::new(); | 98 | let mut incoming = BTreeMap::<ExGuid, u32>::new(); |
| 86 | for id in &pending { | 99 | for id in &pending { |
| ... | @@ -137,15 +150,7 @@ impl ResolvedRevision<'_> { | ... | @@ -137,15 +150,7 @@ impl ResolvedRevision<'_> { |
| 137 | message: "Object references form a cycle", | 150 | message: "Object references form a cycle", |
| 138 | }); | 151 | }); |
| 139 | } | 152 | } |
| 140 | for (id, count) in &incoming { | 153 | Ok(incoming) |
| 141 | if self.objects[id].reference_count != *count { | ||
| 142 | return Err(Error { | ||
| 143 | offset: 0, | ||
| 144 | message: "Stored object reference count disagrees with the reachable graph", | ||
| 145 | }); | ||
| 146 | } | ||
| 147 | } | ||
| 148 | Ok(edges.into_keys().collect()) | ||
| 149 | } | 154 | } |
| 150 | } | 155 | } |
| 151 | 156 |
crates/onestore/src/revisions.rs+44
| ... | @@ -5,10 +5,45 @@ type Result<T> = std::result::Result<T, Error>; | ... | @@ -5,10 +5,45 @@ type Result<T> = std::result::Result<T, Error>; |
| 5 | 5 | ||
| 6 | #[derive(Debug, Default, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)] | 6 | #[derive(Debug, Default, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)] |
| 7 | pub struct ExGuid { | 7 | pub struct ExGuid { |
| 8 | /// GUID bytes in Microsoft's mixed-endian order. | ||
| 8 | pub guid: [u8; 16], | 9 | pub guid: [u8; 16], |
| 9 | pub n: u32, | 10 | pub n: u32, |
| 10 | } | 11 | } |
| 11 | 12 | ||
| 13 | impl std::str::FromStr for ExGuid { | ||
| 14 | type Err = Error; | ||
| 15 | |||
| 16 | /// Parses the display form, accepting either hexadecimal letter case. | ||
| 17 | fn from_str(value: &str) -> Result<Self> { | ||
| 18 | let invalid = || Error { | ||
| 19 | offset: 0, | ||
| 20 | message: "Invalid extended GUID", | ||
| 21 | }; | ||
| 22 | let (guid_text, extension) = value.split_once(',').ok_or_else(invalid)?; | ||
| 23 | if guid_text.len() != 38 || !guid_text.is_ascii() || !(40..=49).contains(&value.len()) { | ||
| 24 | return Err(invalid()); | ||
| 25 | } | ||
| 26 | let mut guid = [0; 16]; | ||
| 27 | for (byte, at) in guid | ||
| 28 | .iter_mut() | ||
| 29 | .zip([1, 3, 5, 7, 10, 12, 15, 17, 20, 22, 25, 27, 29, 31, 33, 35]) | ||
| 30 | { | ||
| 31 | *byte = u8::from_str_radix(&guid_text[at..at + 2], 16).map_err(|_| invalid())?; | ||
| 32 | } | ||
| 33 | guid[..4].reverse(); | ||
| 34 | guid[4..6].reverse(); | ||
| 35 | guid[6..8].reverse(); | ||
| 36 | let id = Self { | ||
| 37 | guid, | ||
| 38 | n: extension.parse().map_err(|_| invalid())?, | ||
| 39 | }; | ||
| 40 | if (id.guid == [0; 16] && id.n != 0) || !id.to_string().eq_ignore_ascii_case(value) { | ||
| 41 | return Err(invalid()); | ||
| 42 | } | ||
| 43 | Ok(id) | ||
| 44 | } | ||
| 45 | } | ||
| 46 | |||
| 12 | impl fmt::Display for ExGuid { | 47 | impl fmt::Display for ExGuid { |
| 13 | fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { | 48 | fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { |
| 14 | let g = &self.guid; | 49 | let g = &self.guid; |
| ... | @@ -40,6 +75,15 @@ impl serde::Serialize for ExGuid { | ... | @@ -40,6 +75,15 @@ impl serde::Serialize for ExGuid { |
| 40 | } | 75 | } |
| 41 | } | 76 | } |
| 42 | 77 | ||
| 78 | impl<'de> serde::Deserialize<'de> for ExGuid { | ||
| 79 | fn deserialize<D: serde::Deserializer<'de>>( | ||
| 80 | deserializer: D, | ||
| 81 | ) -> std::result::Result<Self, D::Error> { | ||
| 82 | let value = <String as serde::Deserialize>::deserialize(deserializer)?; | ||
| 83 | value.parse().map_err(serde::de::Error::custom) | ||
| 84 | } | ||
| 85 | } | ||
| 86 | |||
| 43 | impl Cursor<'_> { | 87 | impl Cursor<'_> { |
| 44 | pub(crate) fn exguid(&mut self) -> Result<ExGuid> { | 88 | pub(crate) fn exguid(&mut self) -> Result<ExGuid> { |
| 45 | let id = ExGuid { | 89 | let id = ExGuid { |
crates/onestore/src/snapshot.rs created+76
| ... | @@ -0,0 +1,76 @@ | ||
| 1 | use crate::{RevisionIndex, Store}; | ||
| 2 | use std::io; | ||
| 3 | |||
| 4 | fn read_exact( | ||
| 5 | read: &mut impl FnMut(u64, &mut [u8]) -> io::Result<usize>, | ||
| 6 | mut offset: u64, | ||
| 7 | mut output: &mut [u8], | ||
| 8 | ) -> io::Result<()> { | ||
| 9 | while !output.is_empty() { | ||
| 10 | match read(offset, output) { | ||
| 11 | Ok(0) => return Err(io::ErrorKind::UnexpectedEof.into()), | ||
| 12 | Ok(count) if count <= output.len() => { | ||
| 13 | offset += count as u64; | ||
| 14 | output = &mut output[count..]; | ||
| 15 | } | ||
| 16 | Ok(_) => return Err(io::ErrorKind::InvalidData.into()), | ||
| 17 | Err(error) if error.kind() == io::ErrorKind::Interrupted => {} | ||
| 18 | Err(error) => return Err(error), | ||
| 19 | } | ||
| 20 | } | ||
| 21 | Ok(()) | ||
| 22 | } | ||
| 23 | |||
| 24 | /// Reads a bounded snapshot; the caller must provide fresh I/O and exclude in-place maintenance. | ||
| 25 | /// Includes unpublished trailing bytes so subsequent commits can validate the physical file. | ||
| 26 | pub fn read_snapshot( | ||
| 27 | mut read: impl FnMut(u64, &mut [u8]) -> io::Result<usize>, | ||
| 28 | limit: usize, | ||
| 29 | ) -> io::Result<Option<Vec<u8>>> { | ||
| 30 | let mut header = [0; 1024]; | ||
| 31 | read_exact(&mut read, 0, &mut header)?; | ||
| 32 | let length = u64::from_le_bytes(header[196..204].try_into().unwrap()); | ||
| 33 | let length = usize::try_from(length) | ||
| 34 | .ok() | ||
| 35 | .filter(|length| (1024..=limit).contains(length)) | ||
| 36 | .ok_or(io::ErrorKind::InvalidData)?; | ||
| 37 | let mut bytes = Vec::new(); | ||
| 38 | bytes.try_reserve_exact(length).map_err(io::Error::other)?; | ||
| 39 | bytes.extend_from_slice(&header); | ||
| 40 | bytes.resize(length, 0); | ||
| 41 | if let Err(error) = read_exact(&mut read, 1024, &mut bytes[1024..]) { | ||
| 42 | // Native writers can shorten unused storage before publishing the new expected length. | ||
| 43 | return if error.kind() == io::ErrorKind::UnexpectedEof { | ||
| 44 | Ok(None) | ||
| 45 | } else { | ||
| 46 | Err(error) | ||
| 47 | }; | ||
| 48 | } | ||
| 49 | let mut tail = [0; 65536]; | ||
| 50 | loop { | ||
| 51 | let size = (limit - bytes.len()).clamp(1, tail.len()); | ||
| 52 | match read(bytes.len() as u64, &mut tail[..size]) { | ||
| 53 | Ok(0) => break, | ||
| 54 | Ok(count) if count <= size && count <= limit - bytes.len() => { | ||
| 55 | bytes.try_reserve_exact(count).map_err(io::Error::other)?; | ||
| 56 | bytes.extend_from_slice(&tail[..count]); | ||
| 57 | } | ||
| 58 | Ok(_) => return Err(io::ErrorKind::InvalidData.into()), | ||
| 59 | Err(error) if error.kind() == io::ErrorKind::Interrupted => {} | ||
| 60 | Err(error) => return Err(error), | ||
| 61 | } | ||
| 62 | } | ||
| 63 | let mut after = [0; 1024]; | ||
| 64 | read_exact(&mut read, 0, &mut after)?; | ||
| 65 | if header != after { | ||
| 66 | return Ok(None); | ||
| 67 | } | ||
| 68 | let parsed = Store::parse(&bytes).and_then(|store| { | ||
| 69 | if !store.checksum_mismatches.is_empty() { | ||
| 70 | return Ok(false); | ||
| 71 | } | ||
| 72 | RevisionIndex::parse(&store)?.validate_current()?; | ||
| 73 | Ok(true) | ||
| 74 | }); | ||
| 75 | Ok(matches!(parsed, Ok(true)).then_some(bytes)) | ||
| 76 | } | ||
crates/onestore/src/store.rs+1-1
| ... | @@ -82,7 +82,7 @@ pub struct Header { | ... | @@ -82,7 +82,7 @@ pub struct Header { |
| 82 | } | 82 | } |
| 83 | 83 | ||
| 84 | impl Header { | 84 | impl Header { |
| 85 | fn parse(data: &[u8]) -> Result<Self> { | 85 | pub(crate) fn parse(data: &[u8]) -> Result<Self> { |
| 86 | let mut c = Cursor { | 86 | let mut c = Cursor { |
| 87 | bytes: data, | 87 | bytes: data, |
| 88 | offset: 0, | 88 | offset: 0, |
crates/onestore/src/write.rs+355-90
| ... | @@ -4,7 +4,13 @@ use crate::{ | ... | @@ -4,7 +4,13 @@ use crate::{ |
| 4 | store::{crc, transaction_crc}, | 4 | store::{crc, transaction_crc}, |
| 5 | }; | 5 | }; |
| 6 | 6 | ||
| 7 | use std::collections::{BTreeMap, BTreeSet}; | 7 | use std::{ |
| 8 | collections::{BTreeMap, BTreeSet}, | ||
| 9 | sync::Arc, | ||
| 10 | }; | ||
| 11 | |||
| 12 | #[cfg(test)] | ||
| 13 | mod tests; | ||
| 8 | 14 | ||
| 9 | type Result<T> = std::result::Result<T, Error>; | 15 | type Result<T> = std::result::Result<T, Error>; |
| 10 | 16 | ||
| ... | @@ -98,109 +104,183 @@ fn compact(id: ExGuid, table: &BTreeMap<u32, [u8; 16]>) -> Result<[u8; 4]> { | ... | @@ -98,109 +104,183 @@ fn compact(id: ExGuid, table: &BTreeMap<u32, [u8; 16]>) -> Result<[u8; 4]> { |
| 98 | Ok(((index << 8) | id.n).to_le_bytes()) | 104 | Ok(((index << 8) | id.n).to_le_bytes()) |
| 99 | } | 105 | } |
| 100 | 106 | ||
| 107 | fn field_length(property: &crate::Property<'_>, set_lengths: &[usize]) -> usize { | ||
| 108 | match &property.value { | ||
| 109 | Value::NoData => 0, | ||
| 110 | Value::Bytes(bytes) => bytes.len() + usize::from(property.id >> 26 & 31 == 7) * 4, | ||
| 111 | Value::References { .. } => usize::from(property.id >> 26 & 1 != 0) * 4, | ||
| 112 | Value::Sets(children) => { | ||
| 113 | let prefix = if property.id >> 26 & 31 == 16 { | ||
| 114 | if children.is_empty() { 4 } else { 8 } | ||
| 115 | } else { | ||
| 116 | 0 | ||
| 117 | }; | ||
| 118 | prefix + children.clone().map(|i| set_lengths[i]).sum::<usize>() | ||
| 119 | } | ||
| 120 | } | ||
| 121 | } | ||
| 122 | |||
| 101 | fn patch_properties( | 123 | fn patch_properties( |
| 102 | blob: &[u8], | 124 | blob: &[u8], |
| 103 | updates: &[(u32, &[u8])], | 125 | updates: &[(u32, &[u8])], |
| 104 | insert: Option<(u32, &[u8])>, | 126 | inserts: &[(u32, &[u8])], |
| 105 | ) -> Result<Vec<u8>> { | 127 | ) -> Result<Vec<u8>> { |
| 106 | let properties = PropertySets::parse(blob)?; | 128 | let properties = PropertySets::parse(blob)?; |
| 129 | let root = &properties.sets[0]; | ||
| 130 | let ids = properties.root_ids.as_ptr().addr() - blob.as_ptr().addr(); | ||
| 131 | let ids_end = ids + properties.root_ids.len(); | ||
| 132 | let body_end = blob.len() - properties.padding.len(); | ||
| 133 | let mut set_lengths = vec![0; properties.sets.len()]; | ||
| 134 | for (i, set) in properties.sets.iter().enumerate().rev() { | ||
| 135 | set_lengths[i] = 2 | ||
| 136 | + set.len() * 4 | ||
| 137 | + set | ||
| 138 | .iter() | ||
| 139 | .map(|p| field_length(p, &set_lengths)) | ||
| 140 | .sum::<usize>(); | ||
| 141 | } | ||
| 142 | let mut offsets = Vec::with_capacity(root.len()); | ||
| 143 | let mut offset = ids_end; | ||
| 144 | for property in root { | ||
| 145 | offsets.push(offset); | ||
| 146 | offset += field_length(property, &set_lengths); | ||
| 147 | } | ||
| 107 | let mut patches = Vec::new(); | 148 | let mut patches = Vec::new(); |
| 108 | for (i, &(property, value)) in updates.iter().chain(insert.iter()).enumerate() { | 149 | let mut added_ids = Vec::new(); |
| 109 | if updates[..i.min(updates.len())] | 150 | let mut added_fields = Vec::new(); |
| 110 | .iter() | 151 | let mut added_references = Vec::new(); |
| 111 | .any(|(id, _)| *id == property) | 152 | let object_header = u32::from_le_bytes(blob[..4].try_into().unwrap()); |
| 112 | { | 153 | let mut object_count = i64::from(object_header & 0xffffff); |
| 154 | let mut seen = BTreeSet::new(); | ||
| 155 | for (i, &(property, value)) in updates.iter().chain(inserts).enumerate() { | ||
| 156 | if !seen.insert(property & 0x7fffffff) { | ||
| 113 | return Err(Error { | 157 | return Err(Error { |
| 114 | offset: 0, | 158 | offset: 0, |
| 115 | message: "Duplicate property update", | 159 | message: "Duplicate property update", |
| 116 | }); | 160 | }); |
| 117 | } | 161 | } |
| 118 | let kind = (property >> 26) & 0x1f; | 162 | let kind = (property >> 26) & 31; |
| 119 | if !(3..=7).contains(&kind) { | 163 | let valid = match kind { |
| 120 | return Err(Error { | 164 | 2 => value.is_empty(), |
| 121 | offset: 0, | 165 | 3..=6 => value.len() == 1 << (kind - 3), |
| 122 | message: "Property does not contain scalar bytes", | 166 | 7 => value.len() < 0x40000000, |
| 123 | }); | 167 | 8 => value.len() == 4, |
| 124 | } | 168 | 9 => value.len().is_multiple_of(4) && value.len() / 4 <= 0xffffff, |
| 125 | if (kind == 7 && value.len() >= 0x40000000) || (kind != 7 && value.len() != 1 << (kind - 3)) | 169 | _ => { |
| 126 | { | 170 | return Err(Error { |
| 171 | offset: 0, | ||
| 172 | message: "Property type cannot be patched", | ||
| 173 | }); | ||
| 174 | } | ||
| 175 | }; | ||
| 176 | if !valid || (kind != 2 && property & 0x80000000 != 0) { | ||
| 127 | return Err(Error { | 177 | return Err(Error { |
| 128 | offset: 0, | 178 | offset: 0, |
| 129 | message: "Replacement has an invalid property length", | 179 | message: "Replacement has an invalid property value", |
| 130 | }); | 180 | }); |
| 131 | } | 181 | } |
| 132 | let matches: Vec<_> = properties.sets[0] | 182 | let matches: Vec<_> = root |
| 133 | .iter() | 183 | .iter() |
| 134 | .filter(|candidate| candidate.id == property) | 184 | .enumerate() |
| 185 | .filter(|(_, p)| p.id & 0x7fffffff == property & 0x7fffffff) | ||
| 135 | .collect(); | 186 | .collect(); |
| 136 | let adding = i == updates.len(); | 187 | let adding = i >= updates.len(); |
| 137 | if matches.len() != usize::from(!adding) { | 188 | if matches.len() != usize::from(!adding) { |
| 138 | return Err(Error { | 189 | return Err(Error { |
| 139 | offset: 0, | 190 | offset: 0, |
| 140 | message: "Property is missing or duplicated", | 191 | message: "Property is missing or duplicated", |
| 141 | }); | 192 | }); |
| 142 | } | 193 | } |
| 143 | let previous = if adding { | ||
| 144 | None | ||
| 145 | } else { | ||
| 146 | let Value::Bytes(previous) = matches[0].value else { | ||
| 147 | return Err(Error { | ||
| 148 | offset: 0, | ||
| 149 | message: "Property does not contain scalar bytes", | ||
| 150 | }); | ||
| 151 | }; | ||
| 152 | if previous == value { | ||
| 153 | continue; | ||
| 154 | } | ||
| 155 | Some(previous) | ||
| 156 | }; | ||
| 157 | let mut encoded = Vec::new(); | 194 | let mut encoded = Vec::new(); |
| 158 | if kind == 7 { | 195 | match kind { |
| 159 | encoded.extend_from_slice(&u32::try_from(value.len()).unwrap().to_le_bytes()); | 196 | 7 => encoded.extend_from_slice(&(value.len() as u32).to_le_bytes()), |
| 197 | 9 => encoded.extend_from_slice(&(value.len() as u32 / 4).to_le_bytes()), | ||
| 198 | _ => {} | ||
| 199 | } | ||
| 200 | if (3..=7).contains(&kind) { | ||
| 201 | encoded.extend_from_slice(value); | ||
| 160 | } | 202 | } |
| 161 | encoded.extend_from_slice(value); | 203 | if adding { |
| 162 | if let Some(previous) = previous { | 204 | added_ids.extend_from_slice(&property.to_le_bytes()); |
| 163 | let start = previous.as_ptr().addr() - blob.as_ptr().addr(); | 205 | added_fields.extend_from_slice(&encoded); |
| 164 | patches.push(( | 206 | if kind >= 8 { |
| 165 | start - if kind == 7 { 4 } else { 0 }, | 207 | object_count += (value.len() / 4) as i64; |
| 166 | start + previous.len(), | 208 | added_references.extend_from_slice(value); |
| 167 | encoded, | 209 | } |
| 168 | )); | ||
| 169 | } else { | 210 | } else { |
| 170 | let count = u16::try_from(properties.sets[0].len() + 1).map_err(|_| Error { | 211 | let (index, previous) = matches[0]; |
| 171 | offset: 0, | 212 | if kind == 2 && previous.id != property { |
| 172 | message: "Root property count exceeds the format limit", | 213 | patches.push(( |
| 173 | })?; | 214 | ids + index * 4, |
| 174 | let ids = properties.root_ids.as_ptr().addr() - blob.as_ptr().addr(); | 215 | ids + index * 4 + 4, |
| 175 | patches.push((ids - 2, ids, count.to_le_bytes().to_vec())); | 216 | index, |
| 176 | let end = ids + properties.root_ids.len(); | 217 | property.to_le_bytes().to_vec(), |
| 177 | patches.push((end, end, property.to_le_bytes().to_vec())); | 218 | )); |
| 178 | let end = blob.len() - properties.padding.len(); | 219 | } |
| 179 | patches.push((end, end, encoded)); | 220 | let start = offsets[index]; |
| 221 | let end = start + field_length(previous, &set_lengths); | ||
| 222 | if blob[start..end] != encoded { | ||
| 223 | patches.push((start, end, index, encoded)); | ||
| 224 | } | ||
| 225 | if let Value::References { compact_ids, .. } = previous.value { | ||
| 226 | object_count += (value.len() / 4) as i64 - (compact_ids.len() / 4) as i64; | ||
| 227 | if compact_ids != value { | ||
| 228 | let start = compact_ids.as_ptr().addr() - blob.as_ptr().addr(); | ||
| 229 | patches.push((start, start + compact_ids.len(), index, value.to_vec())); | ||
| 230 | } | ||
| 231 | } | ||
| 180 | } | 232 | } |
| 181 | } | 233 | } |
| 234 | if !(0..=0xffffff).contains(&object_count) { | ||
| 235 | return Err(Error { | ||
| 236 | offset: 0, | ||
| 237 | message: "Object reference stream exceeds the format limit", | ||
| 238 | }); | ||
| 239 | } | ||
| 240 | let header = (object_header & 0xff000000) | object_count as u32; | ||
| 241 | if header != object_header { | ||
| 242 | patches.push((0, 4, 0, header.to_le_bytes().to_vec())); | ||
| 243 | } | ||
| 244 | if !added_references.is_empty() { | ||
| 245 | let end = 4 + (object_header as usize & 0xffffff) * 4; | ||
| 246 | patches.push((end, end, usize::MAX, added_references)); | ||
| 247 | } | ||
| 248 | if !inserts.is_empty() { | ||
| 249 | let count = u16::try_from(root.len() + inserts.len()).map_err(|_| Error { | ||
| 250 | offset: ids - 2, | ||
| 251 | message: "Root property count exceeds the format limit", | ||
| 252 | })?; | ||
| 253 | patches.push((ids - 2, ids, 0, count.to_le_bytes().to_vec())); | ||
| 254 | patches.push((ids_end, ids_end, usize::MAX - 1, added_ids)); | ||
| 255 | patches.push((body_end, body_end, usize::MAX, added_fields)); | ||
| 256 | } | ||
| 182 | if patches.is_empty() { | 257 | if patches.is_empty() { |
| 183 | return Ok(blob.to_vec()); | 258 | return Ok(blob.to_vec()); |
| 184 | } | 259 | } |
| 185 | patches.sort_by_key(|(start, end, _)| (*start, *end)); | 260 | patches.sort_by_key(|(start, end, order, _)| (*start, *end, *order)); |
| 186 | let mut changed = Vec::new(); | 261 | let mut changed = Vec::new(); |
| 187 | let mut cursor = 0; | 262 | let mut cursor = 0; |
| 188 | for (start, end, value) in patches { | 263 | for (start, end, _, value) in patches { |
| 264 | if start < cursor { | ||
| 265 | return Err(Error { | ||
| 266 | offset: start, | ||
| 267 | message: "Property patches overlap", | ||
| 268 | }); | ||
| 269 | } | ||
| 189 | changed.extend_from_slice(&blob[cursor..start]); | 270 | changed.extend_from_slice(&blob[cursor..start]); |
| 190 | changed.extend_from_slice(&value); | 271 | changed.extend_from_slice(&value); |
| 191 | cursor = end; | 272 | cursor = end; |
| 192 | } | 273 | } |
| 193 | changed.extend_from_slice(&blob[cursor..blob.len() - properties.padding.len()]); | 274 | changed.extend_from_slice(&blob[cursor..body_end]); |
| 194 | changed.resize(changed.len().next_multiple_of(8), 0); | 275 | changed.resize(changed.len().next_multiple_of(8), 0); |
| 195 | PropertySets::parse(&changed)?; | 276 | PropertySets::parse(&changed)?; |
| 196 | |||
| 197 | Ok(changed) | 277 | Ok(changed) |
| 198 | } | 278 | } |
| 199 | 279 | ||
| 200 | pub(crate) struct ObjectEdit<'a> { | 280 | pub(crate) struct ObjectEdit<'a> { |
| 201 | pub object: ExGuid, | 281 | pub object: ExGuid, |
| 202 | pub updates: &'a [(u32, &'a [u8])], | 282 | pub updates: &'a [(u32, &'a [u8])], |
| 203 | pub insert: Option<(u32, &'a [u8])>, | 283 | pub inserts: &'a [(u32, &'a [u8])], |
| 204 | } | 284 | } |
| 205 | 285 | ||
| 206 | /// Replaces a root-level scalar byte property in the default active revision. | 286 | /// Replaces a root-level scalar byte property in the default active revision. |
| ... | @@ -213,13 +293,19 @@ pub fn replace_property_bytes( | ... | @@ -213,13 +293,19 @@ pub fn replace_property_bytes( |
| 213 | property: u32, | 293 | property: u32, |
| 214 | value: &[u8], | 294 | value: &[u8], |
| 215 | ) -> Result<Vec<u8>> { | 295 | ) -> Result<Vec<u8>> { |
| 296 | if !(3..=7).contains(&((property >> 26) & 31)) { | ||
| 297 | return Err(Error { | ||
| 298 | offset: 0, | ||
| 299 | message: "Property does not contain scalar bytes", | ||
| 300 | }); | ||
| 301 | } | ||
| 216 | replace_objects( | 302 | replace_objects( |
| 217 | source, | 303 | source, |
| 218 | space, | 304 | space, |
| 219 | &[ObjectEdit { | 305 | &[ObjectEdit { |
| 220 | object: object_id, | 306 | object: object_id, |
| 221 | updates: &[(property, value)], | 307 | updates: &[(property, value)], |
| 222 | insert: None, | 308 | inserts: &[], |
| 223 | }], | 309 | }], |
| 224 | ) | 310 | ) |
| 225 | } | 311 | } |
| ... | @@ -228,6 +314,96 @@ pub(crate) fn replace_objects( | ... | @@ -228,6 +314,96 @@ pub(crate) fn replace_objects( |
| 228 | source: &[u8], | 314 | source: &[u8], |
| 229 | space: ExGuid, | 315 | space: ExGuid, |
| 230 | edits: &[ObjectEdit<'_>], | 316 | edits: &[ObjectEdit<'_>], |
| 317 | ) -> Result<Vec<u8>> { | ||
| 318 | write_revision(source, space, |revision| { | ||
| 319 | let mut changed = BTreeMap::new(); | ||
| 320 | for edit in edits { | ||
| 321 | if changed.contains_key(&edit.object) { | ||
| 322 | return Err(Error { | ||
| 323 | offset: 0, | ||
| 324 | message: "Duplicate object edit", | ||
| 325 | }); | ||
| 326 | } | ||
| 327 | let object = revision.objects.get(&edit.object).ok_or(Error { | ||
| 328 | offset: 0, | ||
| 329 | message: "Object is absent from the active revision", | ||
| 330 | })?; | ||
| 331 | let ObjectData::Properties(blob) = object.data else { | ||
| 332 | return Err(Error { | ||
| 333 | offset: 0, | ||
| 334 | message: "Object does not contain editable properties", | ||
| 335 | }); | ||
| 336 | }; | ||
| 337 | changed.insert( | ||
| 338 | edit.object, | ||
| 339 | PropertyObject { | ||
| 340 | jcid: object.jcid, | ||
| 341 | bytes: patch_properties(blob, edit.updates, edit.inserts)?, | ||
| 342 | global_ids: Arc::clone(&object.global_ids), | ||
| 343 | }, | ||
| 344 | ); | ||
| 345 | } | ||
| 346 | Ok(changed) | ||
| 347 | }) | ||
| 348 | } | ||
| 349 | |||
| 350 | pub(crate) struct PropertyObject { | ||
| 351 | pub jcid: u32, | ||
| 352 | pub bytes: Vec<u8>, | ||
| 353 | pub global_ids: Arc<BTreeMap<u32, [u8; 16]>>, | ||
| 354 | } | ||
| 355 | |||
| 356 | impl PropertyObject { | ||
| 357 | pub fn from_object(object: &crate::Object<'_>) -> Result<Self> { | ||
| 358 | let ObjectData::Properties(bytes) = object.data else { | ||
| 359 | return Err(Error { | ||
| 360 | offset: 0, | ||
| 361 | message: "Object does not contain editable properties", | ||
| 362 | }); | ||
| 363 | }; | ||
| 364 | Ok(Self { | ||
| 365 | jcid: object.jcid, | ||
| 366 | bytes: bytes.to_vec(), | ||
| 367 | global_ids: Arc::clone(&object.global_ids), | ||
| 368 | }) | ||
| 369 | } | ||
| 370 | |||
| 371 | pub fn set(&mut self, values: &[(u32, &[u8])]) -> Result<()> { | ||
| 372 | let properties = PropertySets::parse(&self.bytes)?; | ||
| 373 | let (updates, inserts): (Vec<_>, Vec<_>) = values.iter().copied().partition(|(id, _)| { | ||
| 374 | properties.sets[0] | ||
| 375 | .iter() | ||
| 376 | .any(|p| p.id & 0x7fffffff == id & 0x7fffffff) | ||
| 377 | }); | ||
| 378 | self.bytes = patch_properties(&self.bytes, &updates, &inserts)?; | ||
| 379 | Ok(()) | ||
| 380 | } | ||
| 381 | |||
| 382 | pub fn reference(&mut self, id: ExGuid) -> Result<[u8; 4]> { | ||
| 383 | if !self.global_ids.values().any(|guid| *guid == id.guid) { | ||
| 384 | let mut index = 0; | ||
| 385 | for key in self.global_ids.keys() { | ||
| 386 | if *key != index { | ||
| 387 | break; | ||
| 388 | } | ||
| 389 | index += 1; | ||
| 390 | } | ||
| 391 | if index >= 0xffffff || id.guid == [0; 16] { | ||
| 392 | return Err(Error { | ||
| 393 | offset: 0, | ||
| 394 | message: "Object identity cannot be added to the global ID table", | ||
| 395 | }); | ||
| 396 | } | ||
| 397 | Arc::make_mut(&mut self.global_ids).insert(index, id.guid); | ||
| 398 | } | ||
| 399 | compact(id, &self.global_ids) | ||
| 400 | } | ||
| 401 | } | ||
| 402 | |||
| 403 | pub(crate) fn write_revision( | ||
| 404 | source: &[u8], | ||
| 405 | space: ExGuid, | ||
| 406 | edit: impl FnOnce(&crate::ResolvedRevision<'_>) -> Result<BTreeMap<ExGuid, PropertyObject>>, | ||
| 231 | ) -> Result<Vec<u8>> { | 407 | ) -> Result<Vec<u8>> { |
| 232 | let store = Store::parse(source)?; | 408 | let store = Store::parse(source)?; |
| 233 | let is_section = store.header.file_type == FileType::Section; | 409 | let is_section = store.header.file_type == FileType::Section; |
| ... | @@ -247,46 +423,136 @@ pub(crate) fn replace_objects( | ... | @@ -247,46 +423,136 @@ pub(crate) fn replace_objects( |
| 247 | offset: 0, | 423 | offset: 0, |
| 248 | message: "Object space has no active default revision", | 424 | message: "Object space has no active default revision", |
| 249 | })?; | 425 | })?; |
| 250 | let revision = index.resolve(space, rid)?; | 426 | let mut revision = index.resolve(space, rid)?; |
| 251 | let reachable = revision.reachable()?; | 427 | let reachable = revision.reachable()?; |
| 252 | let mut changed = BTreeMap::new(); | 428 | let mut replacements = edit(&revision)?; |
| 253 | for (i, edit) in edits.iter().enumerate() { | 429 | for (id, replacement) in &replacements { |
| 254 | if edits[..i] | 430 | if let Some(object) = revision.objects.get(id) { |
| 255 | .iter() | 431 | if !reachable.contains(id) { |
| 256 | .any(|previous| previous.object == edit.object) | 432 | return Err(Error { |
| 257 | { | 433 | offset: 0, |
| 434 | message: "Object is not reachable in the active revision", | ||
| 435 | }); | ||
| 436 | } | ||
| 437 | if object.jcid & 0x100000 != 0 { | ||
| 438 | return Err(Error { | ||
| 439 | offset: 0, | ||
| 440 | message: "Read-only object requires a new identity", | ||
| 441 | }); | ||
| 442 | } | ||
| 443 | if replacement.jcid != object.jcid || !matches!(object.data, ObjectData::Properties(_)) | ||
| 444 | { | ||
| 445 | return Err(Error { | ||
| 446 | offset: 0, | ||
| 447 | message: "An existing object's type cannot be changed", | ||
| 448 | }); | ||
| 449 | } | ||
| 450 | } else if !is_section { | ||
| 258 | return Err(Error { | 451 | return Err(Error { |
| 259 | offset: 0, | 452 | offset: 0, |
| 260 | message: "Duplicate object edit", | 453 | message: "New objects require a section file", |
| 261 | }); | 454 | }); |
| 262 | } | 455 | } |
| 263 | if !reachable.contains(&edit.object) { | 456 | if replacement.jcid & 0x20000 == 0 || replacement.global_ids.keys().any(|i| *i > 0xffffff) { |
| 264 | return Err(Error { | 457 | return Err(Error { |
| 265 | offset: 0, | 458 | offset: 0, |
| 266 | message: "Object is not reachable in the active revision", | 459 | message: "Invalid property object declaration", |
| 267 | }); | 460 | }); |
| 268 | } | 461 | } |
| 269 | let object = &revision.objects[&edit.object]; | 462 | compact(*id, &replacement.global_ids)?; |
| 270 | if object.jcid & 0x100000 != 0 { | 463 | PropertySets::parse(&replacement.bytes)?; |
| 271 | return Err(Error { | 464 | } |
| 272 | offset: 0, | 465 | // Native coalescing of duplicate readonly styles can leave dangling references. |
| 273 | message: "Read-only object requires a new identity", | 466 | let mut aliases = BTreeMap::new(); |
| 274 | }); | 467 | for (id, replacement) in &replacements { |
| 468 | if revision.objects.contains_key(id) | ||
| 469 | || replacement.jcid & 0x100000 == 0 | ||
| 470 | || PropertySets::parse(&replacement.bytes)? | ||
| 471 | .sets | ||
| 472 | .iter() | ||
| 473 | .flatten() | ||
| 474 | .any(|p| matches!(p.value, Value::References { .. })) | ||
| 475 | { | ||
| 476 | continue; | ||
| 275 | } | 477 | } |
| 276 | let ObjectData::Properties(blob) = object.data else { | 478 | let existing = revision.objects.iter().find_map(|(other, object)| { |
| 277 | return Err(Error { | 479 | (reachable.contains(other) |
| 278 | offset: 0, | 480 | && object.jcid == replacement.jcid |
| 279 | message: "Object does not contain editable properties", | 481 | && object.data == ObjectData::Properties(&replacement.bytes)) |
| 280 | }); | 482 | .then_some(*other) |
| 281 | }; | 483 | }); |
| 282 | let bytes = patch_properties(blob, edit.updates, edit.insert)?; | 484 | let existing = existing.or_else(|| { |
| 283 | if bytes != blob { | 485 | replacements.range(..id).find_map(|(other, object)| { |
| 284 | changed.insert(edit.object, bytes); | 486 | (object.jcid == replacement.jcid && object.bytes == replacement.bytes) |
| 487 | .then_some(*aliases.get(other).unwrap_or(other)) | ||
| 488 | }) | ||
| 489 | }); | ||
| 490 | if let Some(existing) = existing { | ||
| 491 | aliases.insert(*id, existing); | ||
| 285 | } | 492 | } |
| 286 | } | 493 | } |
| 287 | if changed.is_empty() { | 494 | for id in aliases.keys() { |
| 495 | replacements.remove(id); | ||
| 496 | } | ||
| 497 | for object in replacements.values_mut() { | ||
| 498 | let mut remapped = Vec::new(); | ||
| 499 | for property in PropertySets::parse(&object.bytes)?.sets.iter().flatten() { | ||
| 500 | if let Value::References { | ||
| 501 | stream: crate::IdStream::Objects, | ||
| 502 | compact_ids, | ||
| 503 | } = property.value | ||
| 504 | { | ||
| 505 | for bytes in compact_ids.chunks_exact(4) { | ||
| 506 | let offset = bytes.as_ptr().addr() - object.bytes.as_ptr().addr(); | ||
| 507 | let id = crate::bytes::Cursor { bytes, offset }.compact(&object.global_ids)?; | ||
| 508 | if let Some(existing) = aliases.get(&id) { | ||
| 509 | remapped.push((offset, *existing)); | ||
| 510 | } | ||
| 511 | } | ||
| 512 | } | ||
| 513 | } | ||
| 514 | for (offset, id) in remapped { | ||
| 515 | let reference = object.reference(id)?; | ||
| 516 | object.bytes[offset..offset + 4].copy_from_slice(&reference); | ||
| 517 | } | ||
| 518 | } | ||
| 519 | replacements.retain(|id, replacement| { | ||
| 520 | !revision.objects.get(id).is_some_and(|object| { | ||
| 521 | object.data == ObjectData::Properties(&replacement.bytes) | ||
| 522 | && object.global_ids == replacement.global_ids | ||
| 523 | }) | ||
| 524 | }); | ||
| 525 | if replacements.is_empty() { | ||
| 288 | return Ok(source.to_vec()); | 526 | return Ok(source.to_vec()); |
| 289 | } | 527 | } |
| 528 | let mut changed: BTreeSet<_> = replacements.keys().copied().collect(); | ||
| 529 | for (id, replacement) in &replacements { | ||
| 530 | revision.objects.insert( | ||
| 531 | *id, | ||
| 532 | crate::Object { | ||
| 533 | jcid: replacement.jcid, | ||
| 534 | reference_count: 0, | ||
| 535 | data: ObjectData::Properties(&replacement.bytes), | ||
| 536 | global_ids: Arc::clone(&replacement.global_ids), | ||
| 537 | }, | ||
| 538 | ); | ||
| 539 | } | ||
| 540 | let incoming = revision.reference_counts()?; | ||
| 541 | if replacements.keys().any(|id| !incoming.contains_key(id)) { | ||
| 542 | return Err(Error { | ||
| 543 | offset: 0, | ||
| 544 | message: "Edited object is not reachable in the resulting revision", | ||
| 545 | }); | ||
| 546 | } | ||
| 547 | for (id, object) in &mut revision.objects { | ||
| 548 | if reachable.contains(id) || incoming.contains_key(id) { | ||
| 549 | let count = incoming.get(id).copied().unwrap_or(0); | ||
| 550 | if object.reference_count != count { | ||
| 551 | object.reference_count = count; | ||
| 552 | changed.insert(*id); | ||
| 553 | } | ||
| 554 | } | ||
| 555 | } | ||
| 290 | 556 | ||
| 291 | // Native cold-open fails on long dependency chains; cap their depth at 512. | 557 | // Native cold-open fails on long dependency chains; cap their depth at 512. |
| 292 | let checkpoint = std::iter::successors(Some(rid), |id| { | 558 | let checkpoint = std::iter::successors(Some(rid), |id| { |
| ... | @@ -297,7 +563,7 @@ pub(crate) fn replace_objects( | ... | @@ -297,7 +563,7 @@ pub(crate) fn replace_objects( |
| 297 | let selected: Vec<_> = revision | 563 | let selected: Vec<_> = revision |
| 298 | .objects | 564 | .objects |
| 299 | .iter() | 565 | .iter() |
| 300 | .filter(|(id, _)| checkpoint || changed.contains_key(id)) | 566 | .filter(|(id, _)| checkpoint || changed.contains(id)) |
| 301 | .collect(); | 567 | .collect(); |
| 302 | let toc_table = if checkpoint && !is_section { | 568 | let toc_table = if checkpoint && !is_section { |
| 303 | if selected.iter().any(|(_, object)| { | 569 | if selected.iter().any(|(_, object)| { |
| ... | @@ -398,8 +664,7 @@ pub(crate) fn replace_objects( | ... | @@ -398,8 +664,7 @@ pub(crate) fn replace_objects( |
| 398 | } | 664 | } |
| 399 | group.push(node(0x73, None, &declaration)?); | 665 | group.push(node(0x73, None, &declaration)?); |
| 400 | } | 666 | } |
| 401 | ObjectData::Properties(previous) => { | 667 | ObjectData::Properties(bytes) => { |
| 402 | let bytes = changed.get(&id).map(Vec::as_slice).unwrap_or(previous); | ||
| 403 | let references = object.references()?; | 668 | let references = object.references()?; |
| 404 | let flags = u8::from(!references.objects.is_empty()) | 669 | let flags = u8::from(!references.objects.is_empty()) |
| 405 | | (u8::from( | 670 | | (u8::from( |
| ... | @@ -422,7 +687,7 @@ pub(crate) fn replace_objects( | ... | @@ -422,7 +687,7 @@ pub(crate) fn replace_objects( |
| 422 | } | 687 | } |
| 423 | } | 688 | } |
| 424 | append(&mut output, &mapped)? | 689 | append(&mut output, &mapped)? |
| 425 | } else if changed.contains_key(&id) { | 690 | } else if replacements.contains_key(&id) { |
| 426 | append(&mut output, bytes)? | 691 | append(&mut output, bytes)? |
| 427 | } else { | 692 | } else { |
| 428 | Chunk { | 693 | Chunk { |
crates/onestore/src/write/tests.rs created+680
| ... | @@ -0,0 +1,680 @@ | ||
| 1 | use super::patch_properties; | ||
| 2 | use crate::{PropertySets, Value, create::properties}; | ||
| 3 | |||
| 4 | #[test] | ||
| 5 | fn document_insertions_and_formatting_respect_readonly_ancestors() { | ||
| 6 | use super::{PropertyObject, write_revision}; | ||
| 7 | use crate::{ExGuid, Insertion, PreparedEdit, RevisionIndex, Store}; | ||
| 8 | use std::collections::BTreeMap; | ||
| 9 | let source = crate::create_section("readonly.one", "Original", "Author").unwrap(); | ||
| 10 | let store = Store::parse(&source).unwrap(); | ||
| 11 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 12 | let (sid, page, outline, paragraph) = index | ||
| 13 | .spaces | ||
| 14 | .iter() | ||
| 15 | .find_map(|(sid, s)| { | ||
| 16 | let raw = index | ||
| 17 | .resolve(*sid, s.labels[&(ExGuid::default(), 1)]) | ||
| 18 | .unwrap(); | ||
| 19 | let by_type = |jcid| { | ||
| 20 | raw.objects | ||
| 21 | .iter() | ||
| 22 | .find_map(|(id, o)| (o.jcid == jcid).then_some(*id)) | ||
| 23 | }; | ||
| 24 | Some(( | ||
| 25 | *sid, | ||
| 26 | by_type(0x6000b)?, | ||
| 27 | by_type(0x6000c)?, | ||
| 28 | by_type(0x6000d)?, | ||
| 29 | )) | ||
| 30 | }) | ||
| 31 | .unwrap(); | ||
| 32 | for blocked in [page, outline, paragraph] { | ||
| 33 | let protected = write_revision(&source, sid, |raw| { | ||
| 34 | let mut object = PropertyObject::from_object(&raw.objects[&blocked])?; | ||
| 35 | object.set(&[(0x88001cde, &[])])?; | ||
| 36 | Ok(BTreeMap::from([(blocked, object)])) | ||
| 37 | }) | ||
| 38 | .unwrap(); | ||
| 39 | let child = Insertion::paragraph(paragraph, None, "Nested", "Author").unwrap(); | ||
| 40 | assert!(PreparedEdit::insert(&protected, sid, &child).is_err()); | ||
| 41 | let raw = index | ||
| 42 | .resolve(sid, index.spaces[&sid].labels[&(ExGuid::default(), 1)]) | ||
| 43 | .unwrap(); | ||
| 44 | let text = raw | ||
| 45 | .objects | ||
| 46 | .iter() | ||
| 47 | .find_map(|(id, object)| (object.jcid == 0x6000e).then_some(*id)) | ||
| 48 | .unwrap(); | ||
| 49 | assert!( | ||
| 50 | PreparedEdit::format( | ||
| 51 | &protected, | ||
| 52 | sid, | ||
| 53 | text, | ||
| 54 | 1..3, | ||
| 55 | &[crate::TextAttribute::Bold(true)] | ||
| 56 | ) | ||
| 57 | .is_err() | ||
| 58 | ); | ||
| 59 | if blocked == page { | ||
| 60 | let outline = Insertion::outline(page, 36.0, 36.0, "Outline", "Author").unwrap(); | ||
| 61 | assert!(PreparedEdit::insert(&protected, sid, &outline).is_err()); | ||
| 62 | } | ||
| 63 | } | ||
| 64 | } | ||
| 65 | |||
| 66 | fn add_paragraph(source: &[u8], number: u32) -> Vec<u8> { | ||
| 67 | use super::{PropertyObject, compact, write_revision}; | ||
| 68 | use crate::{ExGuid, ObjectData, RevisionIndex, Store}; | ||
| 69 | use std::{collections::BTreeMap, sync::Arc}; | ||
| 70 | let store = Store::parse(source).unwrap(); | ||
| 71 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 72 | let space = index | ||
| 73 | .spaces | ||
| 74 | .iter() | ||
| 75 | .find_map(|(sid, s)| { | ||
| 76 | let revision = index | ||
| 77 | .resolve(*sid, s.labels[&(ExGuid::default(), 1)]) | ||
| 78 | .unwrap(); | ||
| 79 | revision | ||
| 80 | .objects | ||
| 81 | .values() | ||
| 82 | .any(|o| o.jcid == 0x6000c) | ||
| 83 | .then_some(*sid) | ||
| 84 | }) | ||
| 85 | .unwrap(); | ||
| 86 | write_revision(source, space, |revision| { | ||
| 87 | let (&outline_id, outline) = revision | ||
| 88 | .objects | ||
| 89 | .iter() | ||
| 90 | .find(|(_, o)| o.jcid == 0x6000c) | ||
| 91 | .unwrap(); | ||
| 92 | let (&author, _) = revision | ||
| 93 | .objects | ||
| 94 | .iter() | ||
| 95 | .find(|(_, o)| o.jcid == 0x120001) | ||
| 96 | .unwrap(); | ||
| 97 | let (&style, _) = revision | ||
| 98 | .objects | ||
| 99 | .iter() | ||
| 100 | .find(|(_, o)| o.jcid == 0x12004d) | ||
| 101 | .unwrap(); | ||
| 102 | let mut guid = [0x69; 16]; | ||
| 103 | guid[..4].copy_from_slice(&number.to_le_bytes()); | ||
| 104 | let paragraph = ExGuid { guid, n: 1 }; | ||
| 105 | let text = ExGuid { guid, n: 2 }; | ||
| 106 | let mut table = (*outline.global_ids).clone(); | ||
| 107 | for guid in [guid, author.guid, style.guid] { | ||
| 108 | if !table.values().any(|previous| *previous == guid) { | ||
| 109 | table.insert(table.last_key_value().map_or(0, |(i, _)| i + 1), guid); | ||
| 110 | } | ||
| 111 | } | ||
| 112 | let table = Arc::new(table); | ||
| 113 | let ObjectData::Properties(blob) = outline.data else { | ||
| 114 | unreachable!() | ||
| 115 | }; | ||
| 116 | let parsed = PropertySets::parse(blob).unwrap(); | ||
| 117 | let Value::References { compact_ids, .. } = parsed.sets[0] | ||
| 118 | .iter() | ||
| 119 | .find(|p| p.id == 0x24001c20) | ||
| 120 | .unwrap() | ||
| 121 | .value | ||
| 122 | else { | ||
| 123 | unreachable!() | ||
| 124 | }; | ||
| 125 | let mut children = compact_ids.to_vec(); | ||
| 126 | children.extend_from_slice(&compact(paragraph, &table).unwrap()); | ||
| 127 | let modified = crate::create::current_timestamps()?.0.to_le_bytes(); | ||
| 128 | let mut changed = BTreeMap::new(); | ||
| 129 | changed.insert( | ||
| 130 | outline_id, | ||
| 131 | PropertyObject { | ||
| 132 | jcid: outline.jcid, | ||
| 133 | bytes: patch_properties( | ||
| 134 | blob, | ||
| 135 | &[(0x24001c20, &children), (0x14001d7a, &modified)], | ||
| 136 | &[], | ||
| 137 | )?, | ||
| 138 | global_ids: Arc::clone(&table), | ||
| 139 | }, | ||
| 140 | ); | ||
| 141 | changed.insert( | ||
| 142 | paragraph, | ||
| 143 | PropertyObject { | ||
| 144 | jcid: 0x6000d, | ||
| 145 | bytes: properties(&[ | ||
| 146 | (0x14001d7a, modified.to_vec()), | ||
| 147 | (0x14001d09, modified.to_vec()), | ||
| 148 | (0x0c001c03, vec![1]), | ||
| 149 | (0x24001c1f, compact(text, &table)?.to_vec()), | ||
| 150 | (0x20001d78, compact(author, &table)?.to_vec()), | ||
| 151 | (0x20001d79, compact(author, &table)?.to_vec()), | ||
| 152 | ])?, | ||
| 153 | global_ids: Arc::clone(&table), | ||
| 154 | }, | ||
| 155 | ); | ||
| 156 | changed.insert( | ||
| 157 | text, | ||
| 158 | PropertyObject { | ||
| 159 | jcid: 0x6000e, | ||
| 160 | bytes: properties(&[ | ||
| 161 | (0x14001d7a, modified.to_vec()), | ||
| 162 | ( | ||
| 163 | 0x1c001c22, | ||
| 164 | format!("Paragraph {number}\0") | ||
| 165 | .encode_utf16() | ||
| 166 | .flat_map(u16::to_le_bytes) | ||
| 167 | .collect(), | ||
| 168 | ), | ||
| 169 | (0x24001e13, compact(style, &table)?.to_vec()), | ||
| 170 | ])?, | ||
| 171 | global_ids: table, | ||
| 172 | }, | ||
| 173 | ); | ||
| 174 | Ok(changed) | ||
| 175 | }) | ||
| 176 | .unwrap() | ||
| 177 | } | ||
| 178 | |||
| 179 | fn restyle(source: &[u8]) -> Vec<u8> { | ||
| 180 | use super::{PropertyObject, compact, write_revision}; | ||
| 181 | use crate::{ExGuid, ObjectData, RevisionIndex, Store}; | ||
| 182 | use std::{collections::BTreeMap, sync::Arc}; | ||
| 183 | let store = Store::parse(source).unwrap(); | ||
| 184 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 185 | let space = index | ||
| 186 | .spaces | ||
| 187 | .iter() | ||
| 188 | .find_map(|(sid, s)| { | ||
| 189 | let revision = index | ||
| 190 | .resolve(*sid, s.labels[&(ExGuid::default(), 1)]) | ||
| 191 | .unwrap(); | ||
| 192 | revision | ||
| 193 | .objects | ||
| 194 | .values() | ||
| 195 | .any(|o| o.jcid == 0x6000c) | ||
| 196 | .then_some(*sid) | ||
| 197 | }) | ||
| 198 | .unwrap(); | ||
| 199 | let style = ExGuid { | ||
| 200 | guid: [0x74; 16], | ||
| 201 | n: 1, | ||
| 202 | }; | ||
| 203 | write_revision(source, space, |revision| { | ||
| 204 | let previous = revision | ||
| 205 | .objects | ||
| 206 | .values() | ||
| 207 | .find(|o| o.jcid == 0x12004d) | ||
| 208 | .unwrap(); | ||
| 209 | let ObjectData::Properties(blob) = previous.data else { | ||
| 210 | unreachable!() | ||
| 211 | }; | ||
| 212 | let mut table = (*previous.global_ids).clone(); | ||
| 213 | table.insert(table.last_key_value().unwrap().0 + 1, style.guid); | ||
| 214 | let mut changed = BTreeMap::from([( | ||
| 215 | style, | ||
| 216 | PropertyObject { | ||
| 217 | jcid: previous.jcid, | ||
| 218 | bytes: patch_properties(blob, &[], &[(0x88001c04, &[])])?, | ||
| 219 | global_ids: Arc::new(table), | ||
| 220 | }, | ||
| 221 | )]); | ||
| 222 | for (id, text) in revision.objects.iter().filter(|(_, o)| o.jcid == 0x6000e) { | ||
| 223 | let ObjectData::Properties(blob) = text.data else { | ||
| 224 | unreachable!() | ||
| 225 | }; | ||
| 226 | let mut table = (*text.global_ids).clone(); | ||
| 227 | table.insert(table.last_key_value().unwrap().0 + 1, style.guid); | ||
| 228 | changed.insert( | ||
| 229 | *id, | ||
| 230 | PropertyObject { | ||
| 231 | jcid: text.jcid, | ||
| 232 | bytes: patch_properties(blob, &[(0x24001e13, &compact(style, &table)?)], &[])?, | ||
| 233 | global_ids: Arc::new(table), | ||
| 234 | }, | ||
| 235 | ); | ||
| 236 | } | ||
| 237 | Ok(changed) | ||
| 238 | }) | ||
| 239 | .unwrap() | ||
| 240 | } | ||
| 241 | |||
| 242 | #[test] | ||
| 243 | fn replaced_readonly_styles_retain_history_with_zero_current_references() { | ||
| 244 | use crate::{ExGuid, RevisionIndex, Store}; | ||
| 245 | let source = add_paragraph( | ||
| 246 | &crate::create_section("style.one", "Original", "Author").unwrap(), | ||
| 247 | 1, | ||
| 248 | ); | ||
| 249 | let changed = restyle(&source); | ||
| 250 | let store = Store::parse(&changed).unwrap(); | ||
| 251 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 252 | index.validate_current().unwrap(); | ||
| 253 | let mut styles = Vec::new(); | ||
| 254 | for (sid, s) in &index.spaces { | ||
| 255 | let revision = index | ||
| 256 | .resolve(*sid, s.labels[&(ExGuid::default(), 1)]) | ||
| 257 | .unwrap(); | ||
| 258 | for (id, object) in &revision.objects { | ||
| 259 | if object.jcid == 0x12004d { | ||
| 260 | styles.push((object.reference_count, id.guid)); | ||
| 261 | } | ||
| 262 | } | ||
| 263 | } | ||
| 264 | styles.sort(); | ||
| 265 | assert_eq!(styles.len(), 2); | ||
| 266 | assert_eq!(styles[0].0, 0); | ||
| 267 | assert_eq!(styles[1], (2, [0x74; 16])); | ||
| 268 | let document = crate::document::Document::parse(&index).unwrap(); | ||
| 269 | let mut count = 0; | ||
| 270 | for space in document.spaces.values() { | ||
| 271 | for revision in space.revisions.values() { | ||
| 272 | for (id, node) in &revision.nodes { | ||
| 273 | if matches!(node.kind, crate::document::Kind::RichText { .. }) { | ||
| 274 | for run in revision.text_runs(*id).unwrap() { | ||
| 275 | assert_eq!(run.format.bold, Some(true)); | ||
| 276 | } | ||
| 277 | count += 1; | ||
| 278 | } | ||
| 279 | } | ||
| 280 | } | ||
| 281 | } | ||
| 282 | assert_eq!(count, 2); | ||
| 283 | } | ||
| 284 | |||
| 285 | #[test] | ||
| 286 | #[ignore = "Writes cold-native candidates to ONESTORE_GROWTH_OUTPUT"] | ||
| 287 | fn export_native_growth_candidates() { | ||
| 288 | let output = std::path::PathBuf::from(std::env::var_os("ONESTORE_GROWTH_OUTPUT").unwrap()); | ||
| 289 | std::fs::create_dir(&output).unwrap(); | ||
| 290 | let mut source = crate::create_section("growth.one", "Original", "Author").unwrap(); | ||
| 291 | for number in 1..=24 { | ||
| 292 | source = add_paragraph(&source, number); | ||
| 293 | } | ||
| 294 | std::fs::write(output.join("growth.one"), &source).unwrap(); | ||
| 295 | std::fs::write(output.join("restyled.one"), restyle(&source)).unwrap(); | ||
| 296 | } | ||
| 297 | |||
| 298 | #[test] | ||
| 299 | fn atomic_graph_growth_preserves_history_and_counts_shared_readonly_objects() { | ||
| 300 | use crate::{ExGuid, RevisionIndex, Store}; | ||
| 301 | let initial = crate::create_section("growth.one", "Original", "Author").unwrap(); | ||
| 302 | let mut source = initial.clone(); | ||
| 303 | for number in 1..=24 { | ||
| 304 | source = add_paragraph(&source, number); | ||
| 305 | let store = Store::parse(&source).unwrap(); | ||
| 306 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 307 | index.validate_current().unwrap(); | ||
| 308 | assert!(store.checksum_mismatches.is_empty()); | ||
| 309 | assert_eq!(store.header.transaction_count, number + 1); | ||
| 310 | let mut text_count = 0; | ||
| 311 | for (sid, s) in &index.spaces { | ||
| 312 | let revision = index | ||
| 313 | .resolve(*sid, s.labels[&(ExGuid::default(), 1)]) | ||
| 314 | .unwrap(); | ||
| 315 | let reachable = revision.reachable().unwrap(); | ||
| 316 | for id in reachable { | ||
| 317 | let object = &revision.objects[&id]; | ||
| 318 | match object.jcid { | ||
| 319 | 0x6000e => text_count += 1, | ||
| 320 | 0x120001 => assert_eq!(object.reference_count, (number + 1) * 2), | ||
| 321 | 0x12004d => assert_eq!(object.reference_count, number + 1), | ||
| 322 | _ => {} | ||
| 323 | } | ||
| 324 | } | ||
| 325 | } | ||
| 326 | assert_eq!(text_count, number + 1); | ||
| 327 | } | ||
| 328 | let initial_store = Store::parse(&initial).unwrap(); | ||
| 329 | let initial_index = RevisionIndex::parse(&initial_store).unwrap(); | ||
| 330 | let store = Store::parse(&source).unwrap(); | ||
| 331 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 332 | for (sid, s) in &initial_index.spaces { | ||
| 333 | let rid = s.labels[&(ExGuid::default(), 1)]; | ||
| 334 | let before = initial_index.resolve(*sid, rid).unwrap(); | ||
| 335 | let after = index.resolve(*sid, rid).unwrap(); | ||
| 336 | assert_eq!(before.roots, after.roots); | ||
| 337 | for (id, object) in &before.objects { | ||
| 338 | assert_eq!(object.data, after.objects[id].data); | ||
| 339 | assert_eq!(object.reference_count, after.objects[id].reference_count); | ||
| 340 | } | ||
| 341 | } | ||
| 342 | } | ||
| 343 | |||
| 344 | #[test] | ||
| 345 | fn changed_graphs_reject_cycles_dangling_and_unreachable_additions() { | ||
| 346 | use super::{PropertyObject, compact, write_revision}; | ||
| 347 | use crate::{ExGuid, ObjectData, RevisionIndex, Store}; | ||
| 348 | use std::{collections::BTreeMap, sync::Arc}; | ||
| 349 | let source = crate::create_section("invalid.one", "Original", "Author").unwrap(); | ||
| 350 | let store = Store::parse(&source).unwrap(); | ||
| 351 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 352 | for (sid, s) in &index.spaces { | ||
| 353 | let revision = index | ||
| 354 | .resolve(*sid, s.labels[&(ExGuid::default(), 1)]) | ||
| 355 | .unwrap(); | ||
| 356 | let Some((&oid, outline)) = revision.objects.iter().find(|(_, o)| o.jcid == 0x6000c) else { | ||
| 357 | continue; | ||
| 358 | }; | ||
| 359 | let ObjectData::Properties(blob) = outline.data else { | ||
| 360 | unreachable!() | ||
| 361 | }; | ||
| 362 | for target in [ | ||
| 363 | oid, | ||
| 364 | ExGuid { | ||
| 365 | guid: oid.guid, | ||
| 366 | n: 255, | ||
| 367 | }, | ||
| 368 | ] { | ||
| 369 | assert!( | ||
| 370 | write_revision(&source, *sid, |_| Ok(BTreeMap::from([( | ||
| 371 | oid, | ||
| 372 | PropertyObject { | ||
| 373 | jcid: outline.jcid, | ||
| 374 | bytes: patch_properties( | ||
| 375 | blob, | ||
| 376 | &[(0x24001c20, &compact(target, &outline.global_ids)?)], | ||
| 377 | &[] | ||
| 378 | )?, | ||
| 379 | global_ids: Arc::clone(&outline.global_ids), | ||
| 380 | } | ||
| 381 | )]))) | ||
| 382 | .is_err() | ||
| 383 | ); | ||
| 384 | } | ||
| 385 | let orphan = ExGuid { | ||
| 386 | guid: oid.guid, | ||
| 387 | n: 255, | ||
| 388 | }; | ||
| 389 | assert!( | ||
| 390 | write_revision(&source, *sid, |_| Ok(BTreeMap::from([( | ||
| 391 | orphan, | ||
| 392 | PropertyObject { | ||
| 393 | jcid: 0x6000e, | ||
| 394 | bytes: properties(&[])?, | ||
| 395 | global_ids: Arc::clone(&outline.global_ids), | ||
| 396 | } | ||
| 397 | )]))) | ||
| 398 | .is_err() | ||
| 399 | ); | ||
| 400 | } | ||
| 401 | } | ||
| 402 | |||
| 403 | #[test] | ||
| 404 | fn property_splices_match_independently_encoded_flat_sets() { | ||
| 405 | let mut seed = 0x749391acb66327d5_u64; | ||
| 406 | let mut next = || { | ||
| 407 | seed ^= seed << 13; | ||
| 408 | seed ^= seed >> 7; | ||
| 409 | seed ^= seed << 17; | ||
| 410 | seed | ||
| 411 | }; | ||
| 412 | for case in 0..20_000 { | ||
| 413 | let mut original = Vec::new(); | ||
| 414 | let mut expected = Vec::new(); | ||
| 415 | let mut updates = Vec::new(); | ||
| 416 | let mut inserts = Vec::new(); | ||
| 417 | for index in 0..next() % 30 { | ||
| 418 | let kind = 2 + (next() % 8) as u32; | ||
| 419 | let mut id = (kind << 26) | index as u32; | ||
| 420 | let length = |random: u64| match kind { | ||
| 421 | 2 => 0, | ||
| 422 | 3..=6 => 1 << (kind - 3), | ||
| 423 | 7 => (random % 20) as usize, | ||
| 424 | 8 => 4, | ||
| 425 | 9 => (random % 5) as usize * 4, | ||
| 426 | _ => unreachable!(), | ||
| 427 | }; | ||
| 428 | let value = (0..length(next())) | ||
| 429 | .map(|_| next() as u8) | ||
| 430 | .collect::<Vec<_>>(); | ||
| 431 | if kind == 2 && next() & 1 != 0 { | ||
| 432 | id |= 0x80000000; | ||
| 433 | } | ||
| 434 | original.push((id, value.clone())); | ||
| 435 | if next() & 1 != 0 { | ||
| 436 | if kind == 2 { | ||
| 437 | id ^= 0x80000000; | ||
| 438 | } | ||
| 439 | let value = (0..length(next())) | ||
| 440 | .map(|_| next() as u8) | ||
| 441 | .collect::<Vec<_>>(); | ||
| 442 | expected.push((id, value.clone())); | ||
| 443 | updates.push((id, value)); | ||
| 444 | } else { | ||
| 445 | expected.push((id, value)); | ||
| 446 | } | ||
| 447 | } | ||
| 448 | for index in 0..next() % 6 { | ||
| 449 | let kind = [2, 7, 8, 9][(next() % 4) as usize]; | ||
| 450 | let mut id = (kind << 26) | (100 + index as u32); | ||
| 451 | if kind == 2 && next() & 1 != 0 { | ||
| 452 | id |= 0x80000000; | ||
| 453 | } | ||
| 454 | let length = match kind { | ||
| 455 | 2 => 0, | ||
| 456 | 7 => next() as usize % 10, | ||
| 457 | 8 => 4, | ||
| 458 | 9 => (next() as usize % 4) * 4, | ||
| 459 | _ => unreachable!(), | ||
| 460 | }; | ||
| 461 | let value = (0..length).map(|_| next() as u8).collect::<Vec<_>>(); | ||
| 462 | inserts.push((id, value.clone())); | ||
| 463 | expected.push((id, value)); | ||
| 464 | } | ||
| 465 | updates.reverse(); | ||
| 466 | let updates: Vec<_> = updates | ||
| 467 | .iter() | ||
| 468 | .map(|(id, value)| (*id, value.as_slice())) | ||
| 469 | .collect(); | ||
| 470 | let inserts: Vec<_> = inserts | ||
| 471 | .iter() | ||
| 472 | .map(|(id, value)| (*id, value.as_slice())) | ||
| 473 | .collect(); | ||
| 474 | let original = properties(&original).unwrap(); | ||
| 475 | let actual = patch_properties(&original, &updates, &inserts).unwrap(); | ||
| 476 | assert_eq!(actual, properties(&expected).unwrap(), "case {case}"); | ||
| 477 | } | ||
| 478 | } | ||
| 479 | |||
| 480 | #[test] | ||
| 481 | fn nested_fields_and_other_reference_streams_remain_byte_exact() { | ||
| 482 | let mut original = 0x40000003_u32.to_le_bytes().to_vec(); | ||
| 483 | original.extend_from_slice(&[1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0]); | ||
| 484 | original.extend_from_slice(&0x40000001_u32.to_le_bytes()); | ||
| 485 | original.extend_from_slice(&[4, 0, 0, 0]); | ||
| 486 | original.extend_from_slice(&1_u32.to_le_bytes()); | ||
| 487 | original.extend_from_slice(&[5, 0, 0, 0]); | ||
| 488 | original.extend_from_slice(&4_u16.to_le_bytes()); | ||
| 489 | for id in [0x24000001_u32, 0x40000002, 0x24000003, 0x1c000004] { | ||
| 490 | original.extend_from_slice(&id.to_le_bytes()); | ||
| 491 | } | ||
| 492 | original.extend_from_slice(&1_u32.to_le_bytes()); | ||
| 493 | let nested_start = original.len(); | ||
| 494 | original.extend_from_slice(&1_u32.to_le_bytes()); | ||
| 495 | original.extend_from_slice(&0x44003456_u32.to_le_bytes()); | ||
| 496 | original.extend_from_slice(&4_u16.to_le_bytes()); | ||
| 497 | for id in [0x20000001_u32, 0x28000002, 0x30000003, 0x1c000004] { | ||
| 498 | original.extend_from_slice(&id.to_le_bytes()); | ||
| 499 | } | ||
| 500 | original.extend_from_slice(&3_u32.to_le_bytes()); | ||
| 501 | original.extend_from_slice(&[91, 92, 93]); | ||
| 502 | let nested_end = original.len(); | ||
| 503 | original.extend_from_slice(&1_u32.to_le_bytes()); | ||
| 504 | original.extend_from_slice(&2_u32.to_le_bytes()); | ||
| 505 | original.extend_from_slice(&[94, 95]); | ||
| 506 | original.resize(original.len().next_multiple_of(8), 0); | ||
| 507 | let changed = patch_properties( | ||
| 508 | &original, | ||
| 509 | &[ | ||
| 510 | (0x24000003, &[]), | ||
| 511 | (0x1c000004, &[96, 97, 98, 99]), | ||
| 512 | (0x24000001, &[6, 0, 0, 0, 7, 0, 0, 0]), | ||
| 513 | ], | ||
| 514 | &[(0x24000005, &[8, 0, 0, 0]), (0x88000006, &[])], | ||
| 515 | ) | ||
| 516 | .unwrap(); | ||
| 517 | let parsed = PropertySets::parse(&changed).unwrap(); | ||
| 518 | let previous = PropertySets::parse(&original).unwrap(); | ||
| 519 | assert_eq!(parsed.sets[1], previous.sets[1]); | ||
| 520 | let nested = &original[nested_start..nested_end]; | ||
| 521 | assert_eq!( | ||
| 522 | changed | ||
| 523 | .windows(nested.len()) | ||
| 524 | .filter(|bytes| *bytes == nested) | ||
| 525 | .count(), | ||
| 526 | 1 | ||
| 527 | ); | ||
| 528 | assert_eq!( | ||
| 529 | &changed[4..20], | ||
| 530 | &[6, 0, 0, 0, 7, 0, 0, 0, 2, 0, 0, 0, 8, 0, 0, 0] | ||
| 531 | ); | ||
| 532 | assert_eq!(&changed[20..36], &original[16..32]); | ||
| 533 | assert_eq!(parsed.sets[0][3].value, Value::Bytes(&[96, 97, 98, 99])); | ||
| 534 | } | ||
| 535 | |||
| 536 | #[test] | ||
| 537 | fn deep_property_splices_do_not_use_the_call_stack() { | ||
| 538 | let mut bytes = 0x80000000_u32.to_le_bytes().to_vec(); | ||
| 539 | for _ in 0..100_000 { | ||
| 540 | bytes.extend_from_slice(&1_u16.to_le_bytes()); | ||
| 541 | bytes.extend_from_slice(&0x44000001_u32.to_le_bytes()); | ||
| 542 | } | ||
| 543 | bytes.extend_from_slice(&0_u16.to_le_bytes()); | ||
| 544 | let changed = patch_properties(&bytes, &[], &[(0x88000002, &[])]).unwrap(); | ||
| 545 | let parsed = PropertySets::parse(&changed).unwrap(); | ||
| 546 | assert_eq!(parsed.sets.len(), 100_001); | ||
| 547 | assert_eq!(parsed.sets[0].len(), 2); | ||
| 548 | assert_eq!( | ||
| 549 | &changed[14..changed.len() - parsed.padding.len()], | ||
| 550 | &bytes[10..] | ||
| 551 | ); | ||
| 552 | } | ||
| 553 | |||
| 554 | #[test] | ||
| 555 | fn invalid_property_splices_are_rejected() { | ||
| 556 | let bytes = properties(&[(0x08000001, vec![]), (0x24000002, vec![])]).unwrap(); | ||
| 557 | for (updates, inserts) in [ | ||
| 558 | (vec![(0x88000001, &[][..]), (0x08000001, &[])], vec![]), | ||
| 559 | (vec![(0x14000003, &[0; 4][..])], vec![]), | ||
| 560 | (vec![(0x24000002, &[0; 3][..])], vec![]), | ||
| 561 | (vec![(0x88000001, &[1][..])], vec![]), | ||
| 562 | (vec![], vec![(0x88000001, &[][..])]), | ||
| 563 | (vec![], vec![(0x20000003, &[][..])]), | ||
| 564 | ] { | ||
| 565 | assert!(patch_properties(&bytes, &updates, &inserts).is_err()); | ||
| 566 | } | ||
| 567 | } | ||
| 568 | |||
| 569 | #[test] | ||
| 570 | fn character_formatting_preserves_inheritance_and_associated_data() { | ||
| 571 | use super::{PropertyObject, write_revision}; | ||
| 572 | use crate::{ | ||
| 573 | ExGuid, PreparedEdit, RevisionIndex, Store, TextAttribute, | ||
| 574 | document::{Document, Kind}, | ||
| 575 | }; | ||
| 576 | use std::collections::BTreeMap; | ||
| 577 | let source = crate::create_section("inherited.one", "abcdef", "Author").unwrap(); | ||
| 578 | let store = Store::parse(&source).unwrap(); | ||
| 579 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 580 | let (sid, text) = index | ||
| 581 | .spaces | ||
| 582 | .iter() | ||
| 583 | .find_map(|(sid, s)| { | ||
| 584 | let raw = index | ||
| 585 | .resolve(*sid, s.labels[&(ExGuid::default(), 1)]) | ||
| 586 | .unwrap(); | ||
| 587 | raw.objects | ||
| 588 | .iter() | ||
| 589 | .find_map(|(id, o)| (o.jcid == 0x6000e).then_some((*sid, *id))) | ||
| 590 | }) | ||
| 591 | .unwrap(); | ||
| 592 | for variant in 0..7 { | ||
| 593 | let fixture = write_revision(&source, sid, |raw| { | ||
| 594 | let mut target = PropertyObject::from_object(&raw.objects[&text])?; | ||
| 595 | target.bytes = properties(&[ | ||
| 596 | (0x1c001c22, crate::create::string("abcdef")), | ||
| 597 | (0x14001d7a, vec![0; 4]), | ||
| 598 | (0x1c001c0a, crate::create::string("Georgia")), | ||
| 599 | (0x88001c05, Vec::new()), | ||
| 600 | ])?; | ||
| 601 | match variant { | ||
| 602 | 1 | 4 | 5 | 6 => { | ||
| 603 | let flag = match variant { | ||
| 604 | 1 => 0x88001e16, | ||
| 605 | 4 => 0x88001e14, | ||
| 606 | 5 => 0x88003401, | ||
| 607 | _ => 0x88001e22, | ||
| 608 | }; | ||
| 609 | target.set(&[(flag, &[])])?; | ||
| 610 | } | ||
| 611 | 2 => { | ||
| 612 | let author = raw | ||
| 613 | .objects | ||
| 614 | .iter() | ||
| 615 | .find_map(|(id, o)| (o.jcid == 0x120001).then_some(*id)) | ||
| 616 | .unwrap(); | ||
| 617 | let reference = target.reference(author)?; | ||
| 618 | target.set(&[(0x24003458, &reference)])?; | ||
| 619 | } | ||
| 620 | 3 => { | ||
| 621 | // An unknown nested run property must survive a style-only edit byte for byte. | ||
| 622 | let parsed = PropertySets::parse(&target.bytes)?; | ||
| 623 | let at = parsed.root_ids.as_ptr().addr() - target.bytes.as_ptr().addr(); | ||
| 624 | let count = parsed.sets[0].len(); | ||
| 625 | let end = target.bytes.len() - parsed.padding.len(); | ||
| 626 | let mut bytes = target.bytes[..end].to_vec(); | ||
| 627 | bytes[at - 2..at].copy_from_slice(&((count + 1) as u16).to_le_bytes()); | ||
| 628 | bytes.splice(at + count * 4..at + count * 4, 0x40003499_u32.to_le_bytes()); | ||
| 629 | bytes.extend_from_slice(&1_u32.to_le_bytes()); | ||
| 630 | bytes.extend_from_slice(&0x44001234_u32.to_le_bytes()); | ||
| 631 | bytes.extend_from_slice(&1_u16.to_le_bytes()); | ||
| 632 | bytes.extend_from_slice(&0x14001234_u32.to_le_bytes()); | ||
| 633 | bytes.extend_from_slice(&0xdeadbeef_u32.to_le_bytes()); | ||
| 634 | bytes.resize(bytes.len().next_multiple_of(8), 0); | ||
| 635 | target.bytes = bytes; | ||
| 636 | } | ||
| 637 | _ => {} | ||
| 638 | } | ||
| 639 | Ok(BTreeMap::from([(text, target)])) | ||
| 640 | }) | ||
| 641 | .unwrap(); | ||
| 642 | let edit = PreparedEdit::format(&fixture, sid, text, 0..6, &[TextAttribute::Bold(true)]); | ||
| 643 | if matches!(variant, 1 | 2 | 4 | 5 | 6) { | ||
| 644 | assert!(edit.is_err()); | ||
| 645 | continue; | ||
| 646 | } | ||
| 647 | let edit = edit.unwrap(); | ||
| 648 | let store = Store::parse(edit.as_bytes()).unwrap(); | ||
| 649 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 650 | let doc = Document::parse(&index).unwrap(); | ||
| 651 | let s = &doc.spaces[&sid]; | ||
| 652 | let view = &s.revisions[&s.contexts[&ExGuid::default()]]; | ||
| 653 | let runs = view.text_runs(text).unwrap(); | ||
| 654 | assert_eq!(runs.len(), 1); | ||
| 655 | assert_eq!(runs[0].format.font.as_deref(), Some("Georgia")); | ||
| 656 | assert_eq!(runs[0].format.italic, Some(true)); | ||
| 657 | assert_eq!(runs[0].format.bold, Some(true)); | ||
| 658 | if variant == 3 { | ||
| 659 | let Kind::RichText { runs, .. } = &view.nodes[&text].kind else { | ||
| 660 | panic!() | ||
| 661 | }; | ||
| 662 | let data = &view.nodes[&text].extra[runs[0].extra_set.unwrap()]; | ||
| 663 | assert_eq!(data.len(), 1); | ||
| 664 | assert_eq!(data[0].id, 0x14001234); | ||
| 665 | assert!( | ||
| 666 | PreparedEdit::format(&fixture, sid, text, 1..3, &[TextAttribute::Bold(true)]) | ||
| 667 | .is_err() | ||
| 668 | ); | ||
| 669 | let previous = Store::parse(&fixture).unwrap(); | ||
| 670 | let previous = RevisionIndex::parse(&previous).unwrap(); | ||
| 671 | let previous_doc = Document::parse(&previous).unwrap(); | ||
| 672 | let s = &previous_doc.spaces[&sid]; | ||
| 673 | let previous_view = &s.revisions[&s.contexts[&ExGuid::default()]]; | ||
| 674 | assert_eq!( | ||
| 675 | format!("{:?}", view.nodes[&text].extra), | ||
| 676 | format!("{:?}", previous_view.nodes[&text].extra) | ||
| 677 | ); | ||
| 678 | } | ||
| 679 | } | ||
| 680 | } | ||
crates/onestore/tests/edit.rs+201-4
| ... | @@ -2,6 +2,8 @@ | ... | @@ -2,6 +2,8 @@ |
| 2 | mod checkpoint; | 2 | mod checkpoint; |
| 3 | #[path = "support/disk.rs"] | 3 | #[path = "support/disk.rs"] |
| 4 | mod disk; | 4 | mod disk; |
| 5 | #[path = "support/trace.rs"] | ||
| 6 | mod trace; | ||
| 5 | 7 | ||
| 6 | use disk::Disk; | 8 | use disk::Disk; |
| 7 | use onestore::{ | 9 | use onestore::{ |
| ... | @@ -44,6 +46,201 @@ fn text_runs(source: &[u8], sid: ExGuid, oid: ExGuid) -> serde_json::Value { | ... | @@ -44,6 +46,201 @@ fn text_runs(source: &[u8], sid: ExGuid, oid: ExGuid) -> serde_json::Value { |
| 44 | .unwrap() | 46 | .unwrap() |
| 45 | } | 47 | } |
| 46 | 48 | ||
| 49 | fn assert_refreshed(source: &[u8], confirmed: &[u8]) { | ||
| 50 | assert_eq!(&source[..212], &confirmed[..212]); | ||
| 51 | assert_eq!(&source[252..], &confirmed[252..]); | ||
| 52 | let before = Store::parse(source).unwrap().header; | ||
| 53 | let after = Store::parse(confirmed).unwrap().header; | ||
| 54 | assert_ne!(before.version_id, after.version_id); | ||
| 55 | assert_ne!(before.deny_read_id, after.deny_read_id); | ||
| 56 | assert_eq!(after.generation, before.generation + 1); | ||
| 57 | } | ||
| 58 | |||
| 59 | #[test] | ||
| 60 | fn prepared_publication_preserves_its_identity_through_every_io_failure() { | ||
| 61 | let source = | ||
| 62 | onestore::create_section("prepared.one", "Fictitious: café 🦀", "Fixture").unwrap(); | ||
| 63 | let (sid, oid) = target(&source); | ||
| 64 | let source = checkpoint::pending(&source, sid, oid, 0x14001d7a); | ||
| 65 | let edit = onestore::PreparedEdit::text(&source, sid, oid, 0..0, "Prepared 🐈 ").unwrap(); | ||
| 66 | let store = Store::parse(edit.as_bytes()).unwrap(); | ||
| 67 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 68 | let planned = index.spaces[&sid].labels[&(ExGuid::default(), 1)]; | ||
| 69 | let persisted: ExGuid = | ||
| 70 | serde_json::from_str(&serde_json::to_string(&planned).unwrap()).unwrap(); | ||
| 71 | let before_store = Store::parse(&source).unwrap(); | ||
| 72 | let before_index = RevisionIndex::parse(&before_store).unwrap(); | ||
| 73 | assert!(!before_index.spaces[&sid].revisions.contains_key(&persisted)); | ||
| 74 | let before = text_runs(&source, sid, oid); | ||
| 75 | let after = text_runs(edit.as_bytes(), sid, oid); | ||
| 76 | for write_limit in [17, 4096] { | ||
| 77 | let disk = |fail_at| Disk { | ||
| 78 | visible: source.clone(), | ||
| 79 | durable: source.clone(), | ||
| 80 | operation: 0, | ||
| 81 | fail_at, | ||
| 82 | write_limit, | ||
| 83 | random: 911, | ||
| 84 | }; | ||
| 85 | let mut success = disk(None); | ||
| 86 | edit.commit(&mut success).unwrap(); | ||
| 87 | assert_eq!(success.durable, edit.as_bytes()); | ||
| 88 | let operations = success.operation; | ||
| 89 | let error = edit.commit(&mut success).unwrap_err(); | ||
| 90 | assert_eq!(error.state, CommitState::NotCommitted); | ||
| 91 | assert_eq!(error.error.kind(), std::io::ErrorKind::ResourceBusy); | ||
| 92 | assert_eq!(success.durable, edit.as_bytes()); | ||
| 93 | for at in 1..=operations { | ||
| 94 | let mut interrupted = disk(Some(at)); | ||
| 95 | let error = edit.commit(&mut interrupted).unwrap_err(); | ||
| 96 | let observed = text_runs(&interrupted.durable, sid, oid); | ||
| 97 | let store = Store::parse(&interrupted.durable).unwrap(); | ||
| 98 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 99 | let present = index.spaces[&sid].revisions.contains_key(&persisted); | ||
| 100 | assert_eq!(observed, if present { &after } else { &before }.clone()); | ||
| 101 | match error.state { | ||
| 102 | CommitState::NotCommitted => assert!(!present), | ||
| 103 | CommitState::Committed => assert!(present), | ||
| 104 | CommitState::Unknown => {} | ||
| 105 | } | ||
| 106 | if present { | ||
| 107 | let snapshot = interrupted.durable.clone(); | ||
| 108 | interrupted.visible.clone_from(&snapshot); | ||
| 109 | interrupted.fail_at = None; | ||
| 110 | interrupted.write_limit = 17; | ||
| 111 | onestore::confirm_snapshot(&mut interrupted, &snapshot).unwrap(); | ||
| 112 | assert_refreshed(&snapshot, &interrupted.durable); | ||
| 113 | } | ||
| 114 | } | ||
| 115 | } | ||
| 116 | } | ||
| 117 | |||
| 118 | #[test] | ||
| 119 | fn snapshot_confirmation_needs_no_surviving_edit_target() { | ||
| 120 | let mut disk = Disk { | ||
| 121 | visible: SOURCE.to_vec(), | ||
| 122 | durable: Vec::new(), | ||
| 123 | operation: 0, | ||
| 124 | fail_at: None, | ||
| 125 | write_limit: 17, | ||
| 126 | random: 911, | ||
| 127 | }; | ||
| 128 | assert!( | ||
| 129 | onestore::replace_text(SOURCE, ExGuid::default(), ExGuid::default(), 0..0, "").is_err() | ||
| 130 | ); | ||
| 131 | onestore::confirm_snapshot(&mut disk, SOURCE).unwrap(); | ||
| 132 | assert_refreshed(SOURCE, &disk.durable); | ||
| 133 | let flush = disk.operation; | ||
| 134 | for (failure, state) in [ | ||
| 135 | (1, CommitState::NotCommitted), | ||
| 136 | (flush, CommitState::Unknown), | ||
| 137 | ] { | ||
| 138 | disk.visible = SOURCE.to_vec(); | ||
| 139 | disk.durable.clear(); | ||
| 140 | disk.operation = 0; | ||
| 141 | disk.fail_at = Some(failure); | ||
| 142 | assert_eq!( | ||
| 143 | onestore::confirm_snapshot(&mut disk, SOURCE) | ||
| 144 | .unwrap_err() | ||
| 145 | .state, | ||
| 146 | state | ||
| 147 | ); | ||
| 148 | } | ||
| 149 | } | ||
| 150 | |||
| 151 | #[test] | ||
| 152 | fn confirmation_notifies_cached_readers_after_interrupted_version_publication() { | ||
| 153 | let source = | ||
| 154 | onestore::create_section("confirmation.one", "Fictitious: before", "Fixture").unwrap(); | ||
| 155 | let (sid, oid) = target(&source); | ||
| 156 | let mut snapshot = onestore::replace_text(&source, sid, oid, 0..0, "Recovered ").unwrap(); | ||
| 157 | snapshot[212..252].copy_from_slice(&source[212..252]); | ||
| 158 | let expected = text_runs(&snapshot, sid, oid); | ||
| 159 | assert_ne!(expected, text_runs(&source, sid, oid)); | ||
| 160 | for write_limit in [1, 17, 40] { | ||
| 161 | let disk = |fail_at| Disk { | ||
| 162 | visible: snapshot.clone(), | ||
| 163 | durable: snapshot.clone(), | ||
| 164 | operation: 0, | ||
| 165 | fail_at, | ||
| 166 | write_limit, | ||
| 167 | random: 911, | ||
| 168 | }; | ||
| 169 | let mut success = disk(None); | ||
| 170 | onestore::confirm_snapshot(&mut success, &snapshot).unwrap(); | ||
| 171 | assert_refreshed(&snapshot, &success.durable); | ||
| 172 | for failure in 1..=success.operation { | ||
| 173 | let mut interrupted = disk(Some(failure)); | ||
| 174 | let error = onestore::confirm_snapshot(&mut interrupted, &snapshot).unwrap_err(); | ||
| 175 | assert_ne!(error.state, CommitState::Committed); | ||
| 176 | assert_eq!(text_runs(&interrupted.durable, sid, oid), expected); | ||
| 177 | assert_eq!(&interrupted.durable[..212], &snapshot[..212]); | ||
| 178 | assert_eq!(&interrupted.durable[252..], &snapshot[252..]); | ||
| 179 | } | ||
| 180 | } | ||
| 181 | } | ||
| 182 | |||
| 183 | #[test] | ||
| 184 | fn confirming_visible_text_requires_flush_without_another_revision() { | ||
| 185 | let (sid, oid) = target(SOURCE); | ||
| 186 | let visible = onestore::replace_text(SOURCE, sid, oid, 0..0, "Recovered ").unwrap(); | ||
| 187 | let mut disk = Disk { | ||
| 188 | visible: visible.clone(), | ||
| 189 | durable: SOURCE.to_vec(), | ||
| 190 | operation: 0, | ||
| 191 | fail_at: None, | ||
| 192 | write_limit: 0, | ||
| 193 | random: 1, | ||
| 194 | }; | ||
| 195 | onestore::commit_text(&mut disk, &visible, sid, oid, 0..0, "").unwrap(); | ||
| 196 | assert_eq!(disk.visible, visible); | ||
| 197 | assert_eq!(disk.durable, visible); | ||
| 198 | let flush = disk.operation; | ||
| 199 | for (failure, state) in [ | ||
| 200 | (1, CommitState::NotCommitted), | ||
| 201 | (flush, CommitState::Unknown), | ||
| 202 | ] { | ||
| 203 | disk.durable = SOURCE.to_vec(); | ||
| 204 | disk.operation = 0; | ||
| 205 | disk.fail_at = Some(failure); | ||
| 206 | let error = onestore::commit_text(&mut disk, &visible, sid, oid, 0..0, "").unwrap_err(); | ||
| 207 | assert_eq!(error.state, state); | ||
| 208 | assert_eq!(disk.visible, visible); | ||
| 209 | } | ||
| 210 | } | ||
| 211 | |||
| 212 | #[test] | ||
| 213 | fn confirmation_rejects_changed_or_truncated_physical_tail_before_any_write() { | ||
| 214 | let (sid, oid) = target(SOURCE); | ||
| 215 | let mut source = SOURCE.to_vec(); | ||
| 216 | source.resize(3 * 1024 * 1024 + 131, 0); | ||
| 217 | let mut disk = trace::Trace { | ||
| 218 | bytes: source.clone(), | ||
| 219 | events: Vec::new(), | ||
| 220 | }; | ||
| 221 | onestore::commit_text(&mut disk, &source, sid, oid, 0..0, "").unwrap(); | ||
| 222 | assert_eq!(disk.events.len(), 1); | ||
| 223 | if let trace::Event::Write(offset, bytes) = &disk.events[0] { | ||
| 224 | panic!("Confirmation wrote {} bytes at {offset}", bytes.len()); | ||
| 225 | } | ||
| 226 | for changed in [65535, 65536, 1048575, 1048576, 2097152, source.len() - 1] { | ||
| 227 | disk.bytes.clone_from(&source); | ||
| 228 | disk.bytes[changed] ^= 1; | ||
| 229 | disk.events.clear(); | ||
| 230 | let error = onestore::commit_text(&mut disk, &source, sid, oid, 0..0, "").unwrap_err(); | ||
| 231 | assert_eq!(error.state, CommitState::NotCommitted); | ||
| 232 | assert_eq!(error.error.kind(), std::io::ErrorKind::ResourceBusy); | ||
| 233 | assert!(disk.events.is_empty()); | ||
| 234 | } | ||
| 235 | for length in [source.len() - 1, source.len() + 1] { | ||
| 236 | disk.bytes.clone_from(&source); | ||
| 237 | disk.bytes.resize(length, 0); | ||
| 238 | let error = onestore::commit_text(&mut disk, &source, sid, oid, 0..0, "").unwrap_err(); | ||
| 239 | assert_eq!(error.state, CommitState::NotCommitted); | ||
| 240 | assert!(disk.events.is_empty()); | ||
| 241 | } | ||
| 242 | } | ||
| 243 | |||
| 47 | fn assert_other_objects_preserved( | 244 | fn assert_other_objects_preserved( |
| 48 | before: &onestore::ResolvedRevision<'_>, | 245 | before: &onestore::ResolvedRevision<'_>, |
| 49 | after: &onestore::ResolvedRevision<'_>, | 246 | after: &onestore::ResolvedRevision<'_>, |
| ... | @@ -398,6 +595,8 @@ fn title_text_and_navigation_caches_publish_together() { | ... | @@ -398,6 +595,8 @@ fn title_text_and_navigation_caches_publish_together() { |
| 398 | for (source, write_limit) in [(source.as_slice(), 17), (checkpoint.as_slice(), 257)] { | 595 | for (source, write_limit) in [(source.as_slice(), 17), (checkpoint.as_slice(), 257)] { |
| 399 | let before = state(source); | 596 | let before = state(source); |
| 400 | for replacement in ["Renamed 🦀 日本語", ""] { | 597 | for replacement in ["Renamed 🦀 日本語", ""] { |
| 598 | let edit = | ||
| 599 | onestore::PreparedEdit::text(source, sid, *oid, 0..end, replacement).unwrap(); | ||
| 401 | let disk = |fail_at| Disk { | 600 | let disk = |fail_at| Disk { |
| 402 | visible: source.to_vec(), | 601 | visible: source.to_vec(), |
| 403 | durable: source.to_vec(), | 602 | durable: source.to_vec(), |
| ... | @@ -407,7 +606,7 @@ fn title_text_and_navigation_caches_publish_together() { | ... | @@ -407,7 +606,7 @@ fn title_text_and_navigation_caches_publish_together() { |
| 407 | random: 42, | 606 | random: 42, |
| 408 | }; | 607 | }; |
| 409 | let mut success = disk(None); | 608 | let mut success = disk(None); |
| 410 | onestore::commit_text(&mut success, source, sid, *oid, 0..end, replacement).unwrap(); | 609 | edit.commit(&mut success).unwrap(); |
| 411 | let after = state(&success.durable); | 610 | let after = state(&success.durable); |
| 412 | assert_eq!(after[0]["text"], replacement); | 611 | assert_eq!(after[0]["text"], replacement); |
| 413 | assert_eq!( | 612 | assert_eq!( |
| ... | @@ -445,9 +644,7 @@ fn title_text_and_navigation_caches_publish_together() { | ... | @@ -445,9 +644,7 @@ fn title_text_and_navigation_caches_publish_together() { |
| 445 | assert_other_objects_preserved(&old, ¤t, *oid); | 644 | assert_other_objects_preserved(&old, ¤t, *oid); |
| 446 | for at in source.len().div_ceil(193)..=success.operation { | 645 | for at in source.len().div_ceil(193)..=success.operation { |
| 447 | let mut interrupted = disk(Some(at)); | 646 | let mut interrupted = disk(Some(at)); |
| 448 | let error = | 647 | let error = edit.commit(&mut interrupted).unwrap_err(); |
| 449 | onestore::commit_text(&mut interrupted, source, sid, *oid, 0..end, replacement) | ||
| 450 | .unwrap_err(); | ||
| 451 | let observed = state(&interrupted.durable); | 648 | let observed = state(&interrupted.durable); |
| 452 | match error.state { | 649 | match error.state { |
| 453 | CommitState::NotCommitted => assert_eq!(observed, before), | 650 | CommitState::NotCommitted => assert_eq!(observed, before), |
crates/onestore/tests/formatting.rs created+407
| ... | @@ -0,0 +1,407 @@ | ||
| 1 | #[path = "support/current.rs"] | ||
| 2 | mod current; | ||
| 3 | #[path = "support/disk.rs"] | ||
| 4 | mod disk; | ||
| 5 | use onestore::{ | ||
| 6 | ExGuid, PreparedEdit, RevisionIndex, Store, TextAttribute as A, | ||
| 7 | document::{Document, Kind}, | ||
| 8 | }; | ||
| 9 | use serde_json::{Value, json}; | ||
| 10 | fn target(source: &[u8]) -> (ExGuid, ExGuid) { | ||
| 11 | let store = Store::parse(source).unwrap(); | ||
| 12 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 13 | let doc = Document::parse(&index).unwrap(); | ||
| 14 | let (sid, page) = doc.pages().unwrap()[0]; | ||
| 15 | let space = &doc.spaces[&sid]; | ||
| 16 | let view = &space.revisions[&space.contexts[&ExGuid::default()]]; | ||
| 17 | let mut pending = view.nodes[&page].children.clone(); | ||
| 18 | while let Some(id) = pending.pop() { | ||
| 19 | let n = &view.nodes[&id]; | ||
| 20 | if matches!(n.kind, Kind::RichText { .. }) { | ||
| 21 | return (sid, id); | ||
| 22 | } | ||
| 23 | pending.extend(&n.children); | ||
| 24 | pending.extend(&n.content); | ||
| 25 | } | ||
| 26 | panic!("Missing text") | ||
| 27 | } | ||
| 28 | fn characters(source: &[u8], sid: ExGuid, id: ExGuid) -> Vec<(char, Value)> { | ||
| 29 | let store = Store::parse(source).unwrap(); | ||
| 30 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 31 | index.validate_current().unwrap(); | ||
| 32 | let doc = Document::parse(&index).unwrap(); | ||
| 33 | let s = &doc.spaces[&sid]; | ||
| 34 | let view = &s.revisions[&s.contexts[&ExGuid::default()]]; | ||
| 35 | view.text_runs(id) | ||
| 36 | .unwrap() | ||
| 37 | .into_iter() | ||
| 38 | .flat_map(|r| { | ||
| 39 | let format = serde_json::to_value(r.format).unwrap(); | ||
| 40 | r.text.chars().map(move |c| (c, format.clone())) | ||
| 41 | }) | ||
| 42 | .collect() | ||
| 43 | } | ||
| 44 | #[test] | ||
| 45 | fn overlapping_unicode_format_edits_match_an_independent_character_model() { | ||
| 46 | let text = "abcdefgh 東京 🦀 café\rSecond\tline"; | ||
| 47 | let original = onestore::create_section("format.one", text, "Author").unwrap(); | ||
| 48 | let (sid, id) = target(&original); | ||
| 49 | for seed in 1..=16_u64 { | ||
| 50 | let mut rng = seed; | ||
| 51 | let mut source = original.clone(); | ||
| 52 | let mut expected = characters(&source, sid, id); | ||
| 53 | let offsets: Vec<u32> = std::iter::once(0) | ||
| 54 | .chain(text.chars().scan(0, |n, c| { | ||
| 55 | *n += c.len_utf16() as u32; | ||
| 56 | Some(*n) | ||
| 57 | })) | ||
| 58 | .collect(); | ||
| 59 | for step in 0..24 { | ||
| 60 | let mut next = || { | ||
| 61 | rng ^= rng << 13; | ||
| 62 | rng ^= rng >> 7; | ||
| 63 | rng ^= rng << 17; | ||
| 64 | rng | ||
| 65 | }; | ||
| 66 | let start = next() as usize % expected.len(); | ||
| 67 | let end = start + 1 + next() as usize % (expected.len() - start); | ||
| 68 | let enabled = next() & 1 != 0; | ||
| 69 | let (attribute, key, value) = match step % 9 { | ||
| 70 | 0 => (A::Bold(enabled), "bold", json!(enabled)), | ||
| 71 | 1 => (A::Italic(enabled), "italic", json!(enabled)), | ||
| 72 | 2 => (A::Underline(enabled), "underline", json!(enabled)), | ||
| 73 | 3 => (A::Strike(enabled), "strike", json!(enabled)), | ||
| 74 | 4 => (A::Font("Arial".into()), "font", json!("Arial")), | ||
| 75 | 5 => (A::FontSize(13.5), "font_size", json!(13.5)), | ||
| 76 | 6 => ( | ||
| 77 | A::Color(Some([0x24, 0x68, 0xac])), | ||
| 78 | "color", | ||
| 79 | json!(0xac6824_u32), | ||
| 80 | ), | ||
| 81 | 7 => (A::Highlight(None), "highlight", json!(0xff000000_u32)), | ||
| 82 | _ => ( | ||
| 83 | A::Highlight(Some([0, 255, 0])), | ||
| 84 | "highlight", | ||
| 85 | json!(0x00ff00_u32), | ||
| 86 | ), | ||
| 87 | }; | ||
| 88 | let edit = PreparedEdit::format( | ||
| 89 | &source, | ||
| 90 | sid, | ||
| 91 | id, | ||
| 92 | offsets[start]..offsets[end], | ||
| 93 | std::slice::from_ref(&attribute), | ||
| 94 | ) | ||
| 95 | .unwrap(); | ||
| 96 | for (_, style) in &mut expected[start..end] { | ||
| 97 | style[key] = value.clone(); | ||
| 98 | } | ||
| 99 | assert_eq!( | ||
| 100 | characters(edit.as_bytes(), sid, id), | ||
| 101 | expected, | ||
| 102 | "seed {seed}, step {step}" | ||
| 103 | ); | ||
| 104 | assert_eq!( | ||
| 105 | PreparedEdit::format( | ||
| 106 | edit.as_bytes(), | ||
| 107 | sid, | ||
| 108 | id, | ||
| 109 | offsets[start]..offsets[end], | ||
| 110 | &[attribute] | ||
| 111 | ) | ||
| 112 | .unwrap() | ||
| 113 | .as_bytes(), | ||
| 114 | edit.as_bytes() | ||
| 115 | ); | ||
| 116 | let old_store = Store::parse(&source).unwrap(); | ||
| 117 | let old = RevisionIndex::parse(&old_store).unwrap(); | ||
| 118 | let new_store = Store::parse(edit.as_bytes()).unwrap(); | ||
| 119 | let new = RevisionIndex::parse(&new_store).unwrap(); | ||
| 120 | let rid = old.spaces[&sid].labels[&(ExGuid::default(), 1)]; | ||
| 121 | assert_eq!( | ||
| 122 | format!("{:?}", old.resolve(sid, rid).unwrap()), | ||
| 123 | format!("{:?}", new.resolve(sid, rid).unwrap()) | ||
| 124 | ); | ||
| 125 | source = edit.as_bytes().to_vec(); | ||
| 126 | } | ||
| 127 | } | ||
| 128 | } | ||
| 129 | #[test] | ||
| 130 | fn script_positions_are_exclusive_and_explicit_false_overrides_true() { | ||
| 131 | let source = onestore::create_section("script.one", "abc", "Author").unwrap(); | ||
| 132 | let (sid, id) = target(&source); | ||
| 133 | let superscript = PreparedEdit::format( | ||
| 134 | &source, | ||
| 135 | sid, | ||
| 136 | id, | ||
| 137 | 0..3, | ||
| 138 | &[A::Superscript(true), A::Bold(true)], | ||
| 139 | ) | ||
| 140 | .unwrap(); | ||
| 141 | let subscript = PreparedEdit::format( | ||
| 142 | superscript.as_bytes(), | ||
| 143 | sid, | ||
| 144 | id, | ||
| 145 | 1..2, | ||
| 146 | &[A::Subscript(true), A::Bold(false)], | ||
| 147 | ) | ||
| 148 | .unwrap(); | ||
| 149 | let chars = characters(subscript.as_bytes(), sid, id); | ||
| 150 | assert_eq!(chars[0].1["superscript"], true); | ||
| 151 | assert_eq!(chars[0].1["subscript"], false); | ||
| 152 | assert_eq!(chars[0].1["bold"], true); | ||
| 153 | assert_eq!(chars[1].1["superscript"], false); | ||
| 154 | assert_eq!(chars[1].1["subscript"], true); | ||
| 155 | assert_eq!(chars[1].1["bold"], false); | ||
| 156 | assert_eq!(chars[0].1, chars[2].1); | ||
| 157 | } | ||
| 158 | #[test] | ||
| 159 | fn empty_paragraph_style_is_used_by_later_text_edits() { | ||
| 160 | let source = onestore::create_section("empty.one", "", "Author").unwrap(); | ||
| 161 | let (sid, id) = target(&source); | ||
| 162 | let formatted = PreparedEdit::format( | ||
| 163 | &source, | ||
| 164 | sid, | ||
| 165 | id, | ||
| 166 | 0..0, | ||
| 167 | &[A::Italic(true), A::FontSize(18.0)], | ||
| 168 | ) | ||
| 169 | .unwrap(); | ||
| 170 | let filled = PreparedEdit::text(formatted.as_bytes(), sid, id, 0..0, "Added 🦀").unwrap(); | ||
| 171 | for (_, format) in characters(filled.as_bytes(), sid, id) { | ||
| 172 | assert_eq!(format["italic"], true); | ||
| 173 | assert_eq!(format["font_size"], 18.0); | ||
| 174 | } | ||
| 175 | } | ||
| 176 | #[test] | ||
| 177 | fn invalid_ranges_attributes_and_fields_are_rejected() { | ||
| 178 | let source = onestore::create_section("invalid.one", "a🦀b", "Author").unwrap(); | ||
| 179 | let (sid, id) = target(&source); | ||
| 180 | for (start, end) in [(2, 3), (1, 2), (0, 8), (2, 2), (3, 1), (1, 1)] { | ||
| 181 | let range = start..end; | ||
| 182 | assert!(PreparedEdit::format(&source, sid, id, range, &[A::Bold(true)]).is_err()); | ||
| 183 | } | ||
| 184 | for attributes in [ | ||
| 185 | vec![], | ||
| 186 | vec![A::Bold(true), A::Bold(false)], | ||
| 187 | vec![A::Superscript(true), A::Subscript(true)], | ||
| 188 | vec![A::Font("".into())], | ||
| 189 | vec![A::Font("a\0b".into())], | ||
| 190 | ] { | ||
| 191 | assert!(PreparedEdit::format(&source, sid, id, 0..4, &attributes).is_err()); | ||
| 192 | } | ||
| 193 | for size in [f32::NAN, f32::INFINITY, 0.0, 5.5, 130.5, 144.0, 144.5, 12.1] { | ||
| 194 | assert!(PreparedEdit::format(&source, sid, id, 0..4, &[A::FontSize(size)]).is_err()); | ||
| 195 | } | ||
| 196 | assert!(PreparedEdit::format(&source, sid, ExGuid::default(), 0..4, &[A::Bold(true)]).is_err()); | ||
| 197 | } | ||
| 198 | #[test] | ||
| 199 | fn formatting_publication_faults_preserve_complete_old_or_new_styles() { | ||
| 200 | let source = onestore::create_section("atomic.one", "Before 🦀 after", "Author").unwrap(); | ||
| 201 | let (sid, id) = target(&source); | ||
| 202 | let edit = PreparedEdit::format( | ||
| 203 | &source, | ||
| 204 | sid, | ||
| 205 | id, | ||
| 206 | 2..10, | ||
| 207 | &[A::Bold(true), A::Color(Some([8, 64, 128]))], | ||
| 208 | ) | ||
| 209 | .unwrap(); | ||
| 210 | let before = current::current(&source); | ||
| 211 | let after = current::current(edit.as_bytes()); | ||
| 212 | for write_limit in [17, 4096] { | ||
| 213 | let disk = |fail_at| disk::Disk { | ||
| 214 | visible: source.clone(), | ||
| 215 | durable: source.clone(), | ||
| 216 | operation: 0, | ||
| 217 | fail_at, | ||
| 218 | write_limit, | ||
| 219 | random: 946, | ||
| 220 | }; | ||
| 221 | let mut success = disk(None); | ||
| 222 | edit.commit(&mut success).unwrap(); | ||
| 223 | assert_eq!(success.durable, edit.as_bytes()); | ||
| 224 | for at in 1..=success.operation { | ||
| 225 | let mut interrupted = disk(Some(at)); | ||
| 226 | let failure = edit.commit(&mut interrupted).unwrap_err(); | ||
| 227 | let actual = current::current(&interrupted.durable); | ||
| 228 | assert!(actual == before || actual == after, "operation {at}"); | ||
| 229 | if failure.state == onestore::CommitState::NotCommitted { | ||
| 230 | assert_eq!(actual, before); | ||
| 231 | } | ||
| 232 | } | ||
| 233 | } | ||
| 234 | } | ||
| 235 | |||
| 236 | #[test] | ||
| 237 | #[ignore = "exports public-API formatting candidates for cold native validation"] | ||
| 238 | fn export_native_formatting_candidates() { | ||
| 239 | use std::{fs, path::PathBuf}; | ||
| 240 | let output = PathBuf::from(std::env::var_os("ONESTORE_FORMAT_OUTPUT").unwrap()); | ||
| 241 | assert!(output.is_absolute()); | ||
| 242 | fs::create_dir(&output).unwrap(); | ||
| 243 | let source = onestore::create_section( | ||
| 244 | "format.one", | ||
| 245 | "Before café 東京 🦀 after", | ||
| 246 | "Formatting author", | ||
| 247 | ) | ||
| 248 | .unwrap(); | ||
| 249 | let (sid, id) = target(&source); | ||
| 250 | let mut manifest = Vec::new(); | ||
| 251 | let mut save = | ||
| 252 | |name: &str, source: &[u8], sid, id, range: std::ops::Range<u32>, attributes: &[A]| { | ||
| 253 | let prepared = | ||
| 254 | PreparedEdit::format(source, sid, id, range.clone(), attributes).unwrap(); | ||
| 255 | fs::write(output.join(format!("{name}.one")), prepared.as_bytes()).unwrap(); | ||
| 256 | manifest.push( | ||
| 257 | json!({"name":name,"space":sid,"object":id,"range":range,"attributes":attributes}), | ||
| 258 | ); | ||
| 259 | prepared.as_bytes().to_vec() | ||
| 260 | }; | ||
| 261 | save( | ||
| 262 | "partial-boolean", | ||
| 263 | &source, | ||
| 264 | sid, | ||
| 265 | id, | ||
| 266 | 2..18, | ||
| 267 | &[ | ||
| 268 | A::Bold(true), | ||
| 269 | A::Italic(true), | ||
| 270 | A::Underline(true), | ||
| 271 | A::Strike(true), | ||
| 272 | ], | ||
| 273 | ); | ||
| 274 | let colored = save( | ||
| 275 | "partial-font-color", | ||
| 276 | &source, | ||
| 277 | sid, | ||
| 278 | id, | ||
| 279 | 3..18, | ||
| 280 | &[ | ||
| 281 | A::Font("Arial".into()), | ||
| 282 | A::FontSize(13.5), | ||
| 283 | A::Color(Some([24, 96, 160])), | ||
| 284 | A::Highlight(Some([255, 255, 0])), | ||
| 285 | ], | ||
| 286 | ); | ||
| 287 | save( | ||
| 288 | "clear-color", | ||
| 289 | &colored, | ||
| 290 | sid, | ||
| 291 | id, | ||
| 292 | 6..14, | ||
| 293 | &[A::Color(None), A::Highlight(None)], | ||
| 294 | ); | ||
| 295 | let scripted = save("subscript", &source, sid, id, 0..23, &[A::Subscript(true)]); | ||
| 296 | save( | ||
| 297 | "superscript", | ||
| 298 | &scripted, | ||
| 299 | sid, | ||
| 300 | id, | ||
| 301 | 6..18, | ||
| 302 | &[A::Superscript(true)], | ||
| 303 | ); | ||
| 304 | let bold = save( | ||
| 305 | "bold", | ||
| 306 | &source, | ||
| 307 | sid, | ||
| 308 | id, | ||
| 309 | 0..23, | ||
| 310 | &[ | ||
| 311 | A::Bold(true), | ||
| 312 | A::Italic(true), | ||
| 313 | A::Underline(true), | ||
| 314 | A::Strike(true), | ||
| 315 | ], | ||
| 316 | ); | ||
| 317 | save( | ||
| 318 | "clear-boolean", | ||
| 319 | &bold, | ||
| 320 | sid, | ||
| 321 | id, | ||
| 322 | 7..14, | ||
| 323 | &[ | ||
| 324 | A::Bold(false), | ||
| 325 | A::Italic(false), | ||
| 326 | A::Underline(false), | ||
| 327 | A::Strike(false), | ||
| 328 | ], | ||
| 329 | ); | ||
| 330 | let empty = onestore::create_section("empty.one", "", "Author").unwrap(); | ||
| 331 | let (empty_sid, empty_id) = target(&empty); | ||
| 332 | let formatted = PreparedEdit::format( | ||
| 333 | &empty, | ||
| 334 | empty_sid, | ||
| 335 | empty_id, | ||
| 336 | 0..0, | ||
| 337 | &[A::FontSize(18.0), A::Italic(true)], | ||
| 338 | ) | ||
| 339 | .unwrap(); | ||
| 340 | let typed = PreparedEdit::text( | ||
| 341 | formatted.as_bytes(), | ||
| 342 | empty_sid, | ||
| 343 | empty_id, | ||
| 344 | 0..0, | ||
| 345 | "Typed café 🦀", | ||
| 346 | ) | ||
| 347 | .unwrap(); | ||
| 348 | fs::write(output.join("empty-then-type.one"), typed.as_bytes()).unwrap(); | ||
| 349 | let native = include_bytes!("../../../corpus/native-ink/20260905-ui/notebook/synthetic.one"); | ||
| 350 | let store = Store::parse(native).unwrap(); | ||
| 351 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 352 | let document = Document::parse(&index).unwrap(); | ||
| 353 | let (native_sid, native_id) = document | ||
| 354 | .spaces | ||
| 355 | .iter() | ||
| 356 | .find_map(|(sid, s)| { | ||
| 357 | let r = &s.revisions[&s.contexts[&ExGuid::default()]]; | ||
| 358 | r.nodes.iter().find_map(|(id, n)| { | ||
| 359 | matches!(&n.kind,Kind::RichText{text,..} if text.starts_with("Fictitious:" )) | ||
| 360 | .then_some((*sid, *id)) | ||
| 361 | }) | ||
| 362 | }) | ||
| 363 | .unwrap(); | ||
| 364 | save( | ||
| 365 | "native-cross-runs", | ||
| 366 | native, | ||
| 367 | native_sid, | ||
| 368 | native_id, | ||
| 369 | 2..26, | ||
| 370 | &[A::Bold(false), A::Italic(true), A::Underline(true)], | ||
| 371 | ); | ||
| 372 | save( | ||
| 373 | "native-partial", | ||
| 374 | native, | ||
| 375 | native_sid, | ||
| 376 | native_id, | ||
| 377 | 3..8, | ||
| 378 | &[A::FontSize(14.0), A::Color(Some([16, 112, 48]))], | ||
| 379 | ); | ||
| 380 | save( | ||
| 381 | "native-font", | ||
| 382 | native, | ||
| 383 | native_sid, | ||
| 384 | native_id, | ||
| 385 | 0..27, | ||
| 386 | &[A::Font("Arial".into())], | ||
| 387 | ); | ||
| 388 | save("small-font", &source, sid, id, 0..23, &[A::FontSize(6.0)]); | ||
| 389 | save("large-font", &source, sid, id, 0..23, &[A::FontSize(130.0)]); | ||
| 390 | for points in [129.5, 130.0] { | ||
| 391 | save( | ||
| 392 | &format!("font-boundary-{points}"), | ||
| 393 | &source, | ||
| 394 | sid, | ||
| 395 | id, | ||
| 396 | 0..23, | ||
| 397 | &[A::FontSize(points)], | ||
| 398 | ); | ||
| 399 | } | ||
| 400 | fs::write(output.join("basic-baseline.one"), &source).unwrap(); | ||
| 401 | fs::write(output.join("native-baseline.one"), native).unwrap(); | ||
| 402 | fs::write( | ||
| 403 | output.join("manifest.json"), | ||
| 404 | serde_json::to_vec_pretty(&manifest).unwrap(), | ||
| 405 | ) | ||
| 406 | .unwrap(); | ||
| 407 | } | ||
crates/onestore/tests/insertion.rs created+377
| ... | @@ -0,0 +1,377 @@ | ||
| 1 | #[path = "support/checkpoint.rs"] | ||
| 2 | mod checkpoint; | ||
| 3 | #[path = "support/current.rs"] | ||
| 4 | mod current; | ||
| 5 | #[path = "support/disk.rs"] | ||
| 6 | mod disk; | ||
| 7 | |||
| 8 | use onestore::{ | ||
| 9 | ExGuid, Insertion, PreparedEdit, RevisionIndex, Store, | ||
| 10 | document::{Document, Kind}, | ||
| 11 | }; | ||
| 12 | |||
| 13 | fn targets(source: &[u8]) -> (ExGuid, ExGuid, ExGuid, ExGuid, ExGuid) { | ||
| 14 | let store = Store::parse(source).unwrap(); | ||
| 15 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 16 | let document = Document::parse(&index).unwrap(); | ||
| 17 | let (sid, page) = document.pages().unwrap()[0]; | ||
| 18 | let space = &document.spaces[&sid]; | ||
| 19 | let view = &space.revisions[&space.contexts[&ExGuid::default()]]; | ||
| 20 | let outline = view.nodes[&page] | ||
| 21 | .children | ||
| 22 | .iter() | ||
| 23 | .copied() | ||
| 24 | .find(|id| matches!(view.nodes[id].kind, Kind::Outline { .. })) | ||
| 25 | .unwrap(); | ||
| 26 | let paragraph = view.nodes[&outline].children[0]; | ||
| 27 | let text = view.nodes[¶graph].content[0]; | ||
| 28 | (sid, page, outline, paragraph, text) | ||
| 29 | } | ||
| 30 | |||
| 31 | #[test] | ||
| 32 | fn paragraph_and_outline_insertions_publish_metadata_and_references_together() { | ||
| 33 | let source = onestore::create_section("insertion.one", "Original", "Original author").unwrap(); | ||
| 34 | let (sid, page, outline, paragraph, original_text) = targets(&source); | ||
| 35 | for (intent, expected_parent, expected_x, expected_y) in [ | ||
| 36 | ( | ||
| 37 | Insertion::paragraph(outline, Some(paragraph), "First 🦀\rSecond", "New author") | ||
| 38 | .unwrap(), | ||
| 39 | outline, | ||
| 40 | None, | ||
| 41 | None, | ||
| 42 | ), | ||
| 43 | ( | ||
| 44 | Insertion::outline(page, 144.0, 18.0, "First 🦀\rSecond", "New author").unwrap(), | ||
| 45 | page, | ||
| 46 | Some(144.0), | ||
| 47 | Some(18.0), | ||
| 48 | ), | ||
| 49 | ] { | ||
| 50 | let prepared = PreparedEdit::insert(&source, sid, &intent).unwrap(); | ||
| 51 | let store = Store::parse(prepared.as_bytes()).unwrap(); | ||
| 52 | assert_eq!( | ||
| 53 | store.header.transaction_count, | ||
| 54 | Store::parse(&source).unwrap().header.transaction_count + 1 | ||
| 55 | ); | ||
| 56 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 57 | index.validate_current().unwrap(); | ||
| 58 | let document = Document::parse(&index).unwrap(); | ||
| 59 | let space = &document.spaces[&sid]; | ||
| 60 | let view = &space.revisions[&space.contexts[&ExGuid::default()]]; | ||
| 61 | assert!( | ||
| 62 | view.nodes[&expected_parent] | ||
| 63 | .children | ||
| 64 | .contains(&intent.object()) | ||
| 65 | ); | ||
| 66 | assert_eq!(view.nodes[&intent.object()].layout.x, expected_x); | ||
| 67 | assert_eq!(view.nodes[&intent.object()].layout.y, expected_y); | ||
| 68 | assert!( | ||
| 69 | matches!(&view.nodes[&original_text].kind, Kind::RichText { text, .. } if text == "Original") | ||
| 70 | ); | ||
| 71 | assert!( | ||
| 72 | matches!(&view.nodes[&intent.text_object()].kind, Kind::RichText { text, .. } if text == "First 🦀\rSecond") | ||
| 73 | ); | ||
| 74 | assert!( | ||
| 75 | matches!(&view.nodes[&page].kind, Kind::Page { alternate_title, .. } if alternate_title.as_deref() == Some("First 🦀")) | ||
| 76 | ); | ||
| 77 | assert!( | ||
| 78 | matches!(&view.nodes[&view.roots[&2]].kind, Kind::Metadata { title, .. } if title.as_deref() == Some("First 🦀")) | ||
| 79 | ); | ||
| 80 | assert!( | ||
| 81 | view.nodes | ||
| 82 | .values() | ||
| 83 | .any(|node| matches!(&node.kind, Kind::Author { name } if name.as_deref() == Some("New author"))) | ||
| 84 | ); | ||
| 85 | let runs = view.text_runs(intent.text_object()).unwrap(); | ||
| 86 | assert_eq!(runs.len(), 1); | ||
| 87 | assert_eq!(runs[0].format.font.as_deref(), Some("Calibri")); | ||
| 88 | assert_eq!(runs[0].format.font_size, Some(11.0)); | ||
| 89 | let previous_store = Store::parse(&source).unwrap(); | ||
| 90 | let previous = RevisionIndex::parse(&previous_store).unwrap(); | ||
| 91 | for (old_sid, old_space) in &previous.spaces { | ||
| 92 | for rid in old_space.revisions.keys() { | ||
| 93 | assert_eq!( | ||
| 94 | format!("{:?}", previous.resolve(*old_sid, *rid).unwrap()), | ||
| 95 | format!("{:?}", index.resolve(*old_sid, *rid).unwrap()) | ||
| 96 | ); | ||
| 97 | } | ||
| 98 | } | ||
| 99 | } | ||
| 100 | } | ||
| 101 | |||
| 102 | #[test] | ||
| 103 | fn repeated_insertions_and_formatting_share_immutable_objects() { | ||
| 104 | let mut source = onestore::create_section("shared.one", "Original", "Same author").unwrap(); | ||
| 105 | let (sid, _, outline, _, _) = targets(&source); | ||
| 106 | let mut texts = Vec::new(); | ||
| 107 | for _ in 0..12 { | ||
| 108 | let insertion = | ||
| 109 | Insertion::paragraph(outline, None, "Repeated paragraph", "Same author").unwrap(); | ||
| 110 | source = PreparedEdit::insert(&source, sid, &insertion) | ||
| 111 | .unwrap() | ||
| 112 | .as_bytes() | ||
| 113 | .to_vec(); | ||
| 114 | source = PreparedEdit::format( | ||
| 115 | &source, | ||
| 116 | sid, | ||
| 117 | insertion.text_object(), | ||
| 118 | 1..8, | ||
| 119 | &[ | ||
| 120 | onestore::TextAttribute::Bold(true), | ||
| 121 | onestore::TextAttribute::FontSize(20.0), | ||
| 122 | ], | ||
| 123 | ) | ||
| 124 | .unwrap() | ||
| 125 | .as_bytes() | ||
| 126 | .to_vec(); | ||
| 127 | texts.push(insertion.text_object()); | ||
| 128 | } | ||
| 129 | let store = Store::parse(&source).unwrap(); | ||
| 130 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 131 | index.validate_current().unwrap(); | ||
| 132 | let raw = index | ||
| 133 | .resolve(sid, index.spaces[&sid].labels[&(ExGuid::default(), 1)]) | ||
| 134 | .unwrap(); | ||
| 135 | let mut unique = std::collections::BTreeSet::new(); | ||
| 136 | let mut counts = Vec::new(); | ||
| 137 | for id in raw.reachable().unwrap() { | ||
| 138 | let object = &raw.objects[&id]; | ||
| 139 | if object.jcid & 0x100000 == 0 { | ||
| 140 | continue; | ||
| 141 | } | ||
| 142 | let onestore::ObjectData::Properties(bytes) = object.data else { | ||
| 143 | panic!() | ||
| 144 | }; | ||
| 145 | assert!( | ||
| 146 | unique.insert((object.jcid, bytes.to_vec())), | ||
| 147 | "Duplicate immutable object" | ||
| 148 | ); | ||
| 149 | counts.push((object.jcid, object.reference_count)); | ||
| 150 | } | ||
| 151 | counts.sort(); | ||
| 152 | assert_eq!(counts, [(0x120001, 26), (0x12004d, 12), (0x12004d, 25)]); | ||
| 153 | let document = Document::parse(&index).unwrap(); | ||
| 154 | let space = &document.spaces[&sid]; | ||
| 155 | let view = &space.revisions[&space.contexts[&ExGuid::default()]]; | ||
| 156 | for text in texts { | ||
| 157 | let runs = view.text_runs(text).unwrap(); | ||
| 158 | assert_eq!(runs.len(), 3); | ||
| 159 | assert_eq!(runs[1].format.bold, Some(true)); | ||
| 160 | assert_eq!(runs[1].format.font_size, Some(20.0)); | ||
| 161 | } | ||
| 162 | } | ||
| 163 | |||
| 164 | #[test] | ||
| 165 | fn serialized_insertions_rebase_with_the_same_objects_and_preserve_remote_edits() { | ||
| 166 | let source = onestore::create_section("rebase.one", "Original", "Author").unwrap(); | ||
| 167 | let (sid, _, outline, paragraph, text) = targets(&source); | ||
| 168 | let intent = Insertion::paragraph(outline, Some(paragraph), "Inserted", "Author").unwrap(); | ||
| 169 | let encoded = serde_json::to_vec(&intent).unwrap(); | ||
| 170 | let restored: Insertion = serde_json::from_slice(&encoded).unwrap(); | ||
| 171 | assert_eq!(intent.object(), restored.object()); | ||
| 172 | assert_eq!(intent.text_object(), restored.text_object()); | ||
| 173 | let original_preparation = PreparedEdit::insert(&source, sid, &intent).unwrap(); | ||
| 174 | let remote = PreparedEdit::text(&source, sid, text, 0..0, "Remote ").unwrap(); | ||
| 175 | let updated = PreparedEdit::insert(remote.as_bytes(), sid, &restored).unwrap(); | ||
| 176 | let store = Store::parse(updated.as_bytes()).unwrap(); | ||
| 177 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 178 | let document = Document::parse(&index).unwrap(); | ||
| 179 | let space = &document.spaces[&sid]; | ||
| 180 | let view = &space.revisions[&space.contexts[&ExGuid::default()]]; | ||
| 181 | assert_eq!( | ||
| 182 | view.nodes[&outline].children, | ||
| 183 | [restored.object(), paragraph] | ||
| 184 | ); | ||
| 185 | assert!( | ||
| 186 | matches!(&view.nodes[&text].kind, Kind::RichText { text, .. } if text == "Remote Original") | ||
| 187 | ); | ||
| 188 | assert!( | ||
| 189 | matches!(&view.nodes[&restored.text_object()].kind, Kind::RichText { text, .. } if text == "Inserted") | ||
| 190 | ); | ||
| 191 | assert!(PreparedEdit::insert(original_preparation.as_bytes(), sid, &restored).is_err()); | ||
| 192 | assert!(PreparedEdit::insert(updated.as_bytes(), sid, &restored).is_err()); | ||
| 193 | } | ||
| 194 | |||
| 195 | #[test] | ||
| 196 | fn repositioning_preserves_intent_identity_and_kind() { | ||
| 197 | let source = onestore::create_section("placement.one", "Original", "Author").unwrap(); | ||
| 198 | let (sid, page, outline, paragraph, _) = targets(&source); | ||
| 199 | let p = Insertion::paragraph(outline, Some(paragraph), "Inserted", "Author").unwrap(); | ||
| 200 | let o = Insertion::outline(page, 144.0, 144.0, "Inserted", "Author").unwrap(); | ||
| 201 | assert!(p.reposition_outline(page, 72.0, 72.0).is_err()); | ||
| 202 | assert!(o.reposition_paragraph(outline, None).is_err()); | ||
| 203 | assert!(p.reposition_paragraph(ExGuid::default(), None).is_err()); | ||
| 204 | assert!(o.reposition_outline(page, f32::NAN, 72.0).is_err()); | ||
| 205 | for (original, moved) in [ | ||
| 206 | (&p, p.reposition_paragraph(outline, None).unwrap()), | ||
| 207 | (&o, o.reposition_outline(page, 288.0, 360.0).unwrap()), | ||
| 208 | ] { | ||
| 209 | let mut before = serde_json::to_value(original).unwrap(); | ||
| 210 | let mut after = serde_json::to_value(&moved).unwrap(); | ||
| 211 | for name in ["parent", "placement"] { | ||
| 212 | before.as_object_mut().unwrap().remove(name); | ||
| 213 | after.as_object_mut().unwrap().remove(name); | ||
| 214 | } | ||
| 215 | assert_eq!(before, after); | ||
| 216 | assert_eq!(original.object(), moved.object()); | ||
| 217 | assert_eq!(original.text_object(), moved.text_object()); | ||
| 218 | let restored: Insertion = | ||
| 219 | serde_json::from_value(serde_json::to_value(&moved).unwrap()).unwrap(); | ||
| 220 | let prepared = PreparedEdit::insert(&source, sid, &restored).unwrap(); | ||
| 221 | let store = Store::parse(prepared.as_bytes()).unwrap(); | ||
| 222 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 223 | index.validate_current().unwrap(); | ||
| 224 | let doc = Document::parse(&index).unwrap(); | ||
| 225 | let space = &doc.spaces[&sid]; | ||
| 226 | let view = &space.revisions[&space.contexts[&ExGuid::default()]]; | ||
| 227 | if original == &p { | ||
| 228 | assert_eq!(view.nodes[&outline].children, [paragraph, moved.object()]); | ||
| 229 | } else { | ||
| 230 | assert_eq!( | ||
| 231 | ( | ||
| 232 | view.nodes[&moved.object()].layout.x, | ||
| 233 | view.nodes[&moved.object()].layout.y | ||
| 234 | ), | ||
| 235 | (Some(288.0), Some(360.0)) | ||
| 236 | ); | ||
| 237 | } | ||
| 238 | } | ||
| 239 | } | ||
| 240 | |||
| 241 | #[test] | ||
| 242 | #[cfg(any(unix, windows))] | ||
| 243 | fn filesystem_insertion_uses_the_prepared_identity_and_rejects_stale_replay() { | ||
| 244 | use std::{fs, io::Write}; | ||
| 245 | let source = onestore::create_section("file.one", "Original", "Author").unwrap(); | ||
| 246 | let (sid, _, outline, _, _) = targets(&source); | ||
| 247 | let intent = Insertion::paragraph(outline, None, "File insertion", "Author").unwrap(); | ||
| 248 | let path = std::env::temp_dir().join(format!("onestore-insertion-{}.one", intent.object())); | ||
| 249 | let mut file = fs::OpenOptions::new() | ||
| 250 | .write(true) | ||
| 251 | .create_new(true) | ||
| 252 | .open(&path) | ||
| 253 | .unwrap(); | ||
| 254 | file.write_all(&source).unwrap(); | ||
| 255 | file.sync_all().unwrap(); | ||
| 256 | drop(file); | ||
| 257 | let edit = PreparedEdit::insert(&source, sid, &intent).unwrap(); | ||
| 258 | let result = edit.commit_file(&path); | ||
| 259 | let written = onestore::read_file(&path).unwrap(); | ||
| 260 | let repeated = edit.commit_file(&path).unwrap_err(); | ||
| 261 | fs::remove_file(path).unwrap(); | ||
| 262 | result.unwrap(); | ||
| 263 | assert_eq!(written, edit.as_bytes()); | ||
| 264 | assert_eq!(repeated.state, onestore::CommitState::NotCommitted); | ||
| 265 | assert_eq!(repeated.error.kind(), std::io::ErrorKind::ResourceBusy); | ||
| 266 | } | ||
| 267 | |||
| 268 | #[test] | ||
| 269 | fn anchors_targets_serialized_identities_and_text_are_validated_before_publication() { | ||
| 270 | let source = onestore::create_section("invalid.one", "Original", "Author").unwrap(); | ||
| 271 | let (sid, page, outline, paragraph, text) = targets(&source); | ||
| 272 | assert!(Insertion::outline(page, f32::NAN, 0.0, "Text", "Author").is_err()); | ||
| 273 | assert!(Insertion::outline(page, 0.0, f32::INFINITY, "Text", "Author").is_err()); | ||
| 274 | for content in ["a\0b", "a\nb", "a\u{fffc}b", "a\u{fddf}b"] { | ||
| 275 | assert!(Insertion::paragraph(outline, None, content, "Author").is_err()); | ||
| 276 | } | ||
| 277 | assert!(Insertion::paragraph(outline, None, "Text", "a\0b").is_err()); | ||
| 278 | for intent in [ | ||
| 279 | Insertion::paragraph(outline, Some(text), "Text", "Author").unwrap(), | ||
| 280 | Insertion::paragraph(text, None, "Text", "Author").unwrap(), | ||
| 281 | Insertion::paragraph(page, None, "Text", "Author").unwrap(), | ||
| 282 | Insertion::outline(paragraph, 0.0, 0.0, "Text", "Author").unwrap(), | ||
| 283 | ] { | ||
| 284 | assert!(PreparedEdit::insert(&source, sid, &intent).is_err()); | ||
| 285 | } | ||
| 286 | let intent = Insertion::paragraph(outline, None, "Text", "Author").unwrap(); | ||
| 287 | let created = PreparedEdit::insert(&source, sid, &intent).unwrap(); | ||
| 288 | let before_missing = | ||
| 289 | Insertion::paragraph(outline, Some(intent.object()), "Anchored", "Author").unwrap(); | ||
| 290 | assert!(PreparedEdit::insert(&source, sid, &before_missing).is_err()); | ||
| 291 | assert!(PreparedEdit::insert(created.as_bytes(), sid, &before_missing).is_ok()); | ||
| 292 | let mut encoded = serde_json::to_value(&intent).unwrap(); | ||
| 293 | encoded["parent"] = serde_json::to_value(intent.object()).unwrap(); | ||
| 294 | let collision: Insertion = serde_json::from_value(encoded).unwrap(); | ||
| 295 | assert!(PreparedEdit::insert(created.as_bytes(), sid, &collision).is_err()); | ||
| 296 | } | ||
| 297 | |||
| 298 | #[test] | ||
| 299 | fn insertions_into_nested_paragraphs_and_native_table_cells_preserve_structure() { | ||
| 300 | let source = onestore::create_section("nested.one", "Original", "Author").unwrap(); | ||
| 301 | let (sid, _, outline, paragraph, _) = targets(&source); | ||
| 302 | let child = Insertion::paragraph(paragraph, None, "Nested", "Author").unwrap(); | ||
| 303 | let changed = PreparedEdit::insert(&source, sid, &child).unwrap(); | ||
| 304 | let store = Store::parse(changed.as_bytes()).unwrap(); | ||
| 305 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 306 | let document = Document::parse(&index).unwrap(); | ||
| 307 | let space = &document.spaces[&sid]; | ||
| 308 | let view = &space.revisions[&space.contexts[&ExGuid::default()]]; | ||
| 309 | assert_eq!(view.nodes[&outline].children, [paragraph]); | ||
| 310 | assert_eq!(view.nodes[¶graph].children, [child.object()]); | ||
| 311 | let source = include_bytes!( | ||
| 312 | "../../../corpus/native/20260905-05/snapshots/07-table/notebook/synthetic.one" | ||
| 313 | ); | ||
| 314 | let store = Store::parse(source).unwrap(); | ||
| 315 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 316 | let document = Document::parse(&index).unwrap(); | ||
| 317 | let (sid, cell) = document | ||
| 318 | .spaces | ||
| 319 | .iter() | ||
| 320 | .find_map(|(sid, space)| { | ||
| 321 | let view = &space.revisions[&space.contexts[&ExGuid::default()]]; | ||
| 322 | view.nodes.iter().find_map(|(id, node)| { | ||
| 323 | matches!(node.kind, Kind::Cell { .. }).then_some((*sid, *id)) | ||
| 324 | }) | ||
| 325 | }) | ||
| 326 | .unwrap(); | ||
| 327 | let insertion = Insertion::paragraph(cell, None, "Added to cell", "Author").unwrap(); | ||
| 328 | let changed = PreparedEdit::insert(source, sid, &insertion).unwrap(); | ||
| 329 | current::current(changed.as_bytes()); | ||
| 330 | } | ||
| 331 | |||
| 332 | #[test] | ||
| 333 | fn insertion_publication_faults_expose_only_complete_graphs_and_title_caches() { | ||
| 334 | let original = onestore::create_section("atomic.one", "Original", "Author").unwrap(); | ||
| 335 | let (sid, _, _, _, text) = targets(&original); | ||
| 336 | let checkpoint = checkpoint::pending(&original, sid, text, 0x14001d7a); | ||
| 337 | for source in [ | ||
| 338 | original.as_slice(), | ||
| 339 | include_bytes!("../../../corpus/append/round-01/tx-255/notebook/synthetic.one"), | ||
| 340 | &checkpoint, | ||
| 341 | ] { | ||
| 342 | let (sid, page, outline, paragraph, _) = targets(source); | ||
| 343 | for intent in [ | ||
| 344 | Insertion::paragraph(outline, Some(paragraph), "First 🦀", "New author").unwrap(), | ||
| 345 | Insertion::outline(page, 0.0, 0.0, "First 🦀", "New author").unwrap(), | ||
| 346 | ] { | ||
| 347 | let edit = PreparedEdit::insert(source, sid, &intent).unwrap(); | ||
| 348 | let before = current::current(source); | ||
| 349 | let after = current::current(edit.as_bytes()); | ||
| 350 | for write_limit in [17, 4096] { | ||
| 351 | let disk = |fail_at| disk::Disk { | ||
| 352 | visible: source.to_vec(), | ||
| 353 | durable: source.to_vec(), | ||
| 354 | operation: 0, | ||
| 355 | fail_at, | ||
| 356 | write_limit, | ||
| 357 | random: 945, | ||
| 358 | }; | ||
| 359 | let mut successful = disk(None); | ||
| 360 | edit.commit(&mut successful).unwrap(); | ||
| 361 | assert_eq!(successful.durable, edit.as_bytes()); | ||
| 362 | for at in 1..=successful.operation { | ||
| 363 | let mut interrupted = disk(Some(at)); | ||
| 364 | let failure = edit.commit(&mut interrupted).unwrap_err(); | ||
| 365 | let state = current::current(&interrupted.durable); | ||
| 366 | assert!(state == before || state == after, "interruption {at}"); | ||
| 367 | if failure.state == onestore::CommitState::NotCommitted { | ||
| 368 | assert_eq!(state, before); | ||
| 369 | } | ||
| 370 | if failure.state == onestore::CommitState::Committed { | ||
| 371 | assert_eq!(state, after); | ||
| 372 | } | ||
| 373 | } | ||
| 374 | } | ||
| 375 | } | ||
| 376 | } | ||
| 377 | } | ||
crates/onestore/tests/revisions.rs+62
| ... | @@ -3,6 +3,68 @@ use std::fs; | ... | @@ -3,6 +3,68 @@ use std::fs; |
| 3 | 3 | ||
| 4 | const TABLE: &str = "../../corpus/native/20260905-05/snapshots/07-table/notebook/synthetic.one"; | 4 | const TABLE: &str = "../../corpus/native/20260905-05/snapshots/07-table/notebook/synthetic.one"; |
| 5 | 5 | ||
| 6 | #[test] | ||
| 7 | fn persisted_identities_preserve_native_byte_order_and_canonical_form() { | ||
| 8 | let text = "{00112233-4455-6677-8899-AABBCCDDEEFF},42"; | ||
| 9 | let id: ExGuid = text.parse().unwrap(); | ||
| 10 | assert_eq!( | ||
| 11 | id.guid, | ||
| 12 | [ | ||
| 13 | 0x33, 0x22, 0x11, 0, 0x55, 0x44, 0x77, 0x66, 0x88, 0x99, 0xaa, 0xbb, 0xcc, 0xdd, 0xee, | ||
| 14 | 0xff | ||
| 15 | ] | ||
| 16 | ); | ||
| 17 | assert_eq!(id.n, 42); | ||
| 18 | assert_eq!(text.to_lowercase().parse::<ExGuid>().unwrap(), id); | ||
| 19 | assert_eq!(id.to_string(), text); | ||
| 20 | let mut random = 7_u64; | ||
| 21 | for n in 0..1024 { | ||
| 22 | let guid = std::array::from_fn(|_| { | ||
| 23 | random = random.wrapping_mul(6364136223846793005).wrapping_add(1); | ||
| 24 | random.to_le_bytes()[0] | ||
| 25 | }); | ||
| 26 | let id = ExGuid { guid, n }; | ||
| 27 | assert_eq!(id.to_string().parse::<ExGuid>().unwrap(), id); | ||
| 28 | assert_eq!( | ||
| 29 | serde_json::from_str::<ExGuid>(&serde_json::to_string(&id).unwrap()).unwrap(), | ||
| 30 | id | ||
| 31 | ); | ||
| 32 | } | ||
| 33 | for id in [ | ||
| 34 | ExGuid::default(), | ||
| 35 | ExGuid { | ||
| 36 | guid: [255; 16], | ||
| 37 | n: u32::MAX, | ||
| 38 | }, | ||
| 39 | ] { | ||
| 40 | assert_eq!(id.to_string().parse::<ExGuid>().unwrap(), id); | ||
| 41 | } | ||
| 42 | for at in 0..text.len() { | ||
| 43 | let mut changed = text.as_bytes().to_vec(); | ||
| 44 | changed[at] = b'?'; | ||
| 45 | assert!( | ||
| 46 | String::from_utf8(changed) | ||
| 47 | .unwrap() | ||
| 48 | .parse::<ExGuid>() | ||
| 49 | .is_err() | ||
| 50 | ); | ||
| 51 | } | ||
| 52 | for changed in [ | ||
| 53 | text.replace(",42", ",+42"), | ||
| 54 | text.replace(",42", ",042"), | ||
| 55 | text.replace(",42", ",4294967296"), | ||
| 56 | text.replace(",42", ",-1"), | ||
| 57 | text.replace("00", "+0"), | ||
| 58 | text.replace("00", "é"), | ||
| 59 | "{00000000-0000-0000-0000-000000000000},1".to_owned(), | ||
| 60 | format!(" {text}"), | ||
| 61 | format!("{text} "), | ||
| 62 | ] { | ||
| 63 | assert!(changed.parse::<ExGuid>().is_err(), "{changed}"); | ||
| 64 | assert!(serde_json::from_str::<ExGuid>(&serde_json::to_string(&changed).unwrap()).is_err()); | ||
| 65 | } | ||
| 66 | } | ||
| 67 | |||
| 6 | #[test] | 68 | #[test] |
| 7 | fn native_encryption_remains_opaque() { | 69 | fn native_encryption_remains_opaque() { |
| 8 | let bytes = | 70 | let bytes = |
crates/onestore/tests/shared_snapshot.rs created+420
| ... | @@ -0,0 +1,420 @@ | ||
| 1 | #[path = "support/current.rs"] | ||
| 2 | mod current; | ||
| 3 | use current::current; | ||
| 4 | |||
| 5 | #[path = "support/checkpoint.rs"] | ||
| 6 | mod checkpoint; | ||
| 7 | #[path = "support/trace.rs"] | ||
| 8 | mod trace; | ||
| 9 | |||
| 10 | use onestore::read_snapshot; | ||
| 11 | use onestore::{ | ||
| 12 | ExGuid, RevisionIndex, Store, | ||
| 13 | document::{Document, Kind}, | ||
| 14 | }; | ||
| 15 | use std::{fs, io}; | ||
| 16 | use trace::{Event, Trace}; | ||
| 17 | |||
| 18 | fn target(bytes: &[u8]) -> (ExGuid, ExGuid, u32) { | ||
| 19 | let store = Store::parse(bytes).unwrap(); | ||
| 20 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 21 | let doc = Document::parse(&index).unwrap(); | ||
| 22 | doc.spaces | ||
| 23 | .iter() | ||
| 24 | .find_map(|(sid, space)| { | ||
| 25 | space.revisions[&space.contexts[&ExGuid::default()]] | ||
| 26 | .nodes | ||
| 27 | .iter() | ||
| 28 | .find_map(|(oid, node)| { | ||
| 29 | if let Kind::RichText { text, .. } = &node.kind | ||
| 30 | && (text.starts_with("Fictitious") || text.starts_with("Transaction")) | ||
| 31 | { | ||
| 32 | return Some((*sid, *oid, text.encode_utf16().count() as u32)); | ||
| 33 | } | ||
| 34 | None | ||
| 35 | }) | ||
| 36 | }) | ||
| 37 | .unwrap() | ||
| 38 | } | ||
| 39 | |||
| 40 | #[test] | ||
| 41 | fn version_cached_readers_cannot_miss_a_completed_publication() { | ||
| 42 | for path in [ | ||
| 43 | "native/20260905-05/snapshots/03-format-unicode/notebook/synthetic.one", | ||
| 44 | "append/round-01/tx-255/notebook/synthetic.one", | ||
| 45 | ] { | ||
| 46 | let source = fs::read(format!("../../corpus/{path}")).unwrap(); | ||
| 47 | let (sid, oid, end) = target(&source); | ||
| 48 | let mut trace = Trace { | ||
| 49 | bytes: source.clone(), | ||
| 50 | events: Vec::new(), | ||
| 51 | }; | ||
| 52 | onestore::commit_text(&mut trace, &source, sid, oid, end..end, " [cached reader]").unwrap(); | ||
| 53 | let final_content = current(&trace.bytes); | ||
| 54 | let mut visible = source; | ||
| 55 | for event in &trace.events { | ||
| 56 | let Event::Write(offset, bytes) = event else { | ||
| 57 | continue; | ||
| 58 | }; | ||
| 59 | visible.resize(visible.len().max(offset + bytes.len()), 0); | ||
| 60 | for (index, byte) in bytes.iter().enumerate() { | ||
| 61 | visible[offset + index] = *byte; | ||
| 62 | if (212..252).contains(&(offset + index)) { | ||
| 63 | let cached_content = current(&visible); | ||
| 64 | let refreshed = if visible[212..252] == trace.bytes[212..252] { | ||
| 65 | cached_content | ||
| 66 | } else { | ||
| 67 | current(&trace.bytes) | ||
| 68 | }; | ||
| 69 | assert_eq!( | ||
| 70 | refreshed, | ||
| 71 | final_content, | ||
| 72 | "{path}: cached header at {}", | ||
| 73 | offset + index | ||
| 74 | ); | ||
| 75 | } | ||
| 76 | } | ||
| 77 | } | ||
| 78 | } | ||
| 79 | } | ||
| 80 | |||
| 81 | #[test] | ||
| 82 | fn published_snapshots_survive_interleaved_commit_io() { | ||
| 83 | let cases = [ | ||
| 84 | ( | ||
| 85 | "unicode", | ||
| 86 | "native/20260905-05/snapshots/03-format-unicode/notebook/synthetic.one", | ||
| 87 | ), | ||
| 88 | ( | ||
| 89 | "attachment", | ||
| 90 | "native/20260905-05/snapshots/06-attachment/notebook/synthetic.one", | ||
| 91 | ), | ||
| 92 | ( | ||
| 93 | "rollover-256", | ||
| 94 | "append/round-01/tx-255/notebook/synthetic.one", | ||
| 95 | ), | ||
| 96 | ( | ||
| 97 | "rollover-65536", | ||
| 98 | "append/round-01/tx-65535/notebook/synthetic.one", | ||
| 99 | ), | ||
| 100 | ( | ||
| 101 | "checkpoint", | ||
| 102 | "native/20260905-05/snapshots/03-format-unicode/notebook/synthetic.one", | ||
| 103 | ), | ||
| 104 | ]; | ||
| 105 | for (name, path) in cases { | ||
| 106 | let mut source = fs::read(format!("../../corpus/{path}")).unwrap(); | ||
| 107 | let (sid, oid, _) = target(&source); | ||
| 108 | if name == "checkpoint" { | ||
| 109 | source = checkpoint::pending(&source, sid, oid, 0x14001d7a); | ||
| 110 | } | ||
| 111 | let (_, _, end) = target(&source); | ||
| 112 | let before = current(&source); | ||
| 113 | let mut trace = Trace { | ||
| 114 | bytes: source.clone(), | ||
| 115 | events: Vec::new(), | ||
| 116 | }; | ||
| 117 | onestore::commit_text(&mut trace, &source, sid, oid, end..end, " [reader café 🦀]") | ||
| 118 | .unwrap(); | ||
| 119 | let after = current(&trace.bytes); | ||
| 120 | assert_ne!(before, after); | ||
| 121 | let mut writes = Vec::new(); | ||
| 122 | for event in &trace.events { | ||
| 123 | if let Event::Write(offset, bytes) = event { | ||
| 124 | let piece = if *offset < 1024 { 17 } else { 4096 }; | ||
| 125 | writes.extend( | ||
| 126 | bytes | ||
| 127 | .chunks(piece) | ||
| 128 | .enumerate() | ||
| 129 | .map(|(i, bytes)| (offset + i * piece, bytes)), | ||
| 130 | ); | ||
| 131 | } | ||
| 132 | } | ||
| 133 | let mut accepted = [0; 2]; | ||
| 134 | let mut retried = 0; | ||
| 135 | let mut interleaved = 0; | ||
| 136 | for run in 0..writes.len() + 1 + 512 { | ||
| 137 | let paused = run <= writes.len(); | ||
| 138 | let mut step = if paused { run } else { 0 }; | ||
| 139 | let mut visible = source.clone(); | ||
| 140 | for &(offset, bytes) in &writes[..step] { | ||
| 141 | visible.resize(visible.len().max(offset + bytes.len()), 0); | ||
| 142 | visible[offset..offset + bytes.len()].copy_from_slice(bytes); | ||
| 143 | } | ||
| 144 | let mut seed = run as u64 + 1; | ||
| 145 | let mut read_calls = 0; | ||
| 146 | let mut overlapped = false; | ||
| 147 | let read_limit = [17, 193, 4096, 65536][run % 4]; | ||
| 148 | let result = read_snapshot( | ||
| 149 | |offset, output| { | ||
| 150 | seed ^= seed << 13; | ||
| 151 | seed ^= seed >> 7; | ||
| 152 | seed ^= seed << 17; | ||
| 153 | if !paused { | ||
| 154 | let count = if seed.is_multiple_of(11) { | ||
| 155 | writes.len() | ||
| 156 | } else { | ||
| 157 | (seed % 4) as usize | ||
| 158 | }; | ||
| 159 | let end = writes.len().min(step + count); | ||
| 160 | for &(at, bytes) in &writes[step..end] { | ||
| 161 | visible.resize(visible.len().max(at + bytes.len()), 0); | ||
| 162 | visible[at..at + bytes.len()].copy_from_slice(bytes); | ||
| 163 | } | ||
| 164 | overlapped |= read_calls > 0 && end > step; | ||
| 165 | step = end; | ||
| 166 | } | ||
| 167 | read_calls += 1; | ||
| 168 | let offset = offset as usize; | ||
| 169 | let count = output | ||
| 170 | .len() | ||
| 171 | .min(read_limit) | ||
| 172 | .min(visible.len().saturating_sub(offset)); | ||
| 173 | if count != 0 { | ||
| 174 | output[..count].copy_from_slice(&visible[offset..offset + count]); | ||
| 175 | } | ||
| 176 | Ok(count) | ||
| 177 | }, | ||
| 178 | trace.bytes.len(), | ||
| 179 | ); | ||
| 180 | interleaved += usize::from(overlapped); | ||
| 181 | if paused && (run == 0 || run == writes.len()) { | ||
| 182 | assert!( | ||
| 183 | matches!(&result, Ok(Some(_))), | ||
| 184 | "{name}: quiescent run {run}" | ||
| 185 | ); | ||
| 186 | } | ||
| 187 | match result { | ||
| 188 | Ok(Some(bytes)) => { | ||
| 189 | let checked = std::panic::catch_unwind(|| { | ||
| 190 | let observed = current(&bytes); | ||
| 191 | assert!( | ||
| 192 | observed == before || observed == after, | ||
| 193 | "{name}: run {run}, write step {step}" | ||
| 194 | ); | ||
| 195 | if run == writes.len() { | ||
| 196 | assert_eq!(observed, after); | ||
| 197 | } | ||
| 198 | observed == after | ||
| 199 | }); | ||
| 200 | match checked { | ||
| 201 | Ok(new) => accepted[usize::from(new)] += 1, | ||
| 202 | Err(failure) => { | ||
| 203 | let time = std::time::SystemTime::now() | ||
| 204 | .duration_since(std::time::UNIX_EPOCH) | ||
| 205 | .unwrap() | ||
| 206 | .as_nanos(); | ||
| 207 | let path = std::path::PathBuf::from(format!( | ||
| 208 | "../../evidence/m9/read-interleaving-failure-{name}-{run}-{time}" | ||
| 209 | )); | ||
| 210 | fs::create_dir_all(&path).unwrap(); | ||
| 211 | fs::write(path.join("source.one"), &source).unwrap(); | ||
| 212 | fs::write(path.join("observed.one"), &bytes).unwrap(); | ||
| 213 | fs::write( | ||
| 214 | path.join("replay.json"), | ||
| 215 | serde_json::to_vec(&serde_json::json!({ | ||
| 216 | "run": run, "read_limit": read_limit, "writes": writes, | ||
| 217 | })) | ||
| 218 | .unwrap(), | ||
| 219 | ) | ||
| 220 | .unwrap(); | ||
| 221 | eprintln!("Replay: {}", path.display()); | ||
| 222 | std::panic::resume_unwind(failure); | ||
| 223 | } | ||
| 224 | } | ||
| 225 | } | ||
| 226 | Ok(None) => retried += 1, | ||
| 227 | Err(error) | ||
| 228 | if matches!( | ||
| 229 | error.kind(), | ||
| 230 | io::ErrorKind::UnexpectedEof | io::ErrorKind::InvalidData | ||
| 231 | ) => | ||
| 232 | { | ||
| 233 | retried += 1 | ||
| 234 | } | ||
| 235 | Err(error) => panic!("{name}: run {run}: {error}"), | ||
| 236 | } | ||
| 237 | } | ||
| 238 | assert!( | ||
| 239 | accepted[0] > 0 && accepted[1] > 0 && interleaved > 0 && retried > 0, | ||
| 240 | "{name}" | ||
| 241 | ); | ||
| 242 | println!( | ||
| 243 | "{name}: old={}, new={}, retry={retried}, overlap={interleaved}", | ||
| 244 | accepted[0], accepted[1] | ||
| 245 | ); | ||
| 246 | } | ||
| 247 | } | ||
| 248 | |||
| 249 | #[test] | ||
| 250 | fn snapshot_rejects_short_io_and_unbounded_allocation() { | ||
| 251 | let source = onestore::create_section("test.one", "read", "test").unwrap(); | ||
| 252 | let mut calls = 0; | ||
| 253 | let result = read_snapshot( | ||
| 254 | |offset, output| { | ||
| 255 | calls += 1; | ||
| 256 | if calls == 1 { | ||
| 257 | return Err(io::ErrorKind::Interrupted.into()); | ||
| 258 | } | ||
| 259 | let offset = offset as usize; | ||
| 260 | let count = output.len().min(7).min(source.len().saturating_sub(offset)); | ||
| 261 | if count != 0 { | ||
| 262 | output[..count].copy_from_slice(&source[offset..offset + count]); | ||
| 263 | } | ||
| 264 | Ok(count) | ||
| 265 | }, | ||
| 266 | source.len(), | ||
| 267 | ) | ||
| 268 | .unwrap() | ||
| 269 | .unwrap(); | ||
| 270 | assert_eq!(current(&source), current(&result)); | ||
| 271 | assert!(calls > source.len() / 7); | ||
| 272 | let error = read_snapshot(|_, output| Ok(output.len() + 1), source.len()).unwrap_err(); | ||
| 273 | assert_eq!(error.kind(), io::ErrorKind::InvalidData); | ||
| 274 | let error = read_snapshot(|_, _| Ok(0), source.len()).unwrap_err(); | ||
| 275 | assert_eq!(error.kind(), io::ErrorKind::UnexpectedEof); | ||
| 276 | let error = read_snapshot( | ||
| 277 | |_, output| { | ||
| 278 | output.copy_from_slice(&source[..1024]); | ||
| 279 | Ok(1024) | ||
| 280 | }, | ||
| 281 | 1023, | ||
| 282 | ) | ||
| 283 | .unwrap_err(); | ||
| 284 | assert_eq!(error.kind(), io::ErrorKind::InvalidData); | ||
| 285 | } | ||
| 286 | |||
| 287 | #[test] | ||
| 288 | fn native_tail_truncation_before_header_update_is_retried() { | ||
| 289 | let compact = onestore::create_section("test.one", "Transaction retained", "test").unwrap(); | ||
| 290 | let mut expanded = compact.clone(); | ||
| 291 | expanded.resize(compact.len() + 216, 0); | ||
| 292 | let length = expanded.len() as u64; | ||
| 293 | expanded[196..204].copy_from_slice(&length.to_le_bytes()); | ||
| 294 | assert_eq!(current(&expanded), current(&compact)); | ||
| 295 | for block in [17, 193, 65536] { | ||
| 296 | for trigger in [0, 1024usize.div_ceil(block)] { | ||
| 297 | let mut visible = expanded.clone(); | ||
| 298 | let mut calls = 0; | ||
| 299 | let result = read_snapshot( | ||
| 300 | |offset, output| { | ||
| 301 | if calls == trigger { | ||
| 302 | visible.truncate(compact.len()); | ||
| 303 | } | ||
| 304 | calls += 1; | ||
| 305 | let offset = offset as usize; | ||
| 306 | let count = output | ||
| 307 | .len() | ||
| 308 | .min(block) | ||
| 309 | .min(visible.len().saturating_sub(offset)); | ||
| 310 | if count != 0 { | ||
| 311 | output[..count].copy_from_slice(&visible[offset..offset + count]); | ||
| 312 | } | ||
| 313 | Ok(count) | ||
| 314 | }, | ||
| 315 | expanded.len(), | ||
| 316 | ) | ||
| 317 | .unwrap(); | ||
| 318 | assert!( | ||
| 319 | result.is_none(), | ||
| 320 | "accepted a snapshot with a stale expected length" | ||
| 321 | ); | ||
| 322 | } | ||
| 323 | } | ||
| 324 | let result = read_snapshot( | ||
| 325 | |offset, output| { | ||
| 326 | let offset = offset as usize; | ||
| 327 | let count = output.len().min(compact.len().saturating_sub(offset)); | ||
| 328 | if count != 0 { | ||
| 329 | output[..count].copy_from_slice(&compact[offset..offset + count]); | ||
| 330 | } | ||
| 331 | Ok(count) | ||
| 332 | }, | ||
| 333 | expanded.len(), | ||
| 334 | ) | ||
| 335 | .unwrap() | ||
| 336 | .unwrap(); | ||
| 337 | assert_eq!(result, compact); | ||
| 338 | } | ||
| 339 | |||
| 340 | #[test] | ||
| 341 | fn unpublished_append_remains_available_for_retry() { | ||
| 342 | let source = onestore::create_section("test.one", "Transaction before", "test").unwrap(); | ||
| 343 | let (sid, oid, end) = target(&source); | ||
| 344 | let mut trace = Trace { | ||
| 345 | bytes: source.clone(), | ||
| 346 | events: Vec::new(), | ||
| 347 | }; | ||
| 348 | onestore::commit_text(&mut trace, &source, sid, oid, end..end, " abandoned").unwrap(); | ||
| 349 | let Event::Write(offset, append) = &trace.events[0] else { | ||
| 350 | panic!() | ||
| 351 | }; | ||
| 352 | assert_eq!(*offset, source.len()); | ||
| 353 | for length in [1, 17, append.len()] { | ||
| 354 | let mut persisted = source.clone(); | ||
| 355 | persisted.extend_from_slice(&append[..length]); | ||
| 356 | let snapshot = read_snapshot( | ||
| 357 | |offset, output| { | ||
| 358 | let offset = offset as usize; | ||
| 359 | let count = output.len().min(persisted.len().saturating_sub(offset)); | ||
| 360 | output[..count].copy_from_slice(&persisted[offset..offset + count]); | ||
| 361 | Ok(count) | ||
| 362 | }, | ||
| 363 | persisted.len(), | ||
| 364 | ) | ||
| 365 | .unwrap() | ||
| 366 | .unwrap(); | ||
| 367 | assert_eq!(snapshot, persisted); | ||
| 368 | assert_eq!(current(&snapshot), current(&source)); | ||
| 369 | let mut retry = Trace { | ||
| 370 | bytes: persisted, | ||
| 371 | events: Vec::new(), | ||
| 372 | }; | ||
| 373 | onestore::commit_text(&mut retry, &snapshot, sid, oid, end..end, " retry").unwrap(); | ||
| 374 | assert_ne!(current(&retry.bytes), current(&source)); | ||
| 375 | } | ||
| 376 | } | ||
| 377 | |||
| 378 | #[test] | ||
| 379 | fn header_comparison_cannot_replace_maintenance_exclusion() { | ||
| 380 | let mut source = onestore::create_section("test.one", "AAAA BBBB", "test").unwrap(); | ||
| 381 | source[212..228].fill(9); | ||
| 382 | source[236..252].fill(7); | ||
| 383 | let encoded: Vec<_> = "AAAA BBBB" | ||
| 384 | .encode_utf16() | ||
| 385 | .flat_map(u16::to_le_bytes) | ||
| 386 | .collect(); | ||
| 387 | let offset = source | ||
| 388 | .windows(encoded.len()) | ||
| 389 | .position(|bytes| bytes == encoded) | ||
| 390 | .unwrap(); | ||
| 391 | let mut changed = source.clone(); | ||
| 392 | let replacement: Vec<_> = "ZZZZ YYYY" | ||
| 393 | .encode_utf16() | ||
| 394 | .flat_map(u16::to_le_bytes) | ||
| 395 | .collect(); | ||
| 396 | changed[offset..offset + replacement.len()].copy_from_slice(&replacement); | ||
| 397 | let before = current(&source); | ||
| 398 | let after = current(&changed); | ||
| 399 | assert_ne!(before, after); | ||
| 400 | let mut visible = &source; | ||
| 401 | let result = read_snapshot( | ||
| 402 | |at, output| { | ||
| 403 | let at = at as usize; | ||
| 404 | if at >= offset + 8 { | ||
| 405 | visible = &changed; | ||
| 406 | } | ||
| 407 | let count = output.len().min(2).min(visible.len().saturating_sub(at)); | ||
| 408 | if count != 0 { | ||
| 409 | output[..count].copy_from_slice(&visible[at..at + count]); | ||
| 410 | } | ||
| 411 | Ok(count) | ||
| 412 | }, | ||
| 413 | source.len(), | ||
| 414 | ) | ||
| 415 | .unwrap() | ||
| 416 | .unwrap(); | ||
| 417 | let hybrid = current(&result); | ||
| 418 | assert_ne!(hybrid, before); | ||
| 419 | assert_ne!(hybrid, after); | ||
| 420 | } | ||
crates/onestore/tests/support/current.rs created+26
| ... | @@ -0,0 +1,26 @@ | ||
| 1 | use onestore::{ExGuid, RevisionIndex, Store, document::Document}; | ||
| 2 | use std::collections::BTreeMap; | ||
| 3 | |||
| 4 | pub fn current(bytes: &[u8]) -> BTreeMap<ExGuid, String> { | ||
| 5 | let store = Store::parse(bytes).unwrap(); | ||
| 6 | assert!(store.checksum_mismatches.is_empty()); | ||
| 7 | let index = RevisionIndex::parse(&store).unwrap(); | ||
| 8 | index.validate_current().unwrap(); | ||
| 9 | Document::parse(&index).unwrap(); | ||
| 10 | index | ||
| 11 | .spaces | ||
| 12 | .iter() | ||
| 13 | .map(|(sid, space)| { | ||
| 14 | let rid = space.labels[&(ExGuid::default(), 1)]; | ||
| 15 | let revision = index.resolve(*sid, rid).unwrap(); | ||
| 16 | for object in revision.objects.values() { | ||
| 17 | if let Some(onestore::FileDataReference::Internal(guid)) = | ||
| 18 | object.file_reference().unwrap() | ||
| 19 | { | ||
| 20 | store.file_data(guid).unwrap(); | ||
| 21 | } | ||
| 22 | } | ||
| 23 | (*sid, format!("{revision:?}")) | ||
| 24 | }) | ||
| 25 | .collect() | ||
| 26 | } | ||
crates/onestore/tests/support/trace.rs created+36
| ... | @@ -0,0 +1,36 @@ | ||
| 1 | use onestore::CommitIo; | ||
| 2 | use std::io; | ||
| 3 | |||
| 4 | pub enum Event { | ||
| 5 | Write(usize, Vec<u8>), | ||
| 6 | Flush, | ||
| 7 | } | ||
| 8 | |||
| 9 | pub struct Trace { | ||
| 10 | pub bytes: Vec<u8>, | ||
| 11 | pub events: Vec<Event>, | ||
| 12 | } | ||
| 13 | |||
| 14 | impl CommitIo for Trace { | ||
| 15 | fn read_at(&mut self, offset: u64, output: &mut [u8]) -> io::Result<usize> { | ||
| 16 | let offset = offset as usize; | ||
| 17 | let count = output.len().min(self.bytes.len().saturating_sub(offset)); | ||
| 18 | if count != 0 { | ||
| 19 | output[..count].copy_from_slice(&self.bytes[offset..offset + count]); | ||
| 20 | } | ||
| 21 | Ok(count) | ||
| 22 | } | ||
| 23 | fn write_at(&mut self, offset: u64, bytes: &[u8]) -> io::Result<usize> { | ||
| 24 | let offset = offset as usize; | ||
| 25 | let count = bytes.len().min(4096); | ||
| 26 | self.bytes.resize(self.bytes.len().max(offset + count), 0); | ||
| 27 | self.bytes[offset..offset + count].copy_from_slice(&bytes[..count]); | ||
| 28 | self.events | ||
| 29 | .push(Event::Write(offset, bytes[..count].to_vec())); | ||
| 30 | Ok(count) | ||
| 31 | } | ||
| 32 | fn flush(&mut self) -> io::Result<()> { | ||
| 33 | self.events.push(Event::Flush); | ||
| 34 | Ok(()) | ||
| 35 | } | ||
| 36 | } | ||
readme.md created+1
| ... | @@ -0,0 +1 @@ | ||
| 1 | # Snowbound — Freeform note taking | ||
tools/codex_usage_report.example.json created+18
| ... | @@ -0,0 +1,18 @@ | ||
| 1 | { | ||
| 2 | "threadIds": ["01a078fb-d5d2-7ad0-97f0-d14db4e4c94e"], | ||
| 3 | "sources": [ | ||
| 4 | "/Users/clo/.codex/sessions/2026/09/06/rollout-2026-09-06T16-09-25-01a078fb-d5d2-7ad0-97f0-d14db4e4c94e.jsonl" | ||
| 5 | ], | ||
| 6 | "boundaries": [ | ||
| 7 | { | ||
| 8 | "name": "Planning discussion", | ||
| 9 | "kind": "discussion", | ||
| 10 | "end": "2026-09-07T01:28:56Z" | ||
| 11 | }, | ||
| 12 | { | ||
| 13 | "name": "Canvas goal", | ||
| 14 | "kind": "goal", | ||
| 15 | "start": "2026-09-07T01:28:56Z" | ||
| 16 | } | ||
| 17 | ] | ||
| 18 | } | ||
tools/codex_usage_report.mjs created+129
| ... | @@ -0,0 +1,129 @@ | ||
| 1 | #!/usr/bin/env node | ||
| 2 | |||
| 3 | import { mkdir, readFile, readdir, writeFile } from "node:fs/promises"; | ||
| 4 | import { homedir } from "node:os"; | ||
| 5 | import { dirname, join, resolve } from "node:path"; | ||
| 6 | |||
| 7 | const rates = { input: 10, cached: 1, output: 50 }; | ||
| 8 | |||
| 9 | function usageCost(usage) { | ||
| 10 | return ((usage.input_tokens - usage.cached_input_tokens) * rates.input | ||
| 11 | + usage.cached_input_tokens * rates.cached | ||
| 12 | + usage.output_tokens * rates.output) / 1_000_000; | ||
| 13 | } | ||
| 14 | |||
| 15 | function duration(ms) { | ||
| 16 | const seconds = Math.round(ms / 1000); | ||
| 17 | const hours = Math.floor(seconds / 3600); | ||
| 18 | const minutes = Math.floor((seconds % 3600) / 60); | ||
| 19 | const remainder = seconds % 60; | ||
| 20 | return hours ? `${hours}h ${minutes}m` : minutes ? `${minutes}m ${remainder}s` : `${remainder}s`; | ||
| 21 | } | ||
| 22 | |||
| 23 | function unionDuration(intervals) { | ||
| 24 | const merged = []; | ||
| 25 | for (const interval of intervals.sort((a, b) => a[0] - b[0])) { | ||
| 26 | const previous = merged.at(-1); | ||
| 27 | if (previous && interval[0] <= previous[1]) previous[1] = Math.max(previous[1], interval[1]); | ||
| 28 | else merged.push([...interval]); | ||
| 29 | } | ||
| 30 | return merged.reduce((total, [start, end]) => total + end - start, 0); | ||
| 31 | } | ||
| 32 | |||
| 33 | async function filesBelow(root) { | ||
| 34 | const entries = await readdir(root, { withFileTypes: true }); | ||
| 35 | const nested = await Promise.all(entries.map((entry) => { | ||
| 36 | const path = join(root, entry.name); | ||
| 37 | return entry.isDirectory() ? filesBelow(path) : entry.name.endsWith(".jsonl") ? [path] : []; | ||
| 38 | })); | ||
| 39 | return nested.flat(); | ||
| 40 | } | ||
| 41 | |||
| 42 | async function loadEvents(paths) { | ||
| 43 | const events = []; | ||
| 44 | for (const path of paths) { | ||
| 45 | for (const line of (await readFile(path, "utf8")).trim().split("\n")) { | ||
| 46 | if (!line) continue; | ||
| 47 | try { events.push(JSON.parse(line)); } catch { /* Ignore a partially-written final line. */ } | ||
| 48 | } | ||
| 49 | } | ||
| 50 | return events; | ||
| 51 | } | ||
| 52 | |||
| 53 | function snapshotBefore(records, time) { | ||
| 54 | return records.filter((record) => Date.parse(record.timestamp) <= time).at(-1)?.payload.thread_token_usage; | ||
| 55 | } | ||
| 56 | |||
| 57 | function taskRows(events, start, end) { | ||
| 58 | const tasks = new Map(); | ||
| 59 | for (const event of events) { | ||
| 60 | if (event.type !== "event_msg" || !["task_started", "task_complete"].includes(event.payload.type)) continue; | ||
| 61 | const prior = tasks.get(event.payload.turn_id) ?? {}; | ||
| 62 | tasks.set(event.payload.turn_id, { ...prior, ...event.payload }); | ||
| 63 | } | ||
| 64 | return [...tasks.values()].filter((task) => task.started_at >= start && task.started_at < end && task.duration_ms); | ||
| 65 | } | ||
| 66 | |||
| 67 | function toolIntervals(events, start, end) { | ||
| 68 | return events.flatMap((event) => { | ||
| 69 | const item = event.type === "event_msg" && event.payload.type === "item_completed" ? event.payload.item : undefined; | ||
| 70 | if (!item || !["CommandExecution", "McpToolCall"].includes(item.type)) return []; | ||
| 71 | const milliseconds = (item.duration?.secs ?? 0) * 1000 + (item.duration?.nanos ?? 0) / 1_000_000; | ||
| 72 | const finished = Date.parse(event.timestamp); | ||
| 73 | return milliseconds && finished >= start && finished < end ? [[finished - milliseconds, finished]] : []; | ||
| 74 | }); | ||
| 75 | } | ||
| 76 | |||
| 77 | function segment(events, records, boundary, previous, finalTime) { | ||
| 78 | const start = boundary.start ? Date.parse(boundary.start) : -Infinity; | ||
| 79 | const end = boundary.end ? Date.parse(boundary.end) : finalTime; | ||
| 80 | const tasks = taskRows(events, start, end); | ||
| 81 | const tools = toolIntervals(events, start, end); | ||
| 82 | const finish = snapshotBefore(records, end); | ||
| 83 | const begin = previous ?? snapshotBefore(records, start); | ||
| 84 | return { | ||
| 85 | name: boundary.name, | ||
| 86 | kind: boundary.kind ?? "segment", | ||
| 87 | cost: finish ? usageCost(finish) - (begin ? usageCost(begin) : 0) : null, | ||
| 88 | durationMs: tasks.reduce((total, task) => total + task.duration_ms, 0), | ||
| 89 | apiTtftMs: tasks.reduce((total, task) => total + (task.time_to_first_token_ms ?? 0), 0), | ||
| 90 | toolWallMs: unionDuration(tools), | ||
| 91 | toolCoreMs: tools.reduce((total, [startTime, endTime]) => total + endTime - startTime, 0), | ||
| 92 | }; | ||
| 93 | } | ||
| 94 | |||
| 95 | function chart(segments) { | ||
| 96 | const maximum = Math.max(...segments.map((segment) => segment.cost ?? 0), 1); | ||
| 97 | return segments.map((segment) => `<div class="row"><span>${escapeHtml(segment.name)}</span><div class="bar"><i style="width:${(segment.cost ?? 0) / maximum * 100}%"></i></div><b>${segment.cost === null ? "—" : `$${segment.cost.toFixed(2)}`}</b></div>`).join("\n"); | ||
| 98 | } | ||
| 99 | |||
| 100 | function escapeHtml(value) { | ||
| 101 | return value.replaceAll("&", "&").replaceAll("<", "<").replaceAll(">", ">"); | ||
| 102 | } | ||
| 103 | |||
| 104 | function html(report) { | ||
| 105 | const rows = report.segments.map((segment) => `<tr><td>${escapeHtml(segment.name)}</td><td>${segment.kind}</td><td>${segment.cost === null ? "—" : `$${segment.cost.toFixed(2)}`}</td><td>${duration(segment.durationMs)}</td><td>${duration(segment.toolWallMs)}</td><td>${duration(segment.toolCoreMs)}</td><td>≥${duration(segment.apiTtftMs)}</td></tr>`).join("\n"); | ||
| 106 | return `<!doctype html><meta charset="utf-8"><title>Codex usage report</title><style>body{font:14px system-ui;margin:40px;max-width:900px;color:#202124}table{border-collapse:collapse;width:100%}th,td{text-align:left;padding:8px;border-bottom:1px solid #ddd}.row{display:grid;grid-template-columns:220px 1fr 90px;gap:12px;align-items:center;margin:8px 0}.bar{background:#edf0f2;height:14px;border-radius:8px;overflow:hidden}.bar i{display:block;height:100%;background:#3874cb}</style><h1>Codex usage report</h1><p>API-equivalent rate: Astra standard ($10/M uncached input, $1/M cached input, $50/M output). Tool core time counts parallel calls separately; API TTFT is a lower bound.</p><h2>Cost</h2>${chart(report.segments)}<h2>Segments</h2><table><thead><tr><th>Segment</th><th>Type</th><th>Cost</th><th>Duration</th><th>Tool wall</th><th>Tool core</th><th>Recorded API</th></tr></thead><tbody>${rows}</tbody></table>`; | ||
| 107 | } | ||
| 108 | |||
| 109 | function usage() { | ||
| 110 | console.error("Usage: node tools/codex_usage_report.mjs report.json [output.html]"); | ||
| 111 | process.exit(1); | ||
| 112 | } | ||
| 113 | |||
| 114 | const [configPath, outputPath] = process.argv.slice(2); | ||
| 115 | if (!configPath) usage(); | ||
| 116 | const config = JSON.parse(await readFile(resolve(configPath), "utf8")); | ||
| 117 | const codexHome = config.codexHome ?? join(homedir(), ".codex"); | ||
| 118 | const paths = config.sources ?? (await Promise.all(["sessions", "archived_sessions"].map((name) => filesBelow(join(codexHome, name))))).flat(); | ||
| 119 | const events = await loadEvents(paths); | ||
| 120 | const selected = events.filter((event) => event.type !== "token_usage_record" || config.threadIds.includes(event.payload.thread_id)); | ||
| 121 | const records = selected.filter((event) => event.type === "token_usage_record").sort((a, b) => Date.parse(a.timestamp) - Date.parse(b.timestamp)); | ||
| 122 | if (!records.length) throw new Error("No token records matched config.threadIds."); | ||
| 123 | const finalTime = Date.parse(records.at(-1).timestamp); | ||
| 124 | const segments = config.boundaries.map((boundary, index) => segment(selected, records, boundary, index ? snapshotBefore(records, Date.parse(config.boundaries[index - 1].end ?? new Date(finalTime).toISOString())) : undefined, finalTime)); | ||
| 125 | const report = { threadIds: config.threadIds, total: usageCost(records.at(-1).payload.thread_token_usage), segments }; | ||
| 126 | if (outputPath) { | ||
| 127 | await mkdir(dirname(resolve(outputPath)), { recursive: true }); | ||
| 128 | await writeFile(resolve(outputPath), html(report)); | ||
| 129 | } else console.log(JSON.stringify(report, null, 2)); | ||
tools/concurrent_rust.py+15-5
| ... | @@ -70,21 +70,31 @@ def verify(logs, initial_transaction, writers, operations, edit=False): | ... | @@ -70,21 +70,31 @@ def verify(logs, initial_transaction, writers, operations, edit=False): |
| 70 | 70 | ||
| 71 | 71 | ||
| 72 | @contextmanager | 72 | @contextmanager |
| 73 | def running_clients(output, source, writers, readers, operations, seed, timeout=600, edit=False): | 73 | def running_clients(output, source, writers, readers, operations, seed, timeout=600, edit=False, executable=CLIENT, environment=None, reader_executable=None): |
| 74 | if not 0 < timeout < 2**64 / 1000: | ||
| 75 | raise ValueError('Choose a finite positive client timeout.') | ||
| 76 | timeout_ms = int(timeout * 1000) | ||
| 77 | if not 0 < timeout_ms < 2**64: | ||
| 78 | raise ValueError('Client timeout does not fit the subprocess clock.') | ||
| 79 | environment = {**(os.environ if environment is None else environment), 'ONESTORE_CLIENT_TIMEOUT_MS': str(timeout_ms)} | ||
| 74 | start, stop = output / 'start', output / 'stop' | 80 | start, stop = output / 'start', output / 'stop' |
| 75 | (output / 'clients.json').write_text(json.dumps({'writers': writers, 'readers': readers, 'operations': operations, | 81 | manifest = {'writers': writers, 'readers': readers, 'operations': operations, |
| 76 | 'seed': seed, 'edit': edit, 'client_sha256': hashlib.sha256(CLIENT.read_bytes()).hexdigest()}, indent=2)) | 82 | 'seed': seed, 'edit': edit, 'timeout_ms': timeout_ms, 'client_sha256': hashlib.sha256(executable.read_bytes()).hexdigest(), 'executable': str(executable)} |
| 83 | if reader_executable is not None: | ||
| 84 | manifest.update(reader_executable=str(reader_executable), reader_sha256=hashlib.sha256(reader_executable.read_bytes()).hexdigest()) | ||
| 85 | (output / 'clients.json').write_text(json.dumps(manifest, indent=2)) | ||
| 77 | processes = {} | 86 | processes = {} |
| 78 | streams = [] | 87 | streams = [] |
| 79 | try: | 88 | try: |
| 80 | for mode, count in [('write', writers), ('read', readers)]: | 89 | for mode, count in [('write', writers), ('read', readers)]: |
| 90 | client = reader_executable if mode == 'read' and reader_executable is not None else executable | ||
| 81 | for i in range(count): | 91 | for i in range(count): |
| 82 | actor = mode[0] + str(i) | 92 | actor = mode[0] + str(i) |
| 83 | out = (output / f'{actor}.jsonl').open('w') | 93 | out = (output / f'{actor}.jsonl').open('w') |
| 84 | err = (output / f'{actor}.stderr').open('w') | 94 | err = (output / f'{actor}.stderr').open('w') |
| 85 | streams.extend([out, err]) | 95 | streams.extend([out, err]) |
| 86 | processes[actor] = subprocess.Popen([CLIENT, 'edit' if edit and mode == 'write' else mode, source, actor, str(operations), start, stop, | 96 | processes[actor] = subprocess.Popen([client, 'edit' if edit and mode == 'write' else mode, source, actor, str(operations), start, stop, |
| 87 | str(seed + i + (10000 if mode == 'read' else 0))], stdout=out, stderr=err) | 97 | str(seed + i + (10000 if mode == 'read' else 0))], stdout=out, stderr=err, env=environment) |
| 88 | deadline = time.monotonic() + timeout | 98 | deadline = time.monotonic() + timeout |
| 89 | while not all((output / f'{actor}.jsonl').stat().st_size for actor in processes): | 99 | while not all((output / f'{actor}.jsonl').stat().st_size for actor in processes): |
| 90 | if any(p.poll() is not None for p in processes.values()) or time.monotonic() > deadline: | 100 | if any(p.poll() is not None for p in processes.values()) or time.monotonic() > deadline: |
tools/crash_recovery.py+15
| ... | @@ -1,12 +1,22 @@ | ... | @@ -1,12 +1,22 @@ |
| 1 | """Independent append-intent accounting across an abrupt storage interruption.""" | 1 | """Independent append-intent accounting across an abrupt storage interruption.""" |
| 2 | from collections import Counter | 2 | from collections import Counter |
| 3 | import re | 3 | import re |
| 4 | from document_model import ordered_pages, walk | ||
| 5 | |||
| 6 | |||
| 7 | def active_text(model): | ||
| 8 | (_, _, revision, page), = ordered_pages(model) | ||
| 9 | text, = [node['kind']['text'] for _, node in walk(revision, page) | ||
| 10 | if node['kind']['type'] == 'RichText' and node['kind']['text'].startswith('Concurrent edits:')] | ||
| 11 | return text | ||
| 4 | 12 | ||
| 5 | 13 | ||
| 6 | def verify_text(baseline, current, logs): | 14 | def verify_text(baseline, current, logs): |
| 7 | events = [event for rows in logs.values() for event in rows] | 15 | events = [event for rows in logs.values() for event in rows] |
| 8 | intents = {event['token'] for event in events if event['event'] == 'intent'} | 16 | intents = {event['token'] for event in events if event['event'] == 'intent'} |
| 9 | possible = set() | 17 | possible = set() |
| 18 | versions = {baseline} | ||
| 19 | predecessors = set() | ||
| 10 | for rows in logs.values(): | 20 | for rows in logs.values(): |
| 11 | outcomes = {event['attempt']: event for event in rows if event['event'] in ('commit', 'retry', 'commit_error')} | 21 | outcomes = {event['attempt']: event for event in rows if event['event'] in ('commit', 'retry', 'commit_error')} |
| 12 | for event in rows: | 22 | for event in rows: |
| ... | @@ -14,12 +24,17 @@ def verify_text(baseline, current, logs): | ... | @@ -14,12 +24,17 @@ def verify_text(baseline, current, logs): |
| 14 | outcome = outcomes.get(event['attempt']) | 24 | outcome = outcomes.get(event['attempt']) |
| 15 | if outcome is None or outcome['event'] == 'commit' or (outcome['event'] == 'commit_error' and outcome['state'] != 'NotCommitted'): | 25 | if outcome is None or outcome['event'] == 'commit' or (outcome['event'] == 'commit_error' and outcome['state'] != 'NotCommitted'): |
| 16 | possible.add(event['token']) | 26 | possible.add(event['token']) |
| 27 | assert event['replacement'] == event['token'] and event['range'] == [len(event['before'].encode('utf-16-le')) // 2] * 2 | ||
| 28 | predecessors.add(event['before']) | ||
| 29 | versions.add(event['before'] + event['token']) | ||
| 30 | assert predecessors <= versions, 'A publication does not follow recorded history' | ||
| 17 | acknowledged = [event for event in events if event['event'] == 'commit' or | 31 | acknowledged = [event for event in events if event['event'] == 'commit' or |
| 18 | (event['event'] == 'commit_error' and event['state'] == 'Committed')] | 32 | (event['event'] == 'commit_error' and event['state'] == 'Committed')] |
| 19 | assert len({event['token'] for event in acknowledged}) == len(acknowledged), 'An edit was acknowledged twice' | 33 | assert len({event['token'] for event in acknowledged}) == len(acknowledged), 'An edit was acknowledged twice' |
| 20 | 34 | ||
| 21 | def tokens(text): | 35 | def tokens(text): |
| 22 | assert text.startswith(baseline), 'Previously retained content changed' | 36 | assert text.startswith(baseline), 'Previously retained content changed' |
| 37 | assert text in versions, 'Text is not a recorded publication result' | ||
| 23 | suffix = text[len(baseline):] | 38 | suffix = text[len(baseline):] |
| 24 | found = re.findall(r' \[w\d+:\d+\]', suffix) | 39 | found = re.findall(r' \[w\d+:\d+\]', suffix) |
| 25 | assert ''.join(found) == suffix, 'Unexpected or partially persisted text' | 40 | assert ''.join(found) == suffix, 'Unexpected or partially persisted text' |
tools/diagnostic/editor.css created+24
| ... | @@ -0,0 +1,24 @@ | ||
| 1 | .diagnostic{position:sticky;top:0;z-index:10;display:flex;align-items:center;gap:12px;flex-wrap:wrap;margin:-30px -40px 24px;padding:12px 20px;background:#f2f6fa;border-bottom:1px solid #aab9c8;font:13px/1.5 system-ui} | ||
| 2 | .diagnostic button,.edit-actions button{font:inherit;padding:6px 12px;cursor:pointer} | ||
| 3 | .diagnostic [role=status]{flex-basis:100%} | ||
| 4 | .diagnostic [role=status]:empty{display:none} | ||
| 5 | .editing [data-text-object]{cursor:pointer;outline:1px dashed #5182b1;outline-offset:2px} | ||
| 6 | .editing [data-text-object]:hover,.editing [data-text-object]:focus{outline:2px solid #175bb2} | ||
| 7 | .editing [data-text-object]:empty::before{content:'Empty text';color:#666;font:12px system-ui} | ||
| 8 | dialog{width:min(720px,90vw);max-height:90vh;overflow:auto;border:1px solid #8a9aab;border-radius:6px;padding:24px;color:#000;background:#fff;font:15px/1.5 system-ui} | ||
| 9 | dialog::backdrop{background:#0005} | ||
| 10 | dialog h2{margin:0 0 12px}dialog label{display:block;margin-top:16px;font-weight:600} | ||
| 11 | dialog textarea{display:block;width:100%;font:16px/1.6 system-ui;resize:vertical;min-height:140px} | ||
| 12 | dialog [hidden]{display:none} | ||
| 13 | dialog fieldset{margin:16px 0;padding:12px;border:1px solid #c1ccd6} | ||
| 14 | dialog fieldset label{margin-top:8px} | ||
| 15 | dialog input,dialog select,.diagnostic select{font:inherit;padding:4px;max-width:100%} | ||
| 16 | dialog select{display:block} | ||
| 17 | .format-grid{display:grid;grid-template-columns:repeat(3,minmax(0,1fr));gap:8px} | ||
| 18 | #coordinates:not([hidden]){display:flex;gap:20px} | ||
| 19 | #coordinates input{width:120px} | ||
| 20 | #format-range{margin:0} | ||
| 21 | dialog input[type=color]{width:64px;height:32px;vertical-align:middle} | ||
| 22 | .edit-actions{display:flex;justify-content:flex-end;gap:12px;margin-top:20px} | ||
| 23 | button:disabled{cursor:default} | ||
| 24 | @media(max-width:750px){.diagnostic{margin:-20px -20px 20px}} | ||
tools/diagnostic/editor.js created+246
| ... | @@ -0,0 +1,246 @@ | ||
| 1 | const route = location.pathname.match(/^\/g\/(\d+)\/report\/(page-\d+\.html)$/); | ||
| 2 | const bar = document.createElement('aside'); | ||
| 3 | bar.className = 'diagnostic'; | ||
| 4 | bar.innerHTML = `<strong>Notebook copy</strong> | ||
| 5 | <button id="edit-mode" type="button" aria-pressed="false">Edit text</button> | ||
| 6 | <label>Text action <select id="text-action"><option value="text">Replace</option><option value="format">Format</option></select></label> | ||
| 7 | <button id="add-paragraph" type="button">Add paragraph</button> | ||
| 8 | <button id="add-outline" type="button">Add outline</button> | ||
| 9 | <a href="/latest">Latest notebook</a> | ||
| 10 | <span id="editor-status" role="status"></span>`; | ||
| 11 | document.querySelector('main').prepend(bar); | ||
| 12 | const status = document.querySelector('#editor-status'); | ||
| 13 | const mode = document.querySelector('#edit-mode'); | ||
| 14 | for (const control of bar.querySelectorAll('button, select')) control.disabled = !route; | ||
| 15 | const dialog = document.createElement('dialog'); | ||
| 16 | dialog.setAttribute('aria-labelledby', 'edit-heading'); | ||
| 17 | dialog.innerHTML = `<form><h2 id="edit-heading">Edit text</h2> | ||
| 18 | <p id="edit-hint"></p> | ||
| 19 | <fieldset id="placement"><legend>Paragraph placement</legend> | ||
| 20 | <label>Parent container <select id="parent"></select></label> | ||
| 21 | <label>Insert before <select id="before"></select></label> | ||
| 22 | </fieldset> | ||
| 23 | <fieldset id="coordinates"><legend>Outline position</legend> | ||
| 24 | <label>X (points) <input id="outline-x" type="number" step="any" value="144"></label> | ||
| 25 | <label>Y (points) <input id="outline-y" type="number" step="any" value="144"></label> | ||
| 26 | </fieldset> | ||
| 27 | <label for="replacement">Text</label><textarea id="replacement" rows="6"></textarea> | ||
| 28 | <label id="author-label">Author <input id="author" value="Diagnostic"></label> | ||
| 29 | <fieldset id="formatting"><legend>Character formatting</legend> | ||
| 30 | <p id="format-range" role="status"></p> | ||
| 31 | <div class="format-grid"></div> | ||
| 32 | <label>Font <input id="font" placeholder="Keep current font"></label> | ||
| 33 | <label>Size (points) <input id="font-size" type="number" min="6" max="130" step="0.5" placeholder="Keep current size"></label> | ||
| 34 | <label for="color-mode">Text color</label><select id="color-mode"><option value="keep">Keep</option><option value="clear">Automatic</option><option value="set">Set color</option></select><input id="color" type="color" value="#123456" aria-label="Text color value"> | ||
| 35 | <label for="highlight-mode">Highlight</label><select id="highlight-mode"><option value="keep">Keep</option><option value="clear">Clear</option><option value="set">Set color</option></select><input id="highlight" type="color" value="#ffff00" aria-label="Highlight color value"> | ||
| 36 | </fieldset> | ||
| 37 | <p id="save-status" role="status"></p> | ||
| 38 | <a id="inspect-latest" target="_blank" rel="noopener" hidden>Open latest page</a> | ||
| 39 | <div class="edit-actions"><button type="button" id="cancel-edit">Cancel</button><button id="save-edit">Save</button></div> | ||
| 40 | </form>`; | ||
| 41 | document.body.append(dialog); | ||
| 42 | for (const name of ['Bold', 'Italic', 'Underline', 'Strike', 'Superscript', 'Subscript']) { | ||
| 43 | const label = document.createElement('label'); | ||
| 44 | label.textContent = name === 'Strike' ? 'Strikethrough' : name; | ||
| 45 | const select = document.createElement('select'); | ||
| 46 | select.dataset.attribute = name; | ||
| 47 | for (const [value, text] of [['', 'Keep'], ['true', 'On'], ['false', 'Off']]) select.add(new Option(text, value)); | ||
| 48 | label.append(select); | ||
| 49 | dialog.querySelector('.format-grid').append(label); | ||
| 50 | } | ||
| 51 | const draft = document.querySelector('#replacement'); | ||
| 52 | const save = document.querySelector('#save-edit'); | ||
| 53 | const cancel = document.querySelector('#cancel-edit'); | ||
| 54 | const message = document.querySelector('#save-status'); | ||
| 55 | const inspect = document.querySelector('#inspect-latest'); | ||
| 56 | const parent = document.querySelector('#parent'); | ||
| 57 | const before = document.querySelector('#before'); | ||
| 58 | let selected = null; | ||
| 59 | let opening = false; | ||
| 60 | let submitting = false; | ||
| 61 | let initial = ''; | ||
| 62 | |||
| 63 | mode.addEventListener('click', () => { | ||
| 64 | const editing = document.body.classList.toggle('editing'); | ||
| 65 | mode.setAttribute('aria-pressed', String(editing)); | ||
| 66 | mode.textContent = editing ? 'Stop editing' : 'Edit text'; | ||
| 67 | for (const element of document.querySelectorAll('[data-text-object]')) { | ||
| 68 | if (editing) { | ||
| 69 | element.tabIndex = 0; | ||
| 70 | element.setAttribute('role', 'button'); | ||
| 71 | element.setAttribute('aria-label', 'Edit text: ' + element.textContent); | ||
| 72 | } else { | ||
| 73 | element.removeAttribute('tabindex'); | ||
| 74 | element.removeAttribute('role'); | ||
| 75 | element.removeAttribute('aria-label'); | ||
| 76 | } | ||
| 77 | } | ||
| 78 | status.textContent = editing ? 'Select a text run.' : ''; | ||
| 79 | }); | ||
| 80 | |||
| 81 | function showEdit(selection, text) { | ||
| 82 | dialog.querySelector('form').reset(); | ||
| 83 | selected = selection; | ||
| 84 | const action = selection.action; | ||
| 85 | const titles = {text: 'Replace text', format: 'Format text', paragraph: 'Add paragraph', outline: 'Add outline'}; | ||
| 86 | document.querySelector('#edit-heading').textContent = titles[action]; | ||
| 87 | document.querySelector('#edit-hint').textContent = { | ||
| 88 | text: 'Replacement text keeps this run’s formatting.', | ||
| 89 | format: 'Select part of the text or leave the whole run selected. Unchanged attributes keep their current values.', | ||
| 90 | paragraph: 'Append to a container or insert before one of its children. Line breaks stay inside the new paragraph.', | ||
| 91 | outline: 'Add a text container to this page. Line breaks stay inside its first paragraph.' | ||
| 92 | }[action]; | ||
| 93 | for (const [id, visible] of [['placement', action === 'paragraph'], ['coordinates', action === 'outline'], ['formatting', action === 'format']]) { | ||
| 94 | const field = document.getElementById(id); | ||
| 95 | field.hidden = !visible; | ||
| 96 | field.disabled = !visible; | ||
| 97 | } | ||
| 98 | document.querySelector('#author-label').hidden = !['paragraph', 'outline'].includes(action); | ||
| 99 | draft.value = text; | ||
| 100 | draft.readOnly = action === 'format'; | ||
| 101 | save.disabled = false; | ||
| 102 | cancel.disabled = false; | ||
| 103 | inspect.hidden = true; | ||
| 104 | message.textContent = ''; | ||
| 105 | status.textContent = ''; | ||
| 106 | dialog.showModal(); | ||
| 107 | draft.focus(); | ||
| 108 | draft.setSelectionRange(0, text.length); | ||
| 109 | document.querySelector('#format-range').textContent = 'Selected: ' + text.length + ' UTF-16 units'; | ||
| 110 | initial = JSON.stringify(requestBody()); | ||
| 111 | } | ||
| 112 | |||
| 113 | async function openEdit(element) { | ||
| 114 | if (opening) return; | ||
| 115 | opening = true; | ||
| 116 | status.textContent = 'Checking…'; | ||
| 117 | const selection = {generation: Number(route[1]), page: route[2], | ||
| 118 | object: element.dataset.textObject, run: Number(element.dataset.run)}; | ||
| 119 | try { | ||
| 120 | const result = await (await fetch('/api/run?' + new URLSearchParams(selection))).json(); | ||
| 121 | if (!result.ok) { | ||
| 122 | status.textContent = 'Edit not supported. ' + result.error; | ||
| 123 | return; | ||
| 124 | } | ||
| 125 | showEdit({...selection, action: document.querySelector('#text-action').value}, result.text); | ||
| 126 | } catch { | ||
| 127 | status.textContent = 'Unable to check this text. Reconnect to the diagnostic server.'; | ||
| 128 | } finally { | ||
| 129 | opening = false; | ||
| 130 | } | ||
| 131 | } | ||
| 132 | |||
| 133 | for (const action of ['paragraph', 'outline']) document.querySelector('#add-' + action).addEventListener('click', async () => { | ||
| 134 | if (opening) return; | ||
| 135 | opening = true; | ||
| 136 | status.textContent = 'Loading…'; | ||
| 137 | const selection = {generation: Number(route[1]), page: route[2], action}; | ||
| 138 | try { | ||
| 139 | const result = await (await fetch('/api/page?' + new URLSearchParams(selection))).json(); | ||
| 140 | if (!result.ok) throw new Error(result.error); | ||
| 141 | if (action === 'paragraph' && !result.targets.length) throw new Error('Add an outline before adding a paragraph.'); | ||
| 142 | parent.replaceChildren(...result.targets.map(target => new Option(target.label, target.object))); | ||
| 143 | parent.onchange = () => { | ||
| 144 | const target = result.targets.find(target => target.object === parent.value); | ||
| 145 | before.replaceChildren(new Option('Append at end', ''), ...(target?.children || []).map(child => new Option(child.label, child.object))); | ||
| 146 | }; | ||
| 147 | parent.onchange(); | ||
| 148 | showEdit({...selection, object: result.object}, ''); | ||
| 149 | } catch (error) { | ||
| 150 | status.textContent = 'Unable to add content. ' + error.message; | ||
| 151 | } finally { | ||
| 152 | opening = false; | ||
| 153 | } | ||
| 154 | }); | ||
| 155 | |||
| 156 | function requestBody() { | ||
| 157 | const body = {...selected}; | ||
| 158 | if (body.action === 'text') body.replacement = draft.value; | ||
| 159 | if (body.action === 'format') { | ||
| 160 | body.start = draft.selectionStart; | ||
| 161 | body.end = draft.selectionEnd; | ||
| 162 | body.attributes = [...dialog.querySelectorAll('[data-attribute]')].filter(select => select.value !== '') | ||
| 163 | .map(select => ({[select.dataset.attribute]: select.value === 'true'})); | ||
| 164 | const font = document.querySelector('#font').value; | ||
| 165 | const size = document.querySelector('#font-size').value; | ||
| 166 | if (font) body.attributes.push({Font: font}); | ||
| 167 | if (size) body.attributes.push({FontSize: Number(size)}); | ||
| 168 | for (const [field, attribute] of [['color', 'Color'], ['highlight', 'Highlight']]) { | ||
| 169 | const mode = document.getElementById(field + '-mode').value; | ||
| 170 | if (mode !== 'keep') body.attributes.push({[attribute]: mode === 'clear' ? null : document.getElementById(field).value.slice(1).match(/../g).map(hex => parseInt(hex, 16))}); | ||
| 171 | } | ||
| 172 | } | ||
| 173 | if (['paragraph', 'outline'].includes(body.action)) { | ||
| 174 | body.text = draft.value.replace(/\n/g, '\r'); | ||
| 175 | body.author = document.querySelector('#author').value; | ||
| 176 | if (body.action === 'paragraph') { | ||
| 177 | body.object = parent.value; | ||
| 178 | body.before = before.value || null; | ||
| 179 | } else { | ||
| 180 | body.x = document.querySelector('#outline-x').valueAsNumber; | ||
| 181 | body.y = document.querySelector('#outline-y').valueAsNumber; | ||
| 182 | } | ||
| 183 | } | ||
| 184 | return body; | ||
| 185 | } | ||
| 186 | |||
| 187 | draft.addEventListener('select', () => { | ||
| 188 | document.querySelector('#format-range').textContent = 'Selected: ' + (draft.selectionEnd - draft.selectionStart) + ' UTF-16 units'; | ||
| 189 | }); | ||
| 190 | document.addEventListener('click', event => { | ||
| 191 | const element = event.target.closest('[data-text-object]'); | ||
| 192 | if (element && document.body.classList.contains('editing')) { | ||
| 193 | event.preventDefault(); | ||
| 194 | openEdit(element); | ||
| 195 | } | ||
| 196 | }); | ||
| 197 | document.addEventListener('keydown', event => { | ||
| 198 | if (event.target.matches('[data-text-object]') && document.body.classList.contains('editing') && ['Enter', ' '].includes(event.key)) { | ||
| 199 | event.preventDefault(); | ||
| 200 | openEdit(event.target); | ||
| 201 | } | ||
| 202 | }); | ||
| 203 | cancel.addEventListener('click', () => dialog.close()); | ||
| 204 | dialog.addEventListener('cancel', event => { if (submitting) event.preventDefault(); }); | ||
| 205 | dialog.addEventListener('close', () => { selected = null; }); | ||
| 206 | window.addEventListener('beforeunload', event => { | ||
| 207 | if (selected && JSON.stringify(requestBody()) !== initial) event.preventDefault(); | ||
| 208 | }); | ||
| 209 | dialog.querySelector('form').addEventListener('submit', async event => { | ||
| 210 | event.preventDefault(); | ||
| 211 | if (save.disabled || !selected) return; | ||
| 212 | const body = requestBody(); | ||
| 213 | save.disabled = true; | ||
| 214 | cancel.disabled = true; | ||
| 215 | submitting = true; | ||
| 216 | for (const control of dialog.querySelectorAll('input, select, textarea')) control.disabled = true; | ||
| 217 | message.textContent = 'Saving…'; | ||
| 218 | let result; | ||
| 219 | try { | ||
| 220 | result = await (await fetch('/api/save', {method: 'POST', headers: { | ||
| 221 | 'Content-Type': 'application/json', 'X-OneNote-Diagnostic': '1' | ||
| 222 | }, body: JSON.stringify(body)})).json(); | ||
| 223 | } catch { | ||
| 224 | result = {ok: false, state: 'Unknown'}; | ||
| 225 | } | ||
| 226 | submitting = false; | ||
| 227 | cancel.disabled = false; | ||
| 228 | for (const control of dialog.querySelectorAll('input, select, textarea')) control.disabled = false; | ||
| 229 | if (result.ok) { | ||
| 230 | selected = null; | ||
| 231 | location.assign(result.location); | ||
| 232 | return; | ||
| 233 | } | ||
| 234 | inspect.href = '/latest?' + new URLSearchParams({generation: selected.generation, page: selected.page}); | ||
| 235 | inspect.hidden = false; | ||
| 236 | if (result.state === 'Unknown') { | ||
| 237 | message.textContent = 'Save outcome unknown. Keep this draft and inspect the latest page before trying again.'; | ||
| 238 | } else if (result.state === 'Committed') { | ||
| 239 | message.textContent = 'Saved, but cleanup or refresh did not finish. Inspect the latest page; do not save this draft again.'; | ||
| 240 | } else if (result.kind === 'ResourceBusy') { | ||
| 241 | message.textContent = 'This section changed. Open the latest page before editing again. Your draft is still here.'; | ||
| 242 | } else { | ||
| 243 | message.textContent = 'Unable to save. ' + (result.error || result.report_error || 'Check this edit and try again.'); | ||
| 244 | save.disabled = false; | ||
| 245 | } | ||
| 246 | }); | ||
tools/native/network.ps1+5-1
| ... | @@ -3,6 +3,10 @@ Set-StrictMode -Version Latest | ... | @@ -3,6 +3,10 @@ Set-StrictMode -Version Latest |
| 3 | $ErrorActionPreference = 'Stop' | 3 | $ErrorActionPreference = 'Stop' |
| 4 | $adapter = @(Get-WmiObject Win32_NetworkAdapterConfiguration | Where-Object { $_.MACAddress -eq $LabMac }) | 4 | $adapter = @(Get-WmiObject Win32_NetworkAdapterConfiguration | Where-Object { $_.MACAddress -eq $LabMac }) |
| 5 | if ($adapter.Count -ne 1) { throw 'The clone lab adapter is ambiguous.' } | 5 | if ($adapter.Count -ne 1) { throw 'The clone lab adapter is ambiguous.' } |
| 6 | $enabled = $adapter[0].EnableDHCP() | ||
| 7 | if ($enabled.ReturnValue -notin @(0, 1)) { | ||
| 8 | throw ('Unable to enable DHCP on the clone lab adapter; result ' + $enabled.ReturnValue) | ||
| 9 | } | ||
| 6 | $release = $adapter[0].ReleaseDHCPLease() | 10 | $release = $adapter[0].ReleaseDHCPLease() |
| 7 | $result = $adapter[0].RenewDHCPLease() | 11 | $result = $adapter[0].RenewDHCPLease() |
| 8 | $deadline = [DateTime]::UtcNow.AddSeconds(60) | 12 | $deadline = [DateTime]::UtcNow.AddSeconds(60) |
| ... | @@ -12,5 +16,5 @@ do { | ... | @@ -12,5 +16,5 @@ do { |
| 12 | if ($addresses.Count -eq 1) { break } | 16 | if ($addresses.Count -eq 1) { break } |
| 13 | Start-Sleep -Milliseconds 250 | 17 | Start-Sleep -Milliseconds 250 |
| 14 | } while ([DateTime]::UtcNow -lt $deadline) | 18 | } while ([DateTime]::UtcNow -lt $deadline) |
| 15 | if ($addresses.Count -ne 1) { throw ('The clone did not receive a lab IPv4 address; renewal result ' + $result.ReturnValue) } | 19 | if ($addresses.Count -ne 1) { throw ('The clone did not receive a lab IPv4 address; renewal result ' + $result.ReturnValue + '; adapter ' + ($adapter[0] | Select-Object MACAddress,DHCPEnabled,IPEnabled,IPAddress | ConvertTo-Json -Compress)) } |
| 16 | @{mac=$LabMac;address=$addresses[0];releaseResult=$release.ReturnValue;renewalResult=$result.ReturnValue} | ConvertTo-Json -Compress | 20 | @{mac=$LabMac;address=$addresses[0];releaseResult=$release.ReturnValue;renewalResult=$result.ReturnValue} | ConvertTo-Json -Compress |
tools/native/probe.ps1+34-8
| ... | @@ -3,33 +3,57 @@ Set-StrictMode -Version Latest | ... | @@ -3,33 +3,57 @@ Set-StrictMode -Version Latest |
| 3 | $ErrorActionPreference = 'Stop' | 3 | $ErrorActionPreference = 'Stop' |
| 4 | $results = New-Object Collections.Generic.List[object] | 4 | $results = New-Object Collections.Generic.List[object] |
| 5 | $index = 0 | 5 | $index = 0 |
| 6 | function Record-Phase([string]$Phase) { | ||
| 7 | @{ name=$file.BaseName; phase=$Phase; utc=[DateTime]::UtcNow.ToString('o') } | | ||
| 8 | ConvertTo-Json -Compress | Add-Content "$Root\progress.jsonl" -Encoding UTF8 | ||
| 9 | } | ||
| 6 | foreach ($file in @(Get-ChildItem "$Root\inputs" -Filter '*.one' | Sort-Object Name)) { | 10 | foreach ($file in @(Get-ChildItem "$Root\inputs" -Filter '*.one' | Sort-Object Name)) { |
| 7 | $case = "C:\one-tests\runs\probe-$index" | ||
| 8 | $output = "$Root\results\$($file.BaseName)" | 11 | $output = "$Root\results\$($file.BaseName)" |
| 9 | New-Item -ItemType Directory "$case\notebook", $output -Force | Out-Null | 12 | New-Item -ItemType Directory $output -Force | Out-Null |
| 10 | Copy-Item $file.FullName "$case\notebook\synthetic.one" | 13 | Record-Phase 'cold-reset' |
| 11 | Get-Process ONENOTE -ErrorAction SilentlyContinue | ForEach-Object { | 14 | $cold = $false |
| 12 | Stop-Process -InputObject $_ -Force | 15 | for ($attempt = 0; $attempt -lt 5; $attempt++) { |
| 13 | if (-not $_.WaitForExit(10000)) { throw 'OneNote did not exit before the cache reset.' } | 16 | $case = "C:\one-tests\runs\probe-$index-$attempt" |
| 17 | New-Item -ItemType Directory "$case\notebook" -Force | Out-Null | ||
| 18 | Copy-Item $file.FullName "$case\notebook\synthetic.one" | ||
| 19 | Get-Process ONENOTE -ErrorAction SilentlyContinue | ForEach-Object { | ||
| 20 | Stop-Process -InputObject $_ -Force -ErrorAction SilentlyContinue | ||
| 21 | if (-not $_.WaitForExit(10000)) { throw 'OneNote did not exit before the cache reset.' } | ||
| 22 | } | ||
| 23 | Start-Sleep -Milliseconds 250 | ||
| 24 | try { & "$PSScriptRoot\cold-current.ps1" -Root $case -CloneHost $CloneHost } | ||
| 25 | catch { | ||
| 26 | if ($_.Exception.Message -ne 'Close OneNote before resetting its test cache.') { throw } | ||
| 27 | continue | ||
| 28 | } | ||
| 29 | if (-not (Get-Process ONENOTE -ErrorAction SilentlyContinue)) { $cold = $true; break } | ||
| 14 | } | 30 | } |
| 15 | & "$PSScriptRoot\cold-current.ps1" -Root $case -CloneHost $CloneHost | 31 | if (-not $cold) { throw 'OneNote kept reopening during the cache reset.' } |
| 32 | Record-Phase 'create-application' | ||
| 16 | $app = New-Object -ComObject OneNote.Application | 33 | $app = New-Object -ComObject OneNote.Application |
| 17 | $notebook = ''; $section = ''; $hierarchy = ''; $failure = $null; $pages = @() | 34 | $notebook = ''; $section = ''; $hierarchy = ''; $failure = $null; $pages = @() |
| 18 | $started = [DateTime]::UtcNow | 35 | $started = [DateTime]::UtcNow |
| 19 | try { | 36 | try { |
| 37 | Record-Phase 'open-notebook' | ||
| 20 | $app.OpenHierarchy("$case\notebook", '', [ref]$notebook, 0) | 38 | $app.OpenHierarchy("$case\notebook", '', [ref]$notebook, 0) |
| 39 | Record-Phase 'open-section' | ||
| 21 | $app.OpenHierarchy("$case\notebook\synthetic.one", '', [ref]$section, 0) | 40 | $app.OpenHierarchy("$case\notebook\synthetic.one", '', [ref]$section, 0) |
| 22 | $deadline = [DateTime]::UtcNow.AddSeconds(30) | 41 | $deadline = [DateTime]::UtcNow.AddSeconds(30) |
| 23 | do { | 42 | do { |
| 43 | Record-Phase 'get-hierarchy' | ||
| 24 | $app.GetHierarchy($section, 4, [ref]$hierarchy, 1) | 44 | $app.GetHierarchy($section, 4, [ref]$hierarchy, 1) |
| 25 | [xml]$tree = $hierarchy | 45 | [xml]$tree = $hierarchy |
| 26 | $pages = @($tree.SelectNodes('//*[local-name()="Page"]')) | 46 | $pages = @($tree.SelectNodes('//*[local-name()="Page"]')) |
| 27 | if ($pages.Count) { break } | 47 | if ($pages.Count -and $tree.DocumentElement.GetAttribute('areAllPagesAvailable') -ne 'false') { break } |
| 28 | Start-Sleep -Milliseconds 250 | 48 | Start-Sleep -Milliseconds 250 |
| 29 | } while ([DateTime]::UtcNow -lt $deadline) | 49 | } while ([DateTime]::UtcNow -lt $deadline) |
| 50 | if (-not $pages.Count -or $tree.DocumentElement.GetAttribute('areAllPagesAvailable') -eq 'false') { | ||
| 51 | throw 'The section did not finish loading its pages.' | ||
| 52 | } | ||
| 30 | $n = 0 | 53 | $n = 0 |
| 31 | foreach ($page in $pages) { | 54 | foreach ($page in $pages) { |
| 32 | $content = '' | 55 | $content = '' |
| 56 | Record-Phase "get-page-$n" | ||
| 33 | $app.GetPageContent($page.GetAttribute('ID'), [ref]$content, 1, 1) | 57 | $app.GetPageContent($page.GetAttribute('ID'), [ref]$content, 1, 1) |
| 34 | [IO.File]::WriteAllText("$output\page-$n.xml", $content, [Text.Encoding]::UTF8) | 58 | [IO.File]::WriteAllText("$output\page-$n.xml", $content, [Text.Encoding]::UTF8) |
| 35 | $n++ | 59 | $n++ |
| ... | @@ -41,11 +65,13 @@ foreach ($file in @(Get-ChildItem "$Root\inputs" -Filter '*.one' | Sort-Object N | ... | @@ -41,11 +65,13 @@ foreach ($file in @(Get-ChildItem "$Root\inputs" -Filter '*.one' | Sort-Object N |
| 41 | seconds=([DateTime]::UtcNow - $started).TotalSeconds; | 65 | seconds=([DateTime]::UtcNow - $started).TotalSeconds; |
| 42 | source_sha256=(Get-FileHash $file.FullName).Hash.ToLowerInvariant() }) | 66 | source_sha256=(Get-FileHash $file.FullName).Hash.ToLowerInvariant() }) |
| 43 | $results | ConvertTo-Json -Depth 5 | Set-Content "$Root\results.json" -Encoding UTF8 | 67 | $results | ConvertTo-Json -Depth 5 | Set-Content "$Root\results.json" -Encoding UTF8 |
| 68 | Record-Phase 'close-notebook' | ||
| 44 | try { if ($notebook) { $app.CloseNotebook($notebook, $false) } } catch {} | 69 | try { if ($notebook) { $app.CloseNotebook($notebook, $false) } } catch {} |
| 45 | [void][Runtime.InteropServices.Marshal]::FinalReleaseComObject($app) | 70 | [void][Runtime.InteropServices.Marshal]::FinalReleaseComObject($app) |
| 46 | $app = $null | 71 | $app = $null |
| 47 | [GC]::Collect() | 72 | [GC]::Collect() |
| 48 | [GC]::WaitForPendingFinalizers() | 73 | [GC]::WaitForPendingFinalizers() |
| 74 | Record-Phase 'complete' | ||
| 49 | } | 75 | } |
| 50 | $index++ | 76 | $index++ |
| 51 | } | 77 | } |
tools/native/stress.ps1+8
| ... | @@ -1,6 +1,7 @@ | ... | @@ -1,6 +1,7 @@ |
| 1 | function Invoke-Stress($app, $section, $command, $output, $shared) { | 1 | function Invoke-Stress($app, $section, $command, $output, $shared) { |
| 2 | $hierarchy = '' | 2 | $hierarchy = '' |
| 3 | $app.GetHierarchy($section, 4, [ref]$hierarchy, 1) | 3 | $app.GetHierarchy($section, 4, [ref]$hierarchy, 1) |
| 4 | [IO.File]::WriteAllText("$output\hierarchy.xml", $hierarchy, [Text.Encoding]::UTF8) | ||
| 4 | [xml]$tree = $hierarchy | 5 | [xml]$tree = $hierarchy |
| 5 | $pages = @($tree.SelectNodes('//*[local-name()="Page"]')) | 6 | $pages = @($tree.SelectNodes('//*[local-name()="Page"]')) |
| 6 | if ($pages.Count -ne 1) { throw 'Expected one stress-test page.' } | 7 | if ($pages.Count -ne 1) { throw 'Expected one stress-test page.' } |
| ... | @@ -23,6 +24,13 @@ function Invoke-Stress($app, $section, $command, $output, $shared) { | ... | @@ -23,6 +24,13 @@ function Invoke-Stress($app, $section, $command, $output, $shared) { |
| 23 | Start-Sleep -Milliseconds 20 | 24 | Start-Sleep -Milliseconds 20 |
| 24 | } | 25 | } |
| 25 | for ($i = 0; $i -lt $command.operations; $i++) { | 26 | for ($i = 0; $i -lt $command.operations; $i++) { |
| 27 | if (($command.PSObject.Properties.Name -contains 'maintenance') -and $command.maintenance -and $i -eq [Math]::Floor($command.operations / 2)) { | ||
| 28 | [IO.File]::WriteAllText("$shared\maintenance-paused-n$($command.actor)", 'paused') | ||
| 29 | while (-not (Test-Path "$shared\maintenance-resume")) { | ||
| 30 | if ([DateTime]::UtcNow -gt $deadline) { throw 'Maintenance pause timed out.' } | ||
| 31 | Start-Sleep -Milliseconds 100 | ||
| 32 | } | ||
| 33 | } | ||
| 26 | Start-Sleep -Milliseconds $random.Next(10, 100) | 34 | Start-Sleep -Milliseconds $random.Next(10, 100) |
| 27 | $started = [DateTime]::UtcNow.Ticks | 35 | $started = [DateTime]::UtcNow.Ticks |
| 28 | $content = '' | 36 | $content = '' |
tools/native_collaboration.py+121-27
| ... | @@ -47,20 +47,27 @@ def verify_final_state(model): | ... | @@ -47,20 +47,27 @@ def verify_final_state(model): |
| 47 | return {'main_space': sid, 'conflict_space': conflict_sid, 'competing_edits': sorted(edits)} | 47 | return {'main_space': sid, 'conflict_space': conflict_sid, 'competing_edits': sorted(edits)} |
| 48 | 48 | ||
| 49 | 49 | ||
| 50 | def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1, rust_writers=4, rust_readers=3, edit=False, seed=710, conflict_clients=0, abrupt=False): | 50 | def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1, rust_writers=4, rust_readers=3, edit=False, seed=710, conflict_clients=0, abrupt=False, embedded_smb=False, maintenance=False, fixture=None, disconnect=False, client_timeout=600, offline=False, offline_outage=False, offline_lost_reply=False, client_profile="debug", document_operations=False, record_writes=False, offline_client_reply=False): |
| 51 | if fixture is not None and not stress_clients: | ||
| 52 | raise ValueError('Use a fixture with stress mode.') | ||
| 51 | if linux_vm.instance_path(server).exists(): | 53 | if linux_vm.instance_path(server).exists(): |
| 52 | raise ValueError('Choose a new Linux VM name; existing machines are not owned by this run.') | 54 | raise ValueError('Choose a new Linux VM name; existing machines are not owned by this run.') |
| 53 | output = output.resolve() | 55 | output = output.resolve() |
| 54 | output.mkdir(parents=True, exist_ok=False) | 56 | output.mkdir(parents=True, exist_ok=False) |
| 55 | mount = output / 'mount' | 57 | mount = output / 'mount' |
| 56 | mount.mkdir() | 58 | mount.mkdir() |
| 57 | mounted = False | ||
| 58 | scripts = output / 'scripts' | 59 | scripts = output / 'scripts' |
| 59 | scripts.mkdir() | 60 | scripts.mkdir() |
| 60 | for name in ('cold.ps1', 'collaborate.ps1', 'network.ps1', 'stress.ps1', 'text.ps1'): | 61 | for name in ('cold.ps1', 'collaborate.ps1', 'network.ps1', 'stress.ps1', 'text.ps1'): |
| 61 | shutil.copyfile(ROOT / 'tools/native' / name, scripts / name) | 62 | shutil.copyfile(ROOT / 'tools/native' / name, scripts / name) |
| 62 | (output / 'run.json').write_text(json.dumps({'server': server, 'stress_clients': stress_clients, 'conflict_clients': conflict_clients, 'stress_operations': stress_operations, 'sync_every': sync_every, 'rust_writers': rust_writers, 'rust_readers': rust_readers, 'edit': edit, 'seed': seed, 'abrupt': abrupt, | 63 | harness = ('native_collaboration.py', 'native_maintenance.py', 'native_disconnect.py', 'native_stress.py', 'offline_history.py', 'offline_document_history.py', 'offline_outage.py', 'verify_offline.py', 'concurrent_rust.py', 'native_runner.py', |
| 63 | 'harness_sha256': {name: hashlib.sha256((ROOT / 'tools' / name).read_bytes()).hexdigest() for name in ('native_collaboration.py', 'native_stress.py', 'concurrent_rust.py', 'native_runner.py')}, 'scripts': {p.name: hashlib.sha256(p.read_bytes()).hexdigest() for p in scripts.iterdir()}}, indent=2)) | 64 | 'crash_recovery.py', 'smb-proxy.py', 'verify_smb_overlap.py', 'w7/crash.py', 'w7/vm.py', 'w7/linux_vm.py') |
| 65 | for name in harness: | ||
| 66 | saved = output / 'harness' / name | ||
| 67 | saved.parent.mkdir(parents=True, exist_ok=True) | ||
| 68 | shutil.copyfile(ROOT / 'tools' / name, saved) | ||
| 69 | (output / 'run.json').write_text(json.dumps({'server': server, 'stress_clients': stress_clients, 'conflict_clients': conflict_clients, 'stress_operations': stress_operations, 'sync_every': sync_every, 'rust_writers': rust_writers, 'rust_readers': rust_readers, 'edit': edit, 'seed': seed, 'abrupt': abrupt, 'embedded_smb': embedded_smb, 'maintenance': maintenance, 'fixture': str(fixture) if fixture is not None else None, | ||
| 70 | 'disconnect': disconnect, 'client_timeout': client_timeout, 'client_profile': client_profile, 'offline': offline, 'document_operations': document_operations, 'record_writes': record_writes, 'offline_outage': offline_outage, 'offline_lost_reply': offline_lost_reply, 'offline_client_reply': offline_client_reply, 'harness_sha256': {name: hashlib.sha256((output / 'harness' / name).read_bytes()).hexdigest() for name in harness}, 'scripts': {p.name: hashlib.sha256(p.read_bytes()).hexdigest() for p in scripts.iterdir()}}, indent=2)) | ||
| 64 | 71 | ||
| 65 | def ssh(text): | 72 | def ssh(text): |
| 66 | result = linux_vm.run_ssh(server, text, timeout=90) | 73 | result = linux_vm.run_ssh(server, text, timeout=90) |
| ... | @@ -76,18 +83,39 @@ def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1, | ... | @@ -76,18 +83,39 @@ def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1, |
| 76 | linux_vm.wait_instance(server, 600) | 83 | linux_vm.wait_instance(server, 600) |
| 77 | config = linux_vm.load_instance(server) | 84 | config = linux_vm.load_instance(server) |
| 78 | (output / 'linux.json').write_text(json.dumps(config, indent=2)) | 85 | (output / 'linux.json').write_text(json.dumps(config, indent=2)) |
| 79 | def reconnect_mount(): | 86 | if embedded_smb: |
| 80 | nonlocal mounted | 87 | subprocess.run(linux_vm.ssh_argv(server, 'cat > /tmp/smb-proxy.py'), |
| 88 | input=(output / 'harness/smb-proxy.py').read_bytes(), check=True) | ||
| 89 | ssh("sudo sed -i '/^\\[global\\]/a smb ports = 1445' /etc/samba/smb.conf && sudo systemctl restart smbd") | ||
| 90 | subprocess.run(linux_vm.ssh_argv(server, 'cat > /tmp/smb-control.json'), | ||
| 91 | input=json.dumps({'record_writes': record_writes}).encode(), check=True) | ||
| 92 | ssh("sudo sh -c 'nohup python3 /tmp/smb-proxy.py /tmp/smb-control.json --port 445 --bind 0.0.0.0 --server 127.0.0.1 --server-port 1445 > /tmp/smb-trace.jsonl 2>&1 < /dev/null &'") | ||
| 93 | ssh("sleep 1; sudo ss -ltn | grep ':445 '") | ||
| 94 | os.environ['ONESTORE_SMB_LAB'] = f'127.0.0.1:{config["samba_port"]}' | ||
| 95 | os.environ['ONESTORE_SMB_SHARE'] = 'agent' | ||
| 96 | def unmount(force=False): | ||
| 97 | for options in ([['-f']] if force else [[], ['-f']]): | ||
| 98 | if not os.path.ismount(mount): return | ||
| 99 | result = subprocess.run(['/sbin/umount', *options, str(mount)], capture_output=True, text=True, timeout=60) | ||
| 100 | with (output / 'unmounts.jsonl').open('a') as log: | ||
| 101 | log.write(json.dumps({'force': bool(options), 'exit': result.returncode, 'stderr': result.stderr}) + '\n') | ||
| 81 | if os.path.ismount(mount): | 102 | if os.path.ismount(mount): |
| 82 | subprocess.run(['/sbin/umount', str(mount)], check=True) | 103 | raise RuntimeError('The owned SMB mount remains attached; preserve its server until it is unmounted.') |
| 83 | mounted = False | 104 | def reconnect_mount(): |
| 105 | unmount() | ||
| 84 | subprocess.run(['/sbin/mount_smbfs', '-N', f'//guest@127.0.0.1:{config["samba_port"]}/agent', mount], check=True, stdin=subprocess.DEVNULL) | 106 | subprocess.run(['/sbin/mount_smbfs', '-N', f'//guest@127.0.0.1:{config["samba_port"]}/agent', mount], check=True, stdin=subprocess.DEVNULL) |
| 85 | mounted = True | ||
| 86 | ssh('mkdir /srv/agent/m6-collaboration') | 107 | ssh('mkdir /srv/agent/m6-collaboration') |
| 87 | source = ROOT / 'corpus/native-ink/cold-ui-ink/notebook' | 108 | source = ROOT / 'corpus/native-ink/cold-ui-ink/notebook' |
| 88 | if stress_clients or conflict_clients or abrupt: | 109 | if stress_clients or conflict_clients or abrupt: |
| 89 | source = output / 'input' | 110 | source = output / 'input' |
| 90 | subprocess.run([ROOT / 'target/debug/examples/create_notebook', source, 'Concurrent edits:', 'Concurrency test'], check=True) | 111 | if fixture is not None: |
| 112 | source.mkdir() | ||
| 113 | for name in ('synthetic.one', 'Open Notebook.onetoc2'): | ||
| 114 | shutil.copyfile(Path(fixture) / name, source / name) | ||
| 115 | elif maintenance: | ||
| 116 | subprocess.run([ROOT / 'target/debug/examples/maintenance_fixture', source, 'Concurrent edits:'], check=True) | ||
| 117 | else: | ||
| 118 | subprocess.run([ROOT / 'target/debug/examples/create_notebook', source, 'Concurrent edits:', 'Concurrency test'], check=True) | ||
| 91 | with tarfile.open(output / 'input.tar', 'w', dereference=True) as archive: | 119 | with tarfile.open(output / 'input.tar', 'w', dereference=True) as archive: |
| 92 | for name in ('synthetic.one', 'Open Notebook.onetoc2'): | 120 | for name in ('synthetic.one', 'Open Notebook.onetoc2'): |
| 93 | archive.add(source / name, arcname=name) | 121 | archive.add(source / name, arcname=name) |
| ... | @@ -127,8 +155,7 @@ def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1, | ... | @@ -127,8 +155,7 @@ def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1, |
| 127 | # Joining workers before stack exit retains ownership even if another boot fails. | 155 | # Joining workers before stack exit retains ownership even if another boot fails. |
| 128 | with ThreadPoolExecutor(max_workers=len(labels)) as pool: | 156 | with ThreadPoolExecutor(max_workers=len(labels)) as pool: |
| 129 | names = dict(zip(labels, pool.map(start_clone, labels))) | 157 | names = dict(zip(labels, pool.map(start_clone, labels))) |
| 130 | clients = [] | 158 | def prepare_client(label): |
| 131 | for label in labels: | ||
| 132 | folder = output / label | 159 | folder = output / label |
| 133 | name = names[label] | 160 | name = names[label] |
| 134 | for local in scripts.iterdir(): | 161 | for local in scripts.iterdir(): |
| ... | @@ -139,10 +166,13 @@ def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1, | ... | @@ -139,10 +166,13 @@ def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1, |
| 139 | command(name, 'powershell -NoProfile -ExecutionPolicy Bypass -File C:\\one-tests\\network.ps1 -LabMac ' + vm.lab_mac(name), folder) | 166 | command(name, 'powershell -NoProfile -ExecutionPolicy Bypass -File C:\\one-tests\\network.ps1 -LabMac ' + vm.lab_mac(name), folder) |
| 140 | command(name, 'ipconfig', folder) | 167 | command(name, 'ipconfig', folder) |
| 141 | command(name, 'dir \\\\192.168.77.1\\agent\\m6-collaboration', folder) | 168 | command(name, 'dir \\\\192.168.77.1\\agent\\m6-collaboration', folder) |
| 142 | clients.append({'name': name, 'folder': folder, 'sequence': 0}) | 169 | client = {'name': name, 'folder': folder, 'sequence': 0} |
| 143 | start_controller(clients[-1]) | 170 | start_controller(client) |
| 144 | command(name, 'ipconfig', folder) | 171 | command(name, 'ipconfig', folder) |
| 145 | print('Collaboration ready:', label, name, flush=True) | 172 | print('Collaboration ready:', label, name, flush=True) |
| 173 | return client | ||
| 174 | with ThreadPoolExecutor(max_workers=len(labels)) as pool: | ||
| 175 | clients = list(pool.map(prepare_client, labels)) | ||
| 146 | 176 | ||
| 147 | def action(client, action, wait=True, **parameters): | 177 | def action(client, action, wait=True, **parameters): |
| 148 | client['sequence'] += 1 | 178 | client['sequence'] += 1 |
| ... | @@ -166,10 +196,19 @@ def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1, | ... | @@ -166,10 +196,19 @@ def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1, |
| 166 | time.sleep(.5) | 196 | time.sleep(.5) |
| 167 | else: raise TimeoutError(f'Native command {sequence} did not complete') | 197 | else: raise TimeoutError(f'Native command {sequence} did not complete') |
| 168 | if action == 'snapshot': | 198 | if action == 'snapshot': |
| 169 | destination = client['folder'] / f'snapshot-{sequence:04}.xml' | 199 | hierarchy = client['folder'] / f'hierarchy-{sequence:04}.xml' |
| 170 | result = windows.do_get(remote + '\\page-0.xml', destination, client['name']) | 200 | result = windows.do_get(remote + '\\hierarchy.xml', hierarchy, client['name']) |
| 171 | if result.get('error'): raise RuntimeError(result['error']) | 201 | if result.get('error'): raise RuntimeError(result['error']) |
| 172 | return texts(ET.parse(destination).getroot()) | 202 | xml = hierarchy.read_text(encoding='utf-8-sig').strip() |
| 203 | if xml == '<?xml version="1.0"?>': return [] | ||
| 204 | pages = [node for node in ET.fromstring(xml).iter() if node.tag.endswith('}Page')] | ||
| 205 | observed = [] | ||
| 206 | for i in range(len(pages)): | ||
| 207 | destination = client['folder'] / f'snapshot-{sequence:04}-{i}.xml' | ||
| 208 | result = windows.do_get(remote + f'\\page-{i}.xml', destination, client['name']) | ||
| 209 | if result.get('error'): raise RuntimeError(result['error']) | ||
| 210 | observed.extend(texts(ET.parse(destination).getroot())) | ||
| 211 | return observed | ||
| 173 | 212 | ||
| 174 | def wait_text(client, expected): | 213 | def wait_text(client, expected): |
| 175 | deadline = time.monotonic() + 120 | 214 | deadline = time.monotonic() + 120 |
| ... | @@ -201,6 +240,7 @@ def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1, | ... | @@ -201,6 +240,7 @@ def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1, |
| 201 | return stream.read() | 240 | return stream.read() |
| 202 | 241 | ||
| 203 | def checkpoint(label): | 242 | def checkpoint(label): |
| 243 | if embedded_smb: reconnect_mount() | ||
| 204 | deadline, previous, incomplete = time.monotonic() + 120, None, 0 | 244 | deadline, previous, incomplete = time.monotonic() + 120, None, 0 |
| 205 | while time.monotonic() < deadline: | 245 | while time.monotonic() < deadline: |
| 206 | snapshot = {name: snapshot_file(name) for name in ('synthetic.one', 'Open Notebook.onetoc2')} | 246 | snapshot = {name: snapshot_file(name) for name in ('synthetic.one', 'Open Notebook.onetoc2')} |
| ... | @@ -228,7 +268,7 @@ def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1, | ... | @@ -228,7 +268,7 @@ def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1, |
| 228 | 268 | ||
| 229 | if abrupt and not conflict_clients: | 269 | if abrupt and not conflict_clients: |
| 230 | from concurrent_rust import running_clients | 270 | from concurrent_rust import running_clients |
| 231 | from crash_recovery import verify_text | 271 | from crash_recovery import active_text, verify_text |
| 232 | import crash | 272 | import crash |
| 233 | 273 | ||
| 234 | action(clients[0], 'prepare-stress', clients=len(clients)) | 274 | action(clients[0], 'prepare-stress', clients=len(clients)) |
| ... | @@ -244,6 +284,8 @@ def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1, | ... | @@ -244,6 +284,8 @@ def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1, |
| 244 | changed = native_text[i] + ' Before server stop.' | 284 | changed = native_text[i] + ' Before server stop.' |
| 245 | action(client, 'edit', expected=native_text[i], text=changed) | 285 | action(client, 'edit', expected=native_text[i], text=changed) |
| 246 | native_text[i] = changed | 286 | native_text[i] = changed |
| 287 | for client in clients: action(client, 'sync') | ||
| 288 | for client in clients: wait_text(client, native_text) | ||
| 247 | deadline = time.monotonic() + 60 | 289 | deadline = time.monotonic() + 60 |
| 248 | while True: | 290 | while True: |
| 249 | commits = sum(line.count('"event":"commit"') for path in folder.glob('w*.jsonl') for line in path.read_text().splitlines()) | 291 | commits = sum(line.count('"event":"commit"') for path in folder.glob('w*.jsonl') for line in path.read_text().splitlines()) |
| ... | @@ -254,6 +296,8 @@ def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1, | ... | @@ -254,6 +296,8 @@ def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1, |
| 254 | assert all(p.poll() is None for p in processes.values()), 'A client stopped before the planned interruption' | 296 | assert all(p.poll() is None for p in processes.values()), 'A client stopped before the planned interruption' |
| 255 | (output / 'server-crash.json').write_text(json.dumps(crash.stop('linux', server), indent=2)) | 297 | (output / 'server-crash.json').write_text(json.dumps(crash.stop('linux', server), indent=2)) |
| 256 | for client in clients: vm.qmp(client['name'], 'set_link', {'name': 'lab', 'up': False}) | 298 | for client in clients: vm.qmp(client['name'], 'set_link', {'name': 'lab', 'up': False}) |
| 299 | for process in processes.values(): process.terminate() | ||
| 300 | unmount(force=True) | ||
| 257 | raise InterruptedError('Recorded server interruption') | 301 | raise InterruptedError('Recorded server interruption') |
| 258 | except InterruptedError: | 302 | except InterruptedError: |
| 259 | pass | 303 | pass |
| ... | @@ -267,11 +311,12 @@ def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1, | ... | @@ -267,11 +311,12 @@ def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1, |
| 267 | subprocess.run(linux_vm.ssh_argv(server, 'cat /srv/agent/m6-collaboration/synthetic.one'), stdout=stream, check=True, timeout=60) | 311 | subprocess.run(linux_vm.ssh_argv(server, 'cat /srv/agent/m6-collaboration/synthetic.one'), stdout=stream, check=True, timeout=60) |
| 268 | subprocess.run([ROOT / 'target/debug/examples/document', recovered / 'synthetic.one', recovered / 'model'], check=True) | 312 | subprocess.run([ROOT / 'target/debug/examples/document', recovered / 'synthetic.one', recovered / 'model'], check=True) |
| 269 | model = json.loads((recovered / 'model/document.json').read_text()) | 313 | model = json.loads((recovered / 'model/document.json').read_text()) |
| 270 | text, = [text for values in reachable_page_text(model).values() for text in values if text.startswith('Concurrent edits:')] | 314 | text = active_text(model) |
| 271 | retained = verify_text('Concurrent edits:', text, logs) | 315 | retained = verify_text('Concurrent edits:', text, logs) |
| 272 | (output / 'server-retention.json').write_text(json.dumps(retained, indent=2)) | 316 | (output / 'server-retention.json').write_text(json.dumps(retained, indent=2)) |
| 273 | reconnect_mount() | 317 | reconnect_mount() |
| 274 | for client in clients: vm.qmp(client['name'], 'set_link', {'name': 'lab', 'up': True}) | 318 | for client in clients: vm.qmp(client['name'], 'set_link', {'name': 'lab', 'up': True}) |
| 319 | for client in clients: action(client, 'sync') | ||
| 275 | for client in clients: wait_text(client, [text, *native_text]) | 320 | for client in clients: wait_text(client, [text, *native_text]) |
| 276 | checkpoint('server-recovered') | 321 | checkpoint('server-recovered') |
| 277 | 322 | ||
| ... | @@ -302,6 +347,14 @@ def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1, | ... | @@ -302,6 +347,14 @@ def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1, |
| 302 | (output / 'native-cache-result.json').write_text(json.dumps({'cache_acknowledged': pending_text, 'retained_after_abrupt_stop': survived, 'server_durability_acknowledged': False}, indent=2)) | 347 | (output / 'native-cache-result.json').write_text(json.dumps({'cache_acknowledged': pending_text, 'retained_after_abrupt_stop': survived, 'server_durability_acknowledged': False}, indent=2)) |
| 303 | for client in clients: wait_text(client, [text, *native_text]) | 348 | for client in clients: wait_text(client, [text, *native_text]) |
| 304 | checkpoint('client-recovered') | 349 | checkpoint('client-recovered') |
| 350 | recovered_model = json.loads((output / 'client-recovered/model/document.json').read_text()) | ||
| 351 | recovered_pages = reachable_page_text(recovered_model) | ||
| 352 | main_space, *conflict_spaces = recovered_pages | ||
| 353 | known = {'Concurrent edits:', *[f'Native {i}:' for i in range(len(clients))], *native_text, pending_text} | ||
| 354 | conflicts = {sid: recovered_pages[sid] for sid in conflict_spaces} | ||
| 355 | assert all(value in known or (value.endswith(']') and text.startswith(value)) | ||
| 356 | for values in conflicts.values() for value in values), 'A recovered conflict contains unrecorded content' | ||
| 357 | (output / 'cache-conflicts.json').write_text(json.dumps(conflicts, indent=2)) | ||
| 305 | 358 | ||
| 306 | continued = output / 'rust-continued' | 359 | continued = output / 'rust-continued' |
| 307 | continued.mkdir() | 360 | continued.mkdir() |
| ... | @@ -310,7 +363,7 @@ def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1, | ... | @@ -310,7 +363,7 @@ def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1, |
| 310 | continuation = {actor: [json.loads(line) for line in (continued / f'{actor}.jsonl').read_text().splitlines()] for actor in processes} | 363 | continuation = {actor: [json.loads(line) for line in (continued / f'{actor}.jsonl').read_text().splitlines()] for actor in processes} |
| 311 | checkpoint('continued') | 364 | checkpoint('continued') |
| 312 | model = json.loads((output / 'continued/model/document.json').read_text()) | 365 | model = json.loads((output / 'continued/model/document.json').read_text()) |
| 313 | final_text, = [value for values in reachable_page_text(model).values() for value in values if value.startswith('Concurrent edits:')] | 366 | final_text = active_text(model) |
| 314 | result = verify_text(text, final_text, continuation) | 367 | result = verify_text(text, final_text, continuation) |
| 315 | for client in clients: wait_text(client, [final_text, *native_text]) | 368 | for client in clients: wait_text(client, [final_text, *native_text]) |
| 316 | (output / 'result.json').write_text(json.dumps({'server': retained, 'continuation': result, 'native_cache_retained': survived, 'expected_text': sorted([final_text, *native_text])}, indent=2)) | 369 | (output / 'result.json').write_text(json.dumps({'server': retained, 'continuation': result, 'native_cache_retained': survived, 'expected_text': sorted([final_text, *native_text])}, indent=2)) |
| ... | @@ -370,7 +423,7 @@ def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1, | ... | @@ -370,7 +423,7 @@ def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1, |
| 370 | for client in clients: action(client, 'sync') | 423 | for client in clients: action(client, 'sync') |
| 371 | elif stress_clients: | 424 | elif stress_clients: |
| 372 | from native_stress import exercise | 425 | from native_stress import exercise |
| 373 | exercise(output, shared, clients, action, wait_action, wait_text, checkpoint, stress_operations, sync_every, rust_writers, rust_readers, edit, seed) | 426 | exercise(output, shared, clients, action, wait_action, wait_text, checkpoint, stress_operations, sync_every, rust_writers, rust_readers, edit, seed, embedded_smb) |
| 374 | else: | 427 | else: |
| 375 | a, b = clients | 428 | a, b = clients |
| 376 | original = 'Fictitious: café, 東京, مرحبا' | 429 | original = 'Fictitious: café, 東京, مرحبا' |
| ... | @@ -450,8 +503,9 @@ def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1, | ... | @@ -450,8 +503,9 @@ def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1, |
| 450 | if abrupt and not conflict_clients: | 503 | if abrupt and not conflict_clients: |
| 451 | checkpoint('crash-closed') | 504 | checkpoint('crash-closed') |
| 452 | model = json.loads((output / 'crash-closed/model/document.json').read_text()) | 505 | model = json.loads((output / 'crash-closed/model/document.json').read_text()) |
| 453 | actual = sorted(value for values in reachable_page_text(model).values() for value in values) | 506 | actual = reachable_page_text(model) |
| 454 | assert actual == sorted([final_text, *native_text]), 'Application closure changed recovered edits' | 507 | assert actual.pop(main_space) == sorted([final_text, *native_text]), 'Application closure changed recovered edits' |
| 508 | assert actual == conflicts, 'Application closure changed recovered conflict pages' | ||
| 455 | elif stress_clients: | 509 | elif stress_clients: |
| 456 | checkpoint('stress-closed') | 510 | checkpoint('stress-closed') |
| 457 | elif conflict_clients: | 511 | elif conflict_clients: |
| ... | @@ -476,9 +530,13 @@ def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1, | ... | @@ -476,9 +530,13 @@ def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1, |
| 476 | (failure / 'capture-error.txt').write_text(str(error)) | 530 | (failure / 'capture-error.txt').write_text(str(error)) |
| 477 | raise | 531 | raise |
| 478 | finally: | 532 | finally: |
| 479 | if mounted: | 533 | if embedded_smb and linux_vm.instance_path(server).exists() and linux_vm.running(server): |
| 480 | result = subprocess.run(['/sbin/umount', str(mount)], capture_output=True, text=True) | 534 | try: |
| 481 | (output / 'unmount.json').write_text(json.dumps({'exit': result.returncode, 'stderr': result.stderr})) | 535 | with (output / 'smb-trace.jsonl').open('wb') as trace: |
| 536 | subprocess.run(linux_vm.ssh_argv(server, 'cat /tmp/smb-trace.jsonl'), stdout=trace, check=True, timeout=30) | ||
| 537 | except Exception as error: | ||
| 538 | (output / 'trace-error.txt').write_text(str(error)) | ||
| 539 | if os.path.ismount(mount): unmount() | ||
| 482 | if linux_vm.instance_path(server).exists(): | 540 | if linux_vm.instance_path(server).exists(): |
| 483 | try: | 541 | try: |
| 484 | if linux_vm.running(server): linux_vm.shutdown(server, 60) | 542 | if linux_vm.running(server): linux_vm.shutdown(server, 60) |
| ... | @@ -488,6 +546,20 @@ def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1, | ... | @@ -488,6 +546,20 @@ def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1, |
| 488 | while linux_vm.running(server) and time.monotonic() < deadline: time.sleep(.1) | 546 | while linux_vm.running(server) and time.monotonic() < deadline: time.sleep(.1) |
| 489 | linux_vm.delete_instance(server) | 547 | linux_vm.delete_instance(server) |
| 490 | (output / 'teardown.json').write_text(json.dumps({'linux_absent': not linux_vm.instance_path(server).exists()}, indent=2)) | 548 | (output / 'teardown.json').write_text(json.dumps({'linux_absent': not linux_vm.instance_path(server).exists()}, indent=2)) |
| 549 | if embedded_smb: | ||
| 550 | from verify_smb_overlap import verify | ||
| 551 | with (output / 'smb-trace.jsonl').open() as trace: | ||
| 552 | overlap = verify(json.loads(line) for line in trace) | ||
| 553 | (output / 'overlap.json').write_text(json.dumps(overlap, indent=2)) | ||
| 554 | if offline_lost_reply: | ||
| 555 | from offline_outage import verify_lost_reply | ||
| 556 | (output / 'offline-lost-reply-verification.json').write_text(json.dumps(verify_lost_reply(output), indent=2)) | ||
| 557 | if offline_outage: | ||
| 558 | from offline_outage import verify_outage | ||
| 559 | (output / 'offline-outage-verification.json').write_text(json.dumps(verify_outage(output), indent=2)) | ||
| 560 | if disconnect: | ||
| 561 | from native_disconnect import verify_disconnect | ||
| 562 | (output / 'disconnect-verification.json').write_text(json.dumps(verify_disconnect(output), indent=2)) | ||
| 491 | 563 | ||
| 492 | 564 | ||
| 493 | if __name__ == '__main__': | 565 | if __name__ == '__main__': |
| ... | @@ -503,7 +575,29 @@ if __name__ == '__main__': | ... | @@ -503,7 +575,29 @@ if __name__ == '__main__': |
| 503 | parser.add_argument('--edit', action='store_true', help='Use random text replacements in every writer.') | 575 | parser.add_argument('--edit', action='store_true', help='Use random text replacements in every writer.') |
| 504 | parser.add_argument('--seed', type=int, default=710) | 576 | parser.add_argument('--seed', type=int, default=710) |
| 505 | parser.add_argument('--abrupt', action='store_true', help='Abrupt server and client stops with preserved-disk recovery and intent accounting.') | 577 | parser.add_argument('--abrupt', action='store_true', help='Abrupt server and client stops with preserved-disk recovery and intent accounting.') |
| 578 | parser.add_argument('--embedded-smb', action='store_true', help='Run Rust stress clients through the embedded SMB adapter.') | ||
| 579 | parser.add_argument('--maintenance', action='store_true', help='Pause the workload for the owned maintenance controller.') | ||
| 580 | parser.add_argument('--offline', action='store_true', help='Use durable local queues and traced offline workers for Rust writers.') | ||
| 581 | parser.add_argument('--document-operations', action='store_true', help='Queue paragraph/outline creation and formatting alongside offline text edits.') | ||
| 582 | parser.add_argument('--record-writes', action='store_true', help='Retain owned lab write payloads for revision replay.') | ||
| 583 | parser.add_argument('--offline-lost-reply', action='store_true', help='Drop a publication reply and require confirmation of its original revision.') | ||
| 584 | parser.add_argument('--offline-client-reply', action='store_true', help='Disconnect only the formatting writer and require peer publication before it reconciles.') | ||
| 585 | parser.add_argument('--offline-outage', action='store_true', help='Queue local edits during an owned SMB outage, then require recovery.') | ||
| 586 | parser.add_argument('--disconnect', action='store_true', help='Interrupt and reconnect the embedded append workload twice.') | ||
| 587 | parser.add_argument('--client-profile', choices=('debug', 'release'), default='debug', help='Cargo build profile for Rust stress clients') | ||
| 588 | parser.add_argument('--client-timeout', type=float, default=600, help='Maximum seconds for the Rust workload, including its start barrier.') | ||
| 589 | parser.add_argument('--fixture', type=Path, help='Copy this fixture directory into the disposable stress notebook.') | ||
| 506 | args = parser.parse_args() | 590 | args = parser.parse_args() |
| 591 | if not 0 < args.client_timeout < 2**64 / 1000: parser.error('Choose a finite positive client timeout.') | ||
| 592 | if args.maintenance and not (args.embedded_smb and args.stress_clients): parser.error('--maintenance requires embedded SMB stress mode.') | ||
| 593 | if args.document_operations and not args.offline: parser.error('--document-operations requires --offline.') | ||
| 594 | if args.record_writes and not args.embedded_smb: parser.error('--record-writes requires --embedded-smb.') | ||
| 595 | if args.offline_client_reply and not (args.offline_lost_reply and args.document_operations): parser.error('--offline-client-reply requires --offline-lost-reply and --document-operations.') | ||
| 596 | if args.offline_lost_reply and (not args.offline or args.offline_outage or args.sync_every or args.stress_operations < 8): parser.error('--offline-lost-reply requires --offline, --sync-every 0, at least eight operations and no --offline-outage.') | ||
| 597 | if args.offline_outage and (not args.offline or args.sync_every or args.stress_operations < 8): parser.error('--offline-outage requires --offline, --sync-every 0 and at least eight operations.') | ||
| 598 | if args.offline and (not args.embedded_smb or not args.stress_clients or args.edit or args.disconnect or args.maintenance): parser.error('--offline requires embedded append stress without disconnect or maintenance.') | ||
| 599 | if args.embedded_smb and not args.stress_clients: parser.error('--embedded-smb requires --stress-clients.') | ||
| 600 | if args.disconnect and (not args.embedded_smb or args.edit or args.maintenance or args.sync_every): parser.error('--disconnect requires embedded append stress with --sync-every 0 and no maintenance.') | ||
| 507 | if args.rust_writers < 2 or args.rust_readers < 1: parser.error('Use at least two Rust writers and one reader.') | 601 | if args.rust_writers < 2 or args.rust_readers < 1: parser.error('Use at least two Rust writers and one reader.') |
| 508 | if args.sync_every < 0 or args.stress_operations <= 0: parser.error('Use a nonnegative sync interval and positive operation count.') | 602 | if args.sync_every < 0 or args.stress_operations <= 0: parser.error('Use a nonnegative sync interval and positive operation count.') |
| 509 | if args.stress_clients and args.stress_clients < 3: parser.error('Stress mode requires at least three native clients.') | 603 | if args.stress_clients and args.stress_clients < 3: parser.error('Stress mode requires at least three native clients.') |
| ... | @@ -511,4 +605,4 @@ if __name__ == '__main__': | ... | @@ -511,4 +605,4 @@ if __name__ == '__main__': |
| 511 | if args.abrupt and (args.stress_clients or args.edit): parser.error('Abrupt recovery uses append intents or the offline-conflict workload.') | 605 | if args.abrupt and (args.stress_clients or args.edit): parser.error('Abrupt recovery uses append intents or the offline-conflict workload.') |
| 512 | def interrupted(_signal, _frame): raise KeyboardInterrupt | 606 | def interrupted(_signal, _frame): raise KeyboardInterrupt |
| 513 | signal.signal(signal.SIGTERM, interrupted) | 607 | signal.signal(signal.SIGTERM, interrupted) |
| 514 | replay(args.output, args.linux, args.stress_clients, args.stress_operations, args.sync_every, args.rust_writers, args.rust_readers, args.edit, args.seed, args.conflict_clients, args.abrupt) | 608 | replay(args.output, args.linux, args.stress_clients, args.stress_operations, args.sync_every, args.rust_writers, args.rust_readers, args.edit, args.seed, args.conflict_clients, args.abrupt, args.embedded_smb, args.maintenance, args.fixture, args.disconnect, args.client_timeout, args.offline, args.offline_outage, args.offline_lost_reply, args.client_profile, args.document_operations, args.record_writes, args.offline_client_reply) |
tools/native_disconnect.py created+137
| ... | @@ -0,0 +1,137 @@ | ||
| 1 | """Interrupt an owned mixed append workload and require progress after reconnection.""" | ||
| 2 | import json | ||
| 3 | import shlex | ||
| 4 | import subprocess | ||
| 5 | import time | ||
| 6 | |||
| 7 | from native_runner import windows | ||
| 8 | import linux_vm | ||
| 9 | from verify_smb_overlap import verify | ||
| 10 | |||
| 11 | |||
| 12 | def interrupt(output, clients, sequences, processes): | ||
| 13 | config = json.loads((output / 'run.json').read_text()) | ||
| 14 | server = config['server'] | ||
| 15 | samples = [] | ||
| 16 | subprocess.run(linux_vm.ssh_argv(server, 'cat > /tmp/verify_smb_overlap.py'), | ||
| 17 | input=(output / 'harness/verify_smb_overlap.py').read_bytes(), check=True) | ||
| 18 | |||
| 19 | def counts(transport=False): | ||
| 20 | result = {} | ||
| 21 | for actor in processes: | ||
| 22 | data = (output / 'rust' / (actor + '.jsonl')).read_text() | ||
| 23 | events = [json.loads(line) for line in data[:data.rfind('\n') + 1].splitlines()] | ||
| 24 | names = ('transport_read_error', 'transport_commit_error') if transport else ('commit' if actor.startswith('w') else 'read',) | ||
| 25 | result[actor] = sum(event['event'] in names for event in events) | ||
| 26 | return result | ||
| 27 | |||
| 28 | def wait_for(predicate, message): | ||
| 29 | deadline = time.monotonic() + 90 | ||
| 30 | while not predicate(): | ||
| 31 | assert all(process.poll() is None for process in processes.values()), 'A Rust client exited during the interruption campaign' | ||
| 32 | if time.monotonic() > deadline: raise TimeoutError(message) | ||
| 33 | time.sleep(.1) | ||
| 34 | |||
| 35 | def ssh(command): | ||
| 36 | result = linux_vm.run_ssh(server, command, timeout=15) | ||
| 37 | with (output / 'disconnect-server.jsonl').open('a') as stream: | ||
| 38 | stream.write(json.dumps({'command': command, 'exit': result.returncode, 'stdout': result.stdout, 'stderr': result.stderr}) + '\n') | ||
| 39 | result.check_returncode() | ||
| 40 | return result.stdout | ||
| 41 | |||
| 42 | def phase(control): | ||
| 43 | text = json.dumps(control) | ||
| 44 | ssh("printf '%s' '" + text + "' > /tmp/smb-control.tmp && mv /tmp/smb-control.tmp /tmp/smb-control.json") | ||
| 45 | wait_for(lambda: text in ssh("grep -F '\"control\":' /tmp/smb-trace.jsonl | tail -n 1"), 'Proxy did not acknowledge the disconnect phase') | ||
| 46 | |||
| 47 | previous = {actor: 0 for actor in processes} | ||
| 48 | for cycle in range(2): | ||
| 49 | wait_for(lambda: all(value >= previous[actor] + 3 for actor, value in counts().items()), 'Clients made no progress before the interruption') | ||
| 50 | before = counts() | ||
| 51 | native = [] | ||
| 52 | for actor, (client, sequence) in enumerate(zip(clients, sequences)): | ||
| 53 | capture = output / f'disconnect-{cycle}-n{actor}.jsonl' | ||
| 54 | result = windows.do_get(f'C:\\one-tests\\runs\\capture\\outbox\\{sequence}\\events.jsonl', capture, client['name']) | ||
| 55 | assert not result.get('error'), result | ||
| 56 | data = capture.read_text(encoding='utf-8-sig') | ||
| 57 | rows = [json.loads(line) for line in data[:data.rfind('\n') + 1].splitlines()] | ||
| 58 | assert 0 < len(rows) < config['stress_operations'], 'A native writer was inactive before the interruption' | ||
| 59 | native.append(len(rows)) | ||
| 60 | before_errors = counts(transport=True) | ||
| 61 | assert all(process.poll() is None for process in processes.values()), 'A Rust client finished before the interruption' | ||
| 62 | try: | ||
| 63 | phase({'phase': f'disconnect-{cycle}', 'cut': 9, 'peer': '10.0.2.2', 'offset': 96, | ||
| 64 | 'direction': 'request' if cycle == 0 else 'response'}) | ||
| 65 | wait_for(lambda: f'"phase": "disconnect-{cycle}"' in ssh("grep -F '\"control\":' /tmp/smb-trace.jsonl | tail -n 1") | ||
| 66 | and int(ssh("grep -c '\"cut\": {' /tmp/smb-trace.jsonl || true").strip()) == cycle + 1, | ||
| 67 | 'The planned write interruption did not occur') | ||
| 68 | time.sleep(3) | ||
| 69 | finally: | ||
| 70 | phase({'phase': f'reconnected-{cycle}'}) | ||
| 71 | wait_for(lambda: all(value >= before[actor] + 3 for actor, value in counts().items()), 'A client failed to progress after reconnecting') | ||
| 72 | script = '\n'.join([ | ||
| 73 | 'import json', 'from verify_smb_overlap import verify, PendingOverlap', | ||
| 74 | "data = open('/tmp/smb-trace.jsonl').read()", | ||
| 75 | "events = [json.loads(line) for line in data[:data.rfind('\\n') + 1].splitlines()]", | ||
| 76 | 'try:', f" result = verify(events, phase='reconnected-{cycle}')", | ||
| 77 | "except PendingOverlap: result = None", 'print(json.dumps(result))']) | ||
| 78 | wait_for(lambda: json.loads(ssh('cd /tmp && python3 -c ' + shlex.quote(script))) is not None, | ||
| 79 | 'Native and Rust guarded I/O did not overlap after reconnection') | ||
| 80 | previous = counts() | ||
| 81 | samples.append({'cycle': cycle, 'before': before, 'after': previous, 'native_before': native, | ||
| 82 | 'before_errors': before_errors, 'after_errors': counts(transport=True)}) | ||
| 83 | (output / 'disconnect-progress.json').write_text(json.dumps(samples, indent=2)) | ||
| 84 | |||
| 85 | |||
| 86 | def verify_disconnect(output): | ||
| 87 | config = json.loads((output / 'run.json').read_text()) | ||
| 88 | assert config['stress_clients'] + config['rust_writers'] + config['rust_readers'] >= 12 | ||
| 89 | samples = json.loads((output / 'disconnect-progress.json').read_text()) | ||
| 90 | assert [sample['cycle'] for sample in samples] == [0, 1] | ||
| 91 | actors = {f'w{i}' for i in range(config['rust_writers'])} | {f'r{i}' for i in range(config['rust_readers'])} | ||
| 92 | events = [json.loads(line) for line in (output / 'smb-trace.jsonl').read_text().splitlines()] | ||
| 93 | assert sum('cut' in event for event in events) == 2, 'Unexpected number of connection interruptions' | ||
| 94 | errors = {} | ||
| 95 | progress = {} | ||
| 96 | progress_limit = 120 | ||
| 97 | started = (output / 'rust/start').stat().st_mtime_ns // 1000 | ||
| 98 | stopped = (output / 'rust/stop').stat().st_mtime_ns // 1000 | ||
| 99 | |||
| 100 | def max_gap(actor, times, units): | ||
| 101 | assert len(times) > 1, 'A client made no progress' | ||
| 102 | gaps = [(end - begin) / units for begin, end in zip(times, times[1:])] | ||
| 103 | assert all(0 <= gap <= progress_limit for gap in gaps), f'{actor}: client progress stalled or went backwards' | ||
| 104 | return max(gaps) | ||
| 105 | |||
| 106 | for i in range(config['stress_clients']): | ||
| 107 | rows = [json.loads(line) for line in (output / f'n{i}/stress-events.jsonl').read_text(encoding='utf-8-sig').splitlines()] | ||
| 108 | assert len(rows) == config['stress_operations'], 'A native writer did not finish' | ||
| 109 | progress[f'n{i}'] = max_gap(f'n{i}', [rows[0]['update_started_ticks'], *[row['updated_ticks'] for row in rows]], 10**7) | ||
| 110 | uncertain = set() | ||
| 111 | for actor in sorted(actors): | ||
| 112 | rows = [json.loads(line) for line in (output / 'rust' / (actor + '.jsonl')).read_text().splitlines()] | ||
| 113 | errors[actor] = sum(row['event'] in ('transport_read_error', 'transport_commit_error') for row in rows) | ||
| 114 | assert errors[actor] and any(row['event'] == 'transport_connected' for row in rows), 'A Rust client did not exercise reconnection' | ||
| 115 | times = [started, *[row['finished_us'] for row in rows if row['event'] == ('commit' if actor.startswith('w') else 'read')]] | ||
| 116 | if actor.startswith('r') and len(times) > 1: times.append(max(times[-1], stopped)) | ||
| 117 | progress[actor] = max_gap(actor, times, 10**6) | ||
| 118 | pending = None | ||
| 119 | for row in rows: | ||
| 120 | if row['event'] == 'transport_commit_error': pending = row | ||
| 121 | if row['event'] != 'transport_reconciled': continue | ||
| 122 | assert pending is not None and pending['token'] == row['token'] | ||
| 123 | if row['published']: assert row.get('flush_confirmed'), 'Visible recovery lacks a durable acknowledgement' | ||
| 124 | if pending['state'] == 'Unknown': uncertain.add('after' if row['published'] else 'before') | ||
| 125 | pending = None | ||
| 126 | assert uncertain == {'before', 'after'}, 'The mixed workload did not resolve both uncertain outcomes' | ||
| 127 | overlap = [] | ||
| 128 | for sample in samples: | ||
| 129 | assert all(set(sample[field]) == actors for field in ('before', 'after', 'before_errors', 'after_errors')) | ||
| 130 | assert all(sample['after'][actor] >= value + 3 for actor, value in sample['before'].items()) | ||
| 131 | assert all(sample['after_errors'][actor] > value for actor, value in sample['before_errors'].items()), 'A client did not encounter this interruption' | ||
| 132 | assert len(sample['native_before']) == config['stress_clients'] | ||
| 133 | assert all(0 < value < config['stress_operations'] for value in sample['native_before']) | ||
| 134 | end = next((i for i, event in enumerate(events) if event.get('control', {}).get('phase') == f'disconnect-{sample["cycle"] + 1}'), len(events)) | ||
| 135 | overlap.append(verify(events[:end], phase=f'reconnected-{sample["cycle"]}')) | ||
| 136 | return {'interruptions': 2, 'transport_errors': errors, 'uncertain_outcomes': sorted(uncertain), 'resumed_overlap': overlap, | ||
| 137 | 'max_progress_gap_seconds': progress, 'progress_limit_seconds': progress_limit} | ||
tools/native_maintenance.py created+210
| ... | @@ -0,0 +1,210 @@ | ||
| 1 | #!/usr/bin/env python3 | ||
| 2 | """Compact a shared section between two halves of a twelve-client editing run.""" | ||
| 3 | import argparse | ||
| 4 | import base64 | ||
| 5 | import json | ||
| 6 | import os | ||
| 7 | from pathlib import Path | ||
| 8 | import signal | ||
| 9 | import subprocess | ||
| 10 | import sys | ||
| 11 | import time | ||
| 12 | import xml.etree.ElementTree as ET | ||
| 13 | |||
| 14 | from native_runner import ROOT, windows | ||
| 15 | import linux_vm | ||
| 16 | from verify_smb_overlap import verify | ||
| 17 | |||
| 18 | |||
| 19 | def maintenance_locks(events): | ||
| 20 | pending, files, peers = {}, {}, {} | ||
| 21 | phase, attempts = None, [] | ||
| 22 | for event in events: | ||
| 23 | assert not event.get('trace_error') and not event.get('encrypted') | ||
| 24 | phase = event.get('control', {}).get('phase', phase) | ||
| 25 | connection = event.get('connection') | ||
| 26 | if event.get('opened'): peers[connection] = event['peer'][0] | ||
| 27 | if event.get('closed'): | ||
| 28 | files = {key: value for key, value in files.items() if key[0] != connection} | ||
| 29 | if 'command' not in event: continue | ||
| 30 | key = connection, event['message'] | ||
| 31 | if event['direction'] == 'request': | ||
| 32 | pending[key] = event, phase | ||
| 33 | continue | ||
| 34 | if event['status'] == '0x103': continue | ||
| 35 | pair = pending.pop(key, None) | ||
| 36 | if pair is None: continue | ||
| 37 | request, issued = pair | ||
| 38 | command = request['command'] | ||
| 39 | if command == 5 and event['status'] == '0x0': | ||
| 40 | files[connection, event['file_id']] = request['path'].replace('\\', '/').lower() | ||
| 41 | elif command == 6: | ||
| 42 | files.pop((connection, request['file_id']), None) | ||
| 43 | elif command == 10 and peers[connection].startswith('192.168.77.'): | ||
| 44 | path = files.get((connection, request['file_id']), '') | ||
| 45 | if path != 'm6-collaboration/synthetic.one': continue | ||
| 46 | for offset, length, flags in request['locks']: | ||
| 47 | if (offset, length) == (0xffffeffc, 4096) and flags & 3 == 2: | ||
| 48 | attempts.append({'phase': issued, 'status': event['status'], 'connection': connection, | ||
| 49 | 'message': event['message'], 'flags': flags}) | ||
| 50 | assert any(a['phase'] == 'maintenance-held' and a['status'] in ('0xc0000054', '0xc0000055') for a in attempts), 'No section maintenance conflict observed while the reader held its guard' | ||
| 51 | assert any(a['phase'] == 'maintenance-released' and a['status'] == '0x0' for a in attempts), 'No section maintenance guard acquired after reader release' | ||
| 52 | return attempts | ||
| 53 | |||
| 54 | |||
| 55 | def run(output, server, fixture, operations, seed): | ||
| 56 | output = output.resolve() | ||
| 57 | assert not output.exists(), 'Choose a new output directory' | ||
| 58 | output.parent.mkdir(parents=True, exist_ok=True) | ||
| 59 | guardian = None | ||
| 60 | with output.with_suffix('.log').open('x') as log: | ||
| 61 | process = subprocess.Popen([sys.executable, ROOT / 'tools/native_collaboration.py', output, | ||
| 62 | '--linux', server, '--stress-clients', '4', '--rust-writers', '4', '--rust-readers', '4', | ||
| 63 | '--stress-operations', str(operations), '--seed', str(seed), '--edit', '--embedded-smb', | ||
| 64 | '--maintenance', '--fixture', fixture], stdout=log, stderr=subprocess.STDOUT) | ||
| 65 | |||
| 66 | def wait_for(predicate, timeout, message): | ||
| 67 | deadline = time.monotonic() + timeout | ||
| 68 | while not predicate(): | ||
| 69 | if process.poll() is not None: raise RuntimeError(f'Workload exited with {process.returncode}: {message}') | ||
| 70 | if guardian is not None and guardian.poll() is not None and guardian.returncode != 0: | ||
| 71 | raise RuntimeError('The reader guardian failed') | ||
| 72 | if time.monotonic() > deadline: raise TimeoutError(message) | ||
| 73 | time.sleep(.2) | ||
| 74 | |||
| 75 | def ssh(command): | ||
| 76 | result = linux_vm.run_ssh(server, command, timeout=30) | ||
| 77 | with (output / 'maintenance-server.jsonl').open('a') as stream: | ||
| 78 | stream.write(json.dumps({'command': command, 'exit': result.returncode, 'stdout': result.stdout, 'stderr': result.stderr}) + '\n') | ||
| 79 | result.check_returncode() | ||
| 80 | return result.stdout | ||
| 81 | |||
| 82 | def phase(name): | ||
| 83 | ssh('printf \'{"phase":"' + name + '"}\' > /tmp/smb-control.json') | ||
| 84 | wait_for(lambda: name in ssh(f'grep -F \'"control": {{"phase": "{name}"}}\' /tmp/smb-trace.jsonl || true'), | ||
| 85 | 10, 'Proxy did not acknowledge the maintenance phase') | ||
| 86 | |||
| 87 | def stat(): | ||
| 88 | inode, size = ssh("stat -c '%i %s' /srv/agent/m6-collaboration/synthetic.one").split() | ||
| 89 | return {'inode': int(inode), 'size': int(size)} | ||
| 90 | |||
| 91 | def ui(name, script): | ||
| 92 | (output / f'{name}.ahk').write_text(script) | ||
| 93 | result = windows.do_exec(script, target=target, timeout_ms=60000, shot_delay_ms=500) | ||
| 94 | screenshot = result.pop('png_b64', None) | ||
| 95 | if screenshot: (output / f'{name}.png').write_bytes(base64.b64decode(screenshot)) | ||
| 96 | (output / f'{name}.json').write_text(json.dumps(result, indent=2)) | ||
| 97 | if result.get('error') or result.get('exit') != 0: raise RuntimeError(str(result)) | ||
| 98 | |||
| 99 | try: | ||
| 100 | wait_for(lambda: (output / 'maintenance-ready').exists(), 900, 'Native clients did not reach the initial checkpoint') | ||
| 101 | (output / 'maintenance-start').touch() | ||
| 102 | shared, rust = output / 'mount/m6-collaboration', output / 'rust' | ||
| 103 | wait_for(lambda: all((shared / f'maintenance-paused-n{i}').exists() and (rust / f'paused-w{i}').exists() for i in range(4)), | ||
| 104 | 300, 'Writers did not reach the maintenance barrier') | ||
| 105 | (rust / 'pause').touch() | ||
| 106 | wait_for(lambda: all((rust / f'paused-r{i}').exists() for i in range(4)), 60, 'Readers did not pause') | ||
| 107 | target = json.loads((output / 'n0/machine.json').read_text())['name'] | ||
| 108 | page = ET.parse(sorted((output / 'n0').glob('snapshot-*.xml'))[-1]).getroot().attrib['ID'] | ||
| 109 | ui('maintenance-options', f'''if A_ScreenWidth != 800 || A_ScreenHeight != 600 | ||
| 110 | throw Error("The maintenance controller requires an 800 by 600 desktop.") | ||
| 111 | ComObject("OneNote.Application").NavigateTo("{page}", "", false) | ||
| 112 | WinWait("ahk_exe ONENOTE.EXE", , 10) | ||
| 113 | WinActivate("ahk_exe ONENOTE.EXE") | ||
| 114 | WinWaitActive("ahk_exe ONENOTE.EXE", , 10) | ||
| 115 | Send("!ft") | ||
| 116 | WinWait("OneNote Options", , 10) | ||
| 117 | WinActivate("OneNote Options") | ||
| 118 | WinWaitActive("OneNote Options", , 10) | ||
| 119 | Sleep(1000) | ||
| 120 | CoordMode("Mouse", "Screen") | ||
| 121 | Click(74, 134) | ||
| 122 | Sleep(1000) | ||
| 123 | ''') | ||
| 124 | hold = output / 'guardian' | ||
| 125 | hold.mkdir() | ||
| 126 | config = json.loads((output / 'linux.json').read_text()) | ||
| 127 | with (hold / 'run.log').open('w') as guardian_log: | ||
| 128 | guardian = subprocess.Popen(['cargo', 'test', '-p', 'onestore-smb', 'live_reader_hold', '--', '--ignored', '--nocapture'], | ||
| 129 | cwd=ROOT, stdout=guardian_log, stderr=subprocess.STDOUT, | ||
| 130 | env={**os.environ, 'ONESTORE_SMB_LAB': f'127.0.0.1:{config["samba_port"]}', | ||
| 131 | 'ONESTORE_SMB_PATH': 'm6-collaboration/synthetic.one', 'ONESTORE_SMB_HOLD': str(hold)}) | ||
| 132 | wait_for(lambda: (hold / 'ready').exists(), 60, 'Reader guard was not acquired') | ||
| 133 | before = stat() | ||
| 134 | phase('maintenance-held') | ||
| 135 | ui('maintenance-denied', '''CoordMode("Mouse", "Screen") | ||
| 136 | WinActivate("OneNote Options") | ||
| 137 | WinWaitActive("OneNote Options", , 10) | ||
| 138 | Click(254, 440) | ||
| 139 | if !WinWait("Microsoft OneNote ahk_class #32770", , 30) | ||
| 140 | throw Error("The maintenance conflict dialog did not appear.") | ||
| 141 | FileAppend(WinGetText("Microsoft OneNote ahk_class #32770"), "*") | ||
| 142 | ''') | ||
| 143 | held = stat() | ||
| 144 | assert held == before, 'Section changed while the reader held its maintenance exclusion guard' | ||
| 145 | (hold / 'release').touch() | ||
| 146 | wait_for(lambda: guardian.poll() is not None, 30, 'Reader guardian did not release') | ||
| 147 | assert guardian.returncode == 0 | ||
| 148 | assert json.loads((hold / 'released.json').read_text())['accepted'] > 0 | ||
| 149 | phase('maintenance-released') | ||
| 150 | ui('maintenance-dismiss', '''WinActivate("Microsoft OneNote ahk_class #32770") | ||
| 151 | ControlClick("Button1", "Microsoft OneNote ahk_class #32770") | ||
| 152 | if !WinWaitClose("Microsoft OneNote ahk_class #32770", , 10) | ||
| 153 | throw Error("The maintenance conflict dialog did not close.") | ||
| 154 | ''') | ||
| 155 | replacements = [] | ||
| 156 | for attempt in range(5): | ||
| 157 | ui(f'maintenance-optimize-{attempt}', '''CoordMode("Mouse", "Screen") | ||
| 158 | WinActivate("OneNote Options") | ||
| 159 | WinWaitActive("OneNote Options", , 10) | ||
| 160 | Click(254, 440) | ||
| 161 | Sleep(3000) | ||
| 162 | if WinExist("Microsoft OneNote ahk_class #32770") { | ||
| 163 | FileAppend(WinGetText("Microsoft OneNote ahk_class #32770"), "*") | ||
| 164 | ControlClick("Button1", "Microsoft OneNote ahk_class #32770") | ||
| 165 | if !WinWaitClose("Microsoft OneNote ahk_class #32770", , 10) | ||
| 166 | throw Error("The maintenance conflict dialog did not close.") | ||
| 167 | } | ||
| 168 | ''') | ||
| 169 | replacements.append(stat()) | ||
| 170 | if replacements[-1]['inode'] != before['inode'] and replacements[-1]['size'] < before['size']: break | ||
| 171 | time.sleep(2) | ||
| 172 | (output / 'maintenance-stat.json').write_text(json.dumps({'before': before, 'held': held, 'attempts': replacements}, indent=2)) | ||
| 173 | assert replacements[-1]['inode'] != before['inode'] and replacements[-1]['size'] < before['size'], 'Native optimization did not replace and shrink the section' | ||
| 174 | ui('maintenance-options-close', '''WinActivate("OneNote Options") | ||
| 175 | CoordMode("Mouse", "Screen") | ||
| 176 | Click(663, 533) | ||
| 177 | WinWaitClose("OneNote Options", , 10) | ||
| 178 | ''') | ||
| 179 | phase('maintenance-resumed') | ||
| 180 | (rust / 'resume').touch() | ||
| 181 | (shared / 'maintenance-resume').touch() | ||
| 182 | wait_for(lambda: process.poll() is not None, 600, 'Post-maintenance workload did not complete') | ||
| 183 | assert process.returncode == 0, 'Mixed workload failed after maintenance' | ||
| 184 | events = [json.loads(line) for line in (output / 'smb-trace.jsonl').read_text().splitlines()] | ||
| 185 | result = {'maintenance_locks': maintenance_locks(events), 'resumed_overlap': verify(events, phase='maintenance-resumed')} | ||
| 186 | (output / 'maintenance-verification.json').write_text(json.dumps(result, indent=2)) | ||
| 187 | finally: | ||
| 188 | if guardian is not None and guardian.poll() is None: | ||
| 189 | (output / 'guardian/release').touch() | ||
| 190 | try: guardian.wait(timeout=30) | ||
| 191 | except subprocess.TimeoutExpired: | ||
| 192 | guardian.terminate() | ||
| 193 | guardian.wait(timeout=30) | ||
| 194 | if process.poll() is None: | ||
| 195 | process.terminate() | ||
| 196 | process.wait(timeout=180) | ||
| 197 | |||
| 198 | |||
| 199 | if __name__ == '__main__': | ||
| 200 | parser = argparse.ArgumentParser(description=__doc__) | ||
| 201 | parser.add_argument('output', type=Path) | ||
| 202 | parser.add_argument('--linux', required=True) | ||
| 203 | parser.add_argument('--fixture', type=Path, required=True) | ||
| 204 | parser.add_argument('--operations', type=int, default=40) | ||
| 205 | parser.add_argument('--seed', type=int, default=908) | ||
| 206 | args = parser.parse_args() | ||
| 207 | if args.operations < 4 or args.operations % 2: parser.error('Use an even operation count of at least four.') | ||
| 208 | def interrupted(_signal, _frame): raise KeyboardInterrupt | ||
| 209 | signal.signal(signal.SIGTERM, interrupted) | ||
| 210 | run(args.output, args.linux, args.fixture.resolve(), args.operations, args.seed) | ||
tools/native_probe.py+6-3
| ... | @@ -9,11 +9,14 @@ import zipfile | ... | @@ -9,11 +9,14 @@ import zipfile |
| 9 | from native_runner import ROOT, clone, command, windows, collect_artifacts | 9 | from native_runner import ROOT, clone, command, windows, collect_artifacts |
| 10 | 10 | ||
| 11 | 11 | ||
| 12 | def run(inputs, output): | 12 | def run(inputs, output, scripts=None): |
| 13 | output.mkdir(parents=True, exist_ok=False) | 13 | output.mkdir(parents=True, exist_ok=False) |
| 14 | files = sorted(inputs.glob('*.one')) | 14 | files = sorted(inputs.glob('*.one')) |
| 15 | assert files, 'No section candidates' | 15 | assert files, 'No section candidates' |
| 16 | scripts = [ROOT / 'tools/native/cold.ps1', ROOT / 'tools/native/probe.ps1'] | 16 | if scripts is None: |
| 17 | (output / 'scripts').mkdir() | ||
| 18 | scripts = [output / 'scripts' / name for name in ('cold.ps1', 'probe.ps1')] | ||
| 19 | for path in scripts: path.write_bytes((ROOT / 'tools/native' / path.name).read_bytes()) | ||
| 17 | (output / 'run.json').write_text(json.dumps({'inputs': {p.name: hashlib.sha256(p.read_bytes()).hexdigest() for p in files}, | 20 | (output / 'run.json').write_text(json.dumps({'inputs': {p.name: hashlib.sha256(p.read_bytes()).hexdigest() for p in files}, |
| 18 | 'scripts': {p.name: hashlib.sha256(p.read_bytes()).hexdigest() for p in scripts}}, indent=2)) | 21 | 'scripts': {p.name: hashlib.sha256(p.read_bytes()).hexdigest() for p in scripts}}, indent=2)) |
| 19 | archive = output / 'inputs.zip' | 22 | archive = output / 'inputs.zip' |
| ... | @@ -28,7 +31,7 @@ def run(inputs, output): | ... | @@ -28,7 +31,7 @@ def run(inputs, output): |
| 28 | if result.get('error'): raise RuntimeError(result['error']) | 31 | if result.get('error'): raise RuntimeError(result['error']) |
| 29 | command(name, r'powershell -NoProfile -Command "Expand-Archive C:\one-tests\inputs.zip C:\one-tests\runs\capture\inputs"', output) | 32 | command(name, r'powershell -NoProfile -Command "Expand-Archive C:\one-tests\inputs.zip C:\one-tests\runs\capture\inputs"', output) |
| 30 | command(name, r'powershell -NoProfile -NonInteractive -ExecutionPolicy Bypass -File C:\one-tests\probe.ps1 -Root C:\one-tests\runs\capture -CloneHost ONE-' + name.upper(), output, (len(files) * 60 + 120) * 1000) | 33 | command(name, r'powershell -NoProfile -NonInteractive -ExecutionPolicy Bypass -File C:\one-tests\probe.ps1 -Root C:\one-tests\runs\capture -CloneHost ONE-' + name.upper(), output, (len(files) * 60 + 120) * 1000) |
| 31 | collect_artifacts(name, output, r'results, C:\one-tests\runs\capture\results.json') | 34 | collect_artifacts(name, output, r'results, C:\one-tests\runs\capture\results.json, C:\one-tests\runs\capture\progress.jsonl') |
| 32 | finally: | 35 | finally: |
| 33 | archive.unlink(missing_ok=True) | 36 | archive.unlink(missing_ok=True) |
| 34 | 37 |
tools/native_stress.py+125-43
| ... | @@ -1,5 +1,6 @@ | ... | @@ -1,5 +1,6 @@ |
| 1 | """Overlapping native and Rust editing histories on one shared section.""" | 1 | """Overlapping native and Rust editing histories on one shared section.""" |
| 2 | import json | 2 | import json |
| 3 | import os | ||
| 3 | import time | 4 | import time |
| 4 | 5 | ||
| 5 | from native_runner import windows | 6 | from native_runner import windows |
| ... | @@ -7,28 +8,36 @@ from concurrent_rust import running_clients | ... | @@ -7,28 +8,36 @@ from concurrent_rust import running_clients |
| 7 | from document_model import ordered_pages, walk | 8 | from document_model import ordered_pages, walk |
| 8 | 9 | ||
| 9 | 10 | ||
| 10 | def edit_history(logs, operations, edit=False): | 11 | def edit_history(logs, operations, edit=False, *, partial=False, offline=False): |
| 11 | links = {} | 12 | if offline: |
| 12 | for actor, events in logs.items(): | 13 | assert not edit, 'Offline acceptance currently uses append intents' |
| 13 | assert events[0]['event'] == 'ready' and events[-1]['event'] == 'done', 'Incomplete client log' | 14 | from offline_history import publication_links |
| 14 | if not actor.startswith('w'): continue | 15 | links = publication_links(logs, operations, partial) |
| 15 | intents = {event['attempt']: event for event in events if event['event'] == 'intent'} | 16 | else: |
| 16 | commits = [event for event in events if event['event'] == 'commit'] | 17 | links = {} |
| 17 | assert sorted(event['operation'] for event in commits) == list(range(operations)), 'Missing or duplicate Rust acknowledgements' | 18 | for actor, events in logs.items(): |
| 18 | for event in commits: | 19 | assert events and events[0]['event'] == 'ready', 'Missing client start' |
| 19 | intent = intents[event['attempt']] | 20 | assert partial or events[-1]['event'] == 'done', 'Incomplete client log' |
| 20 | token = f' [{actor}:{event["operation"]}]' | 21 | if not actor.startswith('w'): continue |
| 21 | offset = len(intent['before'].encode('utf-16-le')) // 2 | 22 | intents = {event['attempt']: event for event in events if event['event'] == 'intent'} |
| 22 | assert event['token'] == intent['token'] == token, 'Acknowledgement differs from intended edit' | 23 | commits = [event for event in events if event['event'] == 'commit'] |
| 23 | assert intent['replacement'] == (' café 🦀' if edit else '') + token, 'Unexpected replacement text' | 24 | assert len(commits) <= operations, 'Unexpected Rust acknowledgement' |
| 24 | assert intent['operation'] == event['operation'] and intent['source_transaction'] == event['source_transaction'], 'Acknowledgement used another intent' | 25 | expected = len(commits) if partial else operations |
| 25 | start, end = intent['range'] | 26 | assert sorted(event['operation'] for event in commits) == list(range(expected)), 'Missing or duplicate Rust acknowledgements' |
| 26 | assert len('Concurrent edits:') <= start <= end <= offset, 'Invalid edit range' | 27 | for event in commits: |
| 27 | if not edit: assert start == end == offset, 'Expected an append intent' | 28 | intent = intents[event['attempt']] |
| 28 | units = intent['before'].encode('utf-16-le') | 29 | token = f' [{actor}:{event["operation"]}]' |
| 29 | after = units[:start * 2].decode('utf-16-le') + intent['replacement'] + units[end * 2:].decode('utf-16-le') | 30 | offset = len(intent['before'].encode('utf-16-le')) // 2 |
| 30 | assert intent['before'] not in links, 'Acknowledged Rust edits branched from the same content' | 31 | assert event['token'] == intent['token'] == token, 'Acknowledgement differs from intended edit' |
| 31 | links[intent['before']] = event, after | 32 | assert intent['replacement'] == (' café 🦀' if edit else '') + token, 'Unexpected replacement text' |
| 33 | assert intent['operation'] == event['operation'] and intent['source_transaction'] == event['source_transaction'], 'Acknowledgement used another intent' | ||
| 34 | start, end = intent['range'] | ||
| 35 | assert len('Concurrent edits:') <= start <= end <= offset, 'Invalid edit range' | ||
| 36 | if not edit: assert start == end == offset, 'Expected an append intent' | ||
| 37 | units = intent['before'].encode('utf-16-le') | ||
| 38 | after = units[:start * 2].decode('utf-16-le') + intent['replacement'] + units[end * 2:].decode('utf-16-le') | ||
| 39 | assert intent['before'] not in links, 'Acknowledged Rust edits branched from the same content' | ||
| 40 | links[intent['before']] = event, after | ||
| 32 | text = 'Concurrent edits:' | 41 | text = 'Concurrent edits:' |
| 33 | versions = {text: 0} | 42 | versions = {text: 0} |
| 34 | ordered = [] | 43 | ordered = [] |
| ... | @@ -79,10 +88,15 @@ def verify_capture(output, capture): | ... | @@ -79,10 +88,15 @@ def verify_capture(output, capture): |
| 79 | operations, edit = config['stress_operations'], config['edit'] | 88 | operations, edit = config['stress_operations'], config['edit'] |
| 80 | actors = [f'w{i}' for i in range(config['rust_writers'])] + [f'r{i}' for i in range(config['rust_readers'])] | 89 | actors = [f'w{i}' for i in range(config['rust_writers'])] + [f'r{i}' for i in range(config['rust_readers'])] |
| 81 | logs = {actor: [json.loads(line) for line in (output / 'rust' / f'{actor}.jsonl').read_text().splitlines()] for actor in actors} | 90 | logs = {actor: [json.loads(line) for line in (output / 'rust' / f'{actor}.jsonl').read_text().splitlines()] for actor in actors} |
| 82 | commits, rust_text = edit_history(logs, operations, edit) | 91 | commits, rust_text = edit_history(logs, operations, edit, offline=config.get('offline', False)) |
| 83 | native = [[json.loads(line) for line in (output / f'n{i}' / 'stress-events.jsonl').read_text(encoding='utf-8-sig').splitlines()] | 92 | native = [[json.loads(line) for line in (output / f'n{i}' / 'stress-events.jsonl').read_text(encoding='utf-8-sig').splitlines()] |
| 84 | for i in range(config['stress_clients'])] | 93 | for i in range(config['stress_clients'])] |
| 85 | expected = [rust_text, *(native_history(events, i, operations, edit) for i, events in enumerate(native))] | 94 | expected = [rust_text, *(native_history(events, i, operations, edit) for i, events in enumerate(native))] |
| 95 | documents = {} | ||
| 96 | if config.get('document_operations'): | ||
| 97 | from offline_document_history import document_history | ||
| 98 | documents = document_history(logs, operations) | ||
| 99 | expected.extend(document['text'] for document in documents.values()) | ||
| 86 | page_file, = capture.glob('page-*.xml') | 100 | page_file, = capture.glob('page-*.xml') |
| 87 | page = ET.parse(page_file).getroot() | 101 | page = ET.parse(page_file).getroot() |
| 88 | paragraphs = native_characters(page, page.findall('one:Outline', ns)) | 102 | paragraphs = native_characters(page, page.findall('one:Outline', ns)) |
| ... | @@ -95,16 +109,31 @@ def verify_capture(output, capture): | ... | @@ -95,16 +109,31 @@ def verify_capture(output, capture): |
| 95 | for field in ('bold', 'italic'): | 109 | for field in ('bold', 'italic'): |
| 96 | assert bool(style.get(field)) == events[-1][field], 'Fresh native formatting differs from its last recorded editing intent' | 110 | assert bool(style.get(field)) == events[-1][field], 'Fresh native formatting differs from its last recorded editing intent' |
| 97 | checks += 1 | 111 | checks += 1 |
| 98 | return {'rust_intents': len(commits), 'native_intents': sum(map(len, native)), | 112 | if documents: |
| 113 | from offline_document_history import verify_native | ||
| 114 | checks += verify_native(paragraphs, documents) | ||
| 115 | return {'rust_intents': len(commits), 'document_intents': len(documents)*2, 'native_intents': sum(map(len, native)), | ||
| 99 | 'exact_paragraphs': len(expected), 'native_intended_format_checks': checks} | 116 | 'exact_paragraphs': len(expected), 'native_intended_format_checks': checks} |
| 100 | 117 | ||
| 101 | 118 | ||
| 102 | def exercise(output, shared, clients, action, wait_action, wait_text, checkpoint, operations, sync_every, rust_writers, rust_readers, edit=False, seed=710): | 119 | def exercise(output, shared, clients, action, wait_action, wait_text, checkpoint, operations, sync_every, rust_writers, rust_readers, edit=False, seed=710, embedded_smb=False): |
| 120 | wait_text(clients[0], ['Concurrent edits:']) | ||
| 103 | action(clients[0], 'prepare-stress', clients=len(clients)) | 121 | action(clients[0], 'prepare-stress', clients=len(clients)) |
| 104 | prefixes = [f'Native {i}:' for i in range(len(clients))] | 122 | prefixes = [f'Native {i}:' for i in range(len(clients))] |
| 105 | for client in clients: | 123 | for client in clients: |
| 106 | wait_text(client, ['Concurrent edits:', *prefixes]) | 124 | wait_text(client, ['Concurrent edits:', *prefixes]) |
| 107 | checkpoint('stress-initial') | 125 | checkpoint('stress-initial') |
| 126 | config = json.loads((output / 'run.json').read_text()) | ||
| 127 | maintenance = config.get('maintenance', False) | ||
| 128 | disconnect = config.get('disconnect', False) | ||
| 129 | offline_outage = config.get('offline_outage', False) | ||
| 130 | offline_lost_reply = config.get('offline_lost_reply', False) | ||
| 131 | if maintenance: | ||
| 132 | (output / 'maintenance-ready').touch() | ||
| 133 | deadline = time.monotonic() + 300 | ||
| 134 | while not (output / 'maintenance-start').exists(): | ||
| 135 | if time.monotonic() > deadline: raise TimeoutError('Maintenance controller did not start the workload') | ||
| 136 | time.sleep(.1) | ||
| 108 | clocks = [] | 137 | clocks = [] |
| 109 | for client in clients: | 138 | for client in clients: |
| 110 | before = time.time_ns() // 1000 | 139 | before = time.time_ns() // 1000 |
| ... | @@ -113,7 +142,7 @@ def exercise(output, shared, clients, action, wait_action, wait_text, checkpoint | ... | @@ -113,7 +142,7 @@ def exercise(output, shared, clients, action, wait_action, wait_text, checkpoint |
| 113 | native = result['utc_us'] | 142 | native = result['utc_us'] |
| 114 | clocks.append({'native_minus_host_us': [native - after - 15625, native - before + 15625]}) | 143 | clocks.append({'native_minus_host_us': [native - after - 15625, native - before + 15625]}) |
| 115 | (output / 'clocks.json').write_text(json.dumps(clocks, indent=2)) | 144 | (output / 'clocks.json').write_text(json.dumps(clocks, indent=2)) |
| 116 | sequences = [action(client, 'stress', wait=False, actor=i, prefix=prefixes[i], operations=operations, seed=seed + 10000 + i, sync_every=sync_every, edit=edit) | 145 | sequences = [action(client, 'stress', wait=False, actor=i, prefix=prefixes[i], operations=operations, seed=seed + 10000 + i, sync_every=sync_every, edit=edit, maintenance=maintenance) |
| 117 | for i, client in enumerate(clients)] | 146 | for i, client in enumerate(clients)] |
| 118 | for client, sequence in zip(clients, sequences): | 147 | for client, sequence in zip(clients, sequences): |
| 119 | deadline = time.monotonic() + 60 | 148 | deadline = time.monotonic() + 60 |
| ... | @@ -125,18 +154,63 @@ def exercise(output, shared, clients, action, wait_action, wait_text, checkpoint | ... | @@ -125,18 +154,63 @@ def exercise(output, shared, clients, action, wait_action, wait_text, checkpoint |
| 125 | folder = output / 'rust' | 154 | folder = output / 'rust' |
| 126 | folder.mkdir() | 155 | folder.mkdir() |
| 127 | start = folder / 'start' | 156 | start = folder / 'start' |
| 128 | with running_clients(folder, shared / 'synthetic.one', rust_writers, rust_readers, operations, seed, edit=edit) as processes: | 157 | from concurrent_rust import ROOT |
| 129 | (shared / 'stress-start').write_text('start') | 158 | binaries = ROOT / 'target' / config.get('client_profile', 'debug') / 'examples' |
| 159 | source = 'm6-collaboration\\synthetic.one' if embedded_smb else shared / 'synthetic.one' | ||
| 160 | executable = binaries / ('smb_concurrent_client' if embedded_smb else 'concurrent_client') | ||
| 161 | if disconnect: executable = binaries / 'smb_reconnect_client' | ||
| 162 | if config.get('offline'): executable = binaries / 'smb_offline_client' | ||
| 163 | environment = {**os.environ, 'ONESTORE_MAINTENANCE_DIR': str(folder)} if maintenance else None | ||
| 164 | reader_executable = None | ||
| 165 | if offline_outage: | ||
| 166 | environment = {**os.environ, 'ONESTORE_OFFLINE_OUTAGE_DIR': str(folder)} | ||
| 167 | reader_executable = binaries / 'smb_reconnect_client' | ||
| 168 | if offline_lost_reply: | ||
| 169 | captures = folder / 'confirmations' | ||
| 170 | captures.mkdir() | ||
| 171 | environment = {**os.environ, 'ONESTORE_OFFLINE_CONFIRM_DIR': str(captures)} | ||
| 172 | reader_executable = binaries / 'smb_reconnect_client' | ||
| 173 | if config.get('document_operations'): | ||
| 174 | environment = {**(environment or os.environ), 'ONESTORE_OFFLINE_DOCUMENTS': '1'} | ||
| 175 | if offline_lost_reply: | ||
| 176 | environment['ONESTORE_OFFLINE_FORMAT_REPLY_DIR'] = str(folder) | ||
| 177 | try: | ||
| 178 | with running_clients(folder, source, rust_writers, rust_readers, operations, seed, timeout=config.get('client_timeout', 600), edit=edit, executable=executable, environment=environment, reader_executable=reader_executable) as processes: | ||
| 179 | (shared / 'stress-start').write_text('start') | ||
| 180 | for client, sequence in zip(clients, sequences): | ||
| 181 | deadline = time.monotonic() + 60 | ||
| 182 | while time.monotonic() < deadline: | ||
| 183 | result = windows.do_cmd(f'if exist C:\\one-tests\\runs\\capture\\outbox\\{sequence}\\editing echo editing', target=client['name']) | ||
| 184 | if 'editing' in result.get('stdout', ''): break | ||
| 185 | time.sleep(.1) | ||
| 186 | else: raise TimeoutError('Native stress client did not acknowledge its first edit') | ||
| 187 | start.touch() | ||
| 188 | if disconnect: | ||
| 189 | from native_disconnect import interrupt | ||
| 190 | interrupt(output, clients, sequences, processes) | ||
| 191 | if offline_outage or offline_lost_reply: | ||
| 192 | from offline_outage import interrupt | ||
| 193 | interrupt(output, clients, sequences, processes) | ||
| 194 | if embedded_smb: | ||
| 195 | import linux_vm | ||
| 196 | server = json.loads((output / 'run.json').read_text())['server'] | ||
| 197 | with (output / 'server-locks.jsonl').open('w') as trace: | ||
| 198 | for _ in range(100): | ||
| 199 | captured = linux_vm.run_ssh(server, 'sudo smbstatus --byterange --json', timeout=5) | ||
| 200 | captured.check_returncode() | ||
| 201 | trace.write(json.dumps(json.loads(captured.stdout)) + '\n') | ||
| 202 | trace.flush() | ||
| 203 | time.sleep(.1) | ||
| 204 | for client, sequence in zip(clients, sequences): | ||
| 205 | wait_action(client, sequence, 'stress') | ||
| 206 | finally: | ||
| 130 | for client, sequence in zip(clients, sequences): | 207 | for client, sequence in zip(clients, sequences): |
| 131 | deadline = time.monotonic() + 60 | 208 | local = client['folder'] / 'stress-events.jsonl' |
| 132 | while time.monotonic() < deadline: | 209 | try: |
| 133 | result = windows.do_cmd(f'if exist C:\\one-tests\\runs\\capture\\outbox\\{sequence}\\editing echo editing', target=client['name']) | 210 | result = windows.do_get(f'C:\\one-tests\\runs\\capture\\outbox\\{sequence}\\events.jsonl', local, client['name']) |
| 134 | if 'editing' in result.get('stdout', ''): break | 211 | except Exception as error: |
| 135 | time.sleep(.1) | 212 | result = {'error': str(error)} |
| 136 | else: raise TimeoutError('Native stress client did not acknowledge its first edit') | 213 | (client['folder'] / 'stress-capture.json').write_text(json.dumps(result, indent=2)) |
| 137 | start.touch() | ||
| 138 | for client, sequence in zip(clients, sequences): | ||
| 139 | wait_action(client, sequence, 'stress') | ||
| 140 | for client, clock in zip(clients, clocks): | 214 | for client, clock in zip(clients, clocks): |
| 141 | before = time.time_ns() // 1000 | 215 | before = time.time_ns() // 1000 |
| 142 | native = windows.do_health(client['name'])['utc_us'] | 216 | native = windows.do_health(client['name'])['utc_us'] |
| ... | @@ -144,23 +218,31 @@ def exercise(output, shared, clients, action, wait_action, wait_text, checkpoint | ... | @@ -144,23 +218,31 @@ def exercise(output, shared, clients, action, wait_action, wait_text, checkpoint |
| 144 | clock['after_native_minus_host_us'] = [native - after - 15625, native - before + 15625] | 218 | clock['after_native_minus_host_us'] = [native - after - 15625, native - before + 15625] |
| 145 | (output / 'clocks.json').write_text(json.dumps(clocks, indent=2)) | 219 | (output / 'clocks.json').write_text(json.dumps(clocks, indent=2)) |
| 146 | rust = {actor: [json.loads(line) for line in (folder / f'{actor}.jsonl').read_text().splitlines()] for actor in processes} | 220 | rust = {actor: [json.loads(line) for line in (folder / f'{actor}.jsonl').read_text().splitlines()] for actor in processes} |
| 147 | commits, expected_rust = edit_history(rust, operations, edit) | 221 | commits, expected_rust = edit_history(rust, operations, edit, offline=config.get('offline', False)) |
| 148 | assert len(commits) == rust_writers * operations, 'Missing Rust acknowledgements' | 222 | assert len(commits) == rust_writers * operations, 'Missing Rust acknowledgements' |
| 149 | native_events = [] | 223 | native_events = [] |
| 150 | for i, (client, sequence) in enumerate(zip(clients, sequences)): | 224 | for client in clients: |
| 151 | local = client['folder'] / 'stress-events.jsonl' | 225 | local = client['folder'] / 'stress-events.jsonl' |
| 152 | result = windows.do_get(f'C:\\one-tests\\runs\\capture\\outbox\\{sequence}\\events.jsonl', local, client['name']) | ||
| 153 | if result.get('error'): raise RuntimeError(str(result)) | ||
| 154 | events = [json.loads(line) for line in local.read_text(encoding='utf-8-sig').splitlines()] | 226 | events = [json.loads(line) for line in local.read_text(encoding='utf-8-sig').splitlines()] |
| 155 | prefixes[i] = native_history(events, i, operations, edit) | ||
| 156 | native_events.append(events) | 227 | native_events.append(events) |
| 228 | prefixes = [native_history(events, i, operations, edit) for i, events in enumerate(native_events)] | ||
| 157 | expected = [expected_rust, *prefixes] | 229 | expected = [expected_rust, *prefixes] |
| 230 | documents = {} | ||
| 231 | if config.get('document_operations'): | ||
| 232 | from offline_document_history import document_history | ||
| 233 | documents = document_history(rust, operations) | ||
| 234 | expected.extend(document['text'] for document in documents.values()) | ||
| 158 | for client in clients: wait_text(client, expected) | 235 | for client in clients: wait_text(client, expected) |
| 159 | checkpoint('stress-final') | 236 | checkpoint('stress-final') |
| 160 | model = json.loads((output / 'stress-final/model/document.json').read_text()) | 237 | model = json.loads((output / 'stress-final/model/document.json').read_text()) |
| 238 | for _, _, revision, _ in ordered_pages(model): | ||
| 239 | assert not revision['nodes'][revision['roots']['1']]['spaces'], 'Disjoint edits created conflict pages' | ||
| 161 | paragraphs = [n['kind']['text'] for _, _, revision, page in ordered_pages(model) | 240 | paragraphs = [n['kind']['text'] for _, _, revision, page in ordered_pages(model) |
| 162 | for _, n in walk(revision, page) if n['kind']['type'] == 'RichText'] | 241 | for _, n in walk(revision, page) if n['kind']['type'] == 'RichText'] |
| 163 | assert sorted(paragraphs) == sorted(expected), 'Final shared state lost, duplicated or added unrecorded content' | 242 | assert sorted(paragraphs) == sorted(expected), 'Final shared state lost, duplicated or added unrecorded content' |
| 243 | if documents: | ||
| 244 | from offline_document_history import verify_model | ||
| 245 | verify_model(model, documents) | ||
| 164 | if edit: | 246 | if edit: |
| 165 | resolved = json.loads((output / 'stress-final/model/text.json').read_text()) | 247 | resolved = json.loads((output / 'stress-final/model/text.json').read_text()) |
| 166 | for i, events in enumerate(native_events): | 248 | for i, events in enumerate(native_events): |
| ... | @@ -180,7 +262,7 @@ def exercise(output, shared, clients, action, wait_action, wait_text, checkpoint | ... | @@ -180,7 +262,7 @@ def exercise(output, shared, clients, action, wait_action, wait_text, checkpoint |
| 180 | earliest_end = (native['updated_ticks'] - 621355968000000000) // 10 - high | 262 | earliest_end = (native['updated_ticks'] - 621355968000000000) // 10 - high |
| 181 | overlap += sum(max(latest_start, rust['started_us']) < min(earliest_end, rust['finished_us']) for rust in commits) | 263 | overlap += sum(max(latest_start, rust['started_us']) < min(earliest_end, rust['finished_us']) for rust in commits) |
| 182 | result = {'native_writers': len(clients), 'rust_writers': rust_writers, 'rust_readers': rust_readers, | 264 | result = {'native_writers': len(clients), 'rust_writers': rust_writers, 'rust_readers': rust_readers, |
| 183 | 'sync_every': sync_every, 'edit': edit, 'seed': seed, 'native_edits': operations * len(clients), 'rust_commits': len(commits), 'rust_reads': len(reads), | 265 | 'sync_every': sync_every, 'edit': edit, 'seed': seed, 'offline': config.get('offline', False), 'native_edits': operations * len(clients), 'rust_commits': len(commits), 'document_commits': len(documents)*2, 'rust_reads': len(reads), |
| 184 | 'clock_bounded_native_rust_call_overlaps': overlap, 'converged_paragraphs': len(expected)} | 266 | 'clock_bounded_native_rust_call_overlaps': overlap, 'converged_paragraphs': len(expected)} |
| 185 | (output / 'result.json').write_text(json.dumps(result, indent=2)) | 267 | (output / 'result.json').write_text(json.dumps(result, indent=2)) |
| 186 | assert overlap, 'No native/Rust call overlap established within clock uncertainty' | 268 | assert overlap, 'No native/Rust call overlap established within clock uncertainty' |
tools/notebook_editor.py created+280
| ... | @@ -0,0 +1,280 @@ | ||
| 1 | #!/usr/bin/env python3 | ||
| 2 | """Serve the HTML diagnostic editor on a new notebook copy.""" | ||
| 3 | import argparse | ||
| 4 | import hashlib | ||
| 5 | from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer | ||
| 6 | import json | ||
| 7 | import mimetypes | ||
| 8 | import os | ||
| 9 | from pathlib import Path | ||
| 10 | import re | ||
| 11 | import shutil | ||
| 12 | import subprocess | ||
| 13 | from threading import Lock | ||
| 14 | import time | ||
| 15 | from urllib.parse import parse_qs, urlsplit | ||
| 16 | import uuid | ||
| 17 | |||
| 18 | from document_model import walk | ||
| 19 | from notebook_report import generate | ||
| 20 | |||
| 21 | ROOT = Path(__file__).resolve().parent.parent | ||
| 22 | BRIDGE = ROOT / 'target/debug/onestore-diagnostic' | ||
| 23 | |||
| 24 | |||
| 25 | def bridge(mode, source, destination, edit=None): | ||
| 26 | try: | ||
| 27 | result = subprocess.run([BRIDGE, mode, source, destination], | ||
| 28 | input=json.dumps(edit) if edit is not None else None, | ||
| 29 | capture_output=True, text=True, timeout=120) | ||
| 30 | if result.returncode: | ||
| 31 | raise RuntimeError(result.stderr or f'Diagnostic process exited {result.returncode}') | ||
| 32 | return json.loads(result.stdout) | ||
| 33 | except (OSError, ValueError, RuntimeError, subprocess.TimeoutExpired) as error: | ||
| 34 | return {'ok': False, 'state': 'Unknown' if mode == 'commit' else 'NotCommitted', | ||
| 35 | 'kind': 'Process', 'error': str(error)} | ||
| 36 | |||
| 37 | |||
| 38 | class Session: | ||
| 39 | def __init__(self, source, output): | ||
| 40 | source = source.resolve(strict=True) | ||
| 41 | self.output = output.resolve() | ||
| 42 | if self.output.is_relative_to(source): | ||
| 43 | raise ValueError('Choose a session directory outside the source notebook.') | ||
| 44 | self.output.mkdir(parents=True, exist_ok=False) | ||
| 45 | self.lock = Lock() | ||
| 46 | hashes = {p.relative_to(source).as_posix(): hashlib.sha256(p.read_bytes()).hexdigest() | ||
| 47 | for p in source.rglob('*') if p.is_file()} | ||
| 48 | shutil.copytree(source, self.output / 'notebook') | ||
| 49 | for name, digest in hashes.items(): | ||
| 50 | copied = self.output / 'notebook' / name | ||
| 51 | if hashlib.sha256(copied.read_bytes()).hexdigest() != digest or hashlib.sha256((source / name).read_bytes()).hexdigest() != digest: | ||
| 52 | raise ValueError('The source changed during copying; start a fresh session.') | ||
| 53 | copied.chmod(copied.stat().st_mode | 0o600) | ||
| 54 | (self.output / 'source.json').write_text(json.dumps({'root': str(source), 'sha256': hashes}, indent=2)) | ||
| 55 | (self.output / 'g').mkdir() | ||
| 56 | (self.output / 'objects').mkdir() | ||
| 57 | self.snapshot() | ||
| 58 | |||
| 59 | def snapshot(self): | ||
| 60 | number = max((int(p.name) for p in (self.output / 'g').iterdir() if p.name.isdecimal()), default=-1) + 1 | ||
| 61 | pending = self.output / 'g' / (str(number) + '-' + uuid.uuid4().hex + '.building') | ||
| 62 | source = pending / 'snapshot' | ||
| 63 | source.mkdir(parents=True) | ||
| 64 | for path in sorted((self.output / 'notebook').rglob('*')): | ||
| 65 | if path.suffix.lower() not in ('.one', '.onetoc2'): continue | ||
| 66 | saved = source / path.relative_to(self.output / 'notebook') | ||
| 67 | saved.parent.mkdir(parents=True, exist_ok=True) | ||
| 68 | result = bridge('snapshot', path, saved) | ||
| 69 | if not result['ok']: raise RuntimeError(result['error']) | ||
| 70 | digest = hashlib.sha256(saved.read_bytes()).hexdigest() | ||
| 71 | blob = self.output / 'objects' / digest | ||
| 72 | if blob.exists(): saved.unlink() | ||
| 73 | else: | ||
| 74 | saved.rename(blob) | ||
| 75 | blob.chmod(0o444) | ||
| 76 | os.link(blob, saved) | ||
| 77 | previous = self.output / 'g' / str(number - 1) / 'report' if number else None | ||
| 78 | generate(source, pending / 'report', editable=True, previous=previous) | ||
| 79 | pending.rename(self.output / 'g' / str(number)) | ||
| 80 | return number | ||
| 81 | |||
| 82 | def page(self, generation, page): | ||
| 83 | if type(generation) is not int or generation < 0: | ||
| 84 | raise ValueError('Choose a page from this report.') | ||
| 85 | folder = self.output / 'g' / str(generation) | ||
| 86 | pages = json.loads((folder / 'report/pages.json').read_text()) | ||
| 87 | row = next(p for p in pages if p['report'] == page) | ||
| 88 | if row['category'] != 'Page': | ||
| 89 | raise ValueError('Choose an active page. Conflicts, templates, deleted pages and history are read-only here.') | ||
| 90 | sources = json.loads((folder / 'report/source.json').read_text()) | ||
| 91 | index = next(i for i, source in enumerate(sources) if source['path'] == row['section']) | ||
| 92 | model = folder / 'report/model' / str(index) | ||
| 93 | document = json.loads((model / 'document.json').read_text()) | ||
| 94 | revision = document['spaces'][row['space']]['revisions'][row['revision']] | ||
| 95 | return row, folder / 'snapshot' / row['section'], revision, model | ||
| 96 | |||
| 97 | def selection(self, generation, page, oid, run): | ||
| 98 | if type(run) is not int or run < 0: | ||
| 99 | raise ValueError('Choose a text run from this report.') | ||
| 100 | row, source, revision, model = self.page(generation, page) | ||
| 101 | node = next(node for key, node in walk(revision, row['object']) if key == oid) | ||
| 102 | if node['kind']['type'] != 'RichText': raise ValueError('Choose a text run.') | ||
| 103 | selected = node['kind']['runs'][run] | ||
| 104 | text = json.loads((model / 'text.json').read_text())[row['space']][row['revision']][oid][run]['text'] | ||
| 105 | request = {'space': row['space'], 'object': oid, | ||
| 106 | 'action': {'type': 'Text', 'start': selected['start'], 'end': selected['end'], 'replacement': text}} | ||
| 107 | return row, source, request | ||
| 108 | |||
| 109 | def location(self, generation, row=None): | ||
| 110 | page = 'index.html' | ||
| 111 | if row: | ||
| 112 | pages = json.loads((self.output / 'g' / str(generation) / 'report/pages.json').read_text()) | ||
| 113 | page = next((p['report'] for p in pages if (p['section'], p['space'], p['object'], p['context']) == | ||
| 114 | (row['section'], row['space'], row['object'], row['context'])), page) | ||
| 115 | return f'/g/{generation}/report/{page}' | ||
| 116 | |||
| 117 | def save(self, data): | ||
| 118 | fields = {'text': {'run', 'replacement'}, 'format': {'run', 'start', 'end', 'attributes'}, | ||
| 119 | 'paragraph': {'before', 'text', 'author'}, 'outline': {'x', 'y', 'text', 'author'}} | ||
| 120 | action = data.get('action') | ||
| 121 | if action not in fields or set(data) != {'generation', 'page', 'object', 'action'} | fields[action]: | ||
| 122 | raise ValueError('Choose text, formatting, a paragraph or an outline to save.') | ||
| 123 | if action in ('text', 'format'): | ||
| 124 | row, source, edit = self.selection(data['generation'], data['page'], data['object'], data['run']) | ||
| 125 | selected = edit['action'] | ||
| 126 | if action == 'text': | ||
| 127 | if not isinstance(data['replacement'], str): raise ValueError('Enter replacement text.') | ||
| 128 | selected['replacement'] = data['replacement'] | ||
| 129 | else: | ||
| 130 | if (type(data['start']) is not int or type(data['end']) is not int | ||
| 131 | or not 0 <= data['start'] <= data['end'] <= selected['end'] - selected['start']): | ||
| 132 | raise ValueError('Select text within this run.') | ||
| 133 | edit['action'] = {'type': 'Format', 'start': selected['start'] + data['start'], | ||
| 134 | 'end': selected['start'] + data['end'], 'attributes': data['attributes']} | ||
| 135 | else: | ||
| 136 | row, source, revision, _ = self.page(data['generation'], data['page']) | ||
| 137 | if data['object'] not in {oid for oid, _ in walk(revision, row['object'])}: | ||
| 138 | raise ValueError('Choose a container on this page.') | ||
| 139 | edit = {'space': row['space'], 'object': data['object'], | ||
| 140 | 'action': {'type': action.title(), **{key: data[key] for key in fields[action]}}} | ||
| 141 | operation = uuid.uuid4().hex | ||
| 142 | result = {'ok': False, 'state': 'NotCommitted'} | ||
| 143 | try: | ||
| 144 | with (self.output / 'operations.jsonl').open('a') as log: | ||
| 145 | intent = {'event': 'intent', 'operation': operation, 'started_ms': time.time_ns() // 1000000, | ||
| 146 | 'selection': {key: data[key] for key in ('generation', 'page', 'object', 'action')}, | ||
| 147 | 'edit': edit, 'section': row['section']} | ||
| 148 | log.write(json.dumps(intent, ensure_ascii=True) + '\n') | ||
| 149 | log.flush(); os.fsync(log.fileno()) | ||
| 150 | result = {'ok': False, 'state': 'Unknown'} | ||
| 151 | result = bridge('commit', self.output / 'notebook' / row['section'], source, edit) | ||
| 152 | log.write(json.dumps({'event': 'outcome', 'operation': operation, 'finished_ms': time.time_ns() // 1000000, **result}) + '\n') | ||
| 153 | log.flush(); os.fsync(log.fileno()) | ||
| 154 | except Exception as error: | ||
| 155 | result = {**result, 'ok': False, 'error': str(error)} | ||
| 156 | try: | ||
| 157 | generation = self.snapshot() | ||
| 158 | result['location'] = self.location(generation, row) | ||
| 159 | except Exception as error: | ||
| 160 | result = {**result, 'ok': False, 'report_error': str(error)} | ||
| 161 | return result | ||
| 162 | |||
| 163 | |||
| 164 | class Handler(BaseHTTPRequestHandler): | ||
| 165 | def redirect(self, location): | ||
| 166 | self.send_response(302) | ||
| 167 | self.send_header('Location', location) | ||
| 168 | self.send_header('Cache-Control', 'no-store') | ||
| 169 | self.send_header('Content-Length', '0') | ||
| 170 | self.end_headers() | ||
| 171 | |||
| 172 | def reply(self, status, value): | ||
| 173 | content = json.dumps(value, ensure_ascii=True).encode() | ||
| 174 | self.send_response(status) | ||
| 175 | self.send_header('Content-Type', 'application/json') | ||
| 176 | self.send_header('Content-Length', str(len(content))) | ||
| 177 | self.send_header('Cache-Control', 'no-store') | ||
| 178 | self.end_headers() | ||
| 179 | self.wfile.write(content) | ||
| 180 | |||
| 181 | def do_GET(self): | ||
| 182 | session = self.server.session | ||
| 183 | url = urlsplit(self.path) | ||
| 184 | try: | ||
| 185 | if url.path == '/': | ||
| 186 | latest = max(int(p.name) for p in (session.output / 'g').iterdir() if p.name.isdecimal()) | ||
| 187 | self.redirect(session.location(latest)) | ||
| 188 | return | ||
| 189 | if url.path in ('/api/run', '/api/page', '/latest'): | ||
| 190 | query = parse_qs(url.query, strict_parsing=True) | ||
| 191 | if url.path == '/latest': | ||
| 192 | row = session.page(int(query['generation'][0]), query['page'][0])[0] if query else None | ||
| 193 | with session.lock: | ||
| 194 | self.redirect(session.location(session.snapshot(), row)) | ||
| 195 | return | ||
| 196 | if url.path == '/api/page': | ||
| 197 | row, _, revision, _ = session.page(int(query['generation'][0]), query['page'][0]) | ||
| 198 | def label(oid): | ||
| 199 | node = revision['nodes'][oid] | ||
| 200 | text = ' '.join(n['kind']['text'] for _, n in walk(revision, oid) if n['kind']['type'] == 'RichText') | ||
| 201 | return node['kind']['type'] + (' · ' + text[:80] if text else '') | ||
| 202 | targets = [] | ||
| 203 | pending = [row['object']] | ||
| 204 | while pending: | ||
| 205 | oid = pending.pop() | ||
| 206 | node = revision['nodes'][oid] | ||
| 207 | if node['kind']['type'] == 'Title': continue | ||
| 208 | if node['kind']['type'] in ('Outline', 'Paragraph', 'OutlineGroup', 'Cell'): | ||
| 209 | targets.append({'object': oid, 'label': label(oid), | ||
| 210 | 'children': [{'object': child, 'label': label(child)} for child in node['children']]}) | ||
| 211 | pending.extend(reversed(node['structure'] + node['content'] + node['children'])) | ||
| 212 | self.reply(200, {'ok': True, 'object': row['object'], 'targets': targets}) | ||
| 213 | return | ||
| 214 | row, source, edit = session.selection(int(query['generation'][0]), query['page'][0], query['object'][0], int(query['run'][0])) | ||
| 215 | result = bridge('check', source, '-', edit) | ||
| 216 | self.reply(200 if result['ok'] else 422, {**result, 'text': edit['action']['replacement']}) | ||
| 217 | return | ||
| 218 | if url.path in ('/editor.js', '/editor.css'): | ||
| 219 | path = ROOT / 'tools/diagnostic' / url.path[1:] | ||
| 220 | else: | ||
| 221 | match = re.fullmatch(r'/g/(\d+)/report/(.+)', url.path) | ||
| 222 | if not match: raise FileNotFoundError() | ||
| 223 | root = session.output / 'g' / match[1] / 'report' | ||
| 224 | path = (root / match[2]).resolve(strict=True) | ||
| 225 | if not path.is_relative_to(root): raise FileNotFoundError() | ||
| 226 | with path.open('rb') as file: | ||
| 227 | self.send_response(200) | ||
| 228 | self.send_header('Content-Type', mimetypes.guess_type(path)[0] or 'application/octet-stream') | ||
| 229 | self.send_header('Content-Length', str(os.fstat(file.fileno()).st_size)) | ||
| 230 | self.send_header('Cache-Control', 'no-store') | ||
| 231 | self.end_headers() | ||
| 232 | shutil.copyfileobj(file, self.wfile) | ||
| 233 | except (ValueError, KeyError, IndexError, StopIteration) as error: | ||
| 234 | self.reply(422, {'ok': False, 'error': str(error) or 'The selected text is unavailable.'}) | ||
| 235 | except OSError: | ||
| 236 | self.reply(404, {'ok': False, 'error': 'Report unavailable. Open the notebook index.'}) | ||
| 237 | except Exception as error: | ||
| 238 | self.reply(503, {'ok': False, 'error': str(error)}) | ||
| 239 | |||
| 240 | def do_POST(self): | ||
| 241 | session = self.server.session | ||
| 242 | host = self.headers.get('Host') | ||
| 243 | allowed = {f'127.0.0.1:{self.server.server_port}', f'localhost:{self.server.server_port}'} | ||
| 244 | if host not in allowed or self.headers.get('Origin', 'http://' + host) != 'http://' + host or self.headers.get('X-OneNote-Diagnostic') != '1': | ||
| 245 | self.reply(403, {'ok': False, 'error': 'Open this editor on its local address.'}) | ||
| 246 | return | ||
| 247 | if self.path != '/api/save': | ||
| 248 | self.reply(404, {'ok': False, 'error': 'Unknown diagnostic action.'}) | ||
| 249 | return | ||
| 250 | try: | ||
| 251 | length = int(self.headers.get('Content-Length', '0')) | ||
| 252 | if not 0 < length <= 65536 or self.headers.get('Content-Type') != 'application/json': | ||
| 253 | raise ValueError('Send a JSON edit smaller than 64 KiB.') | ||
| 254 | data = json.loads(self.rfile.read(length)) | ||
| 255 | with session.lock: | ||
| 256 | result = session.save(data) | ||
| 257 | status = 200 if result['ok'] else 409 if result.get('kind') == 'ResourceBusy' else 422 if result.get('kind') in ('InvalidData', 'Input') else 503 | ||
| 258 | self.reply(status, result) | ||
| 259 | except (ValueError, KeyError, IndexError, StopIteration, TypeError) as error: | ||
| 260 | self.reply(422, {'ok': False, 'state': 'NotCommitted', 'error': str(error) or 'The selected text is unavailable.'}) | ||
| 261 | except Exception as error: | ||
| 262 | self.reply(503, {'ok': False, 'state': 'Unknown' if self.path == '/api/save' else 'NotCommitted', 'error': str(error)}) | ||
| 263 | |||
| 264 | |||
| 265 | if __name__ == '__main__': | ||
| 266 | parser = argparse.ArgumentParser(description=__doc__) | ||
| 267 | parser.add_argument('source', type=Path) | ||
| 268 | parser.add_argument('session', type=Path) | ||
| 269 | parser.add_argument('--port', type=int, default=8782) | ||
| 270 | args = parser.parse_args() | ||
| 271 | server = ThreadingHTTPServer(('127.0.0.1', args.port), Handler) | ||
| 272 | server.session = Session(args.source, args.session) | ||
| 273 | print(f'Diagnostic editor: http://127.0.0.1:{server.server_port}/', flush=True) | ||
| 274 | print(f'Editable notebook copy: {server.session.output / "notebook"}', flush=True) | ||
| 275 | try: | ||
| 276 | server.serve_forever() | ||
| 277 | except KeyboardInterrupt: | ||
| 278 | pass | ||
| 279 | finally: | ||
| 280 | server.server_close() | ||
tools/notebook_report.py+37-12
| ... | @@ -6,6 +6,7 @@ from datetime import datetime, timedelta, timezone | ... | @@ -6,6 +6,7 @@ from datetime import datetime, timedelta, timezone |
| 6 | import hashlib | 6 | import hashlib |
| 7 | from html import escape as esc | 7 | from html import escape as esc |
| 8 | import json | 8 | import json |
| 9 | import os | ||
| 9 | from io import BytesIO | 10 | from io import BytesIO |
| 10 | from pathlib import Path, PureWindowsPath | 11 | from pathlib import Path, PureWindowsPath |
| 11 | from PIL import Image | 12 | from PIL import Image |
| ... | @@ -64,13 +65,16 @@ def css(fmt): | ... | @@ -64,13 +65,16 @@ def css(fmt): |
| 64 | return ';'.join(rules) | 65 | return ';'.join(rules) |
| 65 | 66 | ||
| 66 | 67 | ||
| 67 | def html_page(title, nav, body): | 68 | def html_page(title, nav, body, editable=False): |
| 69 | policy = "default-src 'none'; img-src 'self' data:; style-src 'self' 'unsafe-inline'; media-src 'self'; base-uri 'none'" | ||
| 70 | if editable: | ||
| 71 | policy += "; script-src 'self'; connect-src 'self'" | ||
| 68 | return '''<!doctype html><html lang="en"><meta charset="utf-8"> | 72 | return '''<!doctype html><html lang="en"><meta charset="utf-8"> |
| 69 | <meta name="viewport" content="width=device-width,initial-scale=1"> | 73 | <meta name="viewport" content="width=device-width,initial-scale=1"> |
| 70 | <meta http-equiv="Content-Security-Policy" content="default-src 'none'; img-src 'self' data:; style-src 'unsafe-inline'; media-src 'self'; base-uri 'none'"> | 74 | <meta http-equiv="Content-Security-Policy" content="''' + policy + '''"> |
| 71 | <title>''' + esc(title) + '''</title><style> | 75 | <title>''' + esc(title) + '''</title><style> |
| 72 | *{box-sizing:border-box}body{margin:0;color:#000;background:#fff;font:15px/1.5 system-ui,sans-serif}a{color:#175bb2}nav{position:fixed;inset:0 auto 0 0;width:240px;overflow:auto;background:#f5f6f7;padding:20px}nav a{display:block;padding:3px 0;overflow-wrap:anywhere}nav h2{font-size:14px;margin:20px 0 4px}main{margin-left:240px;padding:30px 40px;max-width:1250px}h1{font-size:28px;line-height:1.2}h2{font-size:18px}p{margin:6px 0}.outline{margin:24px 0;border-top:1px solid #d9dde2;padding-top:12px}.location,.meta{font:12px/1.5 system-ui;color:#666;margin:6px 0}.paragraph{min-height:1.3em;position:relative;overflow-wrap:anywhere}.nested{margin-left:24px}.text{white-space:pre-wrap}.tag{display:inline-block;font:12px system-ui;padding:2px 5px;border:1px solid #aaa;border-radius:3px;margin-right:5px}.list-marker{display:inline-block;min-width:22px;margin-left:-22px}.listed{margin-left:22px}img.content{max-width:100%;height:auto;vertical-align:top}figure{margin:12px 0}figcaption{font-size:12px;color:#666}table{border-collapse:collapse;margin:8px 0;max-width:100%}td{border:1px solid #bbb;padding:5px 8px;vertical-align:top;min-width:30px}table.no-borders td{border-color:transparent}.opaque{border:1px dashed #9aa2ad;padding:12px;margin:12px 0;background:#f7f8fa}details{margin:14px 0}summary{cursor:pointer;font:13px system-ui}pre{white-space:pre-wrap;overflow-wrap:anywhere;font:11px/1.4 ui-monospace,monospace;max-height:560px;overflow:auto}.page-link{padding-left:12px}code{font-size:12px}.references a{margin-right:14px}@media(max-width:750px){nav{position:static;width:auto;max-height:220px}main{margin:0;padding:20px}}@media print{nav{display:none}main{margin:0}details{display:none}} | 76 | *{box-sizing:border-box}body{margin:0;color:#000;background:#fff;font:15px/1.5 system-ui,sans-serif}a{color:#175bb2}nav{position:fixed;inset:0 auto 0 0;width:240px;overflow:auto;background:#f5f6f7;padding:20px}nav a{display:block;padding:3px 0;overflow-wrap:anywhere}nav h2{font-size:14px;margin:20px 0 4px}main{margin-left:240px;padding:30px 40px;max-width:1250px}h1{font-size:28px;line-height:1.2}h2{font-size:18px}p{margin:6px 0}.outline{margin:24px 0;border-top:1px solid #d9dde2;padding-top:12px}.location,.meta{font:12px/1.5 system-ui;color:#666;margin:6px 0}.paragraph{min-height:1.3em;position:relative;overflow-wrap:anywhere}.nested{margin-left:24px}.text{white-space:pre-wrap}.tag{display:inline-block;font:12px system-ui;padding:2px 5px;border:1px solid #aaa;border-radius:3px;margin-right:5px}.list-marker{display:inline-block;min-width:22px;margin-left:-22px}.listed{margin-left:22px}img.content{max-width:100%;height:auto;vertical-align:top}figure{margin:12px 0}figcaption{font-size:12px;color:#666}table{border-collapse:collapse;margin:8px 0;max-width:100%}td{border:1px solid #bbb;padding:5px 8px;vertical-align:top;min-width:30px}table.no-borders td{border-color:transparent}.opaque{border:1px dashed #9aa2ad;padding:12px;margin:12px 0;background:#f7f8fa}details{margin:14px 0}summary{cursor:pointer;font:13px system-ui}pre{white-space:pre-wrap;overflow-wrap:anywhere;font:11px/1.4 ui-monospace,monospace;max-height:560px;overflow:auto}.page-link{padding-left:12px}code{font-size:12px}.references a{margin-right:14px}@media(max-width:750px){nav{position:static;width:auto;max-height:220px}main{margin:0;padding:20px}}@media print{nav{display:none}main{margin:0}details{display:none}} |
| 73 | </style><nav>''' + nav + '</nav><main>' + body + '</main></html>' | 77 | </style>''' + ('<link rel="stylesheet" href="/editor.css"><script src="/editor.js" defer></script>' if editable else '') + '<nav>' + nav + '</nav><main>' + body + '</main></html>' |
| 74 | 78 | ||
| 75 | 79 | ||
| 76 | class Page: | 80 | class Page: |
| ... | @@ -117,7 +121,7 @@ class Page: | ... | @@ -117,7 +121,7 @@ class Page: |
| 117 | body = '' | 121 | body = '' |
| 118 | structured_math = any(c in kind['text'] for c in '\ufdd0\ufdee\ufdef') | 122 | structured_math = any(c in kind['text'] for c in '\ufdd0\ufdee\ufdef') |
| 119 | equation = False | 123 | equation = False |
| 120 | for run in self.text[oid]: | 124 | for run_index, run in enumerate(self.text[oid]): |
| 121 | if run['format']['hidden']: | 125 | if run['format']['hidden']: |
| 122 | continue | 126 | continue |
| 123 | if structured_math and run['format'].get('math'): | 127 | if structured_math and run['format'].get('math'): |
| ... | @@ -129,7 +133,7 @@ class Page: | ... | @@ -129,7 +133,7 @@ class Page: |
| 129 | style = {**run['format'], **tag_format} | 133 | style = {**run['format'], **tag_format} |
| 130 | if (style['superscript'] or style['subscript']) and style['font_size'] is not None: | 134 | if (style['superscript'] or style['subscript']) and style['font_size'] is not None: |
| 131 | style['font_size'] *= 2 / 3 | 135 | style['font_size'] *= 2 / 3 |
| 132 | fragment = '<span style="' + esc(css(style)) + '">' + esc(run['text']) + '</span>' | 136 | fragment = '<span data-text-object="' + esc(oid) + '" data-run="' + str(run_index) + '" style="' + esc(css(style)) + '">' + esc(run['text']) + '</span>' |
| 133 | if run['format']['superscript']: | 137 | if run['format']['superscript']: |
| 134 | fragment = '<sup>' + fragment + '</sup>' | 138 | fragment = '<sup>' + fragment + '</sup>' |
| 135 | if run['format']['subscript']: | 139 | if run['format']['subscript']: |
| ... | @@ -262,19 +266,28 @@ class Page: | ... | @@ -262,19 +266,28 @@ class Page: |
| 262 | return '<div data-object="' + esc(oid) + '">' + body + '</div>' if typ not in ('RichText', 'Row', 'Cell') else body | 266 | return '<div data-object="' + esc(oid) + '">' + body + '</div>' if typ not in ('RichText', 'Row', 'Cell') else body |
| 263 | 267 | ||
| 264 | 268 | ||
| 265 | def generate(source, destination, native=None, versions=(), zone=timezone.utc): | 269 | def generate(source, destination, native=None, versions=(), zone=timezone.utc, editable=False, previous=None): |
| 266 | source = source.resolve(strict=True) | 270 | source = source.resolve(strict=True) |
| 267 | if destination.resolve().is_relative_to(source): | 271 | if destination.resolve().is_relative_to(source): |
| 268 | raise ValueError('Choose an export directory outside the source notebook.') | 272 | raise ValueError('Choose an export directory outside the source notebook.') |
| 269 | destination.mkdir(parents=True, exist_ok=False) | 273 | destination.mkdir(parents=True, exist_ok=False) |
| 270 | (destination / 'model').mkdir(); (destination / 'assets').mkdir() | 274 | (destination / 'model').mkdir(); (destination / 'assets').mkdir() |
| 275 | cached = {row['path']: (row['sha256'], previous / 'model' / str(index)) | ||
| 276 | for index, row in enumerate(json.loads((previous / 'source.json').read_text()))} if previous else {} | ||
| 271 | sections = []; tocs = {}; manifest = [] | 277 | sections = []; tocs = {}; manifest = [] |
| 272 | for index, path in enumerate(sorted(p for p in source.rglob('*') if p.suffix.lower() in ('.one', '.onetoc2'))): | 278 | for index, path in enumerate(sorted(p for p in source.rglob('*') if p.suffix.lower() in ('.one', '.onetoc2'))): |
| 273 | relative = path.relative_to(source) | 279 | relative = path.relative_to(source) |
| 274 | before = path.read_bytes() | 280 | before = path.read_bytes() |
| 275 | manifest.append({'path': relative.as_posix(), 'sha256': hashlib.sha256(before).hexdigest(), 'bytes': len(before)}) | 281 | manifest.append({'path': relative.as_posix(), 'sha256': hashlib.sha256(before).hexdigest(), 'bytes': len(before)}) |
| 276 | exported = destination / 'model' / str(index) | 282 | exported = destination / 'model' / str(index) |
| 277 | subprocess.run([EXPORTER, path, exported], check=True) | 283 | reusable = cached.get(relative.as_posix()) |
| 284 | reused = reusable is not None and reusable[0] == manifest[-1]['sha256'] | ||
| 285 | if reused: | ||
| 286 | exported.mkdir() | ||
| 287 | for name in ('document.json', 'text.json', 'assets.json'): | ||
| 288 | os.link(reusable[1] / name, exported / name) | ||
| 289 | else: | ||
| 290 | subprocess.run([EXPORTER, path, exported], check=True) | ||
| 278 | document = json.loads((exported / 'document.json').read_text()) | 291 | document = json.loads((exported / 'document.json').read_text()) |
| 279 | if path.read_bytes() != before: | 292 | if path.read_bytes() != before: |
| 280 | raise ValueError('A source file changed during export.') | 293 | raise ValueError('A source file changed during export.') |
| ... | @@ -291,6 +304,18 @@ def generate(source, destination, native=None, versions=(), zone=timezone.utc): | ... | @@ -291,6 +304,18 @@ def generate(source, destination, native=None, versions=(), zone=timezone.utc): |
| 291 | } | 304 | } |
| 292 | rows = json.loads((exported / 'assets.json').read_text()) | 305 | rows = json.loads((exported / 'assets.json').read_text()) |
| 293 | for asset in rows: | 306 | for asset in rows: |
| 307 | reference = json.dumps(asset['reference'], sort_keys=True) | ||
| 308 | if reused: | ||
| 309 | name = Path(asset['path']).name | ||
| 310 | assets[reference] = 'assets/' + name | ||
| 311 | if not (destination / 'assets' / name).exists(): | ||
| 312 | os.link(previous / 'assets' / name, destination / 'assets' / name) | ||
| 313 | preview = name + '.png' | ||
| 314 | if name.endswith('.tiff') and reference in image_references: | ||
| 315 | if not (destination / 'assets' / preview).exists(): | ||
| 316 | os.link(previous / 'assets' / preview, destination / 'assets' / preview) | ||
| 317 | previews['assets/' + name] = 'assets/' + preview | ||
| 318 | continue | ||
| 294 | original = exported / asset['path']; data = original.read_bytes() | 319 | original = exported / asset['path']; data = original.read_bytes() |
| 295 | extension = '.png' if data.startswith(b'\x89PNG') else '.jpg' if data.startswith(b'\xff\xd8') else '.gif' if data.startswith(b'GIF8') else '.bmp' if data.startswith(b'BM') else '.tiff' if data.startswith((b'II*\0', b'MM\0*')) else '.bin' | 320 | extension = '.png' if data.startswith(b'\x89PNG') else '.jpg' if data.startswith(b'\xff\xd8') else '.gif' if data.startswith(b'GIF8') else '.bmp' if data.startswith(b'BM') else '.tiff' if data.startswith((b'II*\0', b'MM\0*')) else '.bin' |
| 296 | name = hashlib.sha256(data).hexdigest() + extension | 321 | name = hashlib.sha256(data).hexdigest() + extension |
| ... | @@ -300,7 +325,6 @@ def generate(source, destination, native=None, versions=(), zone=timezone.utc): | ... | @@ -300,7 +325,6 @@ def generate(source, destination, native=None, versions=(), zone=timezone.utc): |
| 300 | else: | 325 | else: |
| 301 | original.rename(target) | 326 | original.rename(target) |
| 302 | asset['path'] = '../../assets/' + name | 327 | asset['path'] = '../../assets/' + name |
| 303 | reference = json.dumps(asset['reference'], sort_keys=True) | ||
| 304 | assets[reference] = 'assets/' + name | 328 | assets[reference] = 'assets/' + name |
| 305 | if extension == '.tiff' and reference in image_references: | 329 | if extension == '.tiff' and reference in image_references: |
| 306 | preview = name + '.png' | 330 | preview = name + '.png' |
| ... | @@ -309,8 +333,9 @@ def generate(source, destination, native=None, versions=(), zone=timezone.utc): | ... | @@ -309,8 +333,9 @@ def generate(source, destination, native=None, versions=(), zone=timezone.utc): |
| 309 | raise ValueError('Multipage TIFF requires frame interpretation before report generation.') | 333 | raise ValueError('Multipage TIFF requires frame interpretation before report generation.') |
| 310 | image.convert('RGBA').save(destination / 'assets' / preview) | 334 | image.convert('RGBA').save(destination / 'assets' / preview) |
| 311 | previews['assets/' + name] = 'assets/' + preview | 335 | previews['assets/' + name] = 'assets/' + preview |
| 312 | (exported / 'assets').rmdir() | 336 | if not reused: |
| 313 | (exported / 'assets.json').write_text(json.dumps(rows, indent=2)) | 337 | (exported / 'assets').rmdir() |
| 338 | (exported / 'assets.json').write_text(json.dumps(rows, indent=2)) | ||
| 314 | if path.suffix.lower() == '.one': | 339 | if path.suffix.lower() == '.one': |
| 315 | sections.append({'path': relative, 'export': exported.relative_to(destination), 'document': document, | 340 | sections.append({'path': relative, 'export': exported.relative_to(destination), 'document': document, |
| 316 | 'text': json.loads((exported / 'text.json').read_text()), 'assets': assets, 'previews': previews}) | 341 | 'text': json.loads((exported / 'text.json').read_text()), 'assets': assets, 'previews': previews}) |
| ... | @@ -443,14 +468,14 @@ def generate(source, destination, native=None, versions=(), zone=timezone.utc): | ... | @@ -443,14 +468,14 @@ def generate(source, destination, native=None, versions=(), zone=timezone.utc): |
| 443 | body += page.render(oid) | 468 | body += page.render(oid) |
| 444 | body += '<details><summary>Document structure and source identities</summary><pre>' + esc(json.dumps(page.space, indent=2, ensure_ascii=False)) + '</pre></details>' | 469 | body += '<details><summary>Document structure and source identities</summary><pre>' + esc(json.dumps(page.space, indent=2, ensure_ascii=False)) + '</pre></details>' |
| 445 | body += '<p class="references"><a href="' + section['export'].as_posix() + '/document.json">Document JSON</a><a href="' + section['export'].as_posix() + '/assets.json">Asset references</a></p>' | 470 | body += '<p class="references"><a href="' + section['export'].as_posix() + '/document.json">Document JSON</a><a href="' + section['export'].as_posix() + '/assets.json">Asset references</a></p>' |
| 446 | (destination / filename).write_text(html_page(title, nav, body)) | 471 | (destination / filename).write_text(html_page(title, nav, body, editable and category == 'Page')) |
| 447 | accounting.append({'section': str(section['path']), 'ordinal': ordinal, 'space': sid, 'revision': rid, 'object': oid, 'title': title, 'report': filename, 'category': category, 'context': context, 'version_modified': version['modified'] if version else None, 'source_report': source_page, 'native_reference': reference.as_posix() if reference else None, 'rendered': dict(page.counts)}) | 472 | accounting.append({'section': str(section['path']), 'ordinal': ordinal, 'space': sid, 'revision': rid, 'object': oid, 'title': title, 'report': filename, 'category': category, 'context': context, 'version_modified': version['modified'] if version else None, 'source_report': source_page, 'native_reference': reference.as_posix() if reference else None, 'rendered': dict(page.counts)}) |
| 448 | (destination / 'source.json').write_text(json.dumps(manifest, indent=2)) | 473 | (destination / 'source.json').write_text(json.dumps(manifest, indent=2)) |
| 449 | (destination / 'pages.json').write_text(json.dumps(accounting, indent=2, ensure_ascii=False)) | 474 | (destination / 'pages.json').write_text(json.dumps(accounting, indent=2, ensure_ascii=False)) |
| 450 | intro = '<h1>Notebook review</h1><p>' + str(len(sections)) + ' sections · ' + str(len(pages)) + ' stored pages</p><p>Readable content follows the stored object order. Outline positions are shown in points. The document structure retains properties and identities that the readable view does not interpret.</p><p><a href="source.json">Source hashes</a> · <a href="pages.json">Page inventory</a></p>' | 475 | intro = '<h1>Notebook review</h1><p>' + str(len(sections)) + ' sections · ' + str(len(pages)) + ' stored pages</p><p>Readable content follows the stored object order. Outline positions are shown in points. The document structure retains properties and identities that the readable view does not interpret.</p><p><a href="source.json">Source hashes</a> · <a href="pages.json">Page inventory</a></p>' |
| 451 | if locked_sections: | 476 | if locked_sections: |
| 452 | intro += '<p>Page counts are unavailable for locked sections: ' + esc(', '.join(locked_sections)) + '.</p>' | 477 | intro += '<p>Page counts are unavailable for locked sections: ' + esc(', '.join(locked_sections)) + '.</p>' |
| 453 | (destination / 'index.html').write_text(html_page('Notebook review', nav, intro)) | 478 | (destination / 'index.html').write_text(html_page('Notebook review', nav, intro, editable)) |
| 454 | 479 | ||
| 455 | 480 | ||
| 456 | if __name__ == '__main__': | 481 | if __name__ == '__main__': |
tools/offline_cache_crash.py created+130
| ... | @@ -0,0 +1,130 @@ | ||
| 1 | #!/usr/bin/env python3 | ||
| 2 | """Interrupt owned offline-cache writers and verify every retained intent and image.""" | ||
| 3 | import argparse | ||
| 4 | import hashlib | ||
| 5 | import json | ||
| 6 | import os | ||
| 7 | from pathlib import Path | ||
| 8 | import select | ||
| 9 | import signal | ||
| 10 | import subprocess | ||
| 11 | import time | ||
| 12 | |||
| 13 | |||
| 14 | def run(binary, output): | ||
| 15 | output.mkdir(parents=True, exist_ok=False) | ||
| 16 | cache = output / "cache.sqlite" | ||
| 17 | source = os.environ.get("ONESTORE_CACHE_PROBE_SOURCE") | ||
| 18 | source_hash = hashlib.sha256(Path(source).read_bytes()).hexdigest() if source else None | ||
| 19 | payload_bytes = int(os.environ.get("ONESTORE_CACHE_PROBE_BYTES", 2 * 1024 * 1024)) | ||
| 20 | assert 0 < payload_bytes <= 2 * 1024 * 1024 | ||
| 21 | (output / "run.json").write_text(json.dumps({ | ||
| 22 | "binary": str(binary), | ||
| 23 | "binary_sha256": hashlib.sha256(binary.read_bytes()).hexdigest(), | ||
| 24 | "controller_sha256": hashlib.sha256(Path(__file__).read_bytes()).hexdigest(), | ||
| 25 | "payload_bytes": payload_bytes, | ||
| 26 | "operation": os.environ.get("ONESTORE_CACHE_PROBE_OPERATION", "text"), | ||
| 27 | "source": source, | ||
| 28 | "source_sha256": source_hash, | ||
| 29 | }, indent=2)) | ||
| 30 | subprocess.run([binary, "init", cache], check=True, timeout=30) | ||
| 31 | retained = [] | ||
| 32 | retained_ids = [] | ||
| 33 | results = [] | ||
| 34 | delays = [None, "ack", "unack", "journal", "database-write", 0, .001, .01, .02, .04, .08, .16, .32, .64, 1.28, 2.56, None, "ack"] | ||
| 35 | for operation, delay in enumerate(delays, 1): | ||
| 36 | with (output / f"owner-{operation}.stderr").open("w") as stderr: | ||
| 37 | owner = subprocess.Popen([binary, "edit", cache], stdin=subprocess.PIPE, | ||
| 38 | stdout=subprocess.PIPE, stderr=stderr, text=True, bufsize=1) | ||
| 39 | try: | ||
| 40 | def expect(prefix): | ||
| 41 | if not select.select([owner.stdout], [], [], 60)[0]: | ||
| 42 | raise TimeoutError(prefix) | ||
| 43 | actual = owner.stdout.readline().strip() | ||
| 44 | assert actual == prefix or actual.startswith(prefix + " "), (prefix, actual, owner.poll()) | ||
| 45 | return actual | ||
| 46 | |||
| 47 | expect("ready") | ||
| 48 | contender = subprocess.run([binary, "read", cache], capture_output=True, text=True, timeout=15) | ||
| 49 | (output / f"contender-{operation}.stderr").write_text(contender.stderr) | ||
| 50 | assert contender.returncode != 0 and "DatabaseBusy" in contender.stderr, contender | ||
| 51 | before_write = cache.stat().st_mtime_ns | ||
| 52 | owner.stdin.write(f"{operation} {'unack' if delay == 'unack' else 'ack'}\n") | ||
| 53 | owner.stdin.flush() | ||
| 54 | expect(f"editing {operation}") | ||
| 55 | acknowledgement = None | ||
| 56 | journal_observation = None | ||
| 57 | if delay == "ack": | ||
| 58 | acknowledgement = expect(f"ack {operation}") | ||
| 59 | elif delay == "unack": | ||
| 60 | expect(f"durable {operation}") | ||
| 61 | elif delay in ("journal", "database-write"): | ||
| 62 | deadline = time.monotonic() + 10 | ||
| 63 | journal = cache.with_name(cache.name + "-journal") | ||
| 64 | while time.monotonic() < deadline: | ||
| 65 | try: | ||
| 66 | with journal.open("rb") as active: | ||
| 67 | header = active.read(28) | ||
| 68 | size = journal.stat().st_size | ||
| 69 | database_changed = cache.stat().st_mtime_ns != before_write | ||
| 70 | if (header[:8] == bytes.fromhex("d9d505f920a163d7") and | ||
| 71 | (delay == "journal" or database_changed)): | ||
| 72 | journal_observation = {"header": header.hex(), "bytes": size, | ||
| 73 | "database_changed": database_changed} | ||
| 74 | break | ||
| 75 | except FileNotFoundError: | ||
| 76 | pass | ||
| 77 | time.sleep(.0001) | ||
| 78 | assert journal_observation, f"No active {delay} phase observed" | ||
| 79 | elif delay is not None: | ||
| 80 | time.sleep(delay) | ||
| 81 | owner.kill() | ||
| 82 | assert owner.wait(timeout=10) == -signal.SIGKILL, "Writer did not terminate at the requested process cut" | ||
| 83 | extra = owner.stdout.read() | ||
| 84 | if f"ack {operation} " in extra: | ||
| 85 | acknowledgement = extra.strip() | ||
| 86 | read = subprocess.run([binary, "read", cache], check=True, capture_output=True, | ||
| 87 | text=True, timeout=60) | ||
| 88 | actual = json.loads(read.stdout) | ||
| 89 | operations = actual["operations"] | ||
| 90 | ids = actual["ids"] | ||
| 91 | assert operations in (retained, retained + [operation]), (retained, operation, actual) | ||
| 92 | assert ids[:len(retained_ids)] == retained_ids, (retained_ids, actual) | ||
| 93 | if acknowledgement: | ||
| 94 | assert operations[-1] == operation | ||
| 95 | assert ids[-1] == int(acknowledgement.split()[2]) | ||
| 96 | if delay == "unack": | ||
| 97 | assert operations[-1] == operation and acknowledgement is None | ||
| 98 | assert actual["complete_payloads"] | ||
| 99 | results.append({"operation": operation, "delay": delay, | ||
| 100 | "process_exit": owner.returncode, | ||
| 101 | "acknowledged": acknowledgement is not None, | ||
| 102 | "retained": operation in operations, | ||
| 103 | "retained_operations": operations, "ids": ids, | ||
| 104 | "section_bytes": actual["section_bytes"], | ||
| 105 | "complete_payloads": True, "exclusive_owner": True, | ||
| 106 | "journal_observation": journal_observation}) | ||
| 107 | retained, retained_ids = operations, ids | ||
| 108 | (output / "results.json").write_text(json.dumps(results, indent=2)) | ||
| 109 | print(json.dumps(results[-1]), flush=True) | ||
| 110 | finally: | ||
| 111 | if owner.poll() is None: | ||
| 112 | owner.kill() | ||
| 113 | owner.wait(timeout=10) | ||
| 114 | owner.stdin.close() | ||
| 115 | owner.stdout.close() | ||
| 116 | assert any(row["acknowledged"] for row in results) | ||
| 117 | assert any(row["retained"] and not row["acknowledged"] for row in results) | ||
| 118 | assert any(not row["retained"] for row in results) | ||
| 119 | subprocess.run([binary, "read", cache, output / "recovered.one"], check=True, timeout=60) | ||
| 120 | if source: | ||
| 121 | assert hashlib.sha256(Path(source).read_bytes()).hexdigest() == source_hash | ||
| 122 | print(f"Passed {len(results)} offline-cache process interruptions", flush=True) | ||
| 123 | |||
| 124 | |||
| 125 | if __name__ == "__main__": | ||
| 126 | parser = argparse.ArgumentParser(description=__doc__) | ||
| 127 | parser.add_argument("output", type=Path) | ||
| 128 | parser.add_argument("--binary", type=Path, default=Path(__file__).resolve().parents[1] / "target/release/examples/cache_probe") | ||
| 129 | args = parser.parse_args() | ||
| 130 | run(args.binary.resolve(), args.output.resolve()) | ||
tools/offline_document_history.py created+163
| ... | @@ -0,0 +1,163 @@ | ||
| 1 | """Account for offline document intents, receipts, reader states and native formatting.""" | ||
| 2 | import uuid | ||
| 3 | |||
| 4 | from document_model import ordered_pages, walk | ||
| 5 | |||
| 6 | |||
| 7 | def identity(insertion, extension): | ||
| 8 | return '{' + str(uuid.UUID(bytes_le=bytes(insertion['guid']))).upper() + '},' + str(extension) | ||
| 9 | |||
| 10 | |||
| 11 | def characters(observed): | ||
| 12 | actual = [(char, run['bold'], run['size'], run['color']) | ||
| 13 | for run in observed['runs'] for char in run['text']] | ||
| 14 | assert ''.join(row[0] for row in actual) == observed['text'], 'Document runs omit or duplicate text' | ||
| 15 | return actual | ||
| 16 | |||
| 17 | |||
| 18 | def document_history(logs, operations): | ||
| 19 | documents = {} | ||
| 20 | for actor, events in logs.items(): | ||
| 21 | if not actor.startswith('w'): continue | ||
| 22 | assert events[0].get('document_operations') is True, 'Writer omitted document operations' | ||
| 23 | edits = [row for row in events if row['event'] == 'local_document_commit'] | ||
| 24 | assert [(row['operation'], row['kind']) for row in edits] == [ | ||
| 25 | (i, kind) for i in range(operations) for kind in ('insert', 'format')], 'Missing or duplicate document intent' | ||
| 26 | ids = [row['id'] for row in edits] | ||
| 27 | assert ids == sorted(set(ids)), 'Document intent IDs are duplicated or unordered' | ||
| 28 | assert not set(ids) & {row['id'] for row in events if row['event'] == 'local_commit'}, 'Text and document intents share an ID' | ||
| 29 | receipts = [row for row in events if row['event'] == 'document_receipt'] | ||
| 30 | reopened = [row for row in events if row['event'] == 'reopened_document_receipt'] | ||
| 31 | assert [row['id'] for row in receipts] == ids, 'Document receipt inventory differs' | ||
| 32 | assert [(r['id'], r['revision']) for r in reopened] == [(r['id'], r['revision']) for r in receipts], 'Document receipts changed across reopen' | ||
| 33 | linked = {} | ||
| 34 | for intent, receipt in zip(edits, receipts, strict=True): | ||
| 35 | attempts = [row for row in events if row['event'] == 'remote_attempt' and row['revision'] == receipt['revision'] | ||
| 36 | and set(row.get('document_changes') or {}) == {intent['object']}] | ||
| 37 | if not attempts and intent['kind'] == 'format': | ||
| 38 | attempts = [row for row in events if row['event'] == 'remote_attempt' and row['state'] == 'Unknown' | ||
| 39 | and set(row.get('document_changes') or {}) == {intent['object']}] | ||
| 40 | assert len(attempts) == 1, 'Document receipt lacks one publication attempt' | ||
| 41 | attempt, = attempts | ||
| 42 | assert attempt['state'] in ('Committed', 'Unknown'), 'Receipt identifies an unpublished document operation' | ||
| 43 | assert intent['object'] in attempt['documents'] and attempt['document_changes'] == {intent['object']: attempt['documents'][intent['object']]}, 'Document publication changed another target' | ||
| 44 | successful = [row for row in events if row['event'] == 'remote_attempt' | ||
| 45 | and row['state'] in ('Committed', 'Unknown') | ||
| 46 | and row.get('document_changes') == attempt['document_changes']] | ||
| 47 | assert successful == [attempt], 'Document intent was published or attempted uncertainly more than once' | ||
| 48 | assert intent['started_us'] <= attempt['started_us'] <= attempt['finished_us'] <= receipt['at_us'] and intent['started_us'] <= intent['finished_us'] <= receipt['at_us'], 'Document acknowledgement order is invalid' | ||
| 49 | if attempt['state'] == 'Unknown': | ||
| 50 | confirmed, observed = False, None | ||
| 51 | for row in events: | ||
| 52 | if row['event'] == 'read': observed = row | ||
| 53 | if (row['event'] != 'remote_confirm' or row['state'] != 'Committed' | ||
| 54 | or receipt['revision'] not in row.get('revisions', {}).get(intent['space'], []) | ||
| 55 | or not attempt['finished_us'] <= row['started_us'] <= row['finished_us'] <= receipt['at_us']): | ||
| 56 | continue | ||
| 57 | if receipt['revision'] != attempt['revision']: | ||
| 58 | assert intent['kind'] == 'format' and attempt['revision'] not in row['revisions'][intent['space']], 'Replacement receipt did not retire the original attempt' | ||
| 59 | assert row.get('current_revisions', {}).get(intent['space']) == receipt['revision'], 'Effect receipt does not identify the confirmed current revision' | ||
| 60 | assert observed and observed['finished_us'] <= row['started_us'] and observed['text'] == row['text'] | ||
| 61 | assert observed.get('documents', {}).get(intent['object']) == attempt['document_changes'][intent['object']], 'Effect confirmation differs from the uncertain formatting intent' | ||
| 62 | confirmed = True | ||
| 63 | assert confirmed, 'Uncertain document publication lacks confirmation' | ||
| 64 | else: | ||
| 65 | assert receipt['revision'] == attempt['revision'] | ||
| 66 | linked[intent['id']] = {**attempt, 'acknowledged_us': receipt['at_us'], 'receipt_revision': receipt['revision']} | ||
| 67 | for inserted, formatted in zip(edits[::2], edits[1::2], strict=True): | ||
| 68 | number = inserted['operation'] | ||
| 69 | text = f'Document {actor}:{number} 🦀' | ||
| 70 | insertion = inserted['insertion'] | ||
| 71 | target = identity(insertion, 2) | ||
| 72 | assert target == inserted['object'] == formatted['object'], 'Dependent formatting addresses another object' | ||
| 73 | assert inserted['space'] == formatted['space'] and inserted['text'] == formatted['text'] == insertion['text'] == text | ||
| 74 | assert insertion['author'] == 'Offline document writer' | ||
| 75 | if number % 2 == 0: | ||
| 76 | assert insertion['placement'] == {'Outline': {'x': 144 + int(actor[1:]) * 240, 'y': 144 + number * 72}}, 'Outline placement differs from intent' | ||
| 77 | else: | ||
| 78 | assert insertion['placement'] == {'Paragraph': {'before': None}} | ||
| 79 | assert insertion['parent'] == identity(edits[(number-1)*2]['insertion'], 1), 'Paragraph lost its outline parent' | ||
| 80 | assert formatted['range'] == [1, len(text.encode('utf-16-le')) // 2 - 2] | ||
| 81 | assert formatted['attributes'] == [{'Bold': True}, {'FontSize': 18 + number % 9}, {'Color': [18, 52, 86]}] | ||
| 82 | old = [(char, False, 11, 0xff000000) for char in text] | ||
| 83 | new = [(char, True, 18 + number % 9, 0x563412) if 0 < i < len(text)-1 else old[i] for i, char in enumerate(text)] | ||
| 84 | assert target not in documents, 'Two insertion intents share an object identity' | ||
| 85 | created, changed = linked[inserted['id']], linked[formatted['id']] | ||
| 86 | assert created['finished_us'] <= changed['started_us'], 'Formatting preceded its insertion' | ||
| 87 | assert characters(created['documents'][target]) == old, 'Insertion publication differs from its local intent' | ||
| 88 | assert characters(changed['documents'][target]) == new, 'Formatting publication differs from its local intent' | ||
| 89 | documents[target] = {'text': text, 'insertion': insertion, 'space': inserted['space'], | ||
| 90 | 'old': old, 'new': new, 'insert': created, 'format': changed} | ||
| 91 | assert documents, 'No document operations were recorded' | ||
| 92 | for actor, events in logs.items(): | ||
| 93 | previous = {} | ||
| 94 | reads = [row for row in events if row['event'] in ('read', 'document_read') and row.get('documents') is not None] | ||
| 95 | assert reads, f'{actor} did not observe document snapshots' | ||
| 96 | assert any(row['documents'] for row in reads), f'{actor} never observed a created document object' | ||
| 97 | for read in reads: | ||
| 98 | observed = read['documents'] | ||
| 99 | assert set(previous) <= set(observed) <= set(documents), 'Reader lost an object or observed an unrecorded insertion' | ||
| 100 | for target, document in documents.items(): | ||
| 101 | if read['started_us'] > document['insert']['acknowledged_us']: | ||
| 102 | assert target in observed, 'Reader missed an acknowledged insertion' | ||
| 103 | if target not in observed: continue | ||
| 104 | assert read['finished_us'] >= document['insert']['started_us'], 'Reader observed a future insertion' | ||
| 105 | actual = characters(observed[target]) | ||
| 106 | assert actual in (document['old'], document['new']), 'Reader observed partial or invented formatting' | ||
| 107 | formatted = actual == document['new'] | ||
| 108 | assert not previous.get(target, False) or formatted, 'Reader reverted acknowledged formatting' | ||
| 109 | if read['started_us'] > document['format']['acknowledged_us']: | ||
| 110 | assert formatted, 'Reader missed acknowledged formatting' | ||
| 111 | if formatted: | ||
| 112 | assert read['finished_us'] >= document['format']['started_us'], 'Reader observed future formatting' | ||
| 113 | previous[target] = formatted | ||
| 114 | return documents | ||
| 115 | |||
| 116 | |||
| 117 | def verify_model(model, documents): | ||
| 118 | children = {} | ||
| 119 | for document in documents.values(): | ||
| 120 | insertion = document['insertion'] | ||
| 121 | if 'Outline' in insertion['placement']: | ||
| 122 | children[identity(insertion, 1)] = [identity(insertion, 3)] | ||
| 123 | for document in documents.values(): | ||
| 124 | insertion = document['insertion'] | ||
| 125 | if 'Paragraph' in insertion['placement']: | ||
| 126 | children[insertion['parent']].append(identity(insertion, 1)) | ||
| 127 | found = set() | ||
| 128 | for sid, _, revision, page in ordered_pages(model): | ||
| 129 | nodes = revision['nodes'] | ||
| 130 | for target, node in walk(revision, page): | ||
| 131 | if target not in documents: continue | ||
| 132 | assert target not in found, 'Inserted text is reachable twice' | ||
| 133 | found.add(target) | ||
| 134 | expected = documents[target] | ||
| 135 | insertion = expected['insertion'] | ||
| 136 | assert sid == expected['space'] and node['kind']['text'] == expected['text'] | ||
| 137 | object_id = identity(insertion, 1) | ||
| 138 | assert object_id in nodes[insertion['parent']]['children'], 'Insertion lost its parent' | ||
| 139 | paragraph = identity(insertion, 3) if 'Outline' in insertion['placement'] else object_id | ||
| 140 | assert nodes[paragraph]['content'] == [target], 'Inserted paragraph content changed' | ||
| 141 | if 'Outline' in insertion['placement']: | ||
| 142 | position = insertion['placement']['Outline'] | ||
| 143 | assert all(nodes[object_id]['layout'][key] == position[key] for key in ('x', 'y')), 'Outline coordinates changed' | ||
| 144 | assert nodes[object_id]['children'] == children[object_id], 'Inserted paragraph order changed' | ||
| 145 | assert found == set(documents), 'Final model omitted an inserted object' | ||
| 146 | |||
| 147 | |||
| 148 | def verify_native(paragraphs, documents): | ||
| 149 | from PIL import ImageColor | ||
| 150 | by_text = {''.join(char for char, _ in paragraph): paragraph for paragraph in paragraphs} | ||
| 151 | checks = 0 | ||
| 152 | for document in documents.values(): | ||
| 153 | actual = by_text[document['text']] | ||
| 154 | for (char, style), (wanted, bold, size, color) in zip(actual, document['new'], strict=True): | ||
| 155 | assert char == wanted and bool(style.get('bold')) == bold, 'Native text or bold differs from intent' | ||
| 156 | assert style.get('font_size', 11) == size, 'Native font size differs from intent' | ||
| 157 | native_color = style.get('color', 'automatic') | ||
| 158 | if color == 0xff000000: | ||
| 159 | assert native_color in ('automatic', None), 'Native automatic color changed' | ||
| 160 | else: | ||
| 161 | assert ImageColor.getrgb(native_color) == (18, 52, 86), 'Native color differs from intent' | ||
| 162 | checks += 3 | ||
| 163 | return checks | ||
tools/offline_history.py created+73
| ... | @@ -0,0 +1,73 @@ | ||
| 1 | """Verify local intent acknowledgements separately from the remote publication chain.""" | ||
| 2 | import re | ||
| 3 | |||
| 4 | |||
| 5 | def tokens(text): | ||
| 6 | assert text.startswith('Concurrent edits:'), 'Unexpected append prefix' | ||
| 7 | tail = text[len('Concurrent edits:'):] | ||
| 8 | found = re.findall(r' \[w[0-9]+:[0-9]+\]', tail) | ||
| 9 | assert ''.join(found) == tail and len(set(found)) == len(found), 'Malformed or duplicated append history' | ||
| 10 | return found | ||
| 11 | |||
| 12 | |||
| 13 | def publication_links(logs, operations, partial=False): | ||
| 14 | local = {} | ||
| 15 | for actor, events in logs.items(): | ||
| 16 | assert events and events[0]['event'] == 'ready', 'Missing client start' | ||
| 17 | assert partial or events[-1]['event'] == 'done', 'Incomplete client log' | ||
| 18 | if not actor.startswith('w'): continue | ||
| 19 | assert events[0].get('offline') is True, 'Expected an offline writer' | ||
| 20 | edits = [event for event in events if event['event'] == 'local_commit'] | ||
| 21 | assert [event['operation'] for event in edits] == list(range(len(edits))), 'Missing or duplicate local acknowledgement' | ||
| 22 | assert len(edits) <= operations and (partial or len(edits) == operations), 'Local operation count differs' | ||
| 23 | assert len({event['id'] for event in edits}) == len(edits), 'Duplicate local intent ID' | ||
| 24 | assert [event['id'] for event in edits] == sorted(event['id'] for event in edits), 'Local IDs went backwards' | ||
| 25 | for event in edits: | ||
| 26 | token = f' [{actor}:{event["operation"]}]' | ||
| 27 | assert event['token'] == token and token not in local, 'Local token differs from its operation' | ||
| 28 | assert event['started_us'] <= event['finished_us'], 'Invalid local acknowledgement interval' | ||
| 29 | prior = tokens(event['before']) | ||
| 30 | own = [item for item in prior if item.startswith(f' [{actor}:')] | ||
| 31 | assert own == [f' [{actor}:{i}]' for i in range(event['operation'])], 'Local view lost or invented its own edit' | ||
| 32 | local[token] = event | ||
| 33 | for event in local.values(): | ||
| 34 | for token in tokens(event['before']): | ||
| 35 | assert token in local and local[token]['started_us'] <= event['finished_us'], 'Local view invented a future token' | ||
| 36 | links = {} | ||
| 37 | seen_revisions = set() | ||
| 38 | for actor, events in logs.items(): | ||
| 39 | if not actor.startswith('w'): continue | ||
| 40 | edits = {event['id']: event for event in events if event['event'] == 'local_commit'} | ||
| 41 | receipts = [event for event in events if event['event'] == 'remote_receipt'] | ||
| 42 | assert len({event['id'] for event in receipts}) == len(receipts), 'Duplicate remote receipt' | ||
| 43 | assert set(event['id'] for event in receipts) <= set(edits), 'Receipt lacks a local intent' | ||
| 44 | assert partial or len(receipts) == len(edits), 'Local success lacks remote acknowledgement' | ||
| 45 | reopened = [event for event in events if event['event'] == 'reopened_receipt'] | ||
| 46 | if not partial: | ||
| 47 | assert [(event['id'], event['revision']) for event in reopened] == [(event['id'], event['revision']) for event in receipts], 'Receipt changed across reopen' | ||
| 48 | for receipt in receipts: | ||
| 49 | intent = edits[receipt['id']] | ||
| 50 | attempts = [event for event in events if event['event'] == 'remote_attempt' and event['revision'] == receipt['revision']] | ||
| 51 | assert len(attempts) == 1, 'Receipt does not identify one publication attempt' | ||
| 52 | attempt, = attempts | ||
| 53 | assert attempt['state'] in ('Committed', 'Unknown'), 'Receipt identifies a proven-unpublished attempt' | ||
| 54 | if attempt['state'] == 'Unknown': | ||
| 55 | assert all(field in attempt and field in intent for field in ('space', 'object')), 'Uncertain target identity is missing' | ||
| 56 | assert attempt['space'] == intent['space'] and attempt['object'] == intent['object'], 'Confirmation identifies another target' | ||
| 57 | confirmed = [event for event in events if event['event'] == 'remote_confirm' | ||
| 58 | and event['state'] == 'Committed' | ||
| 59 | and receipt['revision'] in event.get('revisions', {}).get(intent['space'], []) | ||
| 60 | and event.get('text', '').startswith(attempt['after']) | ||
| 61 | and attempt['finished_us'] <= event['started_us'] <= event['finished_us'] <= receipt['at_us']] | ||
| 62 | assert confirmed, 'Uncertain publication lacks a successful retained-revision confirmation' | ||
| 63 | assert not any(event['event'] == 'remote_attempt' and event['started_us'] >= attempt['finished_us'] | ||
| 64 | and event['after'] == event['before'] + intent['token'] for event in events), 'An uncertain intent was replayed' | ||
| 65 | assert receipt['revision'] not in seen_revisions, 'A revision was acknowledged twice' | ||
| 66 | seen_revisions.add(receipt['revision']) | ||
| 67 | assert intent['started_us'] <= attempt['started_us'] <= attempt['finished_us'] <= receipt['at_us'], 'Receipt precedes its publication' | ||
| 68 | assert attempt['after'] == attempt['before'] + intent['token'], 'Remote publication differs from local intent' | ||
| 69 | tokens(attempt['after']) | ||
| 70 | assert attempt['before'] not in links, 'Remote publications branched from the same content' | ||
| 71 | event = {**attempt, 'event': 'commit', 'operation': intent['operation'], 'token': intent['token'], 'finished_us': receipt['at_us']} | ||
| 72 | links[attempt['before']] = event, attempt['after'] | ||
| 73 | return links | ||
tools/offline_outage.py created+313
| ... | @@ -0,0 +1,313 @@ | ||
| 1 | """Require durable local progress during a confirmed outage of the owned SMB proxy.""" | ||
| 2 | import hashlib | ||
| 3 | import json | ||
| 4 | from pathlib import Path | ||
| 5 | import shlex | ||
| 6 | import time | ||
| 7 | |||
| 8 | from native_runner import windows | ||
| 9 | import linux_vm | ||
| 10 | from verify_smb_overlap import verify | ||
| 11 | |||
| 12 | |||
| 13 | def interrupt(output, clients, sequences, processes): | ||
| 14 | config = json.loads((output / 'run.json').read_text()) | ||
| 15 | folder = output / 'rust' | ||
| 16 | samples = {} | ||
| 17 | |||
| 18 | def logs(): | ||
| 19 | result = {} | ||
| 20 | for actor in processes: | ||
| 21 | text = (folder / f'{actor}.jsonl').read_text() | ||
| 22 | result[actor] = [json.loads(line) for line in text[:text.rfind('\n') + 1].splitlines()] | ||
| 23 | return result | ||
| 24 | |||
| 25 | def wait_for(predicate, message, timeout=90): | ||
| 26 | deadline = time.monotonic() + timeout | ||
| 27 | while not predicate(): | ||
| 28 | assert all(p.poll() in (None, 0) for p in processes.values()), 'A client failed during the offline outage' | ||
| 29 | if time.monotonic() > deadline: raise TimeoutError(message) | ||
| 30 | time.sleep(.05) | ||
| 31 | |||
| 32 | def ssh(command): | ||
| 33 | result = linux_vm.run_ssh(config['server'], command, timeout=15) | ||
| 34 | with (output / 'offline-outage-server.jsonl').open('a') as stream: | ||
| 35 | stream.write(json.dumps({'command': command, 'exit': result.returncode, 'stdout': result.stdout, 'stderr': result.stderr}) + '\n') | ||
| 36 | result.check_returncode() | ||
| 37 | return result.stdout | ||
| 38 | |||
| 39 | def phase(control): | ||
| 40 | text = json.dumps(control) | ||
| 41 | ssh("printf '%s' " + shlex.quote(text) + ' > /tmp/smb-control.tmp && mv /tmp/smb-control.tmp /tmp/smb-control.json') | ||
| 42 | wait_for(lambda: text in ssh("grep -F '\"control\":' /tmp/smb-trace.jsonl | tail -n 1"), 'Proxy did not acknowledge the outage phase') | ||
| 43 | |||
| 44 | def native_counts(label): | ||
| 45 | counts = [] | ||
| 46 | for actor, (client, sequence) in enumerate(zip(clients, sequences)): | ||
| 47 | capture = output / f'offline-{label}-n{actor}.jsonl' | ||
| 48 | result = windows.do_get(f'C:\\one-tests\\runs\\capture\\outbox\\{sequence}\\events.jsonl', capture, client['name']) | ||
| 49 | assert not result.get('error'), result | ||
| 50 | text = capture.read_text(encoding='utf-8-sig') | ||
| 51 | rows = [json.loads(line) for line in text[:text.rfind('\n') + 1].splitlines()] | ||
| 52 | assert 0 < len(rows) < config['stress_operations'], 'Native client was inactive during the outage campaign' | ||
| 53 | counts.append(len(rows)) | ||
| 54 | return counts | ||
| 55 | |||
| 56 | writers = [actor for actor in processes if actor.startswith('w')] | ||
| 57 | readers = [actor for actor in processes if actor.startswith('r')] | ||
| 58 | if config.get('offline_lost_reply'): | ||
| 59 | isolated = config.get('offline_client_reply', False) | ||
| 60 | def counts(): | ||
| 61 | return {actor: sum(row['event'] == ('remote_receipt' if actor in writers else 'read') for row in rows) | ||
| 62 | for actor, rows in logs().items()} | ||
| 63 | if config.get('document_operations'): | ||
| 64 | wait_for(lambda: all((folder / f'offline-paused-{actor}').exists() for actor in writers) | ||
| 65 | and all(counts()[actor] > 0 for actor in readers), 'Clients did not reach the formatting publication barrier') | ||
| 66 | else: | ||
| 67 | wait_for(lambda: all(value >= 3 for value in counts().values()), 'Clients made no progress before the reply cut') | ||
| 68 | samples['before'] = counts() | ||
| 69 | samples['native_before'] = native_counts('reply-before') | ||
| 70 | try: | ||
| 71 | control = {'phase': 'offline-reply-cut', 'cut': 9, 'peer': '10.0.2.2', 'offset': 96, 'direction': 'response'} | ||
| 72 | if isolated: | ||
| 73 | control['scope'] = 'connection' | ||
| 74 | (folder / 'offline-paused-w0.isolate').touch() | ||
| 75 | phase(control) | ||
| 76 | if config.get('document_operations'): | ||
| 77 | samples['format_released_us'] = time.time_ns() // 1000 | ||
| 78 | (folder / 'offline-paused-w0.resume').touch() | ||
| 79 | wait_for(lambda: int(ssh("grep -c '\"cut\": {' /tmp/smb-trace.jsonl || true").strip()) == 1, | ||
| 80 | 'The publication reply was not interrupted') | ||
| 81 | samples['down_started_us'] = time.time_ns() // 1000 | ||
| 82 | if isolated: | ||
| 83 | wait_for(lambda: any(row['event'] == 'confirmation_paused' for row in logs()['w0']), | ||
| 84 | 'The disconnected writer did not retain its uncertain publication') | ||
| 85 | for actor in writers[1:]: (folder / f'offline-paused-{actor}.resume').touch() | ||
| 86 | wait_for(lambda: all(value >= samples['before'][actor] + 3 for actor, value in counts().items() if actor != 'w0'), | ||
| 87 | 'Peers did not advance while the writer was disconnected') | ||
| 88 | wait_for(lambda: (folder / 'offline-retired.one').exists(), | ||
| 89 | 'Native maintenance did not retire the isolated revision') | ||
| 90 | time.sleep(3) | ||
| 91 | if isolated: samples['during'] = counts() | ||
| 92 | samples['native_during'] = native_counts('reply-during') | ||
| 93 | finally: | ||
| 94 | samples['up_started_us'] = time.time_ns() // 1000 | ||
| 95 | phase({'phase': 'offline-reply-reconnected'}) | ||
| 96 | if config.get('document_operations'): | ||
| 97 | for actor in writers: (folder / f'offline-paused-{actor}.resume').touch() | ||
| 98 | if isolated: (folder / 'offline-paused-w0.confirmation-resume').touch() | ||
| 99 | (output / 'offline-lost-reply-progress.json').write_text(json.dumps(samples, indent=2)) | ||
| 100 | wait_for(lambda: all(value >= samples['before'][actor] + 3 for actor, value in counts().items()), | ||
| 101 | 'A client failed to progress after the lost publication reply', timeout=120) | ||
| 102 | samples['after'] = counts() | ||
| 103 | (output / 'offline-lost-reply-progress.json').write_text(json.dumps(samples, indent=2)) | ||
| 104 | return | ||
| 105 | wait_for(lambda: all((folder / f'offline-paused-{actor}').exists() for actor in writers) | ||
| 106 | and (not config.get('document_operations') or all( | ||
| 107 | sum(row['event'] == 'local_document_commit' for row in logs()[actor]) == 2 for actor in writers)) | ||
| 108 | and all(any(row['event'] == 'read' for row in logs()[actor]) for actor in readers), | ||
| 109 | 'Clients did not reach the pre-publication outage barrier') | ||
| 110 | samples['native_before'] = native_counts('before') | ||
| 111 | samples['reader_errors_before'] = {actor: sum(row['event'] == 'transport_read_error' for row in logs()[actor]) for actor in readers} | ||
| 112 | try: | ||
| 113 | phase({'phase': 'offline-down', 'mode': 'down'}) | ||
| 114 | samples['down_started_us'] = time.time_ns() // 1000 | ||
| 115 | (folder / 'offline-outage-down').touch() | ||
| 116 | wait_for(lambda: all(sum(row['event'] == 'local_commit' for row in events) == 8 for actor, events in logs().items() if actor in writers) | ||
| 117 | and (not config.get('document_operations') or all( | ||
| 118 | sum(row['event'] == 'local_document_commit' for row in logs()[actor]) == 16 for actor in writers)) | ||
| 119 | and all(sum(row['event'] == 'transport_read_error' for row in logs()[actor]) > samples['reader_errors_before'][actor] for actor in readers), | ||
| 120 | 'Local queues or disconnected readers failed to progress during the outage') | ||
| 121 | time.sleep(3) | ||
| 122 | samples['native_during'] = native_counts('during') | ||
| 123 | samples['up_started_us'] = time.time_ns() // 1000 | ||
| 124 | finally: | ||
| 125 | phase({'phase': 'offline-reconnected'}) | ||
| 126 | (folder / 'offline-outage-resumed').touch() | ||
| 127 | (output / 'offline-outage-progress.json').write_text(json.dumps(samples, indent=2)) | ||
| 128 | wait_for(lambda: all(sum(row['event'] == 'remote_receipt' for row in logs()[actor]) >= 3 for actor in writers) | ||
| 129 | and all(sum(row['event'] == 'read' and row['started_us'] > samples['up_started_us'] for row in logs()[actor]) >= 3 for actor in readers), | ||
| 130 | 'A client failed to progress after the offline outage') | ||
| 131 | |||
| 132 | |||
| 133 | def native_progress(output, config, sample): | ||
| 134 | down, up = sample['down_started_us'], sample['up_started_us'] | ||
| 135 | assert len(sample['native_before']) == len(sample['native_during']) == config['stress_clients'] | ||
| 136 | native = list(zip(sample['native_before'], sample['native_during'])) | ||
| 137 | assert all(0 < before < during < config['stress_operations'] for before, during in native), 'Native local edits did not advance during the outage' | ||
| 138 | clocks = json.loads((output / 'clocks.json').read_text()) | ||
| 139 | native_inside = [] | ||
| 140 | assert len(clocks) == config['stress_clients'] | ||
| 141 | for index, clock in enumerate(clocks): | ||
| 142 | low = min(clock['native_minus_host_us'][0], clock['after_native_minus_host_us'][0]) | ||
| 143 | high = max(clock['native_minus_host_us'][1], clock['after_native_minus_host_us'][1]) | ||
| 144 | rows = [json.loads(line) for line in (output / f'n{index}/stress-events.jsonl').read_text(encoding='utf-8-sig').splitlines()] | ||
| 145 | inside = sum((row['update_started_ticks'] - 621355968000000000) // 10 - high > down | ||
| 146 | and (row['updated_ticks'] - 621355968000000000) // 10 - low < up for row in rows) | ||
| 147 | assert inside, 'Native timestamps do not establish local edits inside the confirmed outage' | ||
| 148 | native_inside.append(inside) | ||
| 149 | return native_inside | ||
| 150 | |||
| 151 | |||
| 152 | def verify_outage(output): | ||
| 153 | config = json.loads((output / 'run.json').read_text()) | ||
| 154 | assert config['offline'] and config['offline_outage'] and config['embedded_smb'] | ||
| 155 | assert config['stress_clients'] + config['rust_writers'] + config['rust_readers'] >= 12 | ||
| 156 | sample = json.loads((output / 'offline-outage-progress.json').read_text()) | ||
| 157 | down, up = sample['down_started_us'], sample['up_started_us'] | ||
| 158 | assert up - down >= 3_000_000, 'Confirmed outage lasted less than three seconds' | ||
| 159 | native_inside = native_progress(output, config, sample) | ||
| 160 | queues, reconnects = {}, {} | ||
| 161 | for mode, count in [('w', config['rust_writers']), ('r', config['rust_readers'])]: | ||
| 162 | for index in range(count): | ||
| 163 | actor = f'{mode}{index}' | ||
| 164 | events = [json.loads(line) for line in (output / 'rust' / f'{actor}.jsonl').read_text().splitlines()] | ||
| 165 | assert events[0]['event'] == 'ready' and events[-1]['event'] == 'done' | ||
| 166 | connected = [row for row in events if row['event'] == 'transport_connected'] | ||
| 167 | assert any(row['at_us'] > up for row in connected), 'No fresh transport after outage' | ||
| 168 | reconnects[actor] = len(connected) | ||
| 169 | if mode == 'r': | ||
| 170 | assert sum(row['event'] == 'transport_read_error' for row in events) > sample['reader_errors_before'][actor] | ||
| 171 | assert sum(row['event'] == 'read' and row['started_us'] > up for row in events) >= 3 | ||
| 172 | continue | ||
| 173 | local = [row for row in events if row['event'] == 'local_commit'] | ||
| 174 | assert len(local) == config['stress_operations'] | ||
| 175 | assert local[0]['finished_us'] < down | ||
| 176 | queued = [row for row in local if down < row['started_us'] <= row['finished_us'] < up] | ||
| 177 | assert [row['operation'] for row in queued] == list(range(1, 8)), 'Seven local edits were not accepted while SMB was down' | ||
| 178 | queues[actor] = len(queued) | ||
| 179 | paused = [row for row in events if row['event'] == 'publication_paused'] | ||
| 180 | assert len(paused) == 1 and paused[0]['at_us'] < down | ||
| 181 | attempts = [row for row in events if row['event'] == 'remote_attempt'] | ||
| 182 | assert attempts and all(row['started_us'] > up for row in attempts), 'Publication escaped the outage barrier' | ||
| 183 | assert attempts[0]['state'] == 'NotCommitted' and attempts[0]['revision'] == paused[0]['revision'], 'Disconnected pre-I/O attempt was not safely rejected' | ||
| 184 | assert all(row['state'] != 'Unknown' for row in attempts), 'Unexpected uncertain publication requires separate recovery evidence' | ||
| 185 | receipts = [row for row in events if row['event'] == 'remote_receipt'] | ||
| 186 | assert len(receipts) == config['stress_operations'] and all(row['at_us'] > up for row in receipts) | ||
| 187 | if config.get('document_operations'): | ||
| 188 | edits = [row for row in events if row['event'] == 'local_document_commit'] | ||
| 189 | assert [(row['operation'], row['kind']) for row in edits] == [ | ||
| 190 | (operation, kind) for operation in range(config['stress_operations']) for kind in ('insert', 'format')] | ||
| 191 | assert all(row['finished_us'] < down for row in edits[:2]), 'Initial document edits missed the outage barrier' | ||
| 192 | queued = [row for row in edits if down < row['started_us'] <= row['finished_us'] < up] | ||
| 193 | assert [(row['operation'], row['kind']) for row in queued] == [ | ||
| 194 | (operation, kind) for operation in range(1, 8) for kind in ('insert', 'format')], 'Document edits did not persist during the outage' | ||
| 195 | queues[actor] += len(queued) | ||
| 196 | receipts = [row for row in events if row['event'] == 'document_receipt'] | ||
| 197 | assert len(receipts) == config['stress_operations'] * 2 and all(row['at_us'] > up for row in receipts) | ||
| 198 | trace = [json.loads(line) for line in (output / 'smb-trace.jsonl').read_text().splitlines()] | ||
| 199 | controls = [row['control'] for row in trace if row.get('control', {}).get('phase', '').startswith('offline-')] | ||
| 200 | assert controls == [{'phase': 'offline-down', 'mode': 'down'}, {'phase': 'offline-reconnected'}], 'Unexpected outage control sequence' | ||
| 201 | return {'outages': 1, 'confirmed_down_seconds': (up - down) / 1_000_000, | ||
| 202 | 'local_edits_while_down': queues, 'transport_connection_events': reconnects, | ||
| 203 | 'native_local_edits_inside_confirmed_outage': native_inside, | ||
| 204 | 'resumed_guarded_io_overlap': verify(trace, phase='offline-reconnected')} | ||
| 205 | |||
| 206 | |||
| 207 | def verify_lost_reply(output): | ||
| 208 | from offline_history import publication_links, tokens | ||
| 209 | config = json.loads((output / 'run.json').read_text()) | ||
| 210 | assert config['offline'] and config['offline_lost_reply'] and config['embedded_smb'] | ||
| 211 | assert config['stress_clients'] + config['rust_writers'] + config['rust_readers'] >= 12 | ||
| 212 | sample = json.loads((output / 'offline-lost-reply-progress.json').read_text()) | ||
| 213 | isolated = config.get('offline_client_reply', False) | ||
| 214 | assert sample['up_started_us'] - sample['down_started_us'] >= 3_000_000 | ||
| 215 | actors = [*(f'w{i}' for i in range(config['rust_writers'])), *(f'r{i}' for i in range(config['rust_readers']))] | ||
| 216 | assert set(sample['before']) == set(sample['after']) == set(actors) | ||
| 217 | assert all(sample['after'][actor] >= sample['before'][actor] + 3 for actor in actors) | ||
| 218 | if not config.get('document_operations'): | ||
| 219 | assert all(sample['before'][actor] >= 3 for actor in actors) | ||
| 220 | logs = {actor: [json.loads(line) for line in (output / 'rust' / f'{actor}.jsonl').read_text().splitlines()] for actor in actors} | ||
| 221 | publication_links(logs, config['stress_operations']) | ||
| 222 | documents = {} | ||
| 223 | if config.get('document_operations'): | ||
| 224 | from offline_document_history import document_history | ||
| 225 | documents = document_history(logs, config['stress_operations']) | ||
| 226 | for actor, rows in logs.items(): | ||
| 227 | if isolated and actor != 'w0': continue | ||
| 228 | assert any(row['event'] == 'transport_connected' and row['at_us'] > sample['up_started_us'] for row in rows), f'{actor} did not reconnect' | ||
| 229 | unknown = [(actor, row) for actor, rows in logs.items() for row in rows if row['event'] == 'remote_attempt' and row['state'] == 'Unknown'] | ||
| 230 | assert len(unknown) == 1, 'The reply cut did not establish exactly one uncertain publication' | ||
| 231 | actor, attempt = unknown[0] | ||
| 232 | assert attempt['started_us'] < sample['down_started_us'], 'Uncertain attempt started after the reply cut' | ||
| 233 | receipts = [row for row in logs[actor] if row['event'] in ('remote_receipt', 'document_receipt') and row['revision'] == attempt['revision']] | ||
| 234 | if not receipts and documents: | ||
| 235 | target, = attempt['document_changes'] | ||
| 236 | confirmed_revision = documents[target]['format']['receipt_revision'] | ||
| 237 | receipts = [row for row in logs[actor] if row['event'] == 'document_receipt' and row['revision'] == confirmed_revision] | ||
| 238 | assert len(receipts) == 1 and receipts[0]['at_us'] > sample['up_started_us'] | ||
| 239 | peer_progress = {} | ||
| 240 | if isolated: | ||
| 241 | assert sample['during']['w0'] == sample['before']['w0'] | ||
| 242 | retired, = [row for row in logs['r0'] if row['event'] == 'revision_retired'] | ||
| 243 | assert retired['space'] == attempt['space'] and retired['revision'] == attempt['revision'] | ||
| 244 | assert sample['down_started_us'] < retired['started_us'] <= retired['finished_us'] < sample['up_started_us'] | ||
| 245 | assert receipts[0]['revision'] != attempt['revision'], 'The client-disconnect gate did not exercise confirmation after revision retirement' | ||
| 246 | paused, = [row for row in logs['w0'] if row['event'] == 'confirmation_paused'] | ||
| 247 | assert paused['revision'] == attempt['revision'] and attempt['finished_us'] <= paused['at_us'] < sample['up_started_us'] | ||
| 248 | for peer, rows in logs.items(): | ||
| 249 | if peer == 'w0': continue | ||
| 250 | assert sample['during'][peer] >= sample['before'][peer] + 3 | ||
| 251 | if peer.startswith('w'): | ||
| 252 | progress = [row for row in rows if row['event'] == 'remote_attempt' and row['state'] == 'Committed' | ||
| 253 | and sample['down_started_us'] < row['started_us'] <= row['finished_us'] < sample['up_started_us']] | ||
| 254 | else: | ||
| 255 | progress = [row for row in rows if row['event'] == 'read' | ||
| 256 | and sample['down_started_us'] < row['started_us'] <= row['finished_us'] < sample['up_started_us']] | ||
| 257 | assert len(progress) >= 3, f'{peer} has insufficient completed I/O while the writer was disconnected' | ||
| 258 | peer_progress[peer] = len(progress) | ||
| 259 | if config.get('document_operations'): | ||
| 260 | assert actor == 'w0' and sample['format_released_us'] <= attempt['started_us'] | ||
| 261 | intent, = [row for row in logs[actor] if row['event'] == 'local_document_commit' and row['id'] == receipts[0]['id']] | ||
| 262 | assert intent['kind'] == 'format', 'The interrupted publication was not formatting' | ||
| 263 | for writer in (f'w{i}' for i in range(config['rust_writers'])): | ||
| 264 | paused = [row for row in logs[writer] if row['event'] == 'publication_paused'] | ||
| 265 | assert len(paused) == 1 and paused[0]['kind'] == 'format' and paused[0]['at_us'] < sample['format_released_us'] | ||
| 266 | paused, = [row for row in logs[actor] if row['event'] == 'publication_paused'] | ||
| 267 | if paused['revision'] != attempt['revision']: | ||
| 268 | prior, = [row for row in logs[actor] if row['event'] == 'remote_attempt' and row['revision'] == paused['revision']] | ||
| 269 | assert prior['state'] == 'NotCommitted' and sample['format_released_us'] <= prior['started_us'] <= prior['finished_us'] <= attempt['started_us'], 'Paused revision was replaced without proving it unpublished' | ||
| 270 | captures = {} | ||
| 271 | for rows in logs.values(): | ||
| 272 | for row in rows: | ||
| 273 | if row['event'] != 'remote_confirm': continue | ||
| 274 | name = row['capture'] | ||
| 275 | assert Path(name).name == name and name not in captures | ||
| 276 | tokens(row['text']) | ||
| 277 | assert row['started_us'] > sample['up_started_us'] | ||
| 278 | captures[name] = {'sha256': hashlib.sha256((output / 'rust/confirmations' / name).read_bytes()).hexdigest(), 'state': row['state']} | ||
| 279 | assert captures and set(captures) == {path.name for path in (output / 'rust/confirmations').glob('*.one')} | ||
| 280 | trace = [json.loads(line) for line in (output / 'smb-trace.jsonl').read_text().splitlines()] | ||
| 281 | controls = [row['control'] for row in trace if row.get('control', {}).get('phase', '').startswith('offline-')] | ||
| 282 | expected = {'phase': 'offline-reply-cut', 'cut': 9, 'peer': '10.0.2.2', 'offset': 96, 'direction': 'response'} | ||
| 283 | if isolated: expected['scope'] = 'connection' | ||
| 284 | assert controls == [expected, {'phase': 'offline-reply-reconnected'}] | ||
| 285 | cuts = [row['cut'] for row in trace if 'cut' in row] | ||
| 286 | assert len(cuts) == 1 and cuts[0]['direction'] == 'response' and cuts[0]['command'] == 9 and cuts[0]['status'] == '0x0' | ||
| 287 | request, = [row for row in trace if row.get('direction') == 'request' and row['connection'] == cuts[0]['connection'] and row['message'] == cuts[0]['message']] | ||
| 288 | assert request['offset'] == 96 and request['command'] == 9 | ||
| 289 | native_writes = 0 | ||
| 290 | if isolated: | ||
| 291 | peers = {row['connection']: row['peer'][0] for row in trace if row.get('opened')} | ||
| 292 | pending, files = {}, {} | ||
| 293 | for row in trace: | ||
| 294 | if row.get('command') not in (5, 6, 9): continue | ||
| 295 | key = row['connection'], row['message'] | ||
| 296 | if row['direction'] == 'request': | ||
| 297 | pending[key] = row | ||
| 298 | if row['command'] == 6: files.pop((key[0], row['file_id']), None) | ||
| 299 | elif row['status'] == '0x0' and key in pending: | ||
| 300 | request = pending.pop(key) | ||
| 301 | if row['command'] == 5: | ||
| 302 | files[key[0], row['file_id']] = request['path'].lower() | ||
| 303 | elif (row['command'] == 9 and peers[key[0]].startswith('192.168.77.') and row.get('written', 0) > 0 | ||
| 304 | and files.get((key[0], request['file_id']), '').endswith('synthetic.one') | ||
| 305 | and sample['down_started_us'] < request['time'] * 1_000_000 <= row['time'] * 1_000_000 < sample['up_started_us']): | ||
| 306 | native_writes += 1 | ||
| 307 | assert native_writes, 'No successful native writes while the isolated writer awaited reconciliation' | ||
| 308 | return {'reply_cuts': 1, 'uncertain_actor': actor, 'attempted_revision': attempt['revision'], 'confirmed_revision': receipts[0]['revision'], | ||
| 309 | 'peer_operations_during_client_disconnect': peer_progress, | ||
| 310 | 'native_writes_during_client_disconnect': native_writes, | ||
| 311 | 'retired_snapshot_sha256': hashlib.sha256((output / 'rust/offline-retired.one').read_bytes()).hexdigest() if isolated else None, | ||
| 312 | 'confirmation_snapshots': captures, 'native_local_edits_inside_confirmed_outage': native_progress(output, config, sample), | ||
| 313 | 'resumed_guarded_io_overlap': verify(trace, phase='offline-reply-reconnected')} | ||
tools/offline_publication_crash.py created+237
| ... | @@ -0,0 +1,237 @@ | ||
| 1 | #!/usr/bin/env python3 | ||
| 2 | """Kill owned processes across local-cache/remote-file publication boundaries.""" | ||
| 3 | import argparse | ||
| 4 | import hashlib | ||
| 5 | import json | ||
| 6 | import os | ||
| 7 | from pathlib import Path | ||
| 8 | import queue | ||
| 9 | import shutil | ||
| 10 | import signal | ||
| 11 | import sqlite3 | ||
| 12 | import subprocess | ||
| 13 | import threading | ||
| 14 | import time | ||
| 15 | |||
| 16 | ROOT = Path(__file__).resolve().parent.parent | ||
| 17 | BINARY = ROOT / 'target/debug/examples/recovery_probe' | ||
| 18 | TOKEN = ' [offline-recovery]' | ||
| 19 | |||
| 20 | |||
| 21 | def run_command(binary, args, output): | ||
| 22 | result = subprocess.run([str(binary), *map(str, args)], capture_output=True, text=True, timeout=60, | ||
| 23 | env={**os.environ, 'ONESTORE_RECOVERY_PAUSE': ''}) | ||
| 24 | output.with_suffix('.jsonl').write_text(result.stdout) | ||
| 25 | output.with_suffix('.stderr').write_text(result.stderr) | ||
| 26 | result.check_returncode() | ||
| 27 | return [json.loads(line) for line in result.stdout.splitlines()] | ||
| 28 | |||
| 29 | |||
| 30 | def kill_at(binary, args, phase, output, receipt_window=None): | ||
| 31 | events = queue.Queue() | ||
| 32 | database = Path(args[1]) / 'cache.sqlite' | ||
| 33 | journal = database.with_name(database.name + '-journal') | ||
| 34 | def header(): | ||
| 35 | try: | ||
| 36 | with journal.open('rb') as stream: return stream.read(8) | ||
| 37 | except FileNotFoundError: return b'' | ||
| 38 | with output.with_suffix('.jsonl').open('w') as log, output.with_suffix('.stderr').open('w') as error: | ||
| 39 | process = subprocess.Popen([str(binary), *map(str, args)], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=error, text=True, | ||
| 40 | env={**os.environ, 'ONESTORE_RECOVERY_PAUSE': phase}) | ||
| 41 | def collect(): | ||
| 42 | try: | ||
| 43 | for line in process.stdout: | ||
| 44 | log.write(line) | ||
| 45 | log.flush() | ||
| 46 | events.put(json.loads(line)) | ||
| 47 | finally: | ||
| 48 | events.put(None) | ||
| 49 | reader = threading.Thread(target=collect) | ||
| 50 | reader.start() | ||
| 51 | try: | ||
| 52 | deadline = time.monotonic() + 60 | ||
| 53 | while True: | ||
| 54 | event = events.get(timeout=max(0, deadline-time.monotonic())) | ||
| 55 | assert event is not None, f'Process exited before {phase}' | ||
| 56 | if event.get('event') == 'phase' and event['name'] == phase: | ||
| 57 | if receipt_window is not None: | ||
| 58 | assert args[0] == 'sync' and phase == 'publish-after' | ||
| 59 | assert receipt_window in ('journal', 'database') | ||
| 60 | before_mtime = database.stat().st_mtime_ns | ||
| 61 | process.stdin.write('\n') | ||
| 62 | process.stdin.flush() | ||
| 63 | deadline = time.monotonic() + 5 | ||
| 64 | while True: | ||
| 65 | if header() == bytes.fromhex('d9d505f920a163d7') and (receipt_window == 'journal' or database.stat().st_mtime_ns != before_mtime): break | ||
| 66 | assert process.poll() is None, 'Receipt transaction finished before the requested cut' | ||
| 67 | assert time.monotonic() < deadline, 'Receipt transaction window was not observed' | ||
| 68 | time.sleep(.0001) | ||
| 69 | process.kill() | ||
| 70 | break | ||
| 71 | assert process.wait(timeout=10) == -signal.SIGKILL, 'Expected an actual process kill' | ||
| 72 | proof = {'pid': process.pid, 'exit': process.returncode, 'phase': phase} | ||
| 73 | if receipt_window is not None: | ||
| 74 | proof.update(journal_header_after_kill=header().hex(), database_changed=database.stat().st_mtime_ns != before_mtime, receipt_window=receipt_window) | ||
| 75 | assert proof['journal_header_after_kill'] == 'd9d505f920a163d7', 'Receipt transaction committed before the process died' | ||
| 76 | assert receipt_window != 'database' or proof['database_changed'] | ||
| 77 | output.with_suffix('.cut.json').write_text(json.dumps(proof, indent=2)) | ||
| 78 | finally: | ||
| 79 | if process.poll() is None: process.kill() | ||
| 80 | process.wait(timeout=10) | ||
| 81 | process.stdin.close() | ||
| 82 | reader.join(timeout=10) | ||
| 83 | assert not reader.is_alive(), 'Trace reader did not finish' | ||
| 84 | process.stdout.close() | ||
| 85 | |||
| 86 | |||
| 87 | def state(rows, original): | ||
| 88 | found = [row for row in rows if row['event'] == 'state'] | ||
| 89 | assert len(found) == 1, 'Missing independent post-reopen state' | ||
| 90 | result, = found | ||
| 91 | assert result['local_text'] == original + TOKEN, 'Locally acknowledged text was lost or duplicated' | ||
| 92 | assert result['remote_text'] in [original, original+TOKEN], 'Remote current text is partial, duplicated or invented' | ||
| 93 | assert result['status'] in ('pending', 'uncertain', 'published'), 'Intent disappeared or became an unexplained conflict' | ||
| 94 | if result['status'] == 'pending': | ||
| 95 | assert result['revision'] is None, 'Unattempted intent acquired a publication identity' | ||
| 96 | else: | ||
| 97 | assert isinstance(result['revision'], str) and result['revision'], 'Attempted identity was lost' | ||
| 98 | assert (result['revision'] == result['remote_revision']) == (result['remote_text'] == original+TOKEN), 'Publication identity disagrees with the visible effect' | ||
| 99 | if result['status'] == 'published': | ||
| 100 | assert result['pending'] == [] and result['remote_text'] == original+TOKEN | ||
| 101 | assert result['revision'] == result['remote_revision'], 'Receipt identifies another remote revision' | ||
| 102 | else: | ||
| 103 | assert len(result['pending']) == 1, 'Unacknowledged intent was lost or duplicated' | ||
| 104 | pending, = result['pending'] | ||
| 105 | assert pending['id'] == 1 and pending['before'] == original and pending['replacement'] == TOKEN | ||
| 106 | at = len(original.encode('utf-16-le')) // 2 | ||
| 107 | assert pending['range'] == [at, at], 'Durable intent range changed' | ||
| 108 | return result | ||
| 109 | |||
| 110 | |||
| 111 | def save_image(output, data): | ||
| 112 | digest = hashlib.sha256(data).hexdigest() | ||
| 113 | path = output / 'images' / f'{digest}.one' | ||
| 114 | if not path.exists(): path.write_bytes(data) | ||
| 115 | return digest | ||
| 116 | |||
| 117 | |||
| 118 | def confirmation_only(events, before, after): | ||
| 119 | assert not any(row['event'] == 'phase' and row['name'] == 'publish-before' for row in events) | ||
| 120 | assert all(row['offset'] == 212 and row['bytes'] == 40 for row in events if row['event'] == 'write'), 'Recovery republished the remote edit' | ||
| 121 | assert before[:212] == after[:212] and before[252:] == after[252:], 'Confirmation changed content or the transaction count' | ||
| 122 | |||
| 123 | |||
| 124 | def prepare_run(source, output): | ||
| 125 | output.mkdir(parents=True, exist_ok=False) | ||
| 126 | (output / 'images').mkdir() | ||
| 127 | binary = output / 'recovery_probe' | ||
| 128 | shutil.copyfile(BINARY, binary) | ||
| 129 | binary.chmod(0o755) | ||
| 130 | shutil.copyfile(__file__, output / Path(__file__).name) | ||
| 131 | source_hash = hashlib.sha256(source.read_bytes()).hexdigest() | ||
| 132 | (output / 'run.json').write_text(json.dumps({'source': str(source), 'source_sha256': source_hash, | ||
| 133 | 'binary_sha256': hashlib.sha256(binary.read_bytes()).hexdigest(), 'controller_sha256': hashlib.sha256(Path(__file__).read_bytes()).hexdigest()}, indent=2)) | ||
| 134 | return binary, source_hash | ||
| 135 | |||
| 136 | |||
| 137 | def run(source, output): | ||
| 138 | binary, source_hash = prepare_run(source, output) | ||
| 139 | baseline = output / 'baseline' | ||
| 140 | initialized = run_command(binary, ['init', baseline, source], output / 'baseline-init') | ||
| 141 | original = next(row['remote_text'] for row in initialized if row['event'] == 'state') | ||
| 142 | state(initialized, original) | ||
| 143 | published = run_command(binary, ['sync', baseline], output / 'baseline-sync') | ||
| 144 | assert state(published, original)['status'] == 'published' | ||
| 145 | phases = [row['name'] for row in published if row['event'] == 'phase'] | ||
| 146 | assert len(phases) == len(set(phases)), 'Baseline has ambiguous phase names' | ||
| 147 | assert 'publish-after' in phases and any(name.startswith('write-') for name in phases) | ||
| 148 | confirmation = output / 'confirmation-baseline' | ||
| 149 | run_command(binary, ['init', confirmation, source], output / 'confirmation-init') | ||
| 150 | kill_at(binary, ['sync', confirmation], 'publish-after', output / 'confirmation-setup') | ||
| 151 | confirmed = run_command(binary, ['sync', confirmation], output / 'confirmation-sync') | ||
| 152 | confirmation_phases = [row['name'] for row in confirmed if row['event'] == 'phase'] | ||
| 153 | assert len(confirmation_phases) == len(set(confirmation_phases)) and 'confirm-after' in confirmation_phases | ||
| 154 | assert state(confirmed, original)['status'] == 'published' | ||
| 155 | cases = [('local-after', False), *((phase, False) for phase in phases), *((phase, True) for phase in confirmation_phases)] | ||
| 156 | results = [] | ||
| 157 | for index, (phase, confirmation) in enumerate(cases): | ||
| 158 | folder = output / f'case-{index:02}' | ||
| 159 | trace = output / f'case-{index:02}-kill' | ||
| 160 | if phase == 'local-after': | ||
| 161 | kill_at(binary, ['init', folder, source], phase, trace) | ||
| 162 | else: | ||
| 163 | run_command(binary, ['init', folder, source], output / f'case-{index:02}-init') | ||
| 164 | if confirmation: | ||
| 165 | kill_at(binary, ['sync', folder], 'publish-after', output / f'case-{index:02}-setup') | ||
| 166 | kill_at(binary, ['sync', folder], phase, trace) | ||
| 167 | before = state(run_command(binary, ['inspect', folder], output / f'case-{index:02}-inspect'), original) | ||
| 168 | before_bytes = (folder / 'remote.one').read_bytes() | ||
| 169 | first = run_command(binary, ['sync', folder], output / f'case-{index:02}-recover') | ||
| 170 | after = state(first, original) | ||
| 171 | if before['status'] == 'uncertain' and before['remote_text'] == original: | ||
| 172 | assert after['status'] == 'uncertain' and after['revision'] == before['revision'], 'Absent uncertain attempt was replayed' | ||
| 173 | assert not any(row['event'] == 'write' for row in first) | ||
| 174 | else: | ||
| 175 | assert after['status'] == 'published' | ||
| 176 | if before['status'] != 'pending': | ||
| 177 | assert after['revision'] == before['revision'], 'Recovery published another revision' | ||
| 178 | confirmation_only(first, before_bytes, (folder / 'remote.one').read_bytes()) | ||
| 179 | remote_hash = hashlib.sha256((folder / 'remote.one').read_bytes()).hexdigest() | ||
| 180 | repeated = run_command(binary, ['sync', folder], output / f'case-{index:02}-repeat') | ||
| 181 | assert state(repeated, original) == after, 'Repeated recovery changed durable intent state' | ||
| 182 | assert not any(row['event'] == 'write' for row in repeated), 'Repeated recovery published again' | ||
| 183 | assert hashlib.sha256((folder / 'remote.one').read_bytes()).hexdigest() == remote_hash | ||
| 184 | remote_image = save_image(output, (folder / 'remote.one').read_bytes()) | ||
| 185 | connection = sqlite3.connect(folder / 'cache.sqlite') | ||
| 186 | try: | ||
| 187 | assert connection.execute('PRAGMA quick_check').fetchall() == [('ok',)] | ||
| 188 | assert connection.execute('PRAGMA foreign_key_check').fetchall() == [] | ||
| 189 | local_image = save_image(output, connection.execute('SELECT working FROM replica WHERE id=1').fetchone()[0]) | ||
| 190 | finally: | ||
| 191 | connection.close() | ||
| 192 | result = {'case': index, 'phase': phase, 'during_confirmation': confirmation, 'before_status': before['status'], 'after_status': after['status'], | ||
| 193 | 'visible_before_recovery': before['remote_text'] != original, 'remote_image': remote_image, 'local_image': local_image, | ||
| 194 | 'remote_text': after['remote_text'], 'local_text': after['local_text']} | ||
| 195 | results.append(result) | ||
| 196 | (output / 'results.json').write_text(json.dumps(results, indent=2)) | ||
| 197 | print(json.dumps({key: value for key, value in result.items() if not key.endswith('_text')}), flush=True) | ||
| 198 | assert hashlib.sha256(source.read_bytes()).hexdigest() == source_hash, 'The frozen source changed' | ||
| 199 | summary = {'cases': len(results), 'process_kills': 1 + len(results) + sum(confirmation for _, confirmation in cases), 'uncertain_absent_preserved': sum(row['after_status']=='uncertain' for row in results), | ||
| 200 | 'durable_receipts': sum(row['after_status']=='published' for row in results), 'unique_images': len(list((output/'images').glob('*.one')))} | ||
| 201 | (output/'summary.json').write_text(json.dumps(summary, indent=2)) | ||
| 202 | print(json.dumps(summary), flush=True) | ||
| 203 | |||
| 204 | |||
| 205 | def receipt_windows(source, output): | ||
| 206 | binary, source_hash = prepare_run(source, output) | ||
| 207 | results = [] | ||
| 208 | for window in ('journal', 'database'): | ||
| 209 | folder = output / window | ||
| 210 | initialized = run_command(binary, ['init', folder, source], output / (window+'-init')) | ||
| 211 | original = next(row['remote_text'] for row in initialized if row['event'] == 'state') | ||
| 212 | state(initialized, original) | ||
| 213 | kill_at(binary, ['sync', folder], 'publish-after', output / (window+'-kill'), receipt_window=window) | ||
| 214 | before = state(run_command(binary, ['inspect', folder], output / (window+'-inspect')), original) | ||
| 215 | assert before['status'] == 'uncertain' and before['remote_text'] == original+TOKEN, 'Hot-journal recovery lost the pending confirmation' | ||
| 216 | before_bytes = (folder / 'remote.one').read_bytes() | ||
| 217 | recovered = run_command(binary, ['sync', folder], output / (window+'-recover')) | ||
| 218 | after = state(recovered, original) | ||
| 219 | assert after['status'] == 'published' and before['revision'] == after['revision'] | ||
| 220 | confirmation_only(recovered, before_bytes, (folder / 'remote.one').read_bytes()) | ||
| 221 | repeated = run_command(binary, ['sync', folder], output / (window+'-repeat')) | ||
| 222 | assert state(repeated, original) == after and not any(row['event']=='write' for row in repeated) | ||
| 223 | digest = save_image(output, (folder/'remote.one').read_bytes()) | ||
| 224 | results.append({'receipt_window':window, 'remote_image':digest, 'remote_text':after['remote_text'], 'revision':after['revision']}) | ||
| 225 | print(json.dumps({'receipt_window':window, 'status':after['status'], 'remote_image':digest}), flush=True) | ||
| 226 | assert hashlib.sha256(source.read_bytes()).hexdigest() == source_hash | ||
| 227 | (output/'results.json').write_text(json.dumps(results, indent=2)) | ||
| 228 | (output/'summary.json').write_text(json.dumps({'process_kills':2, 'recovered_receipts':2, 'republished_edits':0}, indent=2)) | ||
| 229 | |||
| 230 | |||
| 231 | if __name__ == '__main__': | ||
| 232 | parser = argparse.ArgumentParser(description=__doc__) | ||
| 233 | parser.add_argument('source', type=Path) | ||
| 234 | parser.add_argument('output', type=Path) | ||
| 235 | parser.add_argument('--receipt-windows', action='store_true') | ||
| 236 | args = parser.parse_args() | ||
| 237 | (receipt_windows if args.receipt_windows else run)(args.source.resolve(), args.output.resolve()) | ||
tools/smb-proxy.py+81-14
| ... | @@ -1,5 +1,5 @@ | ... | @@ -1,5 +1,5 @@ |
| 1 | #!/usr/bin/env python3 | 1 | #!/usr/bin/env python3 |
| 2 | """Trace a dedicated test SMB session and cut selected responses before delivery.""" | 2 | """Trace a dedicated test SMB session and cut selected requests or responses.""" |
| 3 | import argparse | 3 | import argparse |
| 4 | import asyncio | 4 | import asyncio |
| 5 | import json | 5 | import json |
| ... | @@ -8,28 +8,44 @@ import struct | ... | @@ -8,28 +8,44 @@ import struct |
| 8 | import time | 8 | import time |
| 9 | 9 | ||
| 10 | 10 | ||
| 11 | def header_fields(offset, data): | ||
| 12 | result = {} | ||
| 13 | for name, start, length in [('transactions', 96, 4), ('version', 212, 16), | ||
| 14 | ('generation', 228, 8), ('deny_read', 236, 16)]: | ||
| 15 | if offset <= start and start + length <= offset + len(data): | ||
| 16 | value = data[start - offset:start - offset + length] | ||
| 17 | result[name] = value.hex() if length == 16 else int.from_bytes(value, 'little') | ||
| 18 | return result | ||
| 19 | |||
| 20 | |||
| 11 | async def main(): | 21 | async def main(): |
| 12 | parser = argparse.ArgumentParser(description=__doc__) | 22 | parser = argparse.ArgumentParser(description=__doc__) |
| 13 | parser.add_argument('control', type=Path) | 23 | parser.add_argument('control', type=Path) |
| 14 | parser.add_argument('--port', type=int, default=11445) | 24 | parser.add_argument('--port', type=int, default=11445) |
| 15 | parser.add_argument('--server', default='10.0.0.1') | 25 | parser.add_argument('--server', default='10.0.0.1') |
| 26 | parser.add_argument('--server-port', type=int, default=445) | ||
| 27 | parser.add_argument('--bind', default='127.0.0.1') | ||
| 16 | args = parser.parse_args() | 28 | args = parser.parse_args() |
| 17 | writers = set() | 29 | writers = set() |
| 18 | state = {'mode': 'up'} | 30 | state = {'mode': 'up'} |
| 19 | previous = None | 31 | previous = None |
| 20 | blocked = False | 32 | blocked = False |
| 21 | matched = 0 | 33 | matched = 0 |
| 34 | connections = 0 | ||
| 35 | record_writes = False | ||
| 22 | 36 | ||
| 23 | def record(**fields): | 37 | def record(**fields): |
| 24 | print(json.dumps({'time': time.time(), **fields}), flush=True) | 38 | print(json.dumps({'time': time.time(), **fields}), flush=True) |
| 25 | 39 | ||
| 26 | async def controls(): | 40 | async def controls(): |
| 27 | nonlocal state, previous, blocked, matched | 41 | nonlocal state, previous, blocked, matched, record_writes |
| 28 | while True: | 42 | while True: |
| 29 | try: | 43 | try: |
| 30 | raw = args.control.read_bytes() | 44 | raw = args.control.read_bytes() |
| 31 | if raw != previous: | 45 | if raw != previous: |
| 32 | state = json.loads(raw) | 46 | state = json.loads(raw) |
| 47 | if 'record_writes' in state: | ||
| 48 | record_writes = bool(state['record_writes']) | ||
| 33 | previous, matched = raw, 0 | 49 | previous, matched = raw, 0 |
| 34 | blocked = state.get('mode') == 'down' | 50 | blocked = state.get('mode') == 'down' |
| 35 | record(control=state) | 51 | record(control=state) |
| ... | @@ -41,13 +57,16 @@ async def main(): | ... | @@ -41,13 +57,16 @@ async def main(): |
| 41 | await asyncio.sleep(0.05) | 57 | await asyncio.sleep(0.05) |
| 42 | 58 | ||
| 43 | async def connection(client, client_writer): | 59 | async def connection(client, client_writer): |
| 44 | nonlocal blocked, matched | 60 | nonlocal blocked, matched, connections |
| 61 | connections += 1 | ||
| 62 | connection_id = connections | ||
| 63 | record(connection=connection_id, peer=client_writer.get_extra_info("peername"), opened=True) | ||
| 45 | if blocked: | 64 | if blocked: |
| 46 | client_writer.close() | 65 | client_writer.close() |
| 47 | return | 66 | return |
| 48 | server_writer = None | 67 | server_writer = None |
| 49 | try: | 68 | try: |
| 50 | server, server_writer = await asyncio.open_connection(args.server, 445) | 69 | server, server_writer = await asyncio.open_connection(args.server, args.server_port) |
| 51 | writers.update((client_writer, server_writer)) | 70 | writers.update((client_writer, server_writer)) |
| 52 | 71 | ||
| 53 | async def forward(reader, writer, direction): | 72 | async def forward(reader, writer, direction): |
| ... | @@ -58,25 +77,68 @@ async def main(): | ... | @@ -58,25 +77,68 @@ async def main(): |
| 58 | at = 0 | 77 | at = 0 |
| 59 | while frame[at:at+4] == b'\xfeSMB': | 78 | while frame[at:at+4] == b'\xfeSMB': |
| 60 | command = struct.unpack_from('<H', frame, at+12)[0] | 79 | command = struct.unpack_from('<H', frame, at+12)[0] |
| 61 | entry = {'direction': direction, 'command': command, | 80 | entry = {'connection': connection_id, 'direction': direction, 'command': command, |
| 62 | 'message': struct.unpack_from('<Q', frame, at+24)[0]} | 81 | 'message': struct.unpack_from('<Q', frame, at+24)[0], |
| 82 | 'credit_charge': struct.unpack_from('<H', frame, at+6)[0], | ||
| 83 | 'credits': struct.unpack_from('<H', frame, at+14)[0]} | ||
| 63 | if direction == 'response': | 84 | if direction == 'response': |
| 64 | entry['status'] = hex(struct.unpack_from('<I', frame, at+8)[0]) | 85 | entry['status'] = hex(struct.unpack_from('<I', frame, at+8)[0]) |
| 65 | elif command in (8, 9): | 86 | elif command in (8, 9): |
| 66 | entry['length'], entry['offset'] = struct.unpack_from('<IQ', frame, at+68) | 87 | entry['length'], entry['offset'] = struct.unpack_from('<IQ', frame, at+68) |
| 67 | if command == 9 and entry['offset'] <= 96 and entry['offset'] + entry['length'] >= 100: | 88 | if command == 9: |
| 68 | data_offset = struct.unpack_from('<H', frame, at+66)[0] | 89 | data_offset = struct.unpack_from('<H', frame, at+66)[0] |
| 69 | entry['transactions'] = struct.unpack_from('<I', frame, at+data_offset+96-entry['offset'])[0] | 90 | data = frame[at+data_offset:at+data_offset+entry['length']] |
| 91 | entry.update(header_fields(entry['offset'], data)) | ||
| 92 | if record_writes: | ||
| 93 | entry['data'] = data.hex() | ||
| 70 | elif command == 10: | 94 | elif command == 10: |
| 71 | count = struct.unpack_from('<H', frame, at+66)[0] | 95 | count = struct.unpack_from('<H', frame, at+66)[0] |
| 72 | entry['locks'] = [struct.unpack_from('<QQI', frame, at+88+24*i) for i in range(count)] | 96 | entry['locks'] = [struct.unpack_from('<QQI', frame, at+88+24*i) for i in range(count)] |
| 97 | if command == 18 and (direction == 'request' or entry['status'] == '0x0'): | ||
| 98 | size = struct.unpack_from('<H', frame, at+64)[0] | ||
| 99 | entry['oplock_body'] = frame[at+64:at+64+size].hex() | ||
| 100 | if direction == 'request': | ||
| 101 | if command in (6, 7, 8, 9, 10): | ||
| 102 | offset = 80 if command in (8, 9) else 72 | ||
| 103 | entry['file_id'] = frame[at+offset:at+offset+16].hex() | ||
| 104 | elif command == 17: | ||
| 105 | entry['info_type'], entry['info_class'] = struct.unpack_from('<BB', frame, at+66) | ||
| 106 | if record_writes: | ||
| 107 | entry['file_id'] = frame[at+80:at+96].hex() | ||
| 108 | length, offset = struct.unpack_from('<IH', frame, at+68) | ||
| 109 | entry['data'] = frame[at+offset:at+offset+length].hex() | ||
| 110 | elif command == 5: | ||
| 111 | entry['oplock'] = frame[at+67] | ||
| 112 | entry['access'] = struct.unpack_from('<I', frame, at+88)[0] | ||
| 113 | entry['share'] = struct.unpack_from('<I', frame, at+96)[0] | ||
| 114 | offset, length = struct.unpack_from('<HH', frame, at+108) | ||
| 115 | entry['path'] = frame[at+offset:at+offset+length].decode('utf-16-le') | ||
| 116 | elif command == 5 and entry['status'] == '0x0': | ||
| 117 | entry['oplock'] = frame[at+66] | ||
| 118 | entry['file_id'] = frame[at+128:at+144].hex() | ||
| 119 | elif command == 9 and entry['status'] == '0x0': | ||
| 120 | entry['written'] = struct.unpack_from('<I', frame, at+68)[0] | ||
| 121 | if direction == 'request': | ||
| 122 | requests[entry['message']] = entry | ||
| 123 | request = entry | ||
| 124 | else: | ||
| 125 | request = requests.get(entry['message'], {}) | ||
| 126 | if entry['status'] != '0x103': requests.pop(entry['message'], None) | ||
| 127 | if command == 8 and entry['status'] == '0x0' and request.get('offset', 252) < 252: | ||
| 128 | data_offset = frame[at+66] | ||
| 129 | length = struct.unpack_from('<I', frame, at+68)[0] | ||
| 130 | entry.update(header_fields(request['offset'], frame[at+data_offset:at+data_offset+length])) | ||
| 73 | record(**entry) | 131 | record(**entry) |
| 74 | if state.get('cut') == command and direction == 'response' and entry['status'] == state.get('status', '0x0'): | 132 | if (state.get('cut') == command and direction == state.get('direction', 'response') |
| 133 | and (direction == 'request' or entry['status'] == state.get('status', '0x0')) | ||
| 134 | and ('peer' not in state or state['peer'] == client_writer.get_extra_info('peername')[0]) | ||
| 135 | and ('offset' not in state or state['offset'] == request.get('offset'))): | ||
| 75 | matched += 1 | 136 | matched += 1 |
| 76 | if matched == state.get('occurrence', 1): | 137 | if matched == state.get('occurrence', 1): |
| 77 | blocked = True | 138 | local = state.get('scope') == 'connection' |
| 139 | blocked = not local | ||
| 78 | record(cut=entry) | 140 | record(cut=entry) |
| 79 | for stream in list(writers): | 141 | for stream in (client_writer, server_writer) if local else list(writers): |
| 80 | stream.transport.abort() | 142 | stream.transport.abort() |
| 81 | return | 143 | return |
| 82 | next_command = struct.unpack_from('<I', frame, at+20)[0] | 144 | next_command = struct.unpack_from('<I', frame, at+20)[0] |
| ... | @@ -88,23 +150,28 @@ async def main(): | ... | @@ -88,23 +150,28 @@ async def main(): |
| 88 | writer.write(prefix + frame) | 150 | writer.write(prefix + frame) |
| 89 | await writer.drain() | 151 | await writer.drain() |
| 90 | 152 | ||
| 153 | requests = {} | ||
| 91 | tasks = [asyncio.create_task(forward(client, server_writer, 'request')), | 154 | tasks = [asyncio.create_task(forward(client, server_writer, 'request')), |
| 92 | asyncio.create_task(forward(server, client_writer, 'response'))] | 155 | asyncio.create_task(forward(server, client_writer, 'response'))] |
| 93 | done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED) | 156 | done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED) |
| 94 | for task in pending: | 157 | for task in pending: |
| 95 | task.cancel() | 158 | task.cancel() |
| 96 | await asyncio.gather(*tasks, return_exceptions=True) | 159 | results = await asyncio.gather(*tasks, return_exceptions=True) |
| 160 | for result in results: | ||
| 161 | if isinstance(result, Exception) and not isinstance(result, (OSError, asyncio.IncompleteReadError)): | ||
| 162 | record(connection=connection_id, trace_error=repr(result)) | ||
| 97 | except (OSError, asyncio.IncompleteReadError) as error: | 163 | except (OSError, asyncio.IncompleteReadError) as error: |
| 98 | record(error=str(error)) | 164 | record(error=str(error)) |
| 99 | finally: | 165 | finally: |
| 166 | record(connection=connection_id, closed=True) | ||
| 100 | for writer in (client_writer, server_writer): | 167 | for writer in (client_writer, server_writer): |
| 101 | if writer: | 168 | if writer: |
| 102 | writers.discard(writer) | 169 | writers.discard(writer) |
| 103 | writer.close() | 170 | writer.close() |
| 104 | 171 | ||
| 105 | server = await asyncio.start_server(connection, '127.0.0.1', args.port) | 172 | server = await asyncio.start_server(connection, args.bind, args.port) |
| 106 | async with server: | 173 | async with server: |
| 107 | record(listening=args.port) | 174 | record(listening=server.sockets[0].getsockname()[1]) |
| 108 | await asyncio.gather(server.serve_forever(), controls()) | 175 | await asyncio.gather(server.serve_forever(), controls()) |
| 109 | 176 | ||
| 110 | 177 |
tools/smb_faults.py created+82
| ... | @@ -0,0 +1,82 @@ | ||
| 1 | #!/usr/bin/env python3 | ||
| 2 | """Run the embedded transport message-loss matrix on an owned Samba VM.""" | ||
| 3 | import argparse | ||
| 4 | import hashlib | ||
| 5 | import json | ||
| 6 | import os | ||
| 7 | from pathlib import Path | ||
| 8 | import shutil | ||
| 9 | import signal | ||
| 10 | import subprocess | ||
| 11 | import time | ||
| 12 | |||
| 13 | from native_runner import ROOT | ||
| 14 | import linux_vm | ||
| 15 | |||
| 16 | |||
| 17 | def run(output, server): | ||
| 18 | if linux_vm.instance_path(server).exists(): | ||
| 19 | raise ValueError('Choose a new Linux VM name.') | ||
| 20 | output = output.resolve() | ||
| 21 | output.mkdir(parents=True, exist_ok=False) | ||
| 22 | sources = ['tools/smb_faults.py', 'tools/smb-proxy.py', 'Cargo.lock', | ||
| 23 | 'crates/onestore-smb/src/lib.rs', 'crates/onestore-smb/src/tests.rs', | ||
| 24 | 'crates/onestore-smb/src/tests/faults.rs', 'crates/onestore/src/commit.rs', | ||
| 25 | 'crates/onestore/src/snapshot.rs'] | ||
| 26 | for name in sources: | ||
| 27 | target = output / 'harness' / name | ||
| 28 | target.parent.mkdir(parents=True, exist_ok=True) | ||
| 29 | shutil.copyfile(ROOT / name, target) | ||
| 30 | (output / 'run.json').write_text(json.dumps({'server': server, | ||
| 31 | 'source_sha256': {name: hashlib.sha256((output / 'harness' / name).read_bytes()).hexdigest() for name in sources}}, indent=2)) | ||
| 32 | process = None | ||
| 33 | try: | ||
| 34 | linux_vm.create_instance(server) | ||
| 35 | linux_vm.launch(server) | ||
| 36 | linux_vm.wait_instance(server, 600) | ||
| 37 | config = linux_vm.load_instance(server) | ||
| 38 | (output / 'linux.json').write_text(json.dumps(config, indent=2)) | ||
| 39 | with (output / 'test.log').open('w') as log: | ||
| 40 | process = subprocess.Popen(['cargo', 'test', '-p', 'onestore-smb', 'live_message_loss', '--', '--ignored', '--nocapture'], | ||
| 41 | cwd=ROOT, stdout=log, stderr=subprocess.STDOUT, start_new_session=True, | ||
| 42 | env={**os.environ, 'ONESTORE_SMB_LAB': f'127.0.0.1:{config["samba_port"]}', | ||
| 43 | 'ONESTORE_SMB_EVIDENCE': str(output / 'cases')}) | ||
| 44 | if process.wait(timeout=900) != 0: raise RuntimeError('The message-loss matrix failed; inspect test.log.') | ||
| 45 | hashes = linux_vm.run_ssh(server, 'sha256sum /srv/agent/fault-*.one', timeout=30) | ||
| 46 | hashes.check_returncode() | ||
| 47 | (output / 'server-sha256.txt').write_text(hashes.stdout) | ||
| 48 | hashes = {Path(path).name: digest for digest, path in (line.split() for line in hashes.stdout.splitlines())} | ||
| 49 | results = json.loads((output / 'cases/results.json').read_text()) | ||
| 50 | for result in results: | ||
| 51 | captured = output / 'cases/recovered' / (result['case'] + '.one') | ||
| 52 | assert hashlib.sha256(captured.read_bytes()).hexdigest() == hashes[result['path']], 'Recovered bytes differ from the independent server read' | ||
| 53 | (output / 'verification.json').write_text(json.dumps({'cases': len(results), 'server_hashes_match': True}, indent=2)) | ||
| 54 | finally: | ||
| 55 | if process is not None and process.poll() is None: | ||
| 56 | os.killpg(process.pid, signal.SIGTERM) | ||
| 57 | process.wait(timeout=30) | ||
| 58 | if linux_vm.instance_path(server).exists(): | ||
| 59 | if linux_vm.running(server): | ||
| 60 | try: | ||
| 61 | with (output / 'server.tar').open('wb') as archive: | ||
| 62 | subprocess.run(linux_vm.ssh_argv(server, 'tar cf - -C /srv/agent .'), stdout=archive, check=True, timeout=30) | ||
| 63 | result = linux_vm.run_ssh(server, 'sudo smbstatus --byterange --json', timeout=10) | ||
| 64 | (output / 'server-locks.json').write_text(result.stdout) | ||
| 65 | finally: | ||
| 66 | try: linux_vm.shutdown(server, 60) | ||
| 67 | finally: | ||
| 68 | if linux_vm.running(server): linux_vm.qmp(server, 'quit') | ||
| 69 | deadline = time.monotonic() + 10 | ||
| 70 | while linux_vm.running(server) and time.monotonic() < deadline: time.sleep(.1) | ||
| 71 | linux_vm.delete_instance(server) | ||
| 72 | (output / 'teardown.json').write_text(json.dumps({'linux_absent': not linux_vm.instance_path(server).exists()}, indent=2)) | ||
| 73 | |||
| 74 | |||
| 75 | if __name__ == '__main__': | ||
| 76 | parser = argparse.ArgumentParser(description=__doc__) | ||
| 77 | parser.add_argument('output', type=Path) | ||
| 78 | parser.add_argument('--linux', required=True) | ||
| 79 | args = parser.parse_args() | ||
| 80 | def interrupted(_signal, _frame): raise KeyboardInterrupt | ||
| 81 | signal.signal(signal.SIGTERM, interrupted) | ||
| 82 | run(args.output, args.linux) | ||
tools/test_concurrent_rust.py+36-1
| ... | @@ -1,9 +1,44 @@ | ... | @@ -1,9 +1,44 @@ |
| 1 | import copy | 1 | import copy |
| 2 | import json | ||
| 3 | from pathlib import Path | ||
| 4 | import sys | ||
| 5 | import tempfile | ||
| 2 | import unittest | 6 | import unittest |
| 3 | from concurrent_rust import verify | 7 | from concurrent_rust import running_clients, verify |
| 4 | 8 | ||
| 5 | 9 | ||
| 6 | class OracleTests(unittest.TestCase): | 10 | class OracleTests(unittest.TestCase): |
| 11 | def test_timeout_and_environment_reach_every_subprocess(self): | ||
| 12 | with tempfile.TemporaryDirectory() as directory: | ||
| 13 | output = Path(directory) | ||
| 14 | executable = output / 'client' | ||
| 15 | executable.write_text(f'#!{sys.executable}\nimport json, os, sys, time\nprint(json.dumps(dict(os.environ)), flush=True)\nwhile not os.path.exists(sys.argv[5]): time.sleep(.001)\n') | ||
| 16 | executable.chmod(0o700) | ||
| 17 | with running_clients(output, 'unused.one', 2, 1, 1, 7, timeout=1.25, | ||
| 18 | executable=executable, environment={'KEPT': 'value'}): | ||
| 19 | (output / 'start').touch() | ||
| 20 | for actor in ['w0', 'w1', 'r0']: | ||
| 21 | environment = json.loads((output / f'{actor}.jsonl').read_text()) | ||
| 22 | self.assertEqual(environment['ONESTORE_CLIENT_TIMEOUT_MS'], '1250') | ||
| 23 | self.assertEqual(environment['KEPT'], 'value') | ||
| 24 | self.assertEqual(json.loads((output / 'clients.json').read_text())['timeout_ms'], 1250) | ||
| 25 | |||
| 26 | def test_distinct_reader_binary_is_launched_and_recorded(self): | ||
| 27 | with tempfile.TemporaryDirectory() as directory: | ||
| 28 | output = Path(directory) | ||
| 29 | for mode in ('writer', 'reader'): | ||
| 30 | executable = output / mode | ||
| 31 | executable.write_text(f'#!{sys.executable}\nimport os, sys, time\nprint({mode!r}, flush=True)\nwhile not os.path.exists(sys.argv[5]): time.sleep(.001)\n') | ||
| 32 | executable.chmod(0o700) | ||
| 33 | with running_clients(output, 'unused.one', 2, 1, 1, 7, | ||
| 34 | executable=output / 'writer', reader_executable=output / 'reader'): | ||
| 35 | (output / 'start').touch() | ||
| 36 | self.assertEqual((output / 'w0.jsonl').read_text().strip(), 'writer') | ||
| 37 | self.assertEqual((output / 'r0.jsonl').read_text().strip(), 'reader') | ||
| 38 | import hashlib | ||
| 39 | manifest = json.loads((output / 'clients.json').read_text()) | ||
| 40 | self.assertEqual(manifest['reader_sha256'], hashlib.sha256((output / 'reader').read_bytes()).hexdigest()) | ||
| 41 | |||
| 7 | def setUp(self): | 42 | def setUp(self): |
| 8 | self.logs = { | 43 | self.logs = { |
| 9 | 'w0': [{'event': 'ready'}, {'event': 'commit', 'source_transaction': 1, | 44 | 'w0': [{'event': 'ready'}, {'event': 'commit', 'source_transaction': 1, |
tools/test_crash_recovery.py+25-4
| ... | @@ -1,15 +1,34 @@ | ... | @@ -1,15 +1,34 @@ |
| 1 | import copy | 1 | import copy |
| 2 | import json | ||
| 3 | from pathlib import Path | ||
| 4 | import subprocess | ||
| 2 | import unittest | 5 | import unittest |
| 3 | 6 | ||
| 4 | from crash_recovery import verify_text | 7 | from crash_recovery import active_text, verify_text |
| 8 | from document_model import EXPORTER, ordered_pages, view, walk | ||
| 5 | 9 | ||
| 6 | 10 | ||
| 7 | class RecoveryOracleTest(unittest.TestCase): | 11 | class RecoveryOracleTest(unittest.TestCase): |
| 12 | def test_current_page_is_distinguished_from_native_conflict_content(self): | ||
| 13 | source = Path(__file__).resolve().parent.parent / 'corpus/collaboration/round-01/offline/notebook/synthetic.one' | ||
| 14 | model = json.loads(subprocess.check_output([EXPORTER, source])) | ||
| 15 | (sid, _, revision, page), = ordered_pages(model) | ||
| 16 | main = next(node for _, node in walk(revision, page) if node['kind']['type'] == 'RichText') | ||
| 17 | main['kind']['text'] = 'Concurrent edits: current' | ||
| 18 | manifest = revision['nodes'][revision['roots']['1']] | ||
| 19 | self.assertTrue(manifest['spaces']) | ||
| 20 | for conflict in manifest['spaces']: | ||
| 21 | self.assertNotEqual(sid, conflict) | ||
| 22 | _, older = view(model, conflict) | ||
| 23 | node = next(node for node in older['nodes'].values() if node['kind']['type'] == 'RichText') | ||
| 24 | node['kind']['text'] = 'Concurrent edits: older' | ||
| 25 | self.assertEqual(active_text(model), 'Concurrent edits: current') | ||
| 26 | |||
| 8 | def test_accounting_and_corruption_controls(self): | 27 | def test_accounting_and_corruption_controls(self): |
| 9 | logs = {'w0': [ | 28 | logs = {'w0': [ |
| 10 | {'event': 'intent', 'token': ' [w0:0]', 'attempt': 1}, | 29 | {'event': 'intent', 'token': ' [w0:0]', 'replacement': ' [w0:0]', 'before': 'Base', 'range': [4, 4], 'attempt': 1}, |
| 11 | {'event': 'commit', 'token': ' [w0:0]', 'attempt': 1, 'started_us': 5, 'finished_us': 10}, | 30 | {'event': 'commit', 'token': ' [w0:0]', 'attempt': 1, 'started_us': 5, 'finished_us': 10}, |
| 12 | {'event': 'intent', 'token': ' [w0:1]', 'attempt': 2}, | 31 | {'event': 'intent', 'token': ' [w0:1]', 'replacement': ' [w0:1]', 'before': 'Base [w0:0]', 'range': [11, 11], 'attempt': 2}, |
| 13 | {'event': 'commit_error', 'token': ' [w0:1]', 'attempt': 2, 'state': 'Unknown', 'started_us': 18, 'finished_us': 20}, | 32 | {'event': 'commit_error', 'token': ' [w0:1]', 'attempt': 2, 'state': 'Unknown', 'started_us': 18, 'finished_us': 20}, |
| 14 | {'event': 'read', 'text': 'Base [w0:0]', 'started_us': 15, 'finished_us': 17}, | 33 | {'event': 'read', 'text': 'Base [w0:0]', 'started_us': 15, 'finished_us': 17}, |
| 15 | ]} | 34 | ]} |
| ... | @@ -28,8 +47,10 @@ class RecoveryOracleTest(unittest.TestCase): | ... | @@ -28,8 +47,10 @@ class RecoveryOracleTest(unittest.TestCase): |
| 28 | verify_text('Base', 'Base [w0:0]', committed_error) | 47 | verify_text('Base', 'Base [w0:0]', committed_error) |
| 29 | rejected = copy.deepcopy(logs) | 48 | rejected = copy.deepcopy(logs) |
| 30 | rejected['w0'][3]['state'] = 'NotCommitted' | 49 | rejected['w0'][3]['state'] = 'NotCommitted' |
| 31 | with self.assertRaisesRegex(AssertionError, 'definitively uncommitted'): | 50 | with self.assertRaisesRegex(AssertionError, 'recorded publication'): |
| 32 | verify_text('Base', 'Base [w0:0] [w0:1]', rejected) | 51 | verify_text('Base', 'Base [w0:0] [w0:1]', rejected) |
| 52 | with self.assertRaisesRegex(AssertionError, 'recorded publication'): | ||
| 53 | verify_text('Base', 'Base [w0:1] [w0:0]', logs) | ||
| 33 | future = copy.deepcopy(logs) | 54 | future = copy.deepcopy(logs) |
| 34 | future['w0'][1]['started_us'] = 18 | 55 | future['w0'][1]['started_us'] = 18 |
| 35 | with self.assertRaisesRegex(AssertionError, 'future edit'): | 56 | with self.assertRaisesRegex(AssertionError, 'future edit'): |
tools/test_native_disconnect.py created+143
| ... | @@ -0,0 +1,143 @@ | ||
| 1 | import copy | ||
| 2 | import json | ||
| 3 | import os | ||
| 4 | from pathlib import Path | ||
| 5 | import tempfile | ||
| 6 | from types import SimpleNamespace | ||
| 7 | import unittest | ||
| 8 | from unittest.mock import patch | ||
| 9 | |||
| 10 | from native_disconnect import interrupt, verify_disconnect | ||
| 11 | |||
| 12 | |||
| 13 | class DisconnectOracle(unittest.TestCase): | ||
| 14 | def setUp(self): | ||
| 15 | temporary = tempfile.TemporaryDirectory() | ||
| 16 | self.addCleanup(temporary.cleanup) | ||
| 17 | self.root = Path(temporary.name) | ||
| 18 | (self.root / 'rust').mkdir() | ||
| 19 | self.config = {'stress_clients': 4, 'rust_writers': 4, 'rust_readers': 4, 'stress_operations': 80} | ||
| 20 | for name, stamp in [('start', 0), ('stop', 10**9)]: | ||
| 21 | path = self.root / 'rust' / name | ||
| 22 | path.touch() | ||
| 23 | os.utime(path, ns=(stamp, stamp)) | ||
| 24 | for i in range(4): | ||
| 25 | folder = self.root / f'n{i}' | ||
| 26 | folder.mkdir() | ||
| 27 | (folder / 'stress-events.jsonl').write_text('\n'.join(json.dumps({ | ||
| 28 | 'update_started_ticks': j * 10**7, 'updated_ticks': (j + 1) * 10**7}) for j in range(80))) | ||
| 29 | actors = [f'{role}{i}' for role in ('w', 'r') for i in range(4)] | ||
| 30 | self.samples = [{'cycle': i, 'before': dict.fromkeys(actors, 3 + 3 * i), | ||
| 31 | 'after': dict.fromkeys(actors, 6 + 3 * i), 'native_before': [5, 6, 7, 8], | ||
| 32 | 'before_errors': dict.fromkeys(actors, i), 'after_errors': dict.fromkeys(actors, i + 1)} | ||
| 33 | for i in range(2)] | ||
| 34 | for actor in actors: | ||
| 35 | (self.root / 'rust' / (actor + '.jsonl')).write_text('\n'.join(json.dumps({'event': event}) for event in | ||
| 36 | ['ready', 'transport_read_error', 'transport_connected', 'transport_read_error', 'transport_connected', 'done'])) | ||
| 37 | with (self.root / 'rust' / (actor + '.jsonl')).open('a') as stream: | ||
| 38 | stream.write('\n' + json.dumps({'event': 'commit' if actor.startswith('w') else 'read', 'finished_us': 1000000})) | ||
| 39 | for actor, published in [('w0', False), ('w1', True)]: | ||
| 40 | with (self.root / 'rust' / (actor + '.jsonl')).open('a') as stream: | ||
| 41 | stream.write('\n' + json.dumps({'event': 'transport_commit_error', 'state': 'Unknown', 'token': actor}) + '\n') | ||
| 42 | stream.write(json.dumps({'event': 'transport_reconciled', 'published': published, 'flush_confirmed': published, 'token': actor}) + '\n') | ||
| 43 | (self.root / 'smb-trace.jsonl').write_text('{"cut":{}}\n{"control":{"phase":"reconnected-0"}}\n{"control":{"phase":"disconnect-1"}}\n{"cut":{}}\n{"control":{"phase":"reconnected-1"}}\n') | ||
| 44 | |||
| 45 | def check(self): | ||
| 46 | (self.root / 'run.json').write_text(json.dumps(self.config)) | ||
| 47 | (self.root / 'disconnect-progress.json').write_text(json.dumps(self.samples)) | ||
| 48 | with patch('native_disconnect.verify', return_value={'guarded': True}) as overlap: | ||
| 49 | result = verify_disconnect(self.root) | ||
| 50 | self.assertEqual([call.kwargs['phase'] for call in overlap.call_args_list], ['reconnected-0', 'reconnected-1']) | ||
| 51 | self.assertEqual([len(call.args[0]) for call in overlap.call_args_list], [2, 5]) | ||
| 52 | return result | ||
| 53 | |||
| 54 | def test_both_interruptions_require_post_reconnect_overlap(self): | ||
| 55 | self.assertEqual(self.check()['interruptions'], 2) | ||
| 56 | |||
| 57 | def test_stalled_and_backward_clients_are_rejected(self): | ||
| 58 | for actor in ['w3', 'r2']: | ||
| 59 | path = self.root / 'rust' / (actor + '.jsonl') | ||
| 60 | original = path.read_text() | ||
| 61 | for stamp in [120000001, -1]: | ||
| 62 | path.write_text(original.replace('"finished_us": 1000000', f'"finished_us": {stamp}')) | ||
| 63 | with self.assertRaisesRegex(AssertionError, 'progress stalled or went backwards'): self.check() | ||
| 64 | path.write_text(original) | ||
| 65 | path = self.root / 'n2/stress-events.jsonl' | ||
| 66 | rows = [json.loads(line) for line in path.read_text().splitlines()] | ||
| 67 | rows[30]['updated_ticks'] += 121 * 10**7 | ||
| 68 | path.write_text('\n'.join(map(json.dumps, rows))) | ||
| 69 | with self.assertRaisesRegex(AssertionError, 'n2: client progress'): self.check() | ||
| 70 | |||
| 71 | def test_reader_progress_must_cover_the_stop_barrier(self): | ||
| 72 | stamp = 121000001000 | ||
| 73 | os.utime(self.root / 'rust/stop', ns=(stamp, stamp)) | ||
| 74 | with self.assertRaisesRegex(AssertionError, 'r0: client progress'): self.check() | ||
| 75 | |||
| 76 | def test_one_idle_or_unaffected_client_is_rejected(self): | ||
| 77 | original = copy.deepcopy(self.samples) | ||
| 78 | for field, value in [('after', 3), ('after_errors', 0)]: | ||
| 79 | self.samples = copy.deepcopy(original) | ||
| 80 | self.samples[0][field]['r3'] = value | ||
| 81 | with self.assertRaises(AssertionError): self.check() | ||
| 82 | |||
| 83 | def test_missing_actor_is_rejected(self): | ||
| 84 | for sample in self.samples: | ||
| 85 | for field in ('before', 'after', 'before_errors', 'after_errors'): del sample[field]['r3'] | ||
| 86 | with self.assertRaises(AssertionError): self.check() | ||
| 87 | |||
| 88 | def test_completed_native_writer_is_rejected(self): | ||
| 89 | self.samples[0]['native_before'][2] = 80 | ||
| 90 | with self.assertRaises(AssertionError): self.check() | ||
| 91 | |||
| 92 | def test_wrong_cut_count_is_rejected(self): | ||
| 93 | (self.root / 'smb-trace.jsonl').write_text('{"cut":{}}\n') | ||
| 94 | with self.assertRaises(AssertionError): self.check() | ||
| 95 | |||
| 96 | def test_missing_reconnection_is_rejected(self): | ||
| 97 | (self.root / 'rust/r2.jsonl').write_text('{"event":"transport_read_error"}\n') | ||
| 98 | with self.assertRaises(AssertionError): self.check() | ||
| 99 | |||
| 100 | def test_visibility_without_flush_confirmation_is_rejected(self): | ||
| 101 | path = self.root / 'rust/w1.jsonl' | ||
| 102 | path.write_text(path.read_text().replace('"flush_confirmed": true', '"flush_confirmed": false')) | ||
| 103 | with self.assertRaisesRegex(AssertionError, 'durable acknowledgement'): self.check() | ||
| 104 | |||
| 105 | def test_one_uncertain_outcome_is_insufficient(self): | ||
| 106 | path = self.root / 'rust/w0.jsonl' | ||
| 107 | path.write_text(path.read_text().replace('"state": "Unknown"', '"state": "NotCommitted"')) | ||
| 108 | with self.assertRaisesRegex(AssertionError, 'both uncertain outcomes'): self.check() | ||
| 109 | |||
| 110 | def test_failed_cut_observation_restores_the_connection(self): | ||
| 111 | (self.root / 'run.json').write_text(json.dumps({**self.config, 'server': 'owned-test'})) | ||
| 112 | (self.root / 'rust/w0.jsonl').write_text('{"event":"commit"}\n' * 3) | ||
| 113 | controls = [] | ||
| 114 | (self.root / 'harness').mkdir() | ||
| 115 | (self.root / 'harness/verify_smb_overlap.py').write_text('test oracle') | ||
| 116 | |||
| 117 | def ssh(server, command, timeout): | ||
| 118 | self.assertEqual(server, 'owned-test') | ||
| 119 | if command.startswith('printf'): | ||
| 120 | controls.append(json.loads(command[command.index('{'):command.index('}') + 1])) | ||
| 121 | output = '' | ||
| 122 | elif (failed_phase_ack and controls[-1]['phase'] == 'disconnect-0') or command.startswith('grep -c'): | ||
| 123 | raise RuntimeError('Fault observation failed') | ||
| 124 | else: output = json.dumps({'control': controls[-1]}) | ||
| 125 | return SimpleNamespace(stdout=output, stderr='', returncode=0, check_returncode=lambda: None) | ||
| 126 | |||
| 127 | def capture(remote, destination, name): | ||
| 128 | destination.write_text('{"operation":0}\n') | ||
| 129 | return {'error': None} | ||
| 130 | |||
| 131 | for failed_phase_ack in [False, True]: | ||
| 132 | controls.clear() | ||
| 133 | with self.subTest(failed_phase_ack=failed_phase_ack), \ | ||
| 134 | patch('native_disconnect.linux_vm.run_ssh', side_effect=ssh), \ | ||
| 135 | patch('native_disconnect.linux_vm.ssh_argv', return_value=['ssh', 'owned-test']), \ | ||
| 136 | patch('native_disconnect.subprocess.run'), \ | ||
| 137 | patch('native_disconnect.windows.do_get', side_effect=capture): | ||
| 138 | with self.assertRaisesRegex(RuntimeError, 'Fault observation failed'): | ||
| 139 | interrupt(self.root, [{'name': 'native'}], [1], {'w0': SimpleNamespace(poll=lambda: None)}) | ||
| 140 | self.assertEqual([control['phase'] for control in controls], ['disconnect-0', 'reconnected-0']) | ||
| 141 | |||
| 142 | |||
| 143 | if __name__ == '__main__': unittest.main() | ||
tools/test_native_maintenance.py created+53
| ... | @@ -0,0 +1,53 @@ | ||
| 1 | import unittest | ||
| 2 | |||
| 3 | from native_maintenance import maintenance_locks | ||
| 4 | |||
| 5 | |||
| 6 | class Maintenance(unittest.TestCase): | ||
| 7 | def history(self, path='m6-collaboration/synthetic.one', host='192.168.77.2', status='0xc0000055'): | ||
| 8 | events = [{'connection': 1, 'opened': True, 'peer': [host, 1]}] | ||
| 9 | def exchange(command, status='0x0', **fields): | ||
| 10 | message = len(events) | ||
| 11 | events.extend([ | ||
| 12 | {'connection': 1, 'direction': 'request', 'command': command, 'message': message, **fields}, | ||
| 13 | {'connection': 1, 'direction': 'response', 'command': command, 'message': message, 'status': status, 'file_id': 'file'}, | ||
| 14 | ]) | ||
| 15 | exchange(5, path=path) | ||
| 16 | for phase, result in [('maintenance-held', status), ('maintenance-released', '0x0')]: | ||
| 17 | events.append({'control': {'phase': phase}}) | ||
| 18 | exchange(10, result, file_id='file', locks=[(0xffffeffc, 4096, 0x12)]) | ||
| 19 | return events | ||
| 20 | |||
| 21 | def test_section_denied_then_acquired(self): | ||
| 22 | self.assertEqual(len(maintenance_locks(self.history())), 2) | ||
| 23 | |||
| 24 | def test_toc_compaction_does_not_count(self): | ||
| 25 | with self.assertRaises(AssertionError): | ||
| 26 | maintenance_locks(self.history(path='m6-collaboration/Open Notebook.onetoc2')) | ||
| 27 | |||
| 28 | def test_rust_maintenance_does_not_count(self): | ||
| 29 | with self.assertRaises(AssertionError): | ||
| 30 | maintenance_locks(self.history(host='10.0.2.2')) | ||
| 31 | |||
| 32 | def test_unrelated_error_does_not_prove_exclusion(self): | ||
| 33 | with self.assertRaises(AssertionError): | ||
| 34 | maintenance_locks(self.history(status='0xc0000001')) | ||
| 35 | |||
| 36 | def test_delayed_response_uses_request_phase(self): | ||
| 37 | events = self.history() | ||
| 38 | events.insert(-1, {'control': {'phase': 'maintenance-resumed'}}) | ||
| 39 | self.assertEqual(len(maintenance_locks(events)), 2) | ||
| 40 | events = self.history() | ||
| 41 | events.insert(-2, {'control': {'phase': 'maintenance-resumed'}}) | ||
| 42 | with self.assertRaises(AssertionError): | ||
| 43 | maintenance_locks(events) | ||
| 44 | |||
| 45 | def test_shared_or_short_lock_is_not_maintenance(self): | ||
| 46 | for lock in [(0xffffeffc, 4096, 0x11), (0xfffffffc, 1, 0x12)]: | ||
| 47 | events = self.history() | ||
| 48 | for event in events: | ||
| 49 | if 'locks' in event: event['locks'] = [lock] | ||
| 50 | with self.assertRaises(AssertionError): maintenance_locks(events) | ||
| 51 | |||
| 52 | |||
| 53 | if __name__ == '__main__': unittest.main() | ||
tools/test_native_stress.py+48
| ... | @@ -1,9 +1,12 @@ | ... | @@ -1,9 +1,12 @@ |
| 1 | import copy | 1 | import copy |
| 2 | from contextlib import contextmanager | ||
| 2 | import json | 3 | import json |
| 3 | from pathlib import Path | 4 | from pathlib import Path |
| 4 | import subprocess | 5 | import subprocess |
| 5 | import tempfile | 6 | import tempfile |
| 6 | import unittest | 7 | import unittest |
| 8 | from unittest.mock import patch | ||
| 9 | import native_stress | ||
| 7 | from native_stress import edit_history, native_history, verify_capture | 10 | from native_stress import edit_history, native_history, verify_capture |
| 8 | from native_collaboration import reachable_page_text | 11 | from native_collaboration import reachable_page_text |
| 9 | from document_model import DEFAULT_CONTEXT | 12 | from document_model import DEFAULT_CONTEXT |
| ... | @@ -50,6 +53,19 @@ class NativeHistoryTests(unittest.TestCase): | ... | @@ -50,6 +53,19 @@ class NativeHistoryTests(unittest.TestCase): |
| 50 | self.logs['r0'] = [{'event': 'ready'}, {'event': 'read', 'text': text, | 53 | self.logs['r0'] = [{'event': 'ready'}, {'event': 'read', 'text': text, |
| 51 | 'started_us': 40, 'finished_us': 50}, {'event': 'done'}] | 54 | 'started_us': 40, 'finished_us': 50}, {'event': 'done'}] |
| 52 | 55 | ||
| 56 | def test_incomplete_capture_requires_explicit_retention_mode(self): | ||
| 57 | self.logs['w0'].pop() | ||
| 58 | with self.assertRaisesRegex(AssertionError, 'Incomplete client log'): | ||
| 59 | edit_history(self.logs, 2) | ||
| 60 | self.assertEqual(len(edit_history(self.logs, 2, partial=True)[0]), 2) | ||
| 61 | self.logs['w1'][2]['operation'] = 1 | ||
| 62 | with self.assertRaisesRegex(AssertionError, 'acknowledgements'): | ||
| 63 | edit_history(self.logs, 2, partial=True) | ||
| 64 | self.logs['w1'][2]['operation'] = 0 | ||
| 65 | self.logs['w1'].pop(2) | ||
| 66 | with self.assertRaisesRegex(AssertionError, 'reader observed'): | ||
| 67 | edit_history(self.logs, 2, partial=True) | ||
| 68 | |||
| 53 | def test_counter_renumbering_preserves_one_content_history(self): | 69 | def test_counter_renumbering_preserves_one_content_history(self): |
| 54 | commits, text = edit_history(self.logs, 1) | 70 | commits, text = edit_history(self.logs, 1) |
| 55 | self.assertEqual(len(commits), 2) | 71 | self.assertEqual(len(commits), 2) |
| ... | @@ -136,3 +152,35 @@ class NativeHistoryTests(unittest.TestCase): | ... | @@ -136,3 +152,35 @@ class NativeHistoryTests(unittest.TestCase): |
| 136 | for changed in (body + native, body.replace(native, ''), body.replace('bold', 'normal')): | 152 | for changed in (body + native, body.replace(native, ''), body.replace('bold', 'normal')): |
| 137 | page.write_text(f'<Page xmlns="http://schemas.microsoft.com/office/onenote/2010/onenote"><Outline>{changed}</Outline></Page>') | 153 | page.write_text(f'<Page xmlns="http://schemas.microsoft.com/office/onenote/2010/onenote"><Outline>{changed}</Outline></Page>') |
| 138 | with self.assertRaises(AssertionError): verify_capture(root, root) | 154 | with self.assertRaises(AssertionError): verify_capture(root, root) |
| 155 | |||
| 156 | |||
| 157 | class FailureArtifacts(unittest.TestCase): | ||
| 158 | def test_failed_clients_preserve_native_logs_without_masking_the_failure(self): | ||
| 159 | @contextmanager | ||
| 160 | def failed_clients(*args, **kwargs): | ||
| 161 | yield {} | ||
| 162 | raise RuntimeError('Rust client exited') | ||
| 163 | |||
| 164 | with tempfile.TemporaryDirectory() as directory: | ||
| 165 | output = Path(directory) | ||
| 166 | (output / 'run.json').write_text(json.dumps({'maintenance': False})) | ||
| 167 | shared = output / 'shared' | ||
| 168 | shared.mkdir() | ||
| 169 | clients = [{'name': f'n{i}', 'folder': output / f'n{i}'} for i in range(2)] | ||
| 170 | for client in clients: client['folder'].mkdir() | ||
| 171 | |||
| 172 | def capture(remote, local, name): | ||
| 173 | self.assertTrue(remote.endswith('\\outbox\\7\\events.jsonl')) | ||
| 174 | if name == 'n0': raise OSError('Native client unavailable') | ||
| 175 | local.write_text('{"operation":0}\n') | ||
| 176 | return {'error': None} | ||
| 177 | |||
| 178 | with patch.object(native_stress, 'running_clients', failed_clients), \ | ||
| 179 | patch.object(native_stress.windows, 'do_health', return_value={'utc_us': 0}), \ | ||
| 180 | patch.object(native_stress.windows, 'do_cmd', return_value={'stdout': 'ready editing'}), \ | ||
| 181 | patch.object(native_stress.windows, 'do_get', side_effect=capture): | ||
| 182 | with self.assertRaisesRegex(RuntimeError, 'Rust client exited'): | ||
| 183 | native_stress.exercise(output, shared, clients, lambda *a, **kw: 7, | ||
| 184 | lambda *a: None, lambda *a: None, lambda *a: None, 40, 1, 4, 4) | ||
| 185 | self.assertEqual(json.loads((output / 'n0/stress-capture.json').read_text())['error'], 'Native client unavailable') | ||
| 186 | self.assertEqual((output / 'n1/stress-events.jsonl').read_text(), '{"operation":0}\n') |
tools/test_notebook_editor.py created+237
| ... | @@ -0,0 +1,237 @@ | ||
| 1 | import json | ||
| 2 | from pathlib import Path | ||
| 3 | import tempfile | ||
| 4 | from threading import Thread | ||
| 5 | import unittest | ||
| 6 | from unittest.mock import patch | ||
| 7 | from urllib.error import HTTPError | ||
| 8 | from urllib.parse import urlencode | ||
| 9 | from urllib.request import Request, urlopen | ||
| 10 | |||
| 11 | from document_model import view, walk | ||
| 12 | import notebook_editor as editor | ||
| 13 | from random_edit_campaign import export, verify | ||
| 14 | |||
| 15 | |||
| 16 | class EditorTest(unittest.TestCase): | ||
| 17 | def setUp(self): | ||
| 18 | self.temporary = tempfile.TemporaryDirectory() | ||
| 19 | self.addCleanup(self.temporary.cleanup) | ||
| 20 | source = editor.ROOT / 'corpus/native/20260905-05/snapshots/06-attachment/notebook' | ||
| 21 | self.session = editor.Session(source, Path(self.temporary.name) / 'session') | ||
| 22 | self.server = editor.ThreadingHTTPServer(('127.0.0.1', 0), editor.Handler) | ||
| 23 | self.server.session = self.session | ||
| 24 | self.thread = Thread(target=self.server.serve_forever) | ||
| 25 | self.thread.start() | ||
| 26 | self.addCleanup(self.stop) | ||
| 27 | self.url = f'http://127.0.0.1:{self.server.server_port}' | ||
| 28 | self.file = self.session.output / 'notebook/synthetic.one' | ||
| 29 | self.before = export(self.file) | ||
| 30 | self.row = json.loads((self.session.output / 'g/0/report/pages.json').read_text())[0] | ||
| 31 | _, revision = view(self.before[0], self.row['space']) | ||
| 32 | self.oid, node = next((oid, node) for oid, node in walk(revision, self.row['object']) | ||
| 33 | if node['kind']['type'] == 'RichText' and node['kind']['text'].startswith('Fictitious')) | ||
| 34 | self.selection = {'generation': 0, 'page': self.row['report'], 'object': self.oid, 'run': 0, 'action': 'text'} | ||
| 35 | self.text = self.before[1][self.row['space']][self.row['revision']][self.oid][0]['text'] | ||
| 36 | |||
| 37 | def stop(self): | ||
| 38 | self.server.shutdown() | ||
| 39 | self.thread.join() | ||
| 40 | self.server.server_close() | ||
| 41 | |||
| 42 | def request(self, path, data=None): | ||
| 43 | request = Request(self.url + path, data=json.dumps(data).encode() if data is not None else None, | ||
| 44 | headers={'Content-Type': 'application/json', 'X-OneNote-Diagnostic': '1'}) | ||
| 45 | try: | ||
| 46 | response = urlopen(request) | ||
| 47 | except HTTPError as error: | ||
| 48 | response = error | ||
| 49 | with response: | ||
| 50 | return response.status, json.loads(response.read()) | ||
| 51 | |||
| 52 | def test_unicode_stale_snapshot_and_unrelated_content(self): | ||
| 53 | status, checked = self.request('/api/run?' + urlencode(self.selection)) | ||
| 54 | self.assertEqual((status, checked['ok'], checked['text']), (200, True, self.text)) | ||
| 55 | replacement = self.text + ' café 🦀 e\u0301 <diagnostic>' | ||
| 56 | status, saved = self.request('/api/save', {**self.selection, 'replacement': replacement}) | ||
| 57 | self.assertEqual((status, saved['state'], saved['ok']), (200, 'Committed', True)) | ||
| 58 | intent, outcome = [json.loads(s) for s in (self.session.output / 'operations.jsonl').read_text().splitlines()] | ||
| 59 | edit = intent['edit'] | ||
| 60 | verify(self.before, export(self.file), {'page': self.row['object'], 'space': edit['space'], 'object': self.oid, | ||
| 61 | 'range': [edit['action']['start'], edit['action']['end']], 'replacement': replacement, 'run_start': edit['action']['start'], | ||
| 62 | 'run_before': self.text, 'started_ms': intent['started_ms'], 'finished_ms': outcome['finished_ms']}) | ||
| 63 | after = self.file.read_bytes() | ||
| 64 | fresh = self.session.output / 'fresh-report' | ||
| 65 | editor.generate(self.session.output / 'g/1/snapshot', fresh, editable=True) | ||
| 66 | cached = self.session.output / 'g/1/report' | ||
| 67 | self.assertEqual({p.relative_to(fresh): p.read_bytes() for p in fresh.rglob('*') if p.is_file()}, | ||
| 68 | {p.relative_to(cached): p.read_bytes() for p in cached.rglob('*') if p.is_file()}) | ||
| 69 | status, stale = self.request('/api/save', {**self.selection, 'replacement': 'another draft'}) | ||
| 70 | self.assertEqual((status, stale['state'], stale['kind']), (409, 'NotCommitted', 'ResourceBusy')) | ||
| 71 | self.assertEqual(self.file.read_bytes(), after) | ||
| 72 | self.assertEqual((self.session.output / 'g/0/snapshot/synthetic.one').read_bytes(), | ||
| 73 | (editor.ROOT / 'corpus/native/20260905-05/snapshots/06-attachment/notebook/synthetic.one').read_bytes()) | ||
| 74 | self.assertEqual((self.session.output / 'g/1/snapshot/synthetic.one').stat().st_ino, | ||
| 75 | (self.session.output / 'g/2/snapshot/synthetic.one').stat().st_ino) | ||
| 76 | with urlopen(self.url + saved['location']) as response: | ||
| 77 | html = response.read().decode() | ||
| 78 | self.assertIn('<diagnostic>', html) | ||
| 79 | self.assertNotIn('<diagnostic>', html) | ||
| 80 | |||
| 81 | def test_invalid_edits_and_selection_never_write(self): | ||
| 82 | before = self.file.read_bytes() | ||
| 83 | for changes in [{'replacement': 'first\nsecond'}, {'replacement': '\ud800'}, | ||
| 84 | {'run': -1}, {'generation': True}, {'object': 'missing'}, {'extra': 'field'}]: | ||
| 85 | status, result = self.request('/api/save', {**self.selection, 'replacement': 'changed', **changes}) | ||
| 86 | self.assertEqual((status, result['state']), (422, 'NotCommitted')) | ||
| 87 | self.assertEqual(self.file.read_bytes(), before) | ||
| 88 | |||
| 89 | def test_unknown_response_does_not_replay_a_real_commit(self): | ||
| 90 | original = editor.bridge | ||
| 91 | calls = [] | ||
| 92 | def uncertain(mode, *args): | ||
| 93 | result = original(mode, *args) | ||
| 94 | if mode == 'commit': | ||
| 95 | calls.append(result) | ||
| 96 | self.assertTrue(result['ok']) | ||
| 97 | return {'ok': False, 'state': 'Unknown', 'error': 'Simulated lost outcome'} | ||
| 98 | return result | ||
| 99 | with patch.object(editor, 'bridge', side_effect=uncertain): | ||
| 100 | status, result = self.request('/api/save', {**self.selection, 'replacement': self.text + ' once'}) | ||
| 101 | self.assertEqual((status, result['state'], len(calls)), (503, 'Unknown', 1)) | ||
| 102 | after = export(self.file) | ||
| 103 | rid, _ = view(after[0], self.row['space']) | ||
| 104 | self.assertEqual(after[1][self.row['space']][rid][self.oid][0]['text'], self.text + ' once') | ||
| 105 | status, stale = self.request('/api/save', {**self.selection, 'replacement': self.text + ' twice'}) | ||
| 106 | self.assertEqual((status, stale['kind']), (409, 'ResourceBusy')) | ||
| 107 | |||
| 108 | def test_refresh_failure_preserves_committed_outcome(self): | ||
| 109 | with patch.object(self.session, 'snapshot', side_effect=OSError('Report storage unavailable')): | ||
| 110 | status, result = self.request('/api/save', {**self.selection, 'replacement': self.text + ' saved'}) | ||
| 111 | self.assertEqual((status, result['state'], result['ok']), (503, 'Committed', False)) | ||
| 112 | self.assertIn('Report storage', result['report_error']) | ||
| 113 | after = export(self.file) | ||
| 114 | rid, _ = view(after[0], self.row['space']) | ||
| 115 | self.assertEqual(after[1][self.row['space']][rid][self.oid][0]['text'], self.text + ' saved') | ||
| 116 | with urlopen(self.url + '/latest?' + urlencode(self.selection)) as response: | ||
| 117 | self.assertIn(' saved', response.read().decode()) | ||
| 118 | |||
| 119 | def test_document_actions_preserve_placement_unicode_and_unselected_formatting(self): | ||
| 120 | status, page = self.request('/api/page?' + urlencode(self.selection)) | ||
| 121 | self.assertEqual(status, 200) | ||
| 122 | outline = next(target for target in page['targets'] if target['label'].startswith('Outline')) | ||
| 123 | base = {'generation': 0, 'page': self.row['report'], 'object': outline['object'], 'action': 'paragraph', | ||
| 124 | 'before': outline['children'][0]['object'], 'text': 'A🦀 café\rsecond line', 'author': 'Diagnostic test'} | ||
| 125 | status, saved = self.request('/api/save', base) | ||
| 126 | self.assertEqual((status, saved['state']), (200, 'Committed')) | ||
| 127 | after = export(self.file) | ||
| 128 | _, revision = view(after[0], self.row['space']) | ||
| 129 | children = revision['nodes'][outline['object']]['children'] | ||
| 130 | self.assertEqual(children[1], base['before']) | ||
| 131 | oid, node = next((key, node) for key, node in walk(revision, children[0]) if node['kind']['type'] == 'RichText') | ||
| 132 | self.assertEqual(node['kind']['text'], base['text']) | ||
| 133 | self.assertEqual(self.before[2], after[2]) | ||
| 134 | attrs = [{'Bold': True}, {'Italic': True}, {'Underline': True}, {'Strike': True}, | ||
| 135 | {'Superscript': True}, {'Subscript': False}, {'Font': 'Arial'}, {'FontSize': 20.5}, | ||
| 136 | {'Color': [18, 52, 86]}, {'Highlight': [255, 255, 0]}] | ||
| 137 | selected = {'generation': 1, 'page': saved['location'].split('/')[-1], 'object': oid, | ||
| 138 | 'run': 0, 'action': 'format', 'start': 1, 'end': 3, 'attributes': attrs} | ||
| 139 | status, formatted = self.request('/api/save', selected) | ||
| 140 | self.assertEqual((status, formatted['state']), (200, 'Committed')) | ||
| 141 | after_format = export(self.file) | ||
| 142 | rid, _ = view(after_format[0], self.row['space']) | ||
| 143 | runs = after_format[1][self.row['space']][rid][oid] | ||
| 144 | self.assertEqual([run['text'] for run in runs], ['A', '🦀', ' café\rsecond line']) | ||
| 145 | original_rid, _ = view(after[0], self.row['space']) | ||
| 146 | original_format = after[1][self.row['space']][original_rid][oid][0]['format'] | ||
| 147 | self.assertEqual(runs[0]['format'], original_format) | ||
| 148 | self.assertEqual(runs[2]['format'], original_format) | ||
| 149 | for key, value in {'bold': True, 'italic': True, 'underline': True, 'strike': True, | ||
| 150 | 'superscript': True, 'subscript': False, 'font': 'Arial', 'font_size': 20.5, | ||
| 151 | 'color': 0x563412, 'highlight': 0xffff}.items(): | ||
| 152 | self.assertEqual(runs[1]['format'][key], value, key) | ||
| 153 | status, cleared = self.request('/api/save', {**selected, 'generation': 2, 'run': 1, 'start': 0, 'end': 2, | ||
| 154 | 'attributes': [{'Subscript': True}, {'Color': None}, {'Highlight': None}]}) | ||
| 155 | self.assertEqual(status, 200) | ||
| 156 | cleared_model = export(self.file) | ||
| 157 | rid, _ = view(cleared_model[0], self.row['space']) | ||
| 158 | fmt = cleared_model[1][self.row['space']][rid][oid][1]['format'] | ||
| 159 | self.assertEqual((fmt['superscript'], fmt['subscript'], fmt['color'], fmt['highlight']), (False, True, 0xff000000, 0xff000000)) | ||
| 160 | status, added = self.request('/api/save', {'generation': 3, 'page': selected['page'], 'action': 'outline', | ||
| 161 | 'object': self.row['object'], 'x': 216.5, 'y': 360, | ||
| 162 | 'text': 'New outline 🦀', 'author': 'Diagnostic test'}) | ||
| 163 | self.assertEqual(status, 200) | ||
| 164 | _, revision = view(export(self.file)[0], self.row['space']) | ||
| 165 | new_outline = revision['nodes'][revision['nodes'][self.row['object']]['children'][-1]] | ||
| 166 | self.assertEqual(new_outline['kind']['type'], 'Outline') | ||
| 167 | self.assertEqual((new_outline['layout']['x'], new_outline['layout']['y']), (216.5, 360)) | ||
| 168 | committed = self.file.read_bytes() | ||
| 169 | for request in (base, selected): | ||
| 170 | status, stale = self.request('/api/save', request) | ||
| 171 | self.assertEqual((status, stale['state'], stale['kind']), (409, 'NotCommitted', 'ResourceBusy')) | ||
| 172 | self.assertEqual(self.file.read_bytes(), committed) | ||
| 173 | |||
| 174 | def test_document_rejections_do_not_publish(self): | ||
| 175 | before = self.file.read_bytes() | ||
| 176 | common = {'generation': 0, 'page': self.row['report'], 'object': self.oid} | ||
| 177 | requests = [ | ||
| 178 | {**common, 'action': 'format', 'run': 0, 'start': 0, 'end': 1, 'attributes': []}, | ||
| 179 | {**common, 'action': 'format', 'run': 0, 'start': True, 'end': 1, 'attributes': [{'Bold': True}]}, | ||
| 180 | {**common, 'action': 'format', 'run': 0, 'start': 0, 'end': 99999, 'attributes': [{'Bold': True}]}, | ||
| 181 | {**common, 'action': 'format', 'run': 0, 'start': 0, 'end': 1, 'attributes': [{'FontSize': 144}]}, | ||
| 182 | {**common, 'action': 'format', 'run': 0, 'start': 0, 'end': 1, 'attributes': [{'Bold': True}, {'Bold': False}]}, | ||
| 183 | {**common, 'action': 'outline', 'x': 1, 'y': 1, 'text': 'No page parent', 'author': 'test'}, | ||
| 184 | {**common, 'action': 'paragraph', 'before': None, 'text': 'No paragraph parent', 'author': 'test'}, | ||
| 185 | ] | ||
| 186 | for request in requests: | ||
| 187 | status, result = self.request('/api/save', request) | ||
| 188 | self.assertEqual((status, result['state']), (422, 'NotCommitted')) | ||
| 189 | self.assertEqual(self.file.read_bytes(), before) | ||
| 190 | |||
| 191 | def test_uncertain_insertion_has_one_publication_and_a_stale_retry(self): | ||
| 192 | request = {'generation': 0, 'page': self.row['report'], 'object': self.row['object'], | ||
| 193 | 'action': 'outline', 'x': 144, 'y': 288, 'text': 'Only once 🦀', 'author': 'test'} | ||
| 194 | original = editor.bridge | ||
| 195 | def uncertain(mode, *args): | ||
| 196 | result = original(mode, *args) | ||
| 197 | return {'ok': False, 'state': 'Unknown'} if mode == 'commit' and result['ok'] else result | ||
| 198 | with patch.object(editor, 'bridge', side_effect=uncertain): | ||
| 199 | status, result = self.request('/api/save', request) | ||
| 200 | self.assertEqual((status, result['state']), (503, 'Unknown')) | ||
| 201 | status, result = self.request('/api/save', request) | ||
| 202 | self.assertEqual((status, result['kind']), (409, 'ResourceBusy')) | ||
| 203 | _, revision = view(export(self.file)[0], self.row['space']) | ||
| 204 | self.assertEqual(sum(n['kind'].get('text') == 'Only once 🦀' for _, n in walk(revision, self.row['object'])), 1) | ||
| 205 | |||
| 206 | def test_generated_fields_are_read_only(self): | ||
| 207 | source = editor.ROOT / 'corpus/m6/native-structure-01/notebook' | ||
| 208 | session = editor.Session(source, Path(self.temporary.name) / 'fields') | ||
| 209 | self.server.session = session | ||
| 210 | pages = json.loads((session.output / 'g/0/report/pages.json').read_text()) | ||
| 211 | document = export(source / 'synthetic.one')[0] | ||
| 212 | for row in pages: | ||
| 213 | _, revision = view(document, row['space']) | ||
| 214 | fields = [oid for oid, node in walk(revision, row['object']) | ||
| 215 | if node['kind']['type'] == 'RichText' and node['kind']['boilerplate']] | ||
| 216 | if fields: break | ||
| 217 | self.assertTrue(fields) | ||
| 218 | status, result = self.request('/api/run?' + urlencode({'generation': 0, 'page': row['report'], 'object': fields[0], 'run': 0})) | ||
| 219 | self.assertEqual((status, result['ok']), (422, False)) | ||
| 220 | |||
| 221 | def test_templates_keep_reader_only_controls(self): | ||
| 222 | session = editor.Session(editor.ROOT / 'corpus/m6/native-template-controls-01/notebook', | ||
| 223 | Path(self.temporary.name) / 'template') | ||
| 224 | self.server.session = session | ||
| 225 | pages = json.loads((session.output / 'g/0/report/pages.json').read_text()) | ||
| 226 | protected = [row for row in pages if row['category'] == 'Default page template'] | ||
| 227 | self.assertTrue(protected) | ||
| 228 | for row in protected: | ||
| 229 | status, result = self.request('/api/run?' + urlencode({'generation': 0, 'page': row['report'], 'object': row['object'], 'run': 0})) | ||
| 230 | self.assertEqual(status, 422) | ||
| 231 | self.assertIn('active page', result['error']) | ||
| 232 | html = (session.output / 'g/0/report' / row['report']).read_text() | ||
| 233 | self.assertNotIn('src="/editor.js"', html) | ||
| 234 | |||
| 235 | |||
| 236 | if __name__ == '__main__': | ||
| 237 | unittest.main() | ||
tools/test_notebook_report.py+33
| ... | @@ -4,7 +4,9 @@ import json | ... | @@ -4,7 +4,9 @@ import json |
| 4 | from pathlib import Path | 4 | from pathlib import Path |
| 5 | import re | 5 | import re |
| 6 | from tempfile import TemporaryDirectory | 6 | from tempfile import TemporaryDirectory |
| 7 | import shutil | ||
| 7 | import unittest | 8 | import unittest |
| 9 | from unittest.mock import patch | ||
| 8 | import xml.etree.ElementTree as ET | 10 | import xml.etree.ElementTree as ET |
| 9 | 11 | ||
| 10 | from PIL import Image | 12 | from PIL import Image |
| ... | @@ -32,6 +34,37 @@ class NotebookReportTest(unittest.TestCase): | ... | @@ -32,6 +34,37 @@ class NotebookReportTest(unittest.TestCase): |
| 32 | references = [a['path'] for p in (output / 'model').glob('*/assets.json') for a in json.loads(p.read_text())] | 34 | references = [a['path'] for p in (output / 'model').glob('*/assets.json') for a in json.loads(p.read_text())] |
| 33 | self.assertIn('../../assets/' + original.name, references) | 35 | self.assertIn('../../assets/' + original.name, references) |
| 34 | 36 | ||
| 37 | def test_reused_models_and_assets_match_a_fresh_report_after_source_order_changes(self): | ||
| 38 | fixture = Path(__file__).resolve().parent.parent / 'corpus/m6/native-features-01/notebook' | ||
| 39 | with TemporaryDirectory() as temporary: | ||
| 40 | root = Path(temporary) | ||
| 41 | source = root / 'notebook' | ||
| 42 | shutil.copytree(fixture, source) | ||
| 43 | generate(source, root / 'first') | ||
| 44 | before = {p.relative_to(root / 'first'): p.read_bytes() for p in (root / 'first').rglob('*') if p.is_file()} | ||
| 45 | with patch('notebook_report.subprocess.run', side_effect=AssertionError('Unchanged model exported again')): | ||
| 46 | generate(source, root / 'same', previous=root / 'first') | ||
| 47 | self.assertEqual(before, {p.relative_to(root / 'same'): p.read_bytes() for p in (root / 'same').rglob('*') if p.is_file()}) | ||
| 48 | shutil.copyfile(source / 'synthetic.one', source / 'a.one') | ||
| 49 | import notebook_report | ||
| 50 | original = notebook_report.subprocess.run | ||
| 51 | with patch('notebook_report.subprocess.run', wraps=original) as run: | ||
| 52 | generate(source, root / 'changed', previous=root / 'first') | ||
| 53 | self.assertEqual([Path(call.args[0][1]).name for call in run.call_args_list], ['a.one']) | ||
| 54 | generate(source, root / 'fresh') | ||
| 55 | self.assertEqual({p.relative_to(root / 'fresh'): p.read_bytes() for p in (root / 'fresh').rglob('*') if p.is_file()}, | ||
| 56 | {p.relative_to(root / 'changed'): p.read_bytes() for p in (root / 'changed').rglob('*') if p.is_file()}) | ||
| 57 | for name, contents in before.items(): | ||
| 58 | self.assertEqual((root / 'first' / name).read_bytes(), contents) | ||
| 59 | old_index = next(i for i, row in enumerate(json.loads((root / 'first/source.json').read_text())) if row['path'] == 'synthetic.one') | ||
| 60 | new_index = next(i for i, row in enumerate(json.loads((root / 'changed/source.json').read_text())) if row['path'] == 'synthetic.one') | ||
| 61 | self.assertEqual((root / f'first/model/{old_index}/document.json').stat().st_ino, | ||
| 62 | (root / f'changed/model/{new_index}/document.json').stat().st_ino) | ||
| 63 | (source / 'a.one').unlink() | ||
| 64 | with patch('notebook_report.subprocess.run', side_effect=AssertionError('Unchanged model exported again')): | ||
| 65 | generate(source, root / 'deleted', previous=root / 'changed') | ||
| 66 | self.assertEqual(before, {p.relative_to(root / 'deleted'): p.read_bytes() for p in (root / 'deleted').rglob('*') if p.is_file()}) | ||
| 67 | |||
| 35 | 68 | ||
| 36 | if __name__ == '__main__': | 69 | if __name__ == '__main__': |
| 37 | unittest.main() | 70 | unittest.main() |
tools/test_offline_confirmation.py created+88
| ... | @@ -0,0 +1,88 @@ | ||
| 1 | import hashlib | ||
| 2 | import json | ||
| 3 | from pathlib import Path | ||
| 4 | import tempfile | ||
| 5 | import unittest | ||
| 6 | from unittest.mock import patch | ||
| 7 | |||
| 8 | from verify_offline_confirmation import verify | ||
| 9 | |||
| 10 | |||
| 11 | class ConfirmationOracle(unittest.TestCase): | ||
| 12 | def test_native_images_must_match_the_confirmation_and_preserved_native_prefix(self): | ||
| 13 | with tempfile.TemporaryDirectory() as folder: | ||
| 14 | root = Path(folder) | ||
| 15 | cold = root / 'cold' | ||
| 16 | (root / 'rust/confirmations').mkdir(parents=True) | ||
| 17 | (root / 'n0').mkdir() | ||
| 18 | (cold / 'results/123-456').mkdir(parents=True) | ||
| 19 | (root / 'run.json').write_text(json.dumps(dict(rust_writers=1, rust_readers=0, stress_clients=1, stress_operations=1))) | ||
| 20 | rows = [dict(event='ready', pid=123), dict(event='remote_attempt', state='Unknown', revision='revision', space='space'), | ||
| 21 | dict(event='remote_confirm', capture='123-456.one', revisions={'space': ['revision']}, text='Concurrent edits: [w0:0]')] | ||
| 22 | (root / 'rust/w0.jsonl').write_text('\n'.join(map(json.dumps, rows))) | ||
| 23 | (root / 'n0/stress-events.jsonl').write_text(json.dumps(dict(operation=0, token=' [n0:0]', before='Native 0:'))) | ||
| 24 | snapshot = root / 'rust/confirmations/123-456.one' | ||
| 25 | snapshot.write_bytes(b'captured snapshot') | ||
| 26 | sha = hashlib.sha256(snapshot.read_bytes()).hexdigest() | ||
| 27 | (cold / 'run.json').write_text(json.dumps(dict(inputs={'123-456.one': sha}))) | ||
| 28 | result = dict(name='123-456', error=None, pages=1, source_sha256=sha, seconds=1) | ||
| 29 | (cold / 'results.json').write_text(json.dumps(result)) | ||
| 30 | (cold / 'teardown.json').write_text(json.dumps(dict(absent=True))) | ||
| 31 | page = cold / 'results/123-456/page-0.xml' | ||
| 32 | def xml(text, native): | ||
| 33 | return f'<one:Page xmlns:one="http://schemas.microsoft.com/office/onenote/2010/onenote"><one:Outline><one:OEChildren><one:OE><one:T>{text}</one:T></one:OE><one:OE><one:T>{native}</one:T></one:OE></one:OEChildren></one:Outline></one:Page>' | ||
| 34 | page.write_text(xml('Concurrent edits: [w0:0]', 'Native 0:')) | ||
| 35 | with patch('verify_offline_confirmation.publication_links') as ledger: | ||
| 36 | self.assertEqual(verify(root, cold)['validated_paragraphs'], 2) | ||
| 37 | self.assertTrue(ledger.called) | ||
| 38 | config = json.loads((root / 'run.json').read_text()) | ||
| 39 | config['stress_operations'] = 2 | ||
| 40 | (root / 'run.json').write_text(json.dumps(config)) | ||
| 41 | with self.assertRaisesRegex(AssertionError, 'Missing native acknowledgements'): | ||
| 42 | verify(root, cold) | ||
| 43 | self.assertFalse(verify(root, cold, partial=True)['complete_workload']) | ||
| 44 | config['stress_operations'] = 1 | ||
| 45 | (root / 'run.json').write_text(json.dumps(config)) | ||
| 46 | for text, native in [('Concurrent edits:', 'Native 0:'), ('Concurrent edits: [w0:0]', 'Native 0: [n0:1]')]: | ||
| 47 | page.write_text(xml(text, native)) | ||
| 48 | with self.assertRaises(AssertionError): verify(root, cold) | ||
| 49 | page.write_text(xml('Concurrent edits: [w0:0]', 'Native 0: [n0:0]')) | ||
| 50 | self.assertEqual(verify(root, cold)['native_images'], 1) | ||
| 51 | snapshot.write_bytes(b'changed') | ||
| 52 | with self.assertRaises(AssertionError): verify(root, cold) | ||
| 53 | snapshot.write_bytes(b'captured snapshot') | ||
| 54 | config['document_operations'] = True | ||
| 55 | (root / 'run.json').write_text(json.dumps(config)) | ||
| 56 | document = dict(text='x', runs=[dict(text='x', bold=True, size=18, color=0x563412)]) | ||
| 57 | rows[1]['document_changes'] = {'target': document} | ||
| 58 | rows[2]['started_us'] = 200 | ||
| 59 | observed = dict(event='read', finished_us=100, text=rows[2]['text'], documents={'target': document}) | ||
| 60 | rows.insert(2, observed) | ||
| 61 | (root / 'rust/w0.jsonl').write_text('\n'.join(map(json.dumps, rows))) | ||
| 62 | markup = '<one:OE><one:T><![CDATA[<span style="font-weight:bold;font-size:18pt;color:#123456">x</span>]]></one:T></one:OE>' | ||
| 63 | content = xml('Concurrent edits: [w0:0]', 'Native 0:').replace('</one:OEChildren>', markup + '</one:OEChildren>') | ||
| 64 | page.write_text(content) | ||
| 65 | with patch('offline_document_history.document_history', return_value={'target': {'text': 'x', 'format': {'receipt_revision': 'revision'}}}) as documents: | ||
| 66 | result = verify(root, cold) | ||
| 67 | self.assertEqual(result['validated_paragraphs'], 3) | ||
| 68 | self.assertEqual(result['native_intended_format_checks'], 3) | ||
| 69 | documents.assert_called_once_with({'w0': rows}, 1) | ||
| 70 | rows[3]['revisions'] = {'space': ['current']} | ||
| 71 | rows[3]['current_revisions'] = {'space': 'current'} | ||
| 72 | documents.return_value['target']['format']['receipt_revision'] = 'current' | ||
| 73 | (root / 'rust/w0.jsonl').write_text('\n'.join(map(json.dumps, rows))) | ||
| 74 | self.assertEqual(verify(root, cold)['confirmed_revision'], 'current') | ||
| 75 | rows[3]['current_revisions'] = {'space': 'unrelated'} | ||
| 76 | (root / 'rust/w0.jsonl').write_text('\n'.join(map(json.dumps, rows))) | ||
| 77 | with self.assertRaisesRegex(AssertionError, 'current effect-confirmation'): verify(root, cold) | ||
| 78 | rows[3]['current_revisions'] = {'space': 'current'} | ||
| 79 | (root / 'rust/w0.jsonl').write_text('\n'.join(map(json.dumps, rows))) | ||
| 80 | page.write_text(content.replace('18pt', '19pt')) | ||
| 81 | with self.assertRaisesRegex(AssertionError, 'font size'): verify(root, cold) | ||
| 82 | page.write_text(content) | ||
| 83 | observed['documents'] = {} | ||
| 84 | (root / 'rust/w0.jsonl').write_text('\n'.join(map(json.dumps, rows))) | ||
| 85 | with self.assertRaisesRegex(AssertionError, 'omitted'): verify(root, cold) | ||
| 86 | |||
| 87 | |||
| 88 | if __name__ == '__main__': unittest.main() | ||
tools/test_offline_document_history.py created+151
| ... | @@ -0,0 +1,151 @@ | ||
| 1 | import copy | ||
| 2 | import unittest | ||
| 3 | import uuid | ||
| 4 | from unittest.mock import patch | ||
| 5 | |||
| 6 | from offline_document_history import document_history, identity, verify_model, verify_native | ||
| 7 | |||
| 8 | |||
| 9 | class DocumentHistoryTests(unittest.TestCase): | ||
| 10 | def setUp(self): | ||
| 11 | self.logs = {'w0': [{'event': 'ready', 'document_operations': True}]} | ||
| 12 | events = self.logs['w0'] | ||
| 13 | observed = {} | ||
| 14 | previous = None | ||
| 15 | for operation in range(2): | ||
| 16 | insertion = {'guid': list(uuid.UUID(int=operation+1).bytes_le), 'text': f'Document w0:{operation} 🦀', | ||
| 17 | 'parent': 'page' if operation == 0 else identity(previous, 1), 'author': 'Offline document writer', | ||
| 18 | 'placement': {'Outline': {'x': 144, 'y': 144}} if operation == 0 else {'Paragraph': {'before': None}}} | ||
| 19 | previous = insertion | ||
| 20 | target = identity(insertion, 2) | ||
| 21 | for step, kind in enumerate(('insert', 'format')): | ||
| 22 | timestamp = 10 + operation*30 + step*10 | ||
| 23 | local_id = operation*3 + step + 2 | ||
| 24 | event = {'event': 'local_document_commit', 'id': local_id, 'operation': operation, 'kind': kind, | ||
| 25 | 'space': 'space', 'object': target, 'text': insertion['text'], 'insertion': insertion if step == 0 else None, | ||
| 26 | 'range': [1, len(insertion['text'].encode('utf-16-le'))//2-2], | ||
| 27 | 'attributes': [{'Bold': True}, {'FontSize': 18+operation}, {'Color': [18, 52, 86]}], | ||
| 28 | 'started_us': timestamp-2, 'finished_us': timestamp-1} | ||
| 29 | events.append(event) | ||
| 30 | runs = [] | ||
| 31 | for index, char in enumerate(insertion['text']): | ||
| 32 | selected = kind == 'format' and 0 < index < len(insertion['text'])-1 | ||
| 33 | runs.append({'text': char, 'bold': selected, 'size': 18+operation if selected else 11, | ||
| 34 | 'color': 0x563412 if selected else 0xff000000}) | ||
| 35 | observed[target] = {'text': insertion['text'], 'runs': runs} | ||
| 36 | revision = f'revision-{local_id}' | ||
| 37 | events.append({'event': 'remote_attempt', 'revision': revision, 'state': 'Committed', | ||
| 38 | 'document_changes': {target: copy.deepcopy(observed[target])}, | ||
| 39 | 'documents': copy.deepcopy(observed), 'started_us': timestamp, 'finished_us': timestamp+1}) | ||
| 40 | events.append({'event': 'document_receipt', 'id': local_id, 'revision': revision, 'at_us': timestamp+2}) | ||
| 41 | events.extend({'event': 'reopened_document_receipt', 'id': row['id'], 'revision': row['revision']} | ||
| 42 | for row in list(events) if row['event'] == 'document_receipt') | ||
| 43 | read = {'event': 'read', 'started_us': 100, 'finished_us': 101, 'documents': observed} | ||
| 44 | events.extend([read, {'event': 'done'}]) | ||
| 45 | self.logs['r0'] = [{'event': 'ready'}, copy.deepcopy(read), {'event': 'done'}] | ||
| 46 | |||
| 47 | def test_document_receipts_and_reader_states_match_the_intents(self): | ||
| 48 | documents = document_history(self.logs, 2) | ||
| 49 | self.assertEqual(len(documents), 2) | ||
| 50 | paragraphs = [[(char, {'bold': bold, 'font_size': size, 'color': 'automatic' if color == 0xff000000 else '#123456'}) | ||
| 51 | for char, bold, size, color in row['new']] for row in documents.values()] | ||
| 52 | self.assertEqual(verify_native(paragraphs, documents), sum(len(row['new'])*3 for row in documents.values())) | ||
| 53 | paragraphs[0][1][1]['font_size'] = 19 | ||
| 54 | with self.assertRaisesRegex(AssertionError, 'font size'): verify_native(paragraphs, documents) | ||
| 55 | |||
| 56 | def test_missing_intents_receipts_or_reopen_records_are_rejected(self): | ||
| 57 | for name in ('local_document_commit', 'document_receipt', 'reopened_document_receipt', 'remote_attempt'): | ||
| 58 | logs = copy.deepcopy(self.logs) | ||
| 59 | events = logs['w0'] | ||
| 60 | events.remove(next(row for row in events if row['event'] == name)) | ||
| 61 | with self.subTest(event=name), self.assertRaises(AssertionError): document_history(logs, 2) | ||
| 62 | |||
| 63 | def test_retired_format_receipt_requires_current_revision_and_exact_observed_effect(self): | ||
| 64 | events = self.logs['w0'] | ||
| 65 | attempt = next(row for row in events if row['event'] == 'remote_attempt' and row['revision'] == 'revision-3') | ||
| 66 | attempt['state'] = 'Unknown' | ||
| 67 | receipt = next(row for row in events if row['event'] == 'document_receipt' and row['id'] == 3) | ||
| 68 | receipt.update(revision='current-revision', at_us=25) | ||
| 69 | next(row for row in events if row['event'] == 'reopened_document_receipt' and row['id'] == 3)['revision'] = receipt['revision'] | ||
| 70 | read = dict(event='read', started_us=21, finished_us=22, text='Concurrent edits:', documents=copy.deepcopy(attempt['documents'])) | ||
| 71 | confirmation = dict(event='remote_confirm', started_us=23, finished_us=24, state='Committed', text=read['text'], | ||
| 72 | revisions={'space': ['current-revision']}, current_revisions={'space': 'current-revision'}) | ||
| 73 | index = events.index(receipt) | ||
| 74 | events[index:index] = [read, confirmation] | ||
| 75 | result = document_history(self.logs, 2) | ||
| 76 | target, = attempt['document_changes'] | ||
| 77 | self.assertEqual(result[target]['format']['receipt_revision'], 'current-revision') | ||
| 78 | for field, value in [('current_revisions', {'space': 'unrelated'}), | ||
| 79 | ('revisions', {'space': ['revision-3', 'current-revision']}), | ||
| 80 | ('state', 'NotCommitted')]: | ||
| 81 | original = confirmation[field] | ||
| 82 | confirmation[field] = value | ||
| 83 | with self.subTest(field=field), self.assertRaises(AssertionError): document_history(self.logs, 2) | ||
| 84 | confirmation[field] = original | ||
| 85 | read['documents'][target]['runs'][1]['size'] = 12 | ||
| 86 | with self.assertRaisesRegex(AssertionError, 'differs from the uncertain formatting intent'): document_history(self.logs, 2) | ||
| 87 | |||
| 88 | def test_model_rejects_reordered_or_reparented_insertions(self): | ||
| 89 | documents = document_history(self.logs, 2) | ||
| 90 | first, second = [row['insertion'] for row in documents.values()] | ||
| 91 | outline, paragraph, appended = identity(first, 1), identity(first, 3), identity(second, 1) | ||
| 92 | nodes = {key: {'structure': [], 'content': [], 'children': [], 'kind': {}, 'layout': {}} | ||
| 93 | for key in ['page', outline, paragraph, appended, *documents]} | ||
| 94 | nodes['page']['children'] = [outline] | ||
| 95 | nodes[outline].update(children=[paragraph, appended], layout={'x': 144, 'y': 144}) | ||
| 96 | for parent, (target, document) in zip([paragraph, appended], documents.items(), strict=True): | ||
| 97 | nodes[parent]['content'] = [target] | ||
| 98 | nodes[target]['kind'] = {'text': document['text']} | ||
| 99 | revision = {'nodes': nodes} | ||
| 100 | with patch('offline_document_history.ordered_pages', return_value=[('space', 'revision', revision, 'page')]): | ||
| 101 | verify_model({}, documents) | ||
| 102 | nodes[outline]['children'].reverse() | ||
| 103 | with self.assertRaisesRegex(AssertionError, 'order'): verify_model({}, documents) | ||
| 104 | nodes[outline]['children'] = [paragraph] | ||
| 105 | nodes['page']['children'].append(appended) | ||
| 106 | with self.assertRaises(AssertionError): verify_model({}, documents) | ||
| 107 | nodes[outline]['children'] = [paragraph, appended] | ||
| 108 | nodes['page']['children'] = [outline] | ||
| 109 | nodes[paragraph]['content'].append(identity(second, 2)) | ||
| 110 | with self.assertRaisesRegex(AssertionError, 'content'): verify_model({}, documents) | ||
| 111 | |||
| 112 | def test_wrong_attributes_targets_and_unconfirmed_receipts_are_rejected(self): | ||
| 113 | for event, field, value in [('local_document_commit', 'object', 'wrong'), | ||
| 114 | ('local_document_commit', 'text', 'changed'), | ||
| 115 | ('document_receipt', 'revision', 'wrong'), | ||
| 116 | ('remote_attempt', 'state', 'Unknown'), | ||
| 117 | ('remote_attempt', 'state', 'NotCommitted')]: | ||
| 118 | logs = copy.deepcopy(self.logs) | ||
| 119 | next(row for row in logs['w0'] if row['event'] == event)[field] = value | ||
| 120 | with self.subTest(event=event, field=field), self.assertRaises(AssertionError): document_history(logs, 2) | ||
| 121 | for field, value in [('attributes', [{'Bold': False}]), ('range', [0, 1])]: | ||
| 122 | logs = copy.deepcopy(self.logs) | ||
| 123 | next(row for row in logs['w0'] if row.get('kind') == 'format')[field] = value | ||
| 124 | with self.subTest(field=field), self.assertRaises(AssertionError): document_history(logs, 2) | ||
| 125 | |||
| 126 | def test_readers_cannot_lose_revert_or_invent_document_content(self): | ||
| 127 | for mutation in ('missing', 'partial', 'future', 'reverted'): | ||
| 128 | logs = copy.deepcopy(self.logs) | ||
| 129 | read = logs['r0'][1] | ||
| 130 | target = next(iter(read['documents'])) | ||
| 131 | if mutation == 'missing': del read['documents'][target] | ||
| 132 | elif mutation == 'partial': read['documents'][target]['runs'][1]['bold'] = False | ||
| 133 | elif mutation == 'future': read.update(started_us=0, finished_us=1) | ||
| 134 | else: | ||
| 135 | old = copy.deepcopy(read) | ||
| 136 | old.update(started_us=102, finished_us=103) | ||
| 137 | for run in old['documents'][target]['runs']: | ||
| 138 | run.update(bold=False, size=11, color=0xff000000) | ||
| 139 | logs['r0'].insert(2, old) | ||
| 140 | with self.subTest(mutation=mutation), self.assertRaises(AssertionError): document_history(logs, 2) | ||
| 141 | |||
| 142 | def test_an_uncertain_document_attempt_cannot_be_replayed_under_another_revision(self): | ||
| 143 | logs = copy.deepcopy(self.logs) | ||
| 144 | first = copy.deepcopy(next(row for row in logs['w0'] if row['event'] == 'remote_attempt')) | ||
| 145 | first.update(state='Unknown', revision='earlier-uncertain', started_us=8, finished_us=9) | ||
| 146 | logs['w0'].insert(2, first) | ||
| 147 | with self.assertRaisesRegex(AssertionError, 'more than once'): document_history(logs, 2) | ||
| 148 | |||
| 149 | |||
| 150 | if __name__ == '__main__': | ||
| 151 | unittest.main() | ||
tools/test_offline_history.py created+156
| ... | @@ -0,0 +1,156 @@ | ||
| 1 | import copy | ||
| 2 | import unittest | ||
| 3 | from native_stress import edit_history | ||
| 4 | |||
| 5 | |||
| 6 | class OfflineHistoryTests(unittest.TestCase): | ||
| 7 | def setUp(self): | ||
| 8 | base = 'Concurrent edits:' | ||
| 9 | self.logs = {} | ||
| 10 | for actor, started, before in [('w1', 5, base), ('w0', 10, base + ' [w1:0]')]: | ||
| 11 | token = f' [{actor}:0]' | ||
| 12 | self.logs[actor] = [ | ||
| 13 | {'event': 'ready', 'offline': True}, | ||
| 14 | {'event': 'local_commit', 'id': 1, 'operation': 0, 'before': base, 'token': token, 'started_us': 1, 'finished_us': 2}, | ||
| 15 | {'event': 'read', 'text': before, 'started_us': started - 1, 'finished_us': started}, | ||
| 16 | {'event': 'remote_attempt', 'revision': actor, 'before': before, 'after': before + token, 'state': 'Committed', 'started_us': started, 'finished_us': started + 1}, | ||
| 17 | {'event': 'remote_receipt', 'id': 1, 'revision': actor, 'at_us': started + 2}, | ||
| 18 | {'event': 'reopened_receipt', 'id': 1, 'revision': actor}, | ||
| 19 | {'event': 'done'}, | ||
| 20 | ] | ||
| 21 | self.logs['r0'] = [{'event': 'ready'}, {'event': 'read', 'text': base + ' [w1:0] [w0:0]', 'started_us': 14, 'finished_us': 15}, {'event': 'done'}] | ||
| 22 | |||
| 23 | def test_private_local_branches_require_one_separate_remote_history(self): | ||
| 24 | commits, text = edit_history(self.logs, 1, offline=True) | ||
| 25 | self.assertEqual([event['token'] for event in commits], [' [w1:0]', ' [w0:0]']) | ||
| 26 | self.assertEqual(text, self.logs['r0'][1]['text']) | ||
| 27 | |||
| 28 | def test_false_receipts_lost_local_intents_and_changed_reopen_state_fail(self): | ||
| 29 | for index, field, value in [(1, 'id', 2), (1, 'token', ' [w0:9]'), (3, 'state', 'Unknown'), | ||
| 30 | (3, 'after', 'Concurrent edits: [w1:0]'), (4, 'at_us', 0), | ||
| 31 | (5, 'revision', 'changed'), (3, 'revision', 'missing')]: | ||
| 32 | with self.subTest(index=index, field=field): | ||
| 33 | logs = copy.deepcopy(self.logs) | ||
| 34 | logs['w0'][index][field] = value | ||
| 35 | with self.assertRaises(AssertionError): edit_history(logs, 1, offline=True) | ||
| 36 | for event in ('local_commit', 'remote_receipt', 'remote_attempt', 'reopened_receipt'): | ||
| 37 | logs = copy.deepcopy(self.logs) | ||
| 38 | logs['w0'] = [item for item in logs['w0'] if item['event'] != event] | ||
| 39 | with self.assertRaises(AssertionError): edit_history(logs, 1, offline=True) | ||
| 40 | |||
| 41 | def test_stale_reads_future_local_tokens_and_duplicate_acknowledgements_fail(self): | ||
| 42 | logs = copy.deepcopy(self.logs) | ||
| 43 | logs['r0'][1]['text'] = 'Concurrent edits:' | ||
| 44 | with self.assertRaises(AssertionError): edit_history(logs, 1, offline=True) | ||
| 45 | logs = copy.deepcopy(self.logs) | ||
| 46 | logs['w0'][1]['before'] += ' [w2:0]' | ||
| 47 | with self.assertRaises(AssertionError): edit_history(logs, 1, offline=True) | ||
| 48 | for event in ('local_commit', 'remote_receipt', 'remote_attempt', 'reopened_receipt'): | ||
| 49 | logs = copy.deepcopy(self.logs) | ||
| 50 | copied = next(item for item in logs['w0'] if item['event'] == event) | ||
| 51 | logs['w0'].insert(-1, copy.deepcopy(copied)) | ||
| 52 | with self.assertRaises(AssertionError): edit_history(logs, 1, offline=True) | ||
| 53 | |||
| 54 | def test_uncertain_receipt_requires_the_original_target_revision_and_successful_confirmation(self): | ||
| 55 | rows = self.logs['w0'] | ||
| 56 | intent = next(row for row in rows if row['event'] == 'local_commit') | ||
| 57 | attempt = next(row for row in rows if row['event'] == 'remote_attempt') | ||
| 58 | receipt = next(row for row in rows if row['event'] == 'remote_receipt') | ||
| 59 | intent.update(space='page-space', object='paragraph') | ||
| 60 | attempt.update(space='page-space', object='paragraph', state='Unknown') | ||
| 61 | receipt['at_us'] = 14 | ||
| 62 | confirmation = dict(event='remote_confirm', state='Committed', started_us=12, finished_us=13, | ||
| 63 | revisions={'page-space': [attempt['revision']]}, text=attempt['after']) | ||
| 64 | rows.insert(4, confirmation) | ||
| 65 | self.assertEqual(len(edit_history(self.logs, 1, offline=True)[0]), 2) | ||
| 66 | original = copy.deepcopy(self.logs) | ||
| 67 | for field, value in [('state', 'NotCommitted'), ('state', 'Unknown'), ('finished_us', 15), | ||
| 68 | ('revisions', {'other-space': ['w0']}), ('revisions', {'page-space': ['wrong']}), ('text', 'Concurrent edits:')]: | ||
| 69 | self.logs = copy.deepcopy(original) | ||
| 70 | next(row for row in self.logs['w0'] if row['event'] == 'remote_confirm')[field] = value | ||
| 71 | with self.subTest(field=field, value=value), self.assertRaises(AssertionError): edit_history(self.logs, 1, offline=True) | ||
| 72 | self.logs = copy.deepcopy(original) | ||
| 73 | replay = dict(attempt, revision='another-attempt', state='NotCommitted', started_us=11, finished_us=12) | ||
| 74 | self.logs['w0'].insert(4, replay) | ||
| 75 | with self.assertRaisesRegex(AssertionError, 'replayed'): edit_history(self.logs, 1, offline=True) | ||
| 76 | |||
| 77 | def test_partial_capture_does_not_promote_pending_local_success(self): | ||
| 78 | logs = copy.deepcopy(self.logs) | ||
| 79 | logs['w0'] = logs['w0'][:2] | ||
| 80 | logs['r0'] = [{'event': 'ready'}] | ||
| 81 | commits, text = edit_history(logs, 1, offline=True, partial=True) | ||
| 82 | self.assertEqual(len(commits), 1) | ||
| 83 | self.assertEqual(text, 'Concurrent edits: [w1:0]') | ||
| 84 | with self.assertRaises(AssertionError): edit_history(logs, 1, offline=True) | ||
| 85 | |||
| 86 | |||
| 87 | class OfflineLedgerTests(unittest.TestCase): | ||
| 88 | def test_independent_ledger_queue_depth_and_progress_checks(self): | ||
| 89 | import json | ||
| 90 | import os | ||
| 91 | from pathlib import Path | ||
| 92 | import sqlite3 | ||
| 93 | import tempfile | ||
| 94 | from verify_offline import verify | ||
| 95 | with tempfile.TemporaryDirectory() as folder: | ||
| 96 | output = Path(folder) | ||
| 97 | (output / 'rust').mkdir() | ||
| 98 | (output / 'run.json').write_text(json.dumps({'offline': True, 'embedded_smb': True, 'edit': False, 'stress_clients': 4, 'rust_writers': 4, 'rust_readers': 4, 'stress_operations': 2})) | ||
| 99 | (output / 'rust/stop').touch() | ||
| 100 | os.utime(output / 'rust/stop', ns=(0, 0)) | ||
| 101 | (output / 'rust/start').touch() | ||
| 102 | os.utime(output / 'rust/start', ns=(0, 0)) | ||
| 103 | logs = {f'w{i}': [{'event': 'ready', 'offline': True}] for i in range(4)} | ||
| 104 | for actor, events in logs.items(): | ||
| 105 | before = 'Concurrent edits:' | ||
| 106 | for op in range(2): | ||
| 107 | token = f' [{actor}:{op}]' | ||
| 108 | events.append({'event': 'local_commit', 'id': op+1, 'operation': op, 'token': token, 'before': before, 'started_us': 1+op*2, 'finished_us': 2+op*2}) | ||
| 109 | before += token | ||
| 110 | before = 'Concurrent edits:' | ||
| 111 | timestamp = 100 | ||
| 112 | for op in range(2): | ||
| 113 | for actor, events in logs.items(): | ||
| 114 | token, revision = f' [{actor}:{op}]', f'{actor}-{op}' | ||
| 115 | events.append({'event': 'remote_attempt', 'before': before, 'after': before+token, 'revision': revision, 'state': 'Committed', 'started_us': timestamp, 'finished_us': timestamp+1}) | ||
| 116 | events.append({'event': 'remote_receipt', 'id': op+1, 'revision': revision, 'at_us': timestamp+2}) | ||
| 117 | before += token | ||
| 118 | timestamp += 10 | ||
| 119 | for actor, events in logs.items(): | ||
| 120 | events.extend({'event': 'reopened_receipt', 'id': op+1, 'revision': f'{actor}-{op}'} for op in range(2)) | ||
| 121 | events.append({'event': 'done'}) | ||
| 122 | connection = sqlite3.connect(output / 'rust' / f'{actor}.sqlite') | ||
| 123 | connection.executescript('CREATE TABLE receipts(edit_id INTEGER, revision TEXT); CREATE TABLE edits(id INTEGER); CREATE TABLE attempt(id INTEGER); CREATE TABLE conflicts(id INTEGER); CREATE TABLE replica(id INTEGER, base BLOB, working BLOB);') | ||
| 124 | connection.executemany('INSERT INTO receipts VALUES (?,?)', [(op+1, f'{actor}-{op}') for op in range(2)]) | ||
| 125 | connection.execute('INSERT INTO replica VALUES (1, ?, ?)', (b'opaque image', b'opaque image')) | ||
| 126 | connection.commit() | ||
| 127 | connection.close() | ||
| 128 | for i in range(4): | ||
| 129 | logs[f'r{i}'] = [{'event': 'ready'}, {'event': 'read', 'text': before, 'started_us': timestamp, 'finished_us': timestamp+1}, {'event': 'done'}] | ||
| 130 | (output / f'n{i}').mkdir() | ||
| 131 | prefix = f'Native {i}:' | ||
| 132 | native = [] | ||
| 133 | for op in range(2): | ||
| 134 | token = f' [n{i}:{op}]' | ||
| 135 | native.append({'operation': op, 'token': token, 'before': prefix, 'updated_ticks': op*10_000_000}) | ||
| 136 | prefix += token | ||
| 137 | (output / f'n{i}/stress-events.jsonl').write_text('\n'.join(json.dumps(event) for event in native)) | ||
| 138 | for actor, events in logs.items(): | ||
| 139 | (output / 'rust' / f'{actor}.jsonl').write_text('\n'.join(json.dumps(event) for event in events)) | ||
| 140 | result = verify(output, max_gap=2) | ||
| 141 | self.assertEqual(result['remote_publications'], 8) | ||
| 142 | self.assertEqual(result['queued_before_publication_lower_bound'], {f'w{i}': 2 for i in range(4)}) | ||
| 143 | with self.assertRaisesRegex(AssertionError, 'progress exceeded'): verify(output, max_gap=.5) | ||
| 144 | os.utime(output / 'rust/stop', ns=(3_000_000_000, 3_000_000_000)) | ||
| 145 | with self.assertRaisesRegex(AssertionError, 'progress exceeded'): verify(output, max_gap=2) | ||
| 146 | os.utime(output / 'rust/stop', ns=(0, 0)) | ||
| 147 | connection = sqlite3.connect(output / 'rust/w0.sqlite') | ||
| 148 | for sql, undo in [("UPDATE receipts SET revision='wrong' WHERE edit_id=1", "UPDATE receipts SET revision='w0-0' WHERE edit_id=1"), | ||
| 149 | ('INSERT INTO attempt VALUES (1)', 'DELETE FROM attempt'), | ||
| 150 | ("UPDATE replica SET working=X'00'", "UPDATE replica SET working=base")]: | ||
| 151 | connection.execute(sql) | ||
| 152 | connection.commit() | ||
| 153 | with self.assertRaises(AssertionError): verify(output) | ||
| 154 | connection.execute(undo) | ||
| 155 | connection.commit() | ||
| 156 | connection.close() | ||
tools/test_offline_outage.py created+216
| ... | @@ -0,0 +1,216 @@ | ||
| 1 | import copy | ||
| 2 | import json | ||
| 3 | from pathlib import Path | ||
| 4 | import tempfile | ||
| 5 | import unittest | ||
| 6 | from unittest.mock import patch | ||
| 7 | |||
| 8 | from offline_outage import verify_outage, verify_lost_reply | ||
| 9 | |||
| 10 | |||
| 11 | class OutageOracle(unittest.TestCase): | ||
| 12 | def setUp(self): | ||
| 13 | temporary = tempfile.TemporaryDirectory() | ||
| 14 | self.addCleanup(temporary.cleanup) | ||
| 15 | self.root = Path(temporary.name) | ||
| 16 | (self.root / 'rust').mkdir() | ||
| 17 | self.config = dict(offline=True, offline_outage=True, embedded_smb=True, stress_clients=4, rust_writers=4, rust_readers=4, stress_operations=8) | ||
| 18 | self.sample = dict(down_started_us=1_000_000, up_started_us=5_000_000, native_before=[1]*4, native_during=[4]*4, reader_errors_before={f'r{i}': 0 for i in range(4)}) | ||
| 19 | (self.root / 'clocks.json').write_text(json.dumps([dict(native_minus_host_us=[-10, 10], after_native_minus_host_us=[-10, 10])]*4)) | ||
| 20 | for i in range(4): | ||
| 21 | (self.root / f'n{i}').mkdir() | ||
| 22 | (self.root / f'n{i}/stress-events.jsonl').write_text(json.dumps(dict(update_started_ticks=621355968020000000, updated_ticks=621355968030000000))) | ||
| 23 | self.logs = {} | ||
| 24 | for i in range(4): | ||
| 25 | self.logs[f'w{i}'] = [dict(event='ready'), dict(event='transport_connected', at_us=0), | ||
| 26 | dict(event='publication_paused', revision=f'w{i}', at_us=100), | ||
| 27 | *[dict(event='local_commit', operation=j, started_us=10 if j == 0 else 2_000_000+j, finished_us=20 if j == 0 else 2_000_010+j) for j in range(8)], | ||
| 28 | dict(event='remote_attempt', started_us=6_000_000, state='NotCommitted', revision=f'w{i}'), | ||
| 29 | dict(event='transport_connected', at_us=6_000_001), | ||
| 30 | *[dict(event='remote_receipt', at_us=7_000_000+j) for j in range(8)], dict(event='done')] | ||
| 31 | self.logs[f'r{i}'] = [dict(event='ready'), dict(event='transport_read_error'), dict(event='transport_connected', at_us=6_000_000), | ||
| 32 | *[dict(event='read', started_us=7_000_000+j, finished_us=7_000_001+j) for j in range(3)], dict(event='done')] | ||
| 33 | self.trace = [dict(control=dict(phase='offline-down', mode='down')), dict(control=dict(phase='offline-reconnected'))] | ||
| 34 | |||
| 35 | def verify(self): | ||
| 36 | (self.root / 'run.json').write_text(json.dumps(self.config)) | ||
| 37 | (self.root / 'offline-outage-progress.json').write_text(json.dumps(self.sample)) | ||
| 38 | (self.root / 'smb-trace.jsonl').write_text('\n'.join(map(json.dumps, self.trace))) | ||
| 39 | for actor, events in self.logs.items(): | ||
| 40 | (self.root / 'rust' / f'{actor}.jsonl').write_text('\n'.join(map(json.dumps, events))) | ||
| 41 | with patch('offline_outage.verify', return_value={'guarded_pairs': 1}) as overlap: | ||
| 42 | result = verify_outage(self.root) | ||
| 43 | overlap.assert_called_once_with(self.trace, phase='offline-reconnected') | ||
| 44 | return result | ||
| 45 | |||
| 46 | def test_confirmed_outage_requires_local_and_native_progress_and_fresh_sessions(self): | ||
| 47 | self.assertEqual(self.verify()['local_edits_while_down'], {f'w{i}': 7 for i in range(4)}) | ||
| 48 | original = copy.deepcopy(self.logs) | ||
| 49 | for event, field, value in [('local_commit', 'started_us', 0), ('publication_paused', 'at_us', 3_000_000), | ||
| 50 | ('remote_attempt', 'state', 'Unknown'), ('remote_attempt', 'started_us', 3_000_000), | ||
| 51 | ('remote_attempt', 'revision', 'other'), ('remote_receipt', 'at_us', 0)]: | ||
| 52 | self.logs = copy.deepcopy(original) | ||
| 53 | row = next(row for row in self.logs['w0'] if row['event'] == event and (event != 'local_commit' or row['operation'] == 1)) | ||
| 54 | row[field] = value | ||
| 55 | with self.subTest(event=event, field=field), self.assertRaises(AssertionError): self.verify() | ||
| 56 | for actor, event in [('r0', 'transport_connected'), ('r0', 'transport_read_error'), ('r0', 'read'), ('w0', 'publication_paused')]: | ||
| 57 | self.logs = copy.deepcopy(original) | ||
| 58 | self.logs[actor] = [row for row in self.logs[actor] if row['event'] != event] | ||
| 59 | with self.subTest(actor=actor, event=event), self.assertRaises(AssertionError): self.verify() | ||
| 60 | |||
| 61 | def test_lost_reply_requires_one_original_revision_receipt_and_captured_confirmation(self): | ||
| 62 | self.config.update(offline_outage=False, offline_lost_reply=True) | ||
| 63 | self.sample.update(before={actor: 3 for actor in self.logs}, after={actor: 6 for actor in self.logs}) | ||
| 64 | attempt = next(row for row in self.logs['w0'] if row['event'] == 'remote_attempt') | ||
| 65 | attempt.update(state='Unknown', started_us=500_000, finished_us=1_100_000) | ||
| 66 | receipt = next(row for row in self.logs['w0'] if row['event'] == 'remote_receipt') | ||
| 67 | receipt['revision'] = attempt['revision'] | ||
| 68 | for row in self.logs['w0']: | ||
| 69 | if row['event'] == 'remote_receipt' and row is not receipt: row['revision'] = 'other' | ||
| 70 | folder = self.root / 'rust/confirmations' | ||
| 71 | folder.mkdir() | ||
| 72 | (folder / 'confirmation.one').write_bytes(b'owned captured image') | ||
| 73 | confirmation = dict(event='remote_confirm', state='Committed', started_us=6_000_000, finished_us=6_000_001, | ||
| 74 | text='Concurrent edits: [w0:0]', capture='confirmation.one') | ||
| 75 | self.logs['w0'].insert(-1, confirmation) | ||
| 76 | self.trace = [dict(control=dict(phase='offline-reply-cut', cut=9, peer='10.0.2.2', offset=96, direction='response')), | ||
| 77 | dict(direction='request', command=9, offset=96, connection=1, message=2), | ||
| 78 | dict(cut=dict(direction='response', command=9, status='0x0', connection=1, message=2)), | ||
| 79 | dict(control=dict(phase='offline-reply-reconnected'))] | ||
| 80 | def check(): | ||
| 81 | (self.root / 'run.json').write_text(json.dumps(self.config)) | ||
| 82 | (self.root / 'offline-lost-reply-progress.json').write_text(json.dumps(self.sample)) | ||
| 83 | (self.root / 'smb-trace.jsonl').write_text('\n'.join(map(json.dumps, self.trace))) | ||
| 84 | for actor, rows in self.logs.items(): | ||
| 85 | (self.root / 'rust' / f'{actor}.jsonl').write_text('\n'.join(map(json.dumps, rows))) | ||
| 86 | with patch('offline_history.publication_links') as ledger, patch('offline_document_history.document_history') as documents, patch('offline_outage.verify', return_value={'guarded_pairs': 1}) as overlap: | ||
| 87 | if self.config.get('offline_client_reply'): | ||
| 88 | documents.return_value = {'target': {'format': {'receipt_revision': 'effect-revision'}}} | ||
| 89 | result = verify_lost_reply(self.root) | ||
| 90 | ledger.assert_called_once_with(self.logs, self.config['stress_operations']) | ||
| 91 | if self.config.get('document_operations'): | ||
| 92 | documents.assert_called_once_with(self.logs, self.config['stress_operations']) | ||
| 93 | else: | ||
| 94 | documents.assert_not_called() | ||
| 95 | overlap.assert_called_once_with(self.trace, phase='offline-reply-reconnected') | ||
| 96 | return result | ||
| 97 | self.assertEqual(check()['confirmed_revision'], 'w0') | ||
| 98 | self.config['document_operations'] = True | ||
| 99 | self.sample['format_released_us'] = 200_000 | ||
| 100 | receipt['event'] = 'document_receipt' | ||
| 101 | receipt['id'] = 17 | ||
| 102 | intent = dict(event='local_document_commit', id=17, kind='format') | ||
| 103 | self.logs['w0'].insert(-1, intent) | ||
| 104 | for actor, rows in self.logs.items(): | ||
| 105 | if actor.startswith('w'): | ||
| 106 | next(row for row in rows if row['event'] == 'publication_paused')['kind'] = 'format' | ||
| 107 | self.assertEqual(check()['confirmed_revision'], 'w0') | ||
| 108 | intent['kind'] = 'insert' | ||
| 109 | with self.assertRaisesRegex(AssertionError, 'not formatting'): check() | ||
| 110 | intent['kind'] = 'format' | ||
| 111 | self.sample['format_released_us'] = 900_000 | ||
| 112 | with self.assertRaises(AssertionError): check() | ||
| 113 | self.sample['format_released_us'] = 200_000 | ||
| 114 | paused = next(row for row in self.logs['w0'] if row['event'] == 'publication_paused') | ||
| 115 | paused['revision'] = 'stale-prepared-revision' | ||
| 116 | prior = dict(event='remote_attempt', revision=paused['revision'], state='NotCommitted', | ||
| 117 | started_us=200_001, finished_us=200_002) | ||
| 118 | self.logs['w0'].insert(-1, prior) | ||
| 119 | self.assertEqual(check()['confirmed_revision'], 'w0') | ||
| 120 | prior['finished_us'] = 600_000 | ||
| 121 | with self.assertRaisesRegex(AssertionError, 'proving it unpublished'): check() | ||
| 122 | self.logs['w0'].remove(prior) | ||
| 123 | paused['revision'] = attempt['revision'] | ||
| 124 | self.config['offline_client_reply'] = True | ||
| 125 | receipt['revision'] = 'effect-revision' | ||
| 126 | attempt.update(space='space', document_changes={'target': {}}) | ||
| 127 | retirement = dict(event='revision_retired', space='space', revision=attempt['revision'], started_us=2_200_000, finished_us=2_300_000) | ||
| 128 | self.logs['r0'].append(retirement) | ||
| 129 | (self.root / 'rust/offline-retired.one').write_bytes(b'retired revision snapshot') | ||
| 130 | self.sample['during'] = {actor: 3 if actor == 'w0' else 6 for actor in self.logs} | ||
| 131 | self.trace[0]['control']['scope'] = 'connection' | ||
| 132 | barrier = dict(event='confirmation_paused', revision=attempt['revision'], at_us=1_200_000) | ||
| 133 | self.logs['w0'].append(barrier) | ||
| 134 | peer_rows = [] | ||
| 135 | for actor, rows in self.logs.items(): | ||
| 136 | if actor == 'w0': continue | ||
| 137 | for i in range(3): | ||
| 138 | row = dict(event='remote_attempt' if actor.startswith('w') else 'read', state='Committed', | ||
| 139 | started_us=2_000_000+i, finished_us=2_000_010+i) | ||
| 140 | rows.append(row) | ||
| 141 | peer_rows.append((rows, row)) | ||
| 142 | wire = [dict(connection=7, opened=True, peer=['192.168.77.12', 445]), | ||
| 143 | dict(connection=7, message=2, command=5, direction='request', path='owned/synthetic.one'), | ||
| 144 | dict(connection=7, message=2, command=5, direction='response', status='0x0', file_id='native-file'), | ||
| 145 | dict(connection=7, message=3, command=9, direction='request', time=2.0, file_id='native-file'), | ||
| 146 | dict(connection=7, message=3, command=9, direction='response', time=2.1, status='0x0', written=4)] | ||
| 147 | self.trace.extend(wire) | ||
| 148 | result = check() | ||
| 149 | self.assertEqual(result['native_writes_during_client_disconnect'], 1) | ||
| 150 | self.assertEqual(result['peer_operations_during_client_disconnect'], {actor: 3 for actor in self.logs if actor != 'w0'}) | ||
| 151 | peer_rows[0][1]['finished_us'] = 6_000_000 | ||
| 152 | with self.assertRaisesRegex(AssertionError, 'insufficient completed I/O'): check() | ||
| 153 | peer_rows[0][1]['finished_us'] = 2_000_010 | ||
| 154 | wire[-1]['time'] = 6.0 | ||
| 155 | with self.assertRaisesRegex(AssertionError, 'No successful native writes'): check() | ||
| 156 | wire[-1]['time'] = 2.1 | ||
| 157 | barrier['revision'] = 'other' | ||
| 158 | with self.assertRaises(AssertionError): check() | ||
| 159 | self.logs['w0'].remove(barrier) | ||
| 160 | self.logs['r0'].remove(retirement) | ||
| 161 | receipt['revision'] = attempt['revision'] | ||
| 162 | for rows, row in peer_rows: rows.remove(row) | ||
| 163 | del self.trace[-len(wire):] | ||
| 164 | del self.trace[0]['control']['scope'] | ||
| 165 | self.config['offline_client_reply'] = False | ||
| 166 | self.config['document_operations'] = False | ||
| 167 | receipt['event'] = 'remote_receipt' | ||
| 168 | self.logs['w0'].remove(intent) | ||
| 169 | attempt['state'] = 'Committed' | ||
| 170 | with self.assertRaises(AssertionError): check() | ||
| 171 | attempt['state'] = 'Unknown' | ||
| 172 | self.trace[1]['offset'] = 100 | ||
| 173 | with self.assertRaises(AssertionError): check() | ||
| 174 | self.trace[1]['offset'] = 96 | ||
| 175 | (folder / 'unrecorded.one').write_bytes(b'extra') | ||
| 176 | with self.assertRaises(AssertionError): check() | ||
| 177 | |||
| 178 | def test_document_outage_requires_both_local_operations_and_delayed_receipts(self): | ||
| 179 | self.config['document_operations'] = True | ||
| 180 | for actor, events in self.logs.items(): | ||
| 181 | if not actor.startswith('w'): continue | ||
| 182 | events[-1:-1] = [dict(event='local_document_commit', operation=operation, kind=kind, | ||
| 183 | started_us=10 if operation == 0 else 2_100_000+operation, | ||
| 184 | finished_us=20 if operation == 0 else 2_100_010+operation) | ||
| 185 | for operation in range(8) for kind in ('insert', 'format')] | ||
| 186 | events[-1:-1] = [dict(event='document_receipt', at_us=7_100_000+i) for i in range(16)] | ||
| 187 | self.assertEqual(self.verify()['local_edits_while_down'], {f'w{i}': 21 for i in range(4)}) | ||
| 188 | original = copy.deepcopy(self.logs) | ||
| 189 | for event, field, value in [('local_document_commit', 'finished_us', 6_000_000), | ||
| 190 | ('local_document_commit', 'kind', 'insert'), | ||
| 191 | ('document_receipt', 'at_us', 0)]: | ||
| 192 | self.logs = copy.deepcopy(original) | ||
| 193 | row = next(row for row in self.logs['w0'] if row['event'] == event | ||
| 194 | and (event != 'local_document_commit' or row['operation'] == 1 and row['kind'] == 'format')) | ||
| 195 | row[field] = value | ||
| 196 | with self.subTest(event=event, field=field), self.assertRaises(AssertionError): self.verify() | ||
| 197 | |||
| 198 | def test_native_edits_outside_confirmed_window_fail(self): | ||
| 199 | (self.root / 'n0/stress-events.jsonl').write_text(json.dumps(dict(update_started_ticks=621355968000000000, updated_ticks=621355968000000100))) | ||
| 200 | with self.assertRaisesRegex(AssertionError, 'timestamps'): self.verify() | ||
| 201 | |||
| 202 | def test_native_stall_short_outage_and_wrong_control_fail(self): | ||
| 203 | self.sample['native_before'].append(1) | ||
| 204 | with self.assertRaises(AssertionError): self.verify() | ||
| 205 | self.sample['native_before'].pop() | ||
| 206 | self.sample['native_during'][0] = 1 | ||
| 207 | with self.assertRaises(AssertionError): self.verify() | ||
| 208 | self.sample['native_during'][0] = 4 | ||
| 209 | self.sample['up_started_us'] = 1_000_001 | ||
| 210 | with self.assertRaises(AssertionError): self.verify() | ||
| 211 | self.sample['up_started_us'] = 5_000_000 | ||
| 212 | self.trace[0]['control'].pop('mode') | ||
| 213 | with self.assertRaises(AssertionError): self.verify() | ||
| 214 | |||
| 215 | |||
| 216 | if __name__ == '__main__': unittest.main() | ||
tools/test_offline_publication_crash.py created+81
| ... | @@ -0,0 +1,81 @@ | ||
| 1 | import copy | ||
| 2 | import hashlib | ||
| 3 | import json | ||
| 4 | from pathlib import Path | ||
| 5 | import tempfile | ||
| 6 | import unittest | ||
| 7 | from unittest.mock import patch | ||
| 8 | from offline_publication_crash import state, TOKEN, confirmation_only | ||
| 9 | from verify_offline_recovery import verify | ||
| 10 | |||
| 11 | |||
| 12 | class RecoveryOracleTests(unittest.TestCase): | ||
| 13 | def test_confirmation_cannot_hide_revision_or_content_writes(self): | ||
| 14 | source = bytes(1024) | ||
| 15 | changed = bytearray(source) | ||
| 16 | changed[212:252] = bytes(range(40)) | ||
| 17 | events = [{'event':'write', 'offset':212, 'bytes':40}] | ||
| 18 | confirmation_only(events, source, changed) | ||
| 19 | for offset in (96, 211, 252, 1023): | ||
| 20 | bad = changed.copy() | ||
| 21 | bad[offset] ^= 1 | ||
| 22 | with self.assertRaises(AssertionError): confirmation_only(events, source, bad) | ||
| 23 | for event in ({'event':'write','offset':96,'bytes':1}, {'event':'phase','name':'publish-before'}): | ||
| 24 | with self.assertRaises(AssertionError): confirmation_only([*events,event], source, changed) | ||
| 25 | |||
| 26 | def test_text_acknowledgements_and_revision_identity_must_agree(self): | ||
| 27 | original = 'Concurrent edits: 🦀' | ||
| 28 | at = len(original.encode('utf-16-le')) // 2 | ||
| 29 | row = {'event': 'state', 'status': 'uncertain', 'revision': 'new', 'remote_revision': 'old', | ||
| 30 | 'local_text': original+TOKEN, 'remote_text': original, | ||
| 31 | 'pending': [{'id': 1, 'before': original, 'replacement': TOKEN, 'range': [at, at]}]} | ||
| 32 | self.assertEqual(state([row], original), row) | ||
| 33 | for field, value in [('revision', None), ('revision', 'old'), ('local_text', original), | ||
| 34 | ('remote_text', original+TOKEN+TOKEN), ('status', 'missing'), ('pending', [])]: | ||
| 35 | changed = copy.deepcopy(row) | ||
| 36 | changed[field] = value | ||
| 37 | with self.subTest(field=field): | ||
| 38 | with self.assertRaises(AssertionError): state([changed], original) | ||
| 39 | for field, value in [('id', 2), ('before', 'other'), ('replacement', 'other'), ('range', [at-1, at-1])]: | ||
| 40 | changed = copy.deepcopy(row) | ||
| 41 | changed['pending'][0][field] = value | ||
| 42 | with self.assertRaises(AssertionError): state([changed], original) | ||
| 43 | row.update(status='published', remote_revision='new', remote_text=original+TOKEN, pending=[]) | ||
| 44 | self.assertEqual(state([row], original), row) | ||
| 45 | for field, value in [('revision', 'old'), ('remote_text', original), ('pending', [{'id': 1}])]: | ||
| 46 | changed = copy.deepcopy(row) | ||
| 47 | changed[field] = value | ||
| 48 | with self.assertRaises(AssertionError): state([changed], original) | ||
| 49 | |||
| 50 | def test_native_inventory_hashes_errors_and_content_are_independently_checked(self): | ||
| 51 | with tempfile.TemporaryDirectory() as folder: | ||
| 52 | run, cold, source = [Path(folder)/name for name in ('run', 'cold', 'source.xml')] | ||
| 53 | (run/'images').mkdir(parents=True) | ||
| 54 | cold.mkdir() | ||
| 55 | data = b'owned fixture bytes' | ||
| 56 | digest = hashlib.sha256(data).hexdigest() | ||
| 57 | (run/'images'/f'{digest}.one').write_bytes(data) | ||
| 58 | expected = ['Concurrent edits:'+TOKEN, 'Other paragraph'] | ||
| 59 | (run/'results.json').write_text(json.dumps([{'remote_image': digest, 'remote_text': expected[0]}])) | ||
| 60 | (cold/'run.json').write_text(json.dumps({'inputs': {digest+'.one': digest}})) | ||
| 61 | record = {'name': digest, 'source_sha256': digest, 'error': None, 'pages': 1, 'seconds': 1} | ||
| 62 | (cold/'results.json').write_text(json.dumps(record)) | ||
| 63 | (cold/'teardown.json').write_text(json.dumps({'absent': True})) | ||
| 64 | (cold/'results'/digest).mkdir(parents=True) | ||
| 65 | (cold/'results'/digest/'page-0.xml').touch() | ||
| 66 | def content(path): | ||
| 67 | return ['Concurrent edits:', 'Other paragraph'] if path == source else expected | ||
| 68 | with patch('verify_offline_recovery.paragraphs', side_effect=content): | ||
| 69 | self.assertEqual(verify(run, cold, source)['exact_images'], 1) | ||
| 70 | for key, value in [('source_sha256', 'wrong'), ('error', 'failed'), ('pages', 0), ('name', 'wrong')]: | ||
| 71 | (cold/'results.json').write_text(json.dumps({**record, key:value})) | ||
| 72 | with self.subTest(key=key): | ||
| 73 | with self.assertRaises(AssertionError): verify(run, cold, source) | ||
| 74 | (cold/'results.json').write_text(json.dumps([record, record])) | ||
| 75 | with self.assertRaises(AssertionError): verify(run, cold, source) | ||
| 76 | (cold/'results.json').write_text(json.dumps(record)) | ||
| 77 | expected[0] = 'Concurrent edits:' | ||
| 78 | with self.assertRaises(AssertionError): verify(run, cold, source) | ||
| 79 | expected[0] += TOKEN | ||
| 80 | (run/'images'/f'{digest}.one').write_bytes(b'changed') | ||
| 81 | with self.assertRaises(AssertionError): verify(run, cold, source) | ||
tools/test_smb_overlap.py created+144
| ... | @@ -0,0 +1,144 @@ | ||
| 1 | import unittest | ||
| 2 | |||
| 3 | from verify_smb_overlap import verify | ||
| 4 | |||
| 5 | |||
| 6 | class Overlap(unittest.TestCase): | ||
| 7 | def history(self, serialized=False, failed_write=False, writer_reads=False): | ||
| 8 | events = [{'connection': 1, 'opened': True, 'peer': ['10.0.2.2', 1]}, | ||
| 9 | {'connection': 2, 'opened': True, 'peer': ['192.168.77.2', 2]}] | ||
| 10 | def exchange(connection, command, status='0x0', **fields): | ||
| 11 | message = len(events) | ||
| 12 | request = {'connection': connection, 'command': command, 'message': message, | ||
| 13 | 'direction': 'request', **fields} | ||
| 14 | response = {'connection': connection, 'command': command, 'message': message, | ||
| 15 | 'direction': 'response', 'status': status, 'file_id': str(connection)} | ||
| 16 | events.extend((request, response)) | ||
| 17 | for connection in (1, 2): | ||
| 18 | exchange(connection, 5, path='synthetic.one', access=0xc0000000 if connection == 2 or writer_reads else 0x80000000) | ||
| 19 | exchange(connection, 10, file_id=str(connection), locks=[(0xfffffffb, 1, 0x11)]) | ||
| 20 | if serialized: | ||
| 21 | exchange(1, 8, file_id='1', length=32) | ||
| 22 | exchange(1, 6, file_id='1') | ||
| 23 | exchange(2, 10, file_id='2', locks=[(0xfffffffd, 1, 0x12)]) | ||
| 24 | if not serialized: | ||
| 25 | exchange(1, 8, file_id='1', length=32) | ||
| 26 | exchange(2, 9, status='0xc0000054' if failed_write else '0x0', file_id='2', length=32) | ||
| 27 | exchange(2, 6, file_id='2') | ||
| 28 | events.append({'connection': 3, 'opened': True, 'peer': ['10.0.2.2', 3]}) | ||
| 29 | exchange(3, 5, path='synthetic.one', access=0xc0000000) | ||
| 30 | exchange(3, 10, file_id='3', locks=[(0xfffffffb, 1, 0x11), (0xfffffffd, 1, 0x12)]) | ||
| 31 | if not serialized: | ||
| 32 | exchange(1, 8, file_id='1', length=32) | ||
| 33 | exchange(3, 9, file_id='3', length=32) | ||
| 34 | return events | ||
| 35 | |||
| 36 | def test_active_read_and_native_write(self): | ||
| 37 | result = verify(self.history()) | ||
| 38 | self.assertEqual(result['active_native_writer_pairs'], 1) | ||
| 39 | self.assertEqual(result['active_rust_writer_pairs'], 1) | ||
| 40 | self.assertEqual(result['overlapping_reads'], 2) | ||
| 41 | self.assertEqual(result['overlapping_writes'], 2) | ||
| 42 | |||
| 43 | def test_progress_before_resume_does_not_satisfy_after_gate(self): | ||
| 44 | events = self.history() | ||
| 45 | events.append({'control': {'phase': 'resumed'}}) | ||
| 46 | with self.assertRaises(AssertionError): | ||
| 47 | verify(events, phase='resumed') | ||
| 48 | with self.assertRaises(AssertionError): | ||
| 49 | verify(self.history(), phase='resumed') | ||
| 50 | self.assertEqual(verify([{'control': {'phase': 'resumed'}}, *self.history()], phase='resumed')['active_rust_writer_pairs'], 1) | ||
| 51 | |||
| 52 | def test_native_only_traffic_does_not_count(self): | ||
| 53 | events = self.history() | ||
| 54 | for event in events: | ||
| 55 | if event.get('opened'): event['peer'][0] = '192.168.77.' + str(event['connection'] + 1) | ||
| 56 | with self.assertRaises(AssertionError): | ||
| 57 | verify(events) | ||
| 58 | |||
| 59 | def test_delayed_io_responses_do_not_count_as_resumed_requests(self): | ||
| 60 | events = self.history() | ||
| 61 | end = next(i for i, event in enumerate(events) if event.get('direction') == 'request' and event['command'] == 6) | ||
| 62 | responses = [event for event in events[:end] if event.get('direction') == 'response' and event['command'] in (8, 9)] | ||
| 63 | events = [event for event in events if event not in responses] | ||
| 64 | events[end - len(responses):end - len(responses)] = [{'control': {'phase': 'resumed'}}, *responses] | ||
| 65 | self.assertEqual(verify(events)['active_native_writer_pairs'], 1) | ||
| 66 | with self.assertRaises(AssertionError): | ||
| 67 | verify(events, phase='resumed') | ||
| 68 | |||
| 69 | def test_serialized_calls_do_not_count(self): | ||
| 70 | with self.assertRaises(AssertionError): | ||
| 71 | verify(self.history(serialized=True)) | ||
| 72 | |||
| 73 | def test_failed_write_does_not_count(self): | ||
| 74 | with self.assertRaises(AssertionError): | ||
| 75 | verify(self.history(failed_write=True)) | ||
| 76 | |||
| 77 | def test_writers_validation_reads_do_not_count(self): | ||
| 78 | with self.assertRaises(AssertionError): | ||
| 79 | verify(self.history(writer_reads=True)) | ||
| 80 | |||
| 81 | def test_lock_failure_does_not_count(self): | ||
| 82 | events = self.history() | ||
| 83 | for event in events: | ||
| 84 | if event.get('direction') == 'response' and event['command'] == 10: | ||
| 85 | event['status'] = '0xc0000055' | ||
| 86 | with self.assertRaises(AssertionError): | ||
| 87 | verify(events) | ||
| 88 | |||
| 89 | def test_separate_lock_lifetimes_do_not_combine(self): | ||
| 90 | events = self.history() | ||
| 91 | inserted = [] | ||
| 92 | for message, flags in [(1000, 4), (1001, 0x12)]: | ||
| 93 | inserted.extend([ | ||
| 94 | {'connection': 3, 'direction': 'request', 'command': 10, 'message': message, | ||
| 95 | 'file_id': '3', 'locks': [(0xfffffffd, 1, flags)]}, | ||
| 96 | {'connection': 3, 'direction': 'response', 'command': 10, 'message': message, | ||
| 97 | 'status': '0x0'}, | ||
| 98 | ]) | ||
| 99 | events[-2:-2] = inserted | ||
| 100 | with self.assertRaises(AssertionError): | ||
| 101 | verify(events) | ||
| 102 | |||
| 103 | def test_renamed_path_does_not_combine_file_versions(self): | ||
| 104 | events = self.history() | ||
| 105 | events[-2:-2] = [ | ||
| 106 | {'connection': 3, 'direction': 'request', 'command': 17, 'message': 1000, | ||
| 107 | 'info_type': 1, 'info_class': 10}, | ||
| 108 | {'connection': 3, 'direction': 'response', 'command': 17, 'message': 1000, 'status': '0x0'}, | ||
| 109 | ] | ||
| 110 | with self.assertRaises(AssertionError): | ||
| 111 | verify(events) | ||
| 112 | |||
| 113 | def test_open_response_crossing_rename_is_not_attributed(self): | ||
| 114 | events = self.history() | ||
| 115 | at = next(i for i, event in enumerate(events) if event.get('connection') == 3 | ||
| 116 | and event.get('command') == 5 and event['direction'] == 'response') | ||
| 117 | inserted = [ | ||
| 118 | {'connection': 4, 'opened': True, 'peer': ['192.168.77.4', 4]}, | ||
| 119 | {'connection': 4, 'direction': 'request', 'command': 17, 'message': 1000, | ||
| 120 | 'info_type': 1, 'info_class': 10}, | ||
| 121 | {'connection': 4, 'direction': 'response', 'command': 17, 'message': 1000, 'status': '0x0'}, | ||
| 122 | ] | ||
| 123 | for message, command, fields in [ | ||
| 124 | (1001, 5, {'path': 'synthetic.one', 'access': 0x80000000}), | ||
| 125 | (1002, 10, {'file_id': '1', 'locks': [(0xfffffffb, 1, 0x11)]}), | ||
| 126 | ]: | ||
| 127 | inserted.extend([ | ||
| 128 | {'connection': 1, 'direction': 'request', 'command': command, 'message': message, **fields}, | ||
| 129 | {'connection': 1, 'direction': 'response', 'command': command, 'message': message, | ||
| 130 | 'status': '0x0', 'file_id': '1'}, | ||
| 131 | ]) | ||
| 132 | events[at:at] = inserted | ||
| 133 | with self.assertRaises(AssertionError): | ||
| 134 | verify(events) | ||
| 135 | |||
| 136 | def test_connection_close_ends_guard(self): | ||
| 137 | events = self.history() | ||
| 138 | events.insert(-2, {'connection': 1, 'closed': True}) | ||
| 139 | with self.assertRaises(AssertionError): | ||
| 140 | verify(events) | ||
| 141 | |||
| 142 | |||
| 143 | if __name__ == '__main__': | ||
| 144 | unittest.main() | ||
tools/test_smb_proxy.py created+196
| ... | @@ -0,0 +1,196 @@ | ||
| 1 | import importlib.util | ||
| 2 | import asyncio | ||
| 3 | import json | ||
| 4 | from pathlib import Path | ||
| 5 | import socket | ||
| 6 | import struct | ||
| 7 | import sys | ||
| 8 | import tempfile | ||
| 9 | import unittest | ||
| 10 | |||
| 11 | spec = importlib.util.spec_from_file_location('smb_proxy', Path(__file__).with_name('smb-proxy.py')) | ||
| 12 | proxy = importlib.util.module_from_spec(spec) | ||
| 13 | spec.loader.exec_module(proxy) | ||
| 14 | |||
| 15 | |||
| 16 | class HeaderTrace(unittest.TestCase): | ||
| 17 | def test_fields_follow_complete_payload_ranges(self): | ||
| 18 | header = bytearray(1024) | ||
| 19 | header[96:100] = (257).to_bytes(4, 'little') | ||
| 20 | header[212:228] = bytes(range(16)) | ||
| 21 | header[228:236] = (999).to_bytes(8, 'little') | ||
| 22 | header[236:252] = bytes(range(16, 32)) | ||
| 23 | expected = {'transactions': 257, 'version': bytes(range(16)).hex(), | ||
| 24 | 'generation': 999, 'deny_read': bytes(range(16, 32)).hex()} | ||
| 25 | self.assertEqual(proxy.header_fields(0, header), expected) | ||
| 26 | self.assertEqual(proxy.header_fields(212, header[212:252]), {k:v for k,v in expected.items() if k != 'transactions'}) | ||
| 27 | self.assertEqual(proxy.header_fields(96, header[96:99]), {}) | ||
| 28 | self.assertEqual(proxy.header_fields(100, header[100:212]), {}) | ||
| 29 | self.assertEqual(proxy.header_fields(228, header[228:251]), {'generation':999}) | ||
| 30 | self.assertEqual(proxy.header_fields(252, bytes(1024)), {}) | ||
| 31 | |||
| 32 | |||
| 33 | class WriteTrace(unittest.IsolatedAsyncioTestCase): | ||
| 34 | async def test_opt_in_payload_and_partial_write_response_are_traced(self): | ||
| 35 | frames = [] | ||
| 36 | |||
| 37 | async def respond(reader, writer): | ||
| 38 | try: | ||
| 39 | for _ in range(3): | ||
| 40 | prefix = await reader.readexactly(4) | ||
| 41 | frame = await reader.readexactly(int.from_bytes(prefix[1:], 'big')) | ||
| 42 | frames.append(frame) | ||
| 43 | reply = bytearray(80) | ||
| 44 | reply[:64] = frame[:64] | ||
| 45 | struct.pack_into('<I', reply, 16, 1) | ||
| 46 | struct.pack_into('<I', reply, 68, 3) | ||
| 47 | writer.write(len(reply).to_bytes(4, 'big') + reply) | ||
| 48 | await writer.drain() | ||
| 49 | finally: | ||
| 50 | writer.close() | ||
| 51 | await writer.wait_closed() | ||
| 52 | |||
| 53 | server = await asyncio.start_server(respond, '127.0.0.1', 0) | ||
| 54 | server_port = server.sockets[0].getsockname()[1] | ||
| 55 | with socket.socket() as reservation: | ||
| 56 | reservation.bind(('127.0.0.1', 0)) | ||
| 57 | proxy_port = reservation.getsockname()[1] | ||
| 58 | with tempfile.TemporaryDirectory() as directory: | ||
| 59 | control = Path(directory) / 'control.json' | ||
| 60 | control.write_text('{}') | ||
| 61 | process = await asyncio.create_subprocess_exec( | ||
| 62 | sys.executable, str(Path(proxy.__file__)), str(control), | ||
| 63 | '--port', str(proxy_port), '--server', '127.0.0.1', '--server-port', str(server_port), | ||
| 64 | stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE) | ||
| 65 | records = [] | ||
| 66 | writer = None | ||
| 67 | try: | ||
| 68 | while not any('control' in row for row in records): | ||
| 69 | records.append(json.loads(await asyncio.wait_for(process.stdout.readline(), 5))) | ||
| 70 | reader, writer = await asyncio.open_connection('127.0.0.1', proxy_port) | ||
| 71 | sent = [] | ||
| 72 | for message, command in enumerate((9, 9, 17)): | ||
| 73 | if message == 1: | ||
| 74 | control.write_text('{"record_writes":true}') | ||
| 75 | while not any(row.get('control', {}).get('record_writes') for row in records): | ||
| 76 | records.append(json.loads(await asyncio.wait_for(process.stdout.readline(), 5))) | ||
| 77 | if message == 2: | ||
| 78 | control.write_text('{}') | ||
| 79 | while True: | ||
| 80 | row = json.loads(await asyncio.wait_for(process.stdout.readline(), 5)) | ||
| 81 | records.append(row) | ||
| 82 | if row.get('control') == {}: | ||
| 83 | break | ||
| 84 | data = b'abcde' if command == 9 else (4096).to_bytes(8, 'little') | ||
| 85 | offset = 112 if command == 9 else 96 | ||
| 86 | frame = bytearray(offset) | ||
| 87 | frame[:4] = b'\xfeSMB' | ||
| 88 | struct.pack_into('<H', frame, 12, command) | ||
| 89 | struct.pack_into('<Q', frame, 24, message) | ||
| 90 | frame[80:96] = bytes(range(16)) | ||
| 91 | if command == 9: | ||
| 92 | struct.pack_into('<HIQ', frame, 66, offset, len(data), 4096) | ||
| 93 | else: | ||
| 94 | struct.pack_into('<BBIH', frame, 66, 1, 20, len(data), offset) | ||
| 95 | frame += data | ||
| 96 | sent.append(bytes(frame)) | ||
| 97 | writer.write(len(frame).to_bytes(4, 'big') + frame) | ||
| 98 | await writer.drain() | ||
| 99 | prefix = await asyncio.wait_for(reader.readexactly(4), 5) | ||
| 100 | await asyncio.wait_for(reader.readexactly(int.from_bytes(prefix[1:], 'big')), 5) | ||
| 101 | while not any(row.get('direction') == 'response' and row['message'] == message for row in records): | ||
| 102 | records.append(json.loads(await asyncio.wait_for(process.stdout.readline(), 5))) | ||
| 103 | self.assertEqual(frames, sent) | ||
| 104 | requests = [row for row in records if row.get('direction') == 'request'] | ||
| 105 | self.assertNotIn('data', requests[0]) | ||
| 106 | self.assertEqual(requests[1]['data'], b'abcde'.hex()) | ||
| 107 | self.assertEqual(requests[1]['offset'], 4096) | ||
| 108 | self.assertEqual(requests[2]['data'], (4096).to_bytes(8, 'little').hex()) | ||
| 109 | self.assertEqual(requests[2]['file_id'], bytes(range(16)).hex()) | ||
| 110 | self.assertEqual([row['written'] for row in records if row.get('direction') == 'response' and row['command'] == 9], [3, 3]) | ||
| 111 | finally: | ||
| 112 | if writer is not None: | ||
| 113 | writer.close() | ||
| 114 | await writer.wait_closed() | ||
| 115 | if process.returncode is None: | ||
| 116 | process.terminate() | ||
| 117 | await asyncio.wait_for(process.wait(), 5) | ||
| 118 | server.close() | ||
| 119 | await server.wait_closed() | ||
| 120 | |||
| 121 | async def test_connection_cut_keeps_existing_and_new_peers_usable(self): | ||
| 122 | async def respond(reader, writer): | ||
| 123 | try: | ||
| 124 | while True: | ||
| 125 | prefix = await reader.readexactly(4) | ||
| 126 | frame = await reader.readexactly(int.from_bytes(prefix[1:], 'big')) | ||
| 127 | reply = bytearray(80) | ||
| 128 | reply[:64] = frame[:64] | ||
| 129 | struct.pack_into('<I', reply, 16, 1) | ||
| 130 | struct.pack_into('<I', reply, 68, 4) | ||
| 131 | writer.write(len(reply).to_bytes(4, 'big') + reply) | ||
| 132 | await writer.drain() | ||
| 133 | except (OSError, asyncio.IncompleteReadError): | ||
| 134 | pass | ||
| 135 | finally: | ||
| 136 | writer.close() | ||
| 137 | await writer.wait_closed() | ||
| 138 | |||
| 139 | async def send(writer, message): | ||
| 140 | frame = bytearray(116) | ||
| 141 | frame[:4] = b'\xfeSMB' | ||
| 142 | struct.pack_into('<H', frame, 12, 9) | ||
| 143 | struct.pack_into('<Q', frame, 24, message) | ||
| 144 | struct.pack_into('<HIQ', frame, 66, 112, 4, 96) | ||
| 145 | writer.write(len(frame).to_bytes(4, 'big') + frame) | ||
| 146 | await writer.drain() | ||
| 147 | |||
| 148 | async def receive(reader): | ||
| 149 | prefix = await asyncio.wait_for(reader.readexactly(4), 5) | ||
| 150 | return await asyncio.wait_for(reader.readexactly(int.from_bytes(prefix[1:], 'big')), 5) | ||
| 151 | |||
| 152 | for scope in ('connection', 'all'): | ||
| 153 | with self.subTest(scope=scope), tempfile.TemporaryDirectory() as directory: | ||
| 154 | server = await asyncio.start_server(respond, '127.0.0.1', 0) | ||
| 155 | control = Path(directory) / 'control.json' | ||
| 156 | control.write_text('{}') | ||
| 157 | process = await asyncio.create_subprocess_exec( | ||
| 158 | sys.executable, str(Path(proxy.__file__)), str(control), '--port', '0', | ||
| 159 | '--server', '127.0.0.1', '--server-port', str(server.sockets[0].getsockname()[1]), | ||
| 160 | stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE) | ||
| 161 | clients, records = [], [] | ||
| 162 | try: | ||
| 163 | while not any('control' in row for row in records): | ||
| 164 | records.append(json.loads(await asyncio.wait_for(process.stdout.readline(), 5))) | ||
| 165 | port, = [row['listening'] for row in records if 'listening' in row] | ||
| 166 | for index in range(2): | ||
| 167 | reader, writer = await asyncio.open_connection('127.0.0.1', port) | ||
| 168 | clients.append((reader, writer)) | ||
| 169 | await send(writer, index) | ||
| 170 | self.assertEqual(struct.unpack_from('<I', await receive(reader), 68)[0], 4) | ||
| 171 | setting = dict(cut=9, offset=96, scope=scope) | ||
| 172 | control.write_text(json.dumps(setting)) | ||
| 173 | while not any(row.get('control') == setting for row in records): | ||
| 174 | records.append(json.loads(await asyncio.wait_for(process.stdout.readline(), 5))) | ||
| 175 | await send(clients[0][1], 2) | ||
| 176 | with self.assertRaises(asyncio.IncompleteReadError): await receive(clients[0][0]) | ||
| 177 | while not any('cut' in row for row in records): | ||
| 178 | records.append(json.loads(await asyncio.wait_for(process.stdout.readline(), 5))) | ||
| 179 | cut, = [row['cut'] for row in records if 'cut' in row] | ||
| 180 | self.assertEqual((cut['direction'], cut['status'], cut['written']), ('response', '0x0', 4)) | ||
| 181 | reader, writer = await asyncio.open_connection('127.0.0.1', port) | ||
| 182 | clients.append((reader, writer)) | ||
| 183 | for reader, writer in clients[1:]: | ||
| 184 | if scope == 'connection': | ||
| 185 | await send(writer, 3) | ||
| 186 | self.assertEqual(struct.unpack_from('<I', await receive(reader), 68)[0], 4) | ||
| 187 | else: | ||
| 188 | with self.assertRaises(asyncio.IncompleteReadError): await receive(reader) | ||
| 189 | finally: | ||
| 190 | for _, writer in clients: | ||
| 191 | writer.close() | ||
| 192 | await writer.wait_closed() | ||
| 193 | if process.returncode is None: process.terminate() | ||
| 194 | await asyncio.wait_for(process.wait(), 5) | ||
| 195 | server.close() | ||
| 196 | await server.wait_closed() | ||
tools/test_verify_smb_faults.py created+97
| ... | @@ -0,0 +1,97 @@ | ||
| 1 | import hashlib | ||
| 2 | import json | ||
| 3 | from pathlib import Path | ||
| 4 | import tempfile | ||
| 5 | import unittest | ||
| 6 | import xml.etree.ElementTree as ET | ||
| 7 | |||
| 8 | from native_xml import ns | ||
| 9 | from verify_smb_faults import verify | ||
| 10 | |||
| 11 | |||
| 12 | class NativeFaultOracle(unittest.TestCase): | ||
| 13 | def setUp(self): | ||
| 14 | temporary = tempfile.TemporaryDirectory() | ||
| 15 | self.addCleanup(temporary.cleanup) | ||
| 16 | self.root = Path(temporary.name) | ||
| 17 | self.cases = self.root / 'cases' | ||
| 18 | self.worker = self.root / 'captures/worker-0' | ||
| 19 | self.worker.mkdir(parents=True) | ||
| 20 | self.records = [{'case': 'sample-000', 'visible': 'before'}, | ||
| 21 | {'case': 'sample-001', 'visible': 'after'}] | ||
| 22 | self.results = [] | ||
| 23 | for role, names in [('source', ['sample']), ('interrupted', ['sample-000', 'sample-001']), | ||
| 24 | ('recovered', ['sample-000', 'sample-001'])]: | ||
| 25 | (self.cases / role).mkdir(parents=True) | ||
| 26 | for name in names: | ||
| 27 | content = (role + name).encode() | ||
| 28 | (self.cases / role / (name + '.one')).write_bytes(content) | ||
| 29 | name = role + '-' + name | ||
| 30 | self.results.append({'name': name, 'error': None, 'pages': 1, | ||
| 31 | 'source_sha256': hashlib.sha256(content).hexdigest()}) | ||
| 32 | replacement = 'before' | ||
| 33 | if name == 'interrupted-sample-001': replacement = 'after' | ||
| 34 | if role == 'recovered': replacement = 'after again' | ||
| 35 | self.page(name, ['Title', replacement, 'Unrelated café 🦀']) | ||
| 36 | (self.cases / 'sample-intent.json').write_text(json.dumps({'before': 'before', 'after': 'after', 'suffix': ' again'})) | ||
| 37 | self.save() | ||
| 38 | |||
| 39 | def save(self): | ||
| 40 | (self.cases / 'results.json').write_text(json.dumps(self.records)) | ||
| 41 | (self.worker / 'results.json').write_text(json.dumps(self.results)) | ||
| 42 | |||
| 43 | def page(self, name, paragraphs): | ||
| 44 | page = ET.Element('{' + ns['one'] + '}Page') | ||
| 45 | outline = ET.SubElement(page, '{' + ns['one'] + '}Outline') | ||
| 46 | for text in paragraphs: | ||
| 47 | ET.SubElement(outline, '{' + ns['one'] + '}T').text = text | ||
| 48 | output = self.worker / 'results' / name | ||
| 49 | output.mkdir(parents=True, exist_ok=True) | ||
| 50 | ET.ElementTree(page).write(output / 'page-0.xml', encoding='utf-8') | ||
| 51 | |||
| 52 | def check(self): | ||
| 53 | return verify(self.root, self.root / 'captures') | ||
| 54 | |||
| 55 | def test_complete_before_and_after_outcomes(self): | ||
| 56 | self.assertEqual(self.check(), {'cases': 2, 'cold_native_opens': 5, 'exact_native_text': True}) | ||
| 57 | |||
| 58 | def test_unrelated_or_partial_edits_are_rejected(self): | ||
| 59 | for paragraphs in [['Title', 'after again', 'Changed'], ['Title', 'afte', 'Unrelated café 🦀'], | ||
| 60 | ['Title', 'after again again', 'Unrelated café 🦀']]: | ||
| 61 | with self.subTest(paragraphs=paragraphs): | ||
| 62 | self.page('recovered-sample-000', paragraphs) | ||
| 63 | with self.assertRaises(AssertionError): self.check() | ||
| 64 | |||
| 65 | def test_wrong_publication_outcome_is_rejected(self): | ||
| 66 | self.records[0]['visible'] = 'after' | ||
| 67 | self.save() | ||
| 68 | with self.assertRaises(AssertionError): self.check() | ||
| 69 | |||
| 70 | def test_input_hash_is_checked(self): | ||
| 71 | (self.cases / 'interrupted/sample-000.one').write_bytes(b'changed') | ||
| 72 | with self.assertRaises(AssertionError): self.check() | ||
| 73 | |||
| 74 | def test_missing_and_duplicate_captures_are_rejected(self): | ||
| 75 | original = self.results[:] | ||
| 76 | for rows in [original[:-1], original + original[:1]]: | ||
| 77 | self.results = rows | ||
| 78 | self.save() | ||
| 79 | with self.assertRaises(AssertionError): self.check() | ||
| 80 | |||
| 81 | def test_extra_artifacts_are_rejected(self): | ||
| 82 | (self.cases / 'interrupted/extra.one').write_bytes(b'extra') | ||
| 83 | with self.assertRaises(AssertionError): self.check() | ||
| 84 | |||
| 85 | def test_duplicate_cases_are_rejected(self): | ||
| 86 | self.records.append(self.records[0]) | ||
| 87 | self.save() | ||
| 88 | with self.assertRaises(AssertionError): self.check() | ||
| 89 | |||
| 90 | def test_native_failure_and_wrong_page_count_are_rejected(self): | ||
| 91 | for error, pages in [('Open failed', 1), (None, 0), (None, 2)]: | ||
| 92 | self.results[0].update(error=error, pages=pages) | ||
| 93 | self.save() | ||
| 94 | with self.assertRaises(AssertionError): self.check() | ||
| 95 | |||
| 96 | |||
| 97 | if __name__ == '__main__': unittest.main() | ||
tools/verify-collaboration.py+1-1
| ... | @@ -27,7 +27,7 @@ for name, text in expected.items(): | ... | @@ -27,7 +27,7 @@ for name, text in expected.items(): |
| 27 | captured = pages(corpus / 'native' / f'stage5-cold-{name}' / 'read') | 27 | captured = pages(corpus / 'native' / f'stage5-cold-{name}' / 'read') |
| 28 | assert len(captured) == 1 and texts(captured[0]) == text, name | 28 | assert len(captured) == 1 and texts(captured[0]) == text, name |
| 29 | 29 | ||
| 30 | subprocess.run(['cargo', 'build', '--quiet', '--example', 'inventory'], cwd=root, check=True) | 30 | subprocess.run(['cargo', 'build', '-p', 'onestore', '--quiet', '--example', 'inventory'], cwd=root, check=True) |
| 31 | for case, main, conflict in [ | 31 | for case, main, conflict in [ |
| 32 | ('live', 'Rust same paragraph.', 'Native same paragraph.'), | 32 | ('live', 'Rust same paragraph.', 'Native same paragraph.'), |
| 33 | ('offline', 'Rust edited during the native outage.', 'Native edited while disconnected.'), | 33 | ('offline', 'Rust edited during the native outage.', 'Native edited while disconnected.'), |
tools/verify-reader.py+1-1
| ... | @@ -12,7 +12,7 @@ root = Path(__file__).resolve().parent.parent | ... | @@ -12,7 +12,7 @@ root = Path(__file__).resolve().parent.parent |
| 12 | private = root / 'corpus/private' | 12 | private = root / 'corpus/private' |
| 13 | paths = sorted((private / 'original').rglob('*.one')) | 13 | paths = sorted((private / 'original').rglob('*.one')) |
| 14 | output = subprocess.check_output( | 14 | output = subprocess.check_output( |
| 15 | ['cargo', 'run', '--quiet', '--example', 'inventory', '--', *map(str, paths)], | 15 | ['cargo', 'run', '-p', 'onestore', '--quiet', '--example', 'inventory', '--', *map(str, paths)], |
| 16 | cwd=root, text=True, | 16 | cwd=root, text=True, |
| 17 | ) | 17 | ) |
| 18 | records = [line.split('\t') for line in output.splitlines()] | 18 | records = [line.split('\t') for line in output.splitlines()] |
tools/verify-writer.py+2-2
| ... | @@ -28,7 +28,7 @@ for before, after in [ | ... | @@ -28,7 +28,7 @@ for before, after in [ |
| 28 | source = before.read_bytes() | 28 | source = before.read_bytes() |
| 29 | assert after.read_bytes()[1024:len(source)] == source[1024:] | 29 | assert after.read_bytes()[1024:len(source)] == source[1024:] |
| 30 | 30 | ||
| 31 | native_saved = subprocess.check_output(['cargo', 'run', '--quiet', '--example', 'inventory', '--', str(corpus / 'native-edit-02/notebook/synthetic.one')], cwd=root, text=True) | 31 | native_saved = subprocess.check_output(['cargo', 'run', '-p', 'onestore', '--quiet', '--example', 'inventory', '--', str(corpus / 'native-edit-02/notebook/synthetic.one')], cwd=root, text=True) |
| 32 | assert [bytes.fromhex(row.split('\t')[2]).decode('ascii') for row in native_saved.splitlines() if row.startswith('ascii\t')] == ['Fictitious plain text.'] | 32 | assert [bytes.fromhex(row.split('\t')[2]).decode('ascii') for row in native_saved.splitlines() if row.startswith('ascii\t')] == ['Fictitious plain text.'] |
| 33 | 33 | ||
| 34 | before = pages(root / 'corpus/native-ink/cold-ui-ink/read')[0] | 34 | before = pages(root / 'corpus/native-ink/cold-ui-ink/read')[0] |
| ... | @@ -93,6 +93,6 @@ after = pages(append / 'native/append-01-complex/read')[0] | ... | @@ -93,6 +93,6 @@ after = pages(append / 'native/append-01-complex/read')[0] |
| 93 | assert fingerprint(normalized(before)) == fingerprint(normalized(after)) | 93 | assert fingerprint(normalized(before)) == fingerprint(normalized(after)) |
| 94 | assert next((root / 'corpus/native-ink/cold-ui-ink/read').glob('*.attachment')).read_bytes() == next((append / 'native/append-01-complex/read').glob('*.attachment')).read_bytes() | 94 | assert next((root / 'corpus/native-ink/cold-ui-ink/read').glob('*.attachment')).read_bytes() == next((append / 'native/append-01-complex/read').glob('*.attachment')).read_bytes() |
| 95 | assert ET.parse(append / 'native/append-01-toc/read/hierarchy.xml').getroot().get('color') == '#336699' | 95 | assert ET.parse(append / 'native/append-01-toc/read/hierarchy.xml').getroot().get('color') == '#336699' |
| 96 | saved = subprocess.check_output(['cargo', 'run', '--quiet', '--example', 'inventory', '--', str(append / 'native-after-511.one')], cwd=root, text=True) | 96 | saved = subprocess.check_output(['cargo', 'run', '-p', 'onestore', '--quiet', '--example', 'inventory', '--', str(append / 'native-after-511.one')], cwd=root, text=True) |
| 97 | assert [bytes.fromhex(row.split('\t')[2]).decode('ascii') for row in saved.splitlines() if row.startswith('ascii\t')] == ['Native after interrupted cleanup.'] | 97 | assert [bytes.fromhex(row.split('\t')[2]).decode('ascii') for row in saved.splitlines() if row.startswith('ascii\t')] == ['Native after interrupted cleanup.'] |
| 98 | print('Passed: append encoding, native metadata and counter recovery, post-recovery native save, and filesystem-backed SMB commit') | 98 | print('Passed: append encoding, native metadata and counter recovery, post-recovery native save, and filesystem-backed SMB commit') |
tools/verify_offline.py created+71
| ... | @@ -0,0 +1,71 @@ | ||
| 1 | #!/usr/bin/env python3 | ||
| 2 | """Audit completed offline/native runs, including the independent SQLite receipt ledger.""" | ||
| 3 | import argparse | ||
| 4 | import hashlib | ||
| 5 | import json | ||
| 6 | from pathlib import Path | ||
| 7 | import sqlite3 | ||
| 8 | |||
| 9 | from native_stress import edit_history, native_history | ||
| 10 | |||
| 11 | |||
| 12 | def verify(output, max_gap=120): | ||
| 13 | config = json.loads((output / 'run.json').read_text()) | ||
| 14 | assert config['offline'] and config['embedded_smb'] and not config['edit'] | ||
| 15 | assert config['stress_clients'] + config['rust_writers'] + config['rust_readers'] >= 12 | ||
| 16 | actors = [*(f'w{i}' for i in range(config['rust_writers'])), *(f'r{i}' for i in range(config['rust_readers']))] | ||
| 17 | logs = {actor: [json.loads(line) for line in (output / 'rust' / f'{actor}.jsonl').read_text().splitlines()] for actor in actors} | ||
| 18 | commits, text = edit_history(logs, config['stress_operations'], offline=True) | ||
| 19 | documents = {} | ||
| 20 | if config.get('document_operations'): | ||
| 21 | from offline_document_history import document_history | ||
| 22 | documents = document_history(logs, config['stress_operations']) | ||
| 23 | started = (output / 'rust/start').stat().st_mtime_ns // 1000 | ||
| 24 | stopped = (output / 'rust/stop').stat().st_mtime_ns // 1000 | ||
| 25 | progress, queues, caches = {}, {}, {} | ||
| 26 | for actor, events in logs.items(): | ||
| 27 | times = [event['at_us'] for event in events if event['event'] in ('remote_receipt', 'document_receipt')] if actor.startswith('w') else [event['finished_us'] for event in events if event['event'] == 'read'] | ||
| 28 | assert times and times == sorted(times), 'Client progress is absent or went backwards' | ||
| 29 | if actor.startswith('r'): times.append(max(times[-1], stopped)) | ||
| 30 | progress[actor] = max(b-a for a, b in zip([started, *times], times)) / 1_000_000 | ||
| 31 | if not actor.startswith('w'): continue | ||
| 32 | edits = {event['id']: event for event in events if event['event'] in ('local_commit', 'local_document_commit')} | ||
| 33 | receipts = {event['id']: event for event in events if event['event'] in ('remote_receipt', 'document_receipt')} | ||
| 34 | attempts = {event['revision']: event for event in events if event['event'] == 'remote_attempt'} | ||
| 35 | publication_starts = {id: documents[edit['object']][edit['kind']]['started_us'] if edit['event'] == 'local_document_commit' | ||
| 36 | else attempts[receipts[id]['revision']]['started_us'] for id, edit in edits.items()} | ||
| 37 | queues[actor] = max(sum(edit['finished_us'] <= at < publication_starts[id] for id, edit in edits.items()) for at in [edit['finished_us'] for edit in edits.values()]) | ||
| 38 | assert queues[actor] >= 2, 'Writer did not establish a durable local queue before publication' | ||
| 39 | path = output / 'rust' / f'{actor}.sqlite' | ||
| 40 | connection = sqlite3.connect(path.resolve().as_uri() + '?mode=ro', uri=True) | ||
| 41 | try: | ||
| 42 | assert connection.execute('PRAGMA quick_check').fetchall() == [('ok',)], 'Cache integrity failed' | ||
| 43 | assert connection.execute('PRAGMA foreign_key_check').fetchall() == [], 'Cache foreign keys failed' | ||
| 44 | for table in ('edits', 'attempt', 'conflicts'): | ||
| 45 | assert connection.execute(f'SELECT count(*) FROM {table}').fetchone() == (0,), 'Completed cache retains unresolved state' | ||
| 46 | persisted = dict(connection.execute('SELECT edit_id, revision FROM receipts')) | ||
| 47 | assert persisted == {id: event['revision'] for id, event in receipts.items()}, 'SQLite receipts differ from observed acknowledgements' | ||
| 48 | base, working = connection.execute('SELECT base, working FROM replica WHERE id=1').fetchone() | ||
| 49 | assert base == working, 'A drained cache retained a divergent local branch' | ||
| 50 | caches[actor] = {'receipts': len(persisted), 'image_bytes': len(base), 'image_sha256': hashlib.sha256(base).hexdigest(), 'database_sha256': hashlib.sha256(path.read_bytes()).hexdigest()} | ||
| 51 | finally: | ||
| 52 | connection.close() | ||
| 53 | for i in range(config['stress_clients']): | ||
| 54 | events = [json.loads(line) for line in (output / f'n{i}/stress-events.jsonl').read_text(encoding='utf-8-sig').splitlines()] | ||
| 55 | native_history(events, i, config['stress_operations'], False) | ||
| 56 | times = [event['updated_ticks'] for event in events] | ||
| 57 | assert times == sorted(times) | ||
| 58 | progress[f'n{i}'] = max((b-a for a, b in zip(times, times[1:])), default=0) / 10_000_000 | ||
| 59 | assert all(gap <= max_gap for gap in progress.values()), f'Client progress exceeded {max_gap}s: {progress}' | ||
| 60 | return {'remote_publications': len(commits), 'document_publications': len(documents)*2, 'remote_text_sha256': hashlib.sha256(text.encode()).hexdigest(), 'maximum_progress_gap_seconds': progress, | ||
| 61 | 'queued_before_publication_lower_bound': queues, 'reviewed_placements': sum(event['event'] == 'reviewed_append' for events in logs.values() for event in events), 'caches': caches} | ||
| 62 | |||
| 63 | |||
| 64 | if __name__ == '__main__': | ||
| 65 | parser = argparse.ArgumentParser(description=__doc__) | ||
| 66 | parser.add_argument('output', type=Path) | ||
| 67 | parser.add_argument('--max-gap', type=float, default=120) | ||
| 68 | args = parser.parse_args() | ||
| 69 | result = verify(args.output, args.max_gap) | ||
| 70 | (args.output / 'offline-verification.json').write_text(json.dumps(result, indent=2)) | ||
| 71 | print(json.dumps(result, indent=2)) | ||
tools/verify_offline_confirmation.py created+95
| ... | @@ -0,0 +1,95 @@ | ||
| 1 | #!/usr/bin/env python3 | ||
| 2 | """Compare cold native reads of confirmation snapshots with the recorded editing history.""" | ||
| 3 | import argparse | ||
| 4 | import hashlib | ||
| 5 | import json | ||
| 6 | from pathlib import Path | ||
| 7 | import xml.etree.ElementTree as ET | ||
| 8 | |||
| 9 | from native_format import native_characters | ||
| 10 | from native_stress import native_history | ||
| 11 | from native_xml import ns | ||
| 12 | from offline_history import publication_links | ||
| 13 | |||
| 14 | |||
| 15 | def verify(output, cold, *, partial=False): | ||
| 16 | config = json.loads((output / 'run.json').read_text()) | ||
| 17 | actors = [*(f'w{i}' for i in range(config['rust_writers'])), *(f'r{i}' for i in range(config['rust_readers']))] | ||
| 18 | logs = {actor: [json.loads(line) for line in (output / 'rust' / f'{actor}.jsonl').read_text().splitlines()] for actor in actors} | ||
| 19 | publication_links(logs, config['stress_operations'], partial=partial) | ||
| 20 | documents = {} | ||
| 21 | if config.get('document_operations'): | ||
| 22 | from offline_document_history import document_history | ||
| 23 | assert not partial, 'Document confirmation audit requires a complete workload' | ||
| 24 | documents = document_history(logs, config['stress_operations']) | ||
| 25 | unknown = [row for rows in logs.values() for row in rows if row['event'] == 'remote_attempt' and row['state'] == 'Unknown'] | ||
| 26 | assert len(unknown) == 1 | ||
| 27 | attempt, = unknown | ||
| 28 | confirmations = {} | ||
| 29 | for actor, rows in logs.items(): | ||
| 30 | observed = None | ||
| 31 | for row in rows: | ||
| 32 | if row['event'] == 'read': observed = row | ||
| 33 | if row['event'] != 'remote_confirm': continue | ||
| 34 | name = row['capture'] | ||
| 35 | assert Path(name).name == name and name.startswith(str(rows[0]['pid']) + '-') | ||
| 36 | assert name not in confirmations | ||
| 37 | if attempt['revision'] not in row['revisions'].get(attempt['space'], []): | ||
| 38 | assert documents and row.get('current_revisions', {}).get(attempt['space']) in row['revisions'].get(attempt['space'], []), 'Retired attempt lacks a current effect-confirmation revision' | ||
| 39 | if documents: | ||
| 40 | assert observed and observed['text'] == row['text'] and observed['finished_us'] <= row['started_us'] | ||
| 41 | assert isinstance(observed.get('documents'), dict) | ||
| 42 | assert all(observed['documents'].get(target) == value for target, value in attempt['document_changes'].items()), 'Confirmation omitted the uncertain document change' | ||
| 43 | confirmations[name] = row, observed['documents'] if documents else {} | ||
| 44 | assert confirmations | ||
| 45 | manifest = json.loads((cold / 'run.json').read_text())['inputs'] | ||
| 46 | assert set(manifest) == set(confirmations) | ||
| 47 | assert set(manifest) == {path.name for path in (output / 'rust/confirmations').glob('*.one')} | ||
| 48 | results = json.loads((cold / 'results.json').read_text(encoding='utf-8-sig')) | ||
| 49 | if isinstance(results, dict): results = [results] | ||
| 50 | assert len(results) == len(confirmations) and len({row['name'] for row in results}) == len(results) | ||
| 51 | native = [] | ||
| 52 | for i in range(config['stress_clients']): | ||
| 53 | rows = [json.loads(line) for line in (output / f'n{i}/stress-events.jsonl').read_text(encoding='utf-8-sig').splitlines()] | ||
| 54 | assert len(rows) <= config['stress_operations'] | ||
| 55 | native_history(rows, i, len(rows) if partial else config['stress_operations'], False) | ||
| 56 | native.append({f'Native {i}:', *(row['before'] + row['token'] for row in rows)}) | ||
| 57 | checks = format_checks = 0 | ||
| 58 | for result in results: | ||
| 59 | name = result['name'] + '.one' | ||
| 60 | assert result['error'] is None and result['pages'] == 1 | ||
| 61 | assert result['source_sha256'] == manifest[name] == hashlib.sha256((output / 'rust/confirmations' / name).read_bytes()).hexdigest() | ||
| 62 | page = ET.parse(cold / 'results' / result['name'] / 'page-0.xml').getroot() | ||
| 63 | formatted = native_characters(page, page.findall('one:Outline', ns)) | ||
| 64 | paragraphs = [''.join(c for c, _ in paragraph) for paragraph in formatted] | ||
| 65 | confirmation, observed = confirmations[name] | ||
| 66 | expected = confirmation['text'] | ||
| 67 | assert len(paragraphs) == len(native) + 1 + len(observed) and paragraphs.count(expected) == 1 | ||
| 68 | paragraphs.remove(expected) | ||
| 69 | if observed: | ||
| 70 | from offline_document_history import characters, verify_native | ||
| 71 | expected_documents = {target: {'text': documents[target]['text'], 'new': characters(value)} for target, value in observed.items()} | ||
| 72 | format_checks += verify_native(formatted, expected_documents) | ||
| 73 | for document in expected_documents.values(): | ||
| 74 | assert paragraphs.count(document['text']) == 1, 'Confirmation duplicated a document paragraph' | ||
| 75 | paragraphs.remove(document['text']) | ||
| 76 | for index, versions in enumerate(native): | ||
| 77 | selected = [text for text in paragraphs if text.startswith(f'Native {index}:')] | ||
| 78 | assert len(selected) == 1 and selected[0] in versions, 'Native confirmation image contains an unrecorded edit or loses a prefix' | ||
| 79 | checks += len(native) + 1 + len(observed) | ||
| 80 | assert json.loads((cold / 'teardown.json').read_text()) == {'absent': True} | ||
| 81 | confirmed_revision = documents[next(iter(attempt['document_changes']))]['format']['receipt_revision'] if documents else attempt['revision'] | ||
| 82 | return {'complete_workload': not partial, 'attempted_revision': attempt['revision'], 'confirmed_revision': confirmed_revision, 'native_images': len(results), 'exact_rust_paragraphs': len(results), | ||
| 83 | 'validated_paragraphs': checks, 'native_intended_format_checks': format_checks, | ||
| 84 | 'maximum_native_export_seconds': max(row['seconds'] for row in results)} | ||
| 85 | |||
| 86 | |||
| 87 | if __name__ == '__main__': | ||
| 88 | parser = argparse.ArgumentParser(description=__doc__) | ||
| 89 | parser.add_argument('run', type=Path) | ||
| 90 | parser.add_argument('cold', type=Path) | ||
| 91 | parser.add_argument('--partial', action='store_true', help='Validate preserved confirmations from an interrupted workload') | ||
| 92 | args = parser.parse_args() | ||
| 93 | result = verify(args.run, args.cold, partial=args.partial) | ||
| 94 | (args.run / 'confirmation-cold-verification.json').write_text(json.dumps(result, indent=2)) | ||
| 95 | print(json.dumps(result, indent=2)) | ||
tools/verify_offline_recovery.py created+56
| ... | @@ -0,0 +1,56 @@ | ||
| 1 | #!/usr/bin/env python3 | ||
| 2 | """Compare every interrupted/recovered image with an independent cold native export.""" | ||
| 3 | import argparse | ||
| 4 | import hashlib | ||
| 5 | import json | ||
| 6 | from pathlib import Path | ||
| 7 | import xml.etree.ElementTree as ET | ||
| 8 | from native_format import native_characters | ||
| 9 | from native_xml import ns | ||
| 10 | |||
| 11 | |||
| 12 | def paragraphs(path): | ||
| 13 | root = ET.parse(path).getroot() | ||
| 14 | return [''.join(char for char, _ in text) for text in native_characters(root, root.findall('one:Outline', ns))] | ||
| 15 | |||
| 16 | |||
| 17 | def verify(run, cold, source_capture): | ||
| 18 | original = paragraphs(source_capture) | ||
| 19 | target, = [text for text in original if text.startswith('Concurrent edits:')] | ||
| 20 | other = [text for text in original if text != target] | ||
| 21 | expected = {} | ||
| 22 | for case in json.loads((run/'results.json').read_text()): | ||
| 23 | for side in ('remote', 'local'): | ||
| 24 | if side+'_image' not in case: continue | ||
| 25 | digest, text = case[side+'_image'], case[side+'_text'] | ||
| 26 | assert text in (target, target+' [offline-recovery]'), 'Recovery oracle is unrelated to the native source' | ||
| 27 | wanted = sorted([text, *other]) | ||
| 28 | if digest in expected: assert expected[digest] == wanted, 'One image has contradictory expected states' | ||
| 29 | expected[digest] = wanted | ||
| 30 | inputs = json.loads((cold/'run.json').read_text())['inputs'] | ||
| 31 | assert inputs == {digest+'.one': digest for digest in expected}, 'Cold input inventory differs from recovery images' | ||
| 32 | records = json.loads((cold/'results.json').read_text(encoding='utf-8-sig')) | ||
| 33 | if isinstance(records, dict): records = [records] | ||
| 34 | assert len(records) == len(expected) and {record['name'] for record in records} == set(expected), 'Cold results omit or duplicate an image' | ||
| 35 | for record in records: | ||
| 36 | digest = record['name'] | ||
| 37 | assert record['source_sha256'] == digest, 'Native input hash differs' | ||
| 38 | assert hashlib.sha256((run/'images'/f'{digest}.one').read_bytes()).hexdigest() == digest, 'Retained image changed' | ||
| 39 | assert record['error'] is None and record['pages'] == 1, f'Native open failed: {record}' | ||
| 40 | pages = list((cold/'results'/digest).glob('page-*.xml')) | ||
| 41 | assert len(pages) == 1 | ||
| 42 | assert sorted(paragraphs(pages[0])) == expected[digest], f'Native content differs for {digest}' | ||
| 43 | assert json.loads((cold/'teardown.json').read_text())['absent'], 'Cold VM remains' | ||
| 44 | return {'exact_images':len(records), 'exact_paragraphs':sum(map(len, expected.values())), 'native_errors':0, | ||
| 45 | 'maximum_native_export_seconds':max(record['seconds'] for record in records)} | ||
| 46 | |||
| 47 | |||
| 48 | if __name__ == '__main__': | ||
| 49 | parser = argparse.ArgumentParser(description=__doc__) | ||
| 50 | parser.add_argument('run', type=Path) | ||
| 51 | parser.add_argument('cold', type=Path) | ||
| 52 | parser.add_argument('source_capture', type=Path) | ||
| 53 | args = parser.parse_args() | ||
| 54 | result = verify(args.run, args.cold, args.source_capture) | ||
| 55 | (args.run/'cold-verification.json').write_text(json.dumps(result, indent=2)) | ||
| 56 | print(json.dumps(result, indent=2)) | ||
tools/verify_smb_faults.py created+81
| ... | @@ -0,0 +1,81 @@ | ||
| 1 | #!/usr/bin/env python3 | ||
| 2 | """Cold-open every SMB fault artifact and compare its complete native text.""" | ||
| 3 | import argparse | ||
| 4 | from concurrent.futures import ThreadPoolExecutor | ||
| 5 | import hashlib | ||
| 6 | import json | ||
| 7 | from pathlib import Path | ||
| 8 | import signal | ||
| 9 | import xml.etree.ElementTree as ET | ||
| 10 | |||
| 11 | from native_format import native_characters | ||
| 12 | from native_xml import ns | ||
| 13 | from native_probe import ROOT, run as probe | ||
| 14 | |||
| 15 | |||
| 16 | def verify(root, captures): | ||
| 17 | cases = root / 'cases' | ||
| 18 | records = json.loads((cases / 'results.json').read_text()) | ||
| 19 | assert records and len({record['case'] for record in records}) == len(records), 'Missing or duplicate fault cases' | ||
| 20 | packets = {f'{role}-{path.stem}': path for role in ('source', 'interrupted', 'recovered') | ||
| 21 | for path in (cases / role).glob('*.one')} | ||
| 22 | expected_names = {f'{role}-{record["case"]}' for record in records for role in ('interrupted', 'recovered')} | ||
| 23 | fixtures = {record['case'].rsplit('-', 1)[0] for record in records} | ||
| 24 | expected_names.update(f'source-{fixture}' for fixture in fixtures) | ||
| 25 | assert set(packets) == expected_names, 'Artifact inventory differs from completed cases' | ||
| 26 | observed = {} | ||
| 27 | for worker in captures.glob('worker-*'): | ||
| 28 | result = json.loads((worker / 'results.json').read_text(encoding='utf-8-sig')) | ||
| 29 | for record in result if isinstance(result, list) else [result]: | ||
| 30 | name = record['name'] | ||
| 31 | assert name in packets and name not in observed, 'Unexpected or duplicate native result' | ||
| 32 | assert record['error'] is None and record['pages'] == 1, f'{name}: native open failed' | ||
| 33 | assert record['source_sha256'] == hashlib.sha256(packets[name].read_bytes()).hexdigest(), f'{name}: native input differs' | ||
| 34 | page, = (worker / 'results' / name).glob('page-*.xml') | ||
| 35 | page = ET.parse(page).getroot() | ||
| 36 | containers = page.findall('one:Title', ns) + page.findall('one:Outline', ns) | ||
| 37 | observed[name] = [''.join(char for char, _ in paragraph) for paragraph in native_characters(page, containers)] | ||
| 38 | assert set(observed) == expected_names, 'Missing native captures' | ||
| 39 | for record in records: | ||
| 40 | fixture = record['case'].rsplit('-', 1)[0] | ||
| 41 | intent = json.loads((cases / f'{fixture}-intent.json').read_text()) | ||
| 42 | baseline = observed[f'source-{fixture}'] | ||
| 43 | assert baseline.count(intent['before']) == 1, 'The native source lacks a unique editing target' | ||
| 44 | for role in ('interrupted', 'recovered'): | ||
| 45 | replacement = intent[record['visible']] if role == 'interrupted' else intent['after'] + intent['suffix'] | ||
| 46 | expected = [replacement if text == intent['before'] else text for text in baseline] | ||
| 47 | assert observed[f'{role}-{record["case"]}'] == expected, f'{role}-{record["case"]}: native content differs from the recorded outcome' | ||
| 48 | return {'cases': len(records), 'cold_native_opens': len(observed), 'exact_native_text': True} | ||
| 49 | |||
| 50 | |||
| 51 | def run(root, output): | ||
| 52 | assert json.loads((root / 'verification.json').read_text())['server_hashes_match'] | ||
| 53 | output.mkdir(parents=True, exist_ok=False) | ||
| 54 | (output / 'scripts').mkdir() | ||
| 55 | scripts = [output / 'scripts' / name for name in ('cold.ps1', 'probe.ps1')] | ||
| 56 | for path in scripts: path.write_bytes((ROOT / 'tools/native' / path.name).read_bytes()) | ||
| 57 | packets = [(role, path) for role in ('source', 'interrupted', 'recovered') | ||
| 58 | for path in sorted((root / 'cases' / role).glob('*.one'))] | ||
| 59 | workers = min(8, len(packets)) | ||
| 60 | inputs = [output / f'input-{i}' for i in range(workers)] | ||
| 61 | for path in inputs: path.mkdir() | ||
| 62 | for i, (role, path) in enumerate(packets): | ||
| 63 | (inputs[i % workers] / f'{role}-{path.name}').symlink_to(path.resolve()) | ||
| 64 | def capture(i): probe(inputs[i], output / f'worker-{i}', scripts) | ||
| 65 | with ThreadPoolExecutor(max_workers=workers) as pool: | ||
| 66 | list(pool.map(capture, range(workers))) | ||
| 67 | result = verify(root, output) | ||
| 68 | (output / 'verification.json').write_text(json.dumps(result, indent=2)) | ||
| 69 | print(json.dumps(result), flush=True) | ||
| 70 | |||
| 71 | |||
| 72 | if __name__ == '__main__': | ||
| 73 | parser = argparse.ArgumentParser(description=__doc__) | ||
| 74 | parser.add_argument('run', type=Path) | ||
| 75 | parser.add_argument('captures', type=Path) | ||
| 76 | parser.add_argument('--verify-only', action='store_true') | ||
| 77 | args = parser.parse_args() | ||
| 78 | def interrupted(_signal, _frame): raise KeyboardInterrupt | ||
| 79 | signal.signal(signal.SIGTERM, interrupted) | ||
| 80 | if args.verify_only: print(json.dumps(verify(args.run.resolve(), args.captures.resolve()), indent=2)) | ||
| 81 | else: run(args.run.resolve(), args.captures.resolve()) | ||
tools/verify_smb_overlap.py created+123
| ... | @@ -0,0 +1,123 @@ | ||
| 1 | """Verify successful I/O inside overlapping SMB reader and writer guards.""" | ||
| 2 | import json | ||
| 3 | from collections import Counter | ||
| 4 | |||
| 5 | |||
| 6 | class PendingOverlap(AssertionError): | ||
| 7 | pass | ||
| 8 | |||
| 9 | |||
| 10 | def renames(request): | ||
| 11 | return request['command'] == 17 and request.get('info_type') == 1 and request.get('info_class') == 10 | ||
| 12 | |||
| 13 | |||
| 14 | def verify(events, phase=None): | ||
| 15 | pending, files, peers, pairs = {}, {}, {}, {} | ||
| 16 | progress = Counter() | ||
| 17 | path_epoch = 0 | ||
| 18 | active_since = -1 if phase is None else None | ||
| 19 | for index, event in enumerate(events): | ||
| 20 | assert not event.get('trace_error'), 'Trace collection failed' | ||
| 21 | assert not event.get('encrypted'), 'Encrypted traffic cannot establish I/O overlap' | ||
| 22 | if phase is not None and event.get('control', {}).get('phase') == phase: | ||
| 23 | active_since = index | ||
| 24 | connection = event.get('connection') | ||
| 25 | if event.get('opened'): | ||
| 26 | peers[connection] = event['peer'][0] | ||
| 27 | if event.get('closed'): | ||
| 28 | files = {key: value for key, value in files.items() if key[0] != connection} | ||
| 29 | if 'command' not in event: | ||
| 30 | continue | ||
| 31 | message = connection, event['message'] | ||
| 32 | if event['direction'] == 'request': | ||
| 33 | if renames(event): | ||
| 34 | files.clear() | ||
| 35 | path_epoch += 1 | ||
| 36 | pending[message] = index, event, path_epoch | ||
| 37 | if event['command'] == 6: | ||
| 38 | files.pop((connection, event['file_id']), None) | ||
| 39 | elif event['command'] == 10: | ||
| 40 | file = files.get((connection, event['file_id'])) | ||
| 41 | if file: | ||
| 42 | for offset, length, flags in event['locks']: | ||
| 43 | if flags & 4: | ||
| 44 | file['locks'] = {at: lock for at, lock in file['locks'].items() | ||
| 45 | if not offset <= at < offset + length} | ||
| 46 | continue | ||
| 47 | if event['status'] == '0x103': | ||
| 48 | continue | ||
| 49 | if message not in pending and event['command'] in (0, 18): | ||
| 50 | continue | ||
| 51 | request_index, request, request_epoch = pending.pop(message) | ||
| 52 | if renames(request): | ||
| 53 | files.clear() | ||
| 54 | path_epoch += 1 | ||
| 55 | if event['status'] != '0x0': | ||
| 56 | continue | ||
| 57 | command = event['command'] | ||
| 58 | if command == 5: | ||
| 59 | if request_epoch != path_epoch or any(renames(request) for _, request, _ in pending.values()): | ||
| 60 | continue | ||
| 61 | files[connection, event['file_id']] = { | ||
| 62 | 'path': request['path'].lower(), 'access': request['access'], | ||
| 63 | 'locks': {}, | ||
| 64 | } | ||
| 65 | continue | ||
| 66 | if command not in (8, 9, 10): | ||
| 67 | continue | ||
| 68 | key = connection, request['file_id'] | ||
| 69 | if key not in files: | ||
| 70 | continue | ||
| 71 | file = files[key] | ||
| 72 | if command == 10: | ||
| 73 | for offset, length, flags in request['locks']: | ||
| 74 | if length != 1 or offset not in (0xfffffffb, 0xfffffffd): | ||
| 75 | continue | ||
| 76 | if flags & 4: | ||
| 77 | file['locks'].pop(offset, None) | ||
| 78 | else: | ||
| 79 | file['locks'][offset] = index, flags & 3 | ||
| 80 | continue | ||
| 81 | if active_since is None or request_index <= active_since or not file['path'].endswith('synthetic.one') or request['length'] == 0: | ||
| 82 | continue | ||
| 83 | native = peers[connection].startswith('192.168.77.') | ||
| 84 | progress[('native' if native else 'host', connection, 'read' if command == 8 else 'write')] += 1 | ||
| 85 | for other_key, other in files.items(): | ||
| 86 | if key[0] == other_key[0] or file['path'] != other['path']: | ||
| 87 | continue | ||
| 88 | reader, writer = (file, other) if command == 8 else (other, file) | ||
| 89 | reader_key, writer_key = (key, other_key) if command == 8 else (other_key, key) | ||
| 90 | reader_lock = reader['locks'].get(0xfffffffb) | ||
| 91 | writer_lock = writer['locks'].get(0xfffffffd) | ||
| 92 | # Only read-only handles distinguish a reader from a writer's own validation reads. | ||
| 93 | if reader['access'] & 0x40000002 or not reader_lock or not writer_lock: | ||
| 94 | continue | ||
| 95 | if reader_lock[1] != 1 or writer_lock[1] != 2: | ||
| 96 | continue | ||
| 97 | if request_index <= max(reader_lock[0], writer_lock[0]): | ||
| 98 | continue | ||
| 99 | pair = reader_key + (reader_lock[0],) + writer_key + (writer_lock[0],) | ||
| 100 | observed = pairs.setdefault(pair, {'read': 0, 'write': 0, | ||
| 101 | 'reader_peer': peers[reader_key[0]], 'writer_peer': peers[writer_key[0]]}) | ||
| 102 | observed['read' if command == 8 else 'write'] += 1 | ||
| 103 | assert active_since is not None, 'Requested trace phase was not observed' | ||
| 104 | both = [pair for pair in pairs.values() if pair['read'] and pair['write']] | ||
| 105 | native_writer_pairs = [pair for pair in both if pair['writer_peer'].startswith('192.168.77.') | ||
| 106 | and not pair['reader_peer'].startswith('192.168.77.')] | ||
| 107 | rust_writer_pairs = [pair for pair in both if not pair['writer_peer'].startswith('192.168.77.')] | ||
| 108 | if not both: raise PendingOverlap('No reader/writer guard pair performed both successful reads and writes while overlapping') | ||
| 109 | if not native_writer_pairs: raise PendingOverlap('No active Rust reader overlapped successful native writes') | ||
| 110 | if not rust_writer_pairs: raise PendingOverlap('No active reader overlapped successful Rust writes') | ||
| 111 | return {'active_guard_pairs': len(both), 'active_native_writer_pairs': len(native_writer_pairs), | ||
| 112 | 'active_rust_writer_pairs': len(rust_writer_pairs), | ||
| 113 | 'overlapping_reads': sum(pair['read'] for pair in both), | ||
| 114 | 'overlapping_writes': sum(pair['write'] for pair in both), | ||
| 115 | 'connections': [{'kind': kind, 'connection': connection, 'operation': operation, 'count': count} | ||
| 116 | for (kind, connection, operation), count in sorted(progress.items())]} | ||
| 117 | |||
| 118 | |||
| 119 | if __name__ == '__main__': | ||
| 120 | import sys | ||
| 121 | with open(sys.argv[1]) as stream: | ||
| 122 | result = verify(json.loads(line) for line in stream) | ||
| 123 | print(json.dumps(result, indent=2)) | ||