| 1 | # notebook |
| 2 | |
| 3 | The application-facing crate: notebook discovery, a durable local replica with |
| 4 | reconnect reconciliation, external-asset caching, recovery export and optional |
| 5 | embedded SMB access. A replica keeps one section's queue in a local SQLite database: |
| 6 | the image the queued edits apply to (the base, in 16 KiB chunks), the edits as |
| 7 | `onestore::op` ops in publication batches, and picture and attachment bytes once |
| 8 | each by hash. Local success does not acknowledge publication to a shared notebook. |
| 9 | |
| 10 | ```text |
| 11 | editor ── Edit ──► section thread ─ Section::apply, INSERT edit ─┐ |
| 12 | (the parsed section: base + sealed + open) │ one fsync per burst |
| 13 | sync thread: remote.stamp() == base ? publish(sealed batch) ──────┤ base chunks += transaction |
| 14 | otherwise read once, replay the queue on it ─────────┘ base := remote, conflict pages queued |
| 15 | ``` |
| 16 | |
| 17 | ## Edits |
| 18 | |
| 19 | `Replica::apply(author, edit)` applies an `onestore::op::Edit` to the section the |
| 20 | queue leaves and returns its id once written; a refused edit returns `Rejected` |
| 21 | and changes nothing. `page(space)` and `pages()` read what the queue leaves, for |
| 22 | opening and reloading. Edits collect in an open batch until the sync thread seals |
| 23 | it, so edits arriving while a publication is in flight publish together. |
| 24 | |
| 25 | ```no_run |
| 26 | use onestore::{ExGuid, op::{Edit, Op, PageOp}}; |
| 27 | use notebook::Replica; |
| 28 | # fn example(path: &std::path::Path, source: &[u8], space: ExGuid, text: ExGuid) |
| 29 | # -> Result<(), Box<dyn std::error::Error>> { |
| 30 | let cache = Replica::create(path, source)?; |
| 31 | let typed = PageOp::Text { text, range: 0..0, with: "Hello ".into() }; |
| 32 | let id = cache.apply("Author", Edit { at: 133_000_000_000_000_000, ops: vec![Op::Page { space, op: typed }] })?; |
| 33 | drop(cache); |
| 34 | |
| 35 | let reopened = Replica::open(path)?; |
| 36 | assert_eq!(reopened.pending()?.last().map(|edit| edit.id), Some(id)); |
| 37 | # Ok(()) |
| 38 | # } |
| 39 | ``` |
| 40 | |
| 41 | Page reads never wait for the sync thread: a rebase or reread of the queue runs on a |
| 42 | new thread while the current one keeps answering `page` and `pages` from the section as |
| 43 | it was, then hands its requests over; edits sent meanwhile apply to the rebuilt section. |
| 44 | |
| 45 | Share one `Replica` between threads. Keep the cache on a local filesystem: the |
| 46 | connection holds exclusive ownership between transactions, and a second open fails |
| 47 | busy. No network wait occurs in a local edit. After a database error the section |
| 48 | thread rereads the queue. |
| 49 | |
| 50 | ## Sessions |
| 51 | |
| 52 | `session::Notebook::open(root, cache_dir)` discovers a notebook directory, reading only |
| 53 | the files its folders list otherwise than when last opened (`discover::Cache`, kept under |
| 54 | `cache_dir/listings`), and |
| 55 | `session::Section::open(file, cache_dir, notify)` opens one section file through |
| 56 | a replica named by the file's location and the section's document identity, so the same |
| 57 | file reopens the same queue after a relaunch and a copy of it elsewhere has its own |
| 58 | (`notebook::location`: `folder`, `local`, `smb`, `moved` for a notebook the app moves, |
| 59 | and `forget` for one it deletes). |
| 60 | `Notebook::section(path, notify)` opens a mounted notebook's section; a replica already |
| 61 | named by the identity its discovery read resumes without reading the file, which the |
| 62 | worker checks next, as on a share. |
| 63 | A section is `Send + Sync`: `apply(author, edit)` returns at once and reports a refusal as `Event::Rejected`; `events()` drains |
| 64 | remote changes (`Changed(spaces)`), publication outcomes, unreachable files and |
| 65 | failures; `notify` runs on a background thread whenever an event waits. |
| 66 | `conflicts()` lists each page's conflict pages, which `page` reads and `delete_pages` |
| 67 | removes; `release(id, archive, Resolution)` ends an uncertain attempt. The author an |
| 68 | edit names is the host's: the app passes the account's full name. |
| 69 | `import_page` creates a page holding a copy of another; `delete_pages` removes pages. |
| 70 | `set_pause(pause)` has edits publish once `pause` passes without another (at most 30 s after the |
| 71 | first waiting), as a notebook on a cloud drive does; `wake()` publishes them at once. |
| 72 | `sync_status()` gives when the section file was last reached, why it could not be since |
| 73 | and how many edits wait for it. `set_offline(true)` works offline as OneNote does: the |
| 74 | worker stops connecting and edits queue until `wake()` (Sync Now) or `set_offline(false)`. |
| 75 | |
| 76 | `session::Background` keeps the sections no session holds in sync, as OneNote 2010 keeps |
| 77 | every section of an open notebook: `Background::smb(root, limit, connect, notify)` on a |
| 78 | share, `Notebook::background(watched, copies, notify)` for a mounted notebook, then |
| 79 | `watch(notebook.replicas())`. On a share, one CHANGE_NOTIFY on the notebook's folder |
| 80 | (`smb::Client::watch`) reports what changed; a host watching a mounted folder passes the |
| 81 | changed paths to `touched`, and says whether it reports every change as `watched`. A |
| 82 | reported section is checked a second later; a reported folder is listed a second later and |
| 83 | only its sections listed otherwise are checked. A failing section is tried every 31 seconds, |
| 84 | and any other every `Background::BACKSTOP` while a watch reports, `UNWATCHED` otherwise. On |
| 85 | connecting, and on working online again, every folder is listed: a section whose file lists as |
| 86 | discovery or the last check found it needs no reading, the others are checked one at a time |
| 87 | 100 ms apart. A check reads the file's stamp; a section whose replica has edits waiting, or |
| 88 | whose file moved past the replica's base, has the replica opened for the synchronization steps |
| 89 | that publish or rebase it and closed again, and a section without a replica gets one from the |
| 90 | file, its offline copy, on a share or with `copies`. `hold(path, section)` leaves an open |
| 91 | section to its session: its worker no longer polls while a watch reports, the watch wakes it |
| 92 | instead, and once its replica is released the background checks the file at once. A replica |
| 93 | a session holds is otherwise skipped (`Error::busy`), so opening a section may wait out one |
| 94 | step. `status()` gives each section's `SyncStatus`, `changed()` the sections another client |
| 95 | changed, `wake` and `set_offline` follow Sync Now and Work Offline, and `discard` stops the |
| 96 | thread and deletes the replicas holding nothing unpublished, as when the notebook closes. |
| 97 | `Notebook::replica_path(path)` names a section's replica for either kind of notebook. |
| 98 | |
| 99 | `Section::resume(file, replica, notify)` starts from an owned `Replica` without |
| 100 | consulting the remote file; with the `smb` feature, `Section::resume_smb(path, |
| 101 | replica, limit, connect, notify)` binds a share-relative path, `connect` running on |
| 102 | the worker again after transport failure. |
| 103 | |
| 104 | ## Reconciliation |
| 105 | |
| 106 | `sync_once(&mut remote)` returns what one step did as `Synced { edit, changed }`: |
| 107 | the state the step's batch reached, named by its newest edit, and the pages a remote |
| 108 | change replaced. While the remote's stamp (header and length) equals the base's, the |
| 109 | oldest sealed batch publishes as its `Transaction`, and the base's chunks take its |
| 110 | writes; nothing is read. When the stamp moved, the remote image is read once and the |
| 111 | queue replays on it (`merge.rs`): an op whose objects the remote left alone applies |
| 112 | as it is; a text op shifts past the remote's changes to the same text when its |
| 113 | range stays clear of them; a move, deletion or property the remote already made is |
| 114 | done; a page the remote already holds as the local edits leave it drops their ops. |
| 115 | Anything else conflicts, as in OneNote 2010: the op is dropped with every later op naming |
| 116 | what it named, the remote's version stays the page, and the local version becomes a |
| 117 | conflict page under it (`SectionOp::Conflict`, queued as one more edit and published with |
| 118 | the rest), its conflicting objects marked, named for the author of the first edit that did |
| 119 | not replay. Page lists merge as OneNote 2010 merges page series: a page the remote moved |
| 120 | keeps the remote's place, a page moved here goes before the next page in the local order |
| 121 | the remote left in place, or last, and a page the remote removed comes back as a new copy |
| 122 | of the local one, placed the same way (`corpus/conflict-page/native-pages`, |
| 123 | `native-restore`). Nothing blocks the queue. |
| 124 | |
| 125 | A remote a file provider keeps may list versions of the file beside it (`Remote::versions`, |
| 126 | `version`, `retire`), as iCloud Drive keeps a commit that lost to another device's. Each step |
| 127 | merges them first (`resolve.rs`): per object space, the newest revision of the version's that |
| 128 | the file holds, or holds merged, is where the two last agreed; the section opened at those |
| 129 | revisions (`onestore::Section::open_at`) is the base the version's changes lower from |
| 130 | (`lower_page`), and they replay on the file through the rebase rules above, conflict pages where |
| 131 | they clash. The revisions the merge writes are named after the version's it merged |
| 132 | (`Section::seal_as`), so a version merged once, here or on another device, merges as nothing |
| 133 | again. It is published on the file's stamp, then retired; one of another section, or one that |
| 134 | cannot be read, is retired with `keep` for the remote to keep beside the file. |
| 135 | |
| 136 | Publication attempts are recorded before network I/O. An attempt confirms only |
| 137 | when the remote holds its revisions, or every page it changed as it changed them, |
| 138 | followed by `Remote::confirm`; otherwise it stays `AwaitingConfirmation` and is never |
| 139 | replayed. `release(id, archive, Resolution)` exports the queue first, then |
| 140 | publishes the batch again (`Mine`) or abandons every unpublished edit (`Theirs`). |
| 141 | A durable `Published` receipt survives reopening. |
| 142 | |
| 143 | A schema-14 cache is converted when opened: it is exported to |
| 144 | `<cache>.v14-recovery`, each queued page is lowered against the page the conversion |
| 145 | has so far, and every converted page must equal the page in the old working image, |
| 146 | or the conversion rolls back and the open fails naming the archive; a page it held in |
| 147 | conflict becomes a conflict page. A schema-15 cache's batches lose the review conflict |
| 148 | they recorded when opened. |
| 149 | |
| 150 | With the optional `smb` feature, `SmbRemote::new(client, path, limit)` binds an |
| 151 | `notebook::smb::Client` to one share-relative file and snapshot limit. Remote identity uses the logical root |
| 152 | object space, which survives the tested native compaction that replaces the file ID. |
| 153 | |
| 154 | An `Arc<Replica>` can own one background worker. Supply a connection factory, poll |
| 155 | interval and observer; successful publications drain immediately, durable local |
| 156 | edits wake the worker, and `wake()` requests an immediate reachability retry. |
| 157 | While nothing is queued, or the queue waits on a remote that has not changed since, |
| 158 | a remote whose `Remote::stamp` holds is not read again; a worker a `Background` holds reads |
| 159 | the stamp only when the notebook's watch wakes it. |
| 160 | Transport failures discard the old connection and retry through the factory; |
| 161 | Read contention and `NotCommitted` operations with `WouldBlock` or `ResourceBusy` |
| 162 | reuse the connection. Contended `NotCommitted` operations use randomized backoff, |
| 163 | capped at one second, to separate competing retry cycles. Cancellation interrupts this delay; local wake notifications |
| 164 | remain coalesced until its end. |
| 165 | `RemoteIo` distinguishes connection/read failures from |
| 166 | local `Io` errors. Cache/document errors stop the worker. |
| 167 | |
| 168 | ```no_run |
| 169 | # #[cfg(feature = "smb")] |
| 170 | # fn example(cache: std::sync::Arc<notebook::Replica>, username: String, password: String) |
| 171 | # -> Result<(), Box<dyn std::error::Error>> { |
| 172 | use notebook::SmbRemote; |
| 173 | use notebook::smb::{Client, Credentials}; |
| 174 | use std::time::Duration; |
| 175 | |
| 176 | let worker = cache.start_sync( |
| 177 | Duration::from_secs(2), |
| 178 | move || { |
| 179 | Client::connect( |
| 180 | "server:445", "notes", |
| 181 | Credentials { username: &username, password: &password, domain: "" }, |
| 182 | Duration::from_secs(5), |
| 183 | ).map(|client| SmbRemote::new(client, "Personal/Video.one", 64 * 1024 * 1024)) |
| 184 | }, |
| 185 | |result| { |
| 186 | if let Err(error) = result { eprintln!("{error}"); } |
| 187 | }, |
| 188 | )?; |
| 189 | // Retain `worker` while synchronization should run; local edits wake it automatically. |
| 190 | worker.stop()?; |
| 191 | # Ok(()) |
| 192 | # } |
| 193 | ``` |
| 194 | |
| 195 | `stop()` cancels future steps and joins the worker, returning a fatal cache error |
| 196 | or worker panic. Dropping it requests cancellation without waiting. Remote |
| 197 | operations and callbacks must have bounded execution times if shutdown latency |
| 198 | matters. Credentials belong to the factory, not the cache database. |
| 199 | |
| 200 | `create` refuses an existing path; `open_or_create(path, key, source)` opens the cache at |
| 201 | `path`, reading `source` only to make it where there is none, and opens one another thread |
| 202 | made meanwhile. Opening validates database integrity and replays |
| 203 | the queue on its base. SQLite runs in WAL mode under exclusive locking with FULL |
| 204 | synchronization and fullfsync, each queried back: every commit is durable, and the only |
| 205 | file beside the cache is `<cache>-wal`, which holds committed pages until a checkpoint |
| 206 | and must travel with the cache when it is copied. Recovery archives are single files. |
| 207 | The cache contains notebook content. |
| 208 | |
| 209 | The SMB-enabled `smb_offline_client` example is an owned-lab workload for |
| 210 | `tools/native_collaboration.py --offline --embedded-smb`; its append-specific |
| 211 | conflict policy lives in the client. |
| 212 | |
| 213 | ## Recovery archives |
| 214 | |
| 215 | `export_recovery(new_path)` captures the base and remote images, the edit queue, |
| 216 | uncertain attempts, receipts, downloaded media and the edit-ID sequence in one |
| 217 | SQLite snapshot. It refuses existing destinations and leaves the live queue |
| 218 | unchanged. Export to a local directory from a background thread: copying holds |
| 219 | the cache mutex while capturing the database. Failure after the final rename can |
| 220 | leave a complete archive at the requested path; it never acknowledges a remote |
| 221 | edit. Archives contain notebook content and use a separate database identity, so |
| 222 | `Replica::open` rejects them as writable caches. |
| 223 | |
| 224 | ```no_run |
| 225 | use notebook::{Recovery, Replica}; |
| 226 | # fn example(cache: &Replica) -> Result<(), Box<dyn std::error::Error>> { |
| 227 | cache.export_recovery("review.sqlite")?; |
| 228 | let review = Recovery::open("review.sqlite")?; |
| 229 | let counts = review.summary()?; |
| 230 | let local = review.snapshot()?; |
| 231 | let remote = review.remote_snapshot()?; |
| 232 | let pending = review.pending()?; |
| 233 | let receipts = review.receipts()?; |
| 234 | # Ok(()) |
| 235 | # } |
| 236 | ``` |
| 237 | |
| 238 | `Recovery` provides read-only inspection and no synchronization or restore method. |
| 239 | `status(id)` preserves the same state interpretation as the live replica. |
| 240 | `recovery_summary()` on the live replica and `summary()` on an archive return |
| 241 | counts and byte sizes without notebook text, paths, authors or credentials. |
| 242 | Opening an archive validates its schema and images without migration. A recovery |
| 243 | archive is evidence for a reviewed recovery decision, not a second active queue. |
| 244 | |
| 245 | ## Downloaded media |
| 246 | |
| 247 | `fetch_asset(source, section, filename, limit)` resolves a declared external |
| 248 | payload through `notebook::discover::Source` and durably caches its exact bytes. |
| 249 | The section path is relative to the source root; the filename comes from a |
| 250 | `FileDataReference::External` in the retained working or remote image. Local and |
| 251 | SMB sources use the same API. Downloads release the cache mutex during network |
| 252 | I/O and recheck the reference before committing; edits can continue meanwhile. |
| 253 | |
| 254 | `cached_asset(filename, limit)` reads previously downloaded bytes without network |
| 255 | access. `None` means never downloaded; `Some(Vec::new())` is a downloaded empty |
| 256 | payload. Each read checks the stored SHA-256 and enforces the byte limit before |
| 257 | loading the payload. Cache contents describe the prior download, not current |
| 258 | server reachability or presence. A failed fetch returns its error without |
| 259 | silently substituting cached bytes. Different bytes for an already cached file |
| 260 | identity return `AssetChanged` and preserve the previous download. |
| 261 | |
| 262 | Downloads do not change pending edits, publication attempts or receipts. Recovery |
| 263 | archives include cached media and expose the same bounded `cached_asset` lookup. |
| 264 | |
| 265 | ## Discovery |
| 266 | |
| 267 | `notebook::discover` provides read-only notebook discovery over a caller-supplied |
| 268 | root, keeping directory traversal out of the single-file storage parser. |
| 269 | |
| 270 | Discovery returns ordered sections and nested groups with file identities and |
| 271 | share-relative paths. Section display-name overrides remain distinct from file |
| 272 | names. TOC references whose identities are absent from the directory remain |
| 273 | inspectable; a cached filename never substitutes for an identity match. |
| 274 | Reserved `_onefiles` directories are excluded from section-group traversal; |
| 275 | OneNote's `OneNote_RecycleBin` is listed as the section group OneNote shows. |
| 276 | Dot files and folders (`.DS_Store`, AppleDouble `._` companions, `.snowbound`) and Office's |
| 277 | `~$` owner files are skipped wherever they are, never read as sections. |
| 278 | Encrypted sections and valid storage with an unreadable document graph retain |
| 279 | their identity as `Locked` or `Unreadable`, without being presented as empty pages. |
| 280 | A child folder or section file that is denied or gone while listing (a folder |
| 281 | another client holds delete-pending reads as denied) is kept as `unavailable` and |
| 282 | tried again on the next discovery; the root itself, malformed storage, other failed |
| 283 | reads and ambiguous identities reject the discovery. |
| 284 | |
| 285 | `Cache::discover` keeps each file's listing (size and last write time), stamp and what |
| 286 | discovery took from it; the next discovery reads only the files listed otherwise, taking one |
| 287 | from a copy where `Source::copy` has it as it stands (on a share, a replica whose base has the |
| 288 | file's stamp), and `found(path)` gives a section's listing and stamp for `Background::watch`. |
| 289 | `take` hands on, once, the sections it read from the source, up to a memory bound, which |
| 290 | `Notebook::replicas` gives `Background::watch` so that its first check need not read them |
| 291 | again. A file written during discovery is read again next time; a failed discovery leaves the |
| 292 | cache as it was. |
| 293 | |
| 294 | Each file read must be a consistent, bounded snapshot. The result is an |
| 295 | observation across multiple files, not an atomic notebook transaction or |
| 296 | authorization to publish an edit. Refresh rejects observed topology changes. Of |
| 297 | several section files with one identity (a `Name 2.one` copy), each lists, as OneNote opens |
| 298 | each; all but the one its folder's TOC names (else the one with the shortest path) are |
| 299 | marked `copy`, have replicas of their own, and are never placed or listed anew by structure |
| 300 | operations, which touch only a TOC entry OneNote gave the copy's name. Of several groups |
| 301 | with one TOC identity, the one its parent's TOC names, else the newest, lists; the others |
| 302 | list in `unavailable` as `Reason::Copy`. iOS's |
| 303 | `.Name.one.icloud` placeholders, and macOS's dataless files, list there under their real names |
| 304 | as `Reason::Evicted`, unless listed unchanged since discovery last read them. |
| 305 | Retain the last accepted catalog if discovery fails; a connection failure does not mean |
| 306 | files were deleted. |
| 307 | |
| 308 | `read_external_asset` resolves a validated UUID `.onebin` filename beneath the |
| 309 | selected section's sibling `_onefiles` folder and returns exact bounded bytes. |
| 310 | Missing files, permissions and size failures retain their I/O error kinds; an |
| 311 | empty payload is a successful empty buffer. Embedded payloads remain available |
| 312 | directly from the core document model. |
| 313 | |
| 314 | ## Notebook structure |
| 315 | |
| 316 | `Notebook::create`, `create_section`, `create_group`, `rename`, `move_entry`, |
| 317 | `set_section_color`, `set_color`, `reorder` and `delete` change a notebook the way OneNote does: the |
| 318 | table of contents (`Open Notebook.onetoc2`, created when a folder has none) gains, |
| 319 | renames, reorders or loses entries; a new notebook holds "New Section 1", and a new |
| 320 | section an empty section's file plus the page its `PageCreation` makes, in OneNote's |
| 321 | new-section colour order; a section's colour lives in its own metadata, the notebook's |
| 322 | in its root table of contents (`corpus/section-color`); a deleted |
| 323 | section moves into `OneNote_RecycleBin`, a group with its own TOC, and a deleted group's |
| 324 | sections move there too before its folders go. `recycle_pages` keeps copies of pages a |
| 325 | section is about to delete in `OneNote_RecycleBin/OneNote_DeletedPages.one`, with their |
| 326 | identities, titles, dates and creation times, as OneNote does |
| 327 | (`corpus/notebook-management`); `unrecycle_pages` takes them out again by identity, as |
| 328 | OneNote's Undo of a page delete does. A bin or deleted-pages file its TOC does not list is |
| 329 | placed and listed again, a bin whose TOC OneNote named otherwise keeps it, and an |
| 330 | unavailable bin refuses the delete (`corpus/recycle-bin-repair`). `empty_recycle_bin` is OneNote's |
| 331 | Empty Recycle Bin: Deleted Pages loses every page in one revision and each binned section's |
| 332 | file goes, the bin's TOC left as it was (`corpus/recycle-bin-view`). An entry a TOC still |
| 333 | lists for a file gone from its folder gives way to the file an edit gives its name. |
| 334 | `rename_folder` renames the notebook's own folder, which OneNote 2010 opens again from its |
| 335 | new name with no file inside changed (it drops the old one from its list): on a share it |
| 336 | first takes and lets go of OneNote's writer locks on every section and TOC, refusing with |
| 337 | `WouldBlock` while another writer holds one, then the replicas and listing follow. |
| 338 | Edits keep entries in the order their ordering numbers give; a rename or colour keeps |
| 339 | the numbers. Every created, renamed or moved file is placed with |
| 340 | `onestore::place_file`, which sets the header's ancestor to the parent TOC's |
| 341 | identity and the name CRC OneNote checks on open; a file without them is |
| 342 | re-identified and listed anew. They run over `Storage`: a mounted directory |
| 343 | (`Notebook::open`) or an SMB share (`Notebook::open_smb` with a |
| 344 | `smb::Client`, which gained create, directory creation, rename, delete and |
| 345 | header placement under native writer coordination); `tools/test_smb_structure.py |
| 346 | VM OUTPUT` drives them against a disposable Samba lab VM. |
| 347 | |
| 348 | `Notebook::find_page(url)` resolves a stored internal link to a section path |
| 349 | and page space by identity: the linked section first, then every readable |
| 350 | section, so a link follows its page when the section is renamed or moved and |
| 351 | when the page itself was moved to another section. Other URLs and unknown |
| 352 | targets are `None`. |
| 353 | |
| 354 | `Notebook::read_section(path)` reads a section file as stored, without opening a |
| 355 | replica, and `session::stored_pages(image)` builds its pages with each page's |
| 356 | `LastModifiedTime` and the author of its latest change: the app's search indexes the |
| 357 | sections it has not opened this way, and unread changes leave out the reader's own. |
| 358 | |
| 359 | A password-protected section lists as `Locked`. `Notebook::unlock(path, password)` opens |
| 360 | its key (a wrong password is `Error::Protected(PasswordMismatch)`), and |
| 361 | `section_unlocked(path, &key, notify)` (`section_unlocked_with`, or |
| 362 | `Replica::open_or_create(path, Some(&key), source)` for a host that opens replicas itself) |
| 363 | opens it as any section opens: edits queue, publish, merge and become conflict pages as an ordinary section's do, |
| 364 | each revision sealed under the section's key. The replica keeps none of it in the clear: |
| 365 | its base and transactions are the file's own ciphertext, and its queued edits and payloads |
| 366 | are sealed with AES-256-GCM (a random nonce each, the row's kind as associated data) under |
| 367 | a key HMAC-SHA256 derives from the section's, payloads named by HMAC rather than SHA-256. |
| 368 | Only the author's name, payload names and revision identities stay readable, and the |
| 369 | queue opens only under the key (a recovery archive with `Recovery::open_unlocked`). |
| 370 | `stored_pages_unlocked(image, &key)` reads a protected section's pages for search. The |
| 371 | background never opens a locked section. |
| 372 | |
| 373 | `Notebook::set_password(path, key, password)` sets, changes or removes a password as |
| 374 | OneNote 2010 does: the section is written anew under new identities |
| 375 | (`onestore::protected::rekey`), next to the file as a dot file that `Storage::supersede` |
| 376 | puts in its place under the writers' coordination while the file's stamp holds (on a share as |
| 377 | OneNote's maintenance does it: the old file aside, the new one in, the old one deleted), and |
| 378 | its TOC entry takes the new identity (`TocEdit::Reidentify`). The old replica, a cache of the |
| 379 | old file, is deleted, so the section's session must be closed and its edits published first; |
| 380 | a section whose edits wait refuses. |
| 381 | |
| 382 | Another device learns of the password when it reads the notebook again (`open`, `refresh`): |
| 383 | a replica of the file the protected section superseded, found by the placement the rewritten |
| 384 | header keeps, is deleted when nothing waits in it. Edits that wait stay until `unlock` on that |
| 385 | device queues them under the key, each page they changed coming back as a copy, as a page |
| 386 | another client removed does. A session open meanwhile stops as `SyncState::Protected`. |
| 387 | |
| 388 | `Section::import_page(page, author)` copies a page, usually read from another |
| 389 | section, to the end of this one as one `SectionOp::Import` queued like the |
| 390 | user's own edits, under fresh identities (`Page::copy`); payloads travel with |
| 391 | the model, so the copy publishes offline later like any edit. Content outside |
| 392 | the model refuses to copy. A move is an import here followed by |
| 393 | `Section::delete_pages` there. |
| 394 | |
| 395 | `Notebook::refresh` rereads the directory on the caller's schedule and reports |
| 396 | what another client changed as `Change`s keyed by file identity: a renamed or |
| 397 | moved section is `Moved`, not removed and added, and a deleted section moves |
| 398 | into `OneNote_RecycleBin`; a folder whose surviving entries changed sequence |
| 399 | is `Reordered`. A failed read returns the error and keeps |
| 400 | the previous catalog, so an unreachable share never reads as an emptied |
| 401 | notebook. |
| 402 | |
| 403 | ## Packages |
| 404 | |
| 405 | `package` reads and writes OneNote packages (`.onepkg`), cabinet files of a notebook's tables |
| 406 | of contents and sections at their paths in its folder. `notebook_files(notebook, image)` gathers |
| 407 | what OneNote 2010's Save As packs (each folder's TOC and sections, groups after, the recycle bin |
| 408 | and copies left out, its TOC entry kept), taking a section the caller holds edits for from |
| 409 | `image` (a replica's `snapshot`); `pack` writes them MSZIP-compressed, each file outside any |
| 410 | notebook (no `guidAncestor` or `crcName`), as OneNote's are. `read` takes MSZIP or OneNote's LZX, |
| 411 | refusing a path leaving the folder or more than a byte limit unpacked, and `unpack(files, root)` |
| 412 | writes a new folder as Unpack Notebook does: every file takes an identity of its own |
| 413 | (`onestore::reidentify`) and is placed under its folder's TOC, whose entries follow it where |
| 414 | OneNote lists each anew beside the stale one. `section_copy` and `page_section` are Save As's |
| 415 | section and page: a copy under a new identity, and a new section holding the page with its |
| 416 | identity, both outside any notebook (`corpus/notebook-package`). |
| 417 | |
| 418 | ## Snowbound's folder |
| 419 | |
| 420 | `sidecar` keeps Snowbound-only data in the notebook's `.snowbound` folder, which |
| 421 | OneNote 2010 skips because it has the Windows hidden attribute: `Storage::hide` sets it |
| 422 | on a share with SET_INFO FileBasicInformation (always, since Samba reports a dot name hidden |
| 423 | without storing it), `SetFileAttributesW` on Windows, or macOS's |
| 424 | `UF_HIDDEN`, which an smbfs mount passes to the share; Linux keeps no such attribute. Discovery skips dot names, and a notebook |
| 425 | without the folder is whole. |
| 426 | |
| 427 | ```text |
| 428 | .snowbound/ |
| 429 | ├ tags.json [{ name, shape, art, mapped }]: a tag's name and symbol → its art |
| 430 | ├ tags/ <SHA-256>.png | .svg, written through a temporary name, never rewritten |
| 431 | └ themes.json { themes: [{ id, name, styles, modified, deleted }], |
| 432 | assignments: [{ scope, theme, assigned }] }: style themes and who wears them |
| 433 | ``` |
| 434 | |
| 435 | `Notebook::map_tag_art(name, shape, bytes, extension)` makes the folder and hides it, |
| 436 | keeps the picture, then rereads `tags.json`, merges its mapping and replaces the file, |
| 437 | reading it back and merging again until its own holds. `sidecar::merge` keeps every tag |
| 438 | either side mapped; one tag mapped by both keeps the later `mapped`, then the greater art |
| 439 | name. A mapping two writers replace within one round trip of each other can still lose |
| 440 | one side's new tag until that side maps it again. `tag_art` reads the mappings, passing |
| 441 | over entries that name no picture, and `tag_art_file` returns a picture once its bytes |
| 442 | match its name. |
| 443 | |
| 444 | `sidecar::themes` keeps style themes: each gives OneNote 2010's eleven gallery styles |
| 445 | (`STYLES`: `h1`…`h6`, `PageTitle`, `cite`, `blockquote`, `code`, `p`) a font, size, |
| 446 | weight, slant, colour and spacing, and `Theme::sheet` turns them into the paragraph style |
| 447 | definitions a page stores. `built_in()` lists the shipped themes, which never change once |
| 448 | shipped. A scope is the notebook, a section by file identity or a page by its |
| 449 | notebook-management identity; `Themes::effective(section, page)` takes the page's, else the |
| 450 | section's, else the notebook's. `Notebook::themes` reads the file and `save_themes(change)` |
| 451 | merges a change in as `map_tag_art` does: themes by id and assignments by scope, the later |
| 452 | timestamp winning, then the greater entry. `onestore::op::restyle(page, sheet)` gives the |
| 453 | `Restyle` ops that put a page in a sheet. |
| 454 | |
| 455 | ## SMB (feature `smb`) |
| 456 | |
| 457 | `notebook::smb` provides blocking SMB access for OneNote sections and |
| 458 | table-of-contents files. The core `onestore` crate remains independent of network runtimes. This is an |
| 459 | experimental Rust API with native interoperability evidence in the repository's |
| 460 | [Milestone 9](../../evidence/MILESTONE9.md). |
| 461 | |
| 462 | ```no_run |
| 463 | # #[cfg(feature = "smb")] { |
| 464 | use notebook::smb::{Client, Credentials}; |
| 465 | use std::time::Duration; |
| 466 | |
| 467 | let client = Client::connect( |
| 468 | "server:445", |
| 469 | "notes", |
| 470 | Credentials { username: "user", password: "password", domain: "" }, |
| 471 | Duration::from_secs(5), |
| 472 | )?; |
| 473 | let snapshot = client.read("Personal/Video.one", 64 * 1024 * 1024)?; |
| 474 | let store = onestore::Store::parse(&snapshot)?; |
| 475 | let revisions = onestore::RevisionIndex::parse(&store)?; |
| 476 | let document = onestore::document::Document::parse(&revisions)?; |
| 477 | # } |
| 478 | # Ok::<(), Box<dyn std::error::Error>>(()) |
| 479 | ``` |
| 480 | |
| 481 | Paths are relative to the share. The read limit bounds the complete physical |
| 482 | snapshot. Call from a background thread outside a Tokio runtime. |
| 483 | `Client::commit_transaction` publishes a `Transaction` from `Section::seal`; its errors |
| 484 | retain `onestore::CommitState`. `Client::stamp` reads a file's header and length in one |
| 485 | round trip without coordination, for polling. `Client::confirm` checks and flushes a |
| 486 | stamp, then refreshes its header version metadata without adding a revision. The caller |
| 487 | must first establish which intents that image contains and reread before another commit. |
| 488 | |
| 489 | A commit is eight round trips while its appended bytes and patches fit fifteen 64 KiB |
| 490 | writes: open; the coordination locks, identity queries and the stamp check's reads as one |
| 491 | compound request; the path's identity; the appended bytes and patches, the header, the |
| 492 | counter (twice at a carry) and the version, each as its writes and a flush in one |
| 493 | compound; close. The server |
| 494 | performs a compound in order and answers its flush once the writes are durable, so the |
| 495 | writes carry no write-through flag. |
| 496 | |
| 497 | Readers use shared native guards while writers publish under native write-open |
| 498 | and byte-lock exclusion. Maintenance is excluded during each operation; pathname |
| 499 | identity is checked after acquiring the guards. Connection loss retires the |
| 500 | client. Reconnect for subsequent operations, and reconcile an `Unknown` edit |
| 501 | before retrying it. The transport does not automatically replay requests. |
| 502 | |
| 503 | `Client::read_dir(path, entry_limit)` enumerates a directory, including the share |
| 504 | root with an empty path. It follows every response page and returns no partial |
| 505 | list on interruption, entry-limit overflow or close failure. Entries retain exact |
| 506 | Unicode names, observed sizes, last write times and MS-FSCC attributes, including |
| 507 | directory/reparse flags. Concurrent directory changes are not an atomic snapshot; repeated names |
| 508 | are rejected with `ResourceBusy`. Notebook identities come from the files, not |
| 509 | directory names or sizes. Missing paths, denied access and non-directory paths |
| 510 | have distinct I/O error kinds. |
| 511 | |
| 512 | `smb::shares(address, credentials, timeout)` lists the disk shares a server offers the |
| 513 | account, over `IPC$` and srvsvc. `smb::Refusal::of(&error)` names why `Client::connect`, |
| 514 | `shares` or `read_dir` failed as a person remedies it: unreachable, SMB1 only, sign-in |
| 515 | refused, no such share, denied, no such folder. A server that turns SMB2's negotiation |
| 516 | away is asked for SMB1's `NT LM 0.12` once, which an SMB1-only server agrees to. |
| 517 | |
| 518 | `Client::watch(path, changed)` arms one CHANGE_NOTIFY with WATCH_TREE on a directory, as |
| 519 | OneNote 2010 watches a notebook's folder, and keeps it armed on the client's runtime without |
| 520 | probing the connection, so an idle watch sends nothing. `changed` hears each batch of changed |
| 521 | paths relative to the directory, `""` when the server lost count, and an error when the watch |
| 522 | ends with its connection. |
| 523 | |
| 524 | An external payload on a share (`discover::Smb`) is read under a read-only share handle that excludes writes and deletion. Empty files succeed; limits, |
| 525 | interrupted reads and failed close never return partial bytes. This payload read |
| 526 | does not parse a OneStore header or acquire its reader-coordination bytes. |
| 527 | |
| 528 | `python3 tools/test_smb_directory.py VM OUTPUT` checks a caller-owned disposable |
| 529 | Linux lab VM against its filesystem listing and interrupts directory requests, |
| 530 | responses and close. It creates synthetic files in that VM; the caller retains |
| 531 | responsibility for VM teardown. The parser also has bounded-record/truncation |
| 532 | tests independent of the server. |
| 533 | |
| 534 | Device and simulator builds link for iOS. Native acceptance uses disposable |
| 535 | OneNote 2010 clients and Samba; it does not establish on-device execution or |
| 536 | physical power-loss durability. |
| 537 | |
| 538 | ## Live presence and Live Share (feature `live`) |
| 539 | |
| 540 | `live::Live::start(hello, room, reach, relay, events)` listens on a TCP port and, with a |
| 541 | `Reach`, advertises `_snowbound._tcp` by mDNS on every network or on loopback alone, |
| 542 | connecting to the peers in the same `Room` that it finds: a notebook's or a share's random |
| 543 | secret (`Room::Notebook`), or a code typed on both (`Room::join("7KQ-4MZ-9XR", password)`). |
| 544 | With a `relay` (`wss://live.example.net`, `crates/relay`) it also joins the room there and |
| 545 | meets its peers through it; the end sharing a code (`Room::share`) has the relay number its |
| 546 | secret, `code()` then has the whole code, and it burns a code after too many wrong tries. |
| 547 | Codes are Crockford base32 (`live::code`): two symbols numbering the room, six of secret |
| 548 | (30 bits) and a check symbol that refuses a typo before it spends one of the relay's tries. |
| 549 | `connect(address)` meets a peer discovery did not find. Peers meet through SPAKE2 on the |
| 550 | room's secret, then every frame is AES-256-GCM under the keys it agreed: its number, which is |
| 551 | also its nonce, then a message kind and a CBOR map (`live::wire`). A frame lost, repeated, |
| 552 | reordered or forged on the way fails where it lands; the connection is dropped as broken, |
| 553 | nothing from it after the fault is applied, and the ends meet again from scratch. A reader |
| 554 | skips kinds and map keys it doesn't know, so later versions add both freely. |
| 555 | `set_presence` says which section, page and caret this end has (text object and UTF-16 |
| 556 | offset, as ops address text); a connection sends only the newest. `peers()` lists each |
| 557 | connected peer's `Hello` (name, picture) and presence; `events` hears when they change, who |
| 558 | was met and left, and every other frame with the `Line` to answer on. `leave(reason)` says |
| 559 | `Bye` first; dropping the `Live` leaves at once. `Notebook::presence_room()` is the secret of |
| 560 | a notebook's presence room, kept in `.snowbound/live.json` and made where it has none. |
| 561 | |
| 562 | `live::share` is Live Share. `Host::start(storage, hello, sharing, name, reach, relay, events)` |
| 563 | serves `Notebook::into_storage()` to the peers in the share's room and welcomes whoever knows |
| 564 | `Sharing::code` (and its password) from the code's room; `code()` replaces a burned code with |
| 565 | a new secret, `guests()` lists who is connected, `touched(paths)` passes the host's own changes |
| 566 | on, and `stop()` lets every guest go. `join(hello, code, password, reach, relay)` returns the |
| 567 | `Welcome` (the share and its secret), or a `Refusal` saying why not: a wrong code, no one |
| 568 | sharing it, an expired code, too many wrong tries, or no relay. `Guest::start` joins the share; |
| 569 | `Notebook::open_hosted(guest, cache)`, `Background::hosted(guest, notify)` and |
| 570 | `Section::resume_hosted(path, replica, guest, notify)` then work as on a share, with |
| 571 | `Guest::host()` and `stopped()` for whether the host is there and still sharing. |
| 572 | |
| 573 | ## Queue measurement |
| 574 | |
| 575 | `cargo run -p notebook --release --example queue_scale -- NEW_DIRECTORY 1000` |
| 576 | measures alternating-page edits, cache reopen, recovery export and one local-file |
| 577 | publication of the queue. `keystroke_probe PARAGRAPHS KEYSTROKES` types into a large |
| 578 | page and reports the bytes the cache and the section file write per keystroke. |