| 1 | # onestore |
| 2 | |
| 3 | An experimental native Rust library for OneNote revision stores (`.one` and |
| 4 | `.onetoc2`). It reads committed object graphs, creates a small notebook without |
| 5 | a template, appends property, text, paragraph, outline and formatting edits with a |
| 6 | recoverable commit protocol, and interprets MS-ONE document structure, formatting, media, and historical pages. |
| 7 | The storage gates are recorded in [PROGRESS.md](../../evidence/PROGRESS.md); document-model |
| 8 | verification is recorded in [M6-ACCEPTANCE.md](../../evidence/M6-ACCEPTANCE.md). Concurrent-editing |
| 9 | verification is recorded in [MILESTONE7.md](../../evidence/MILESTONE7.md). Crash recovery and the |
| 10 | read/write HTML diagnostic editor are recorded in [MILESTONE8.md](../../evidence/MILESTONE8.md). |
| 11 | Embedded SMB coordination, durable offline editing and document-growth acceptance |
| 12 | are recorded in [MILESTONE9.md](../../evidence/MILESTONE9.md#document-writer-and-offline-acceptance). |
| 13 | |
| 14 | Use disposable copies for notebook editing. Header version notification now |
| 15 | follows durable transaction publication, fixing a native cached-reader race. |
| 16 | The [lost-reply acceptance](../../evidence/MILESTONE9.md#lost-reply-acceptance-with-version-notification-published-last) |
| 17 | records the failure, reduced regression model, twelve-client repeat and cold |
| 18 | OneNote verification of all 3200 editing intents. |
| 19 | |
| 20 | ## Workspace |
| 21 | |
| 22 | `crates/onestore` contains the library, examples and integration tests. New Rust |
| 23 | prototypes belong in sibling directories under `crates/` and depend on |
| 24 | `onestore = { path = "../onestore" }`. The root manifest discovers these crates. |
| 25 | The consumer boundary and API tradeoffs are recorded in [API-AUDIT.md](../../evidence/API-AUDIT.md). |
| 26 | `onestore-diagnostic` in the `notebook` crate backs the [HTML diagnostic editor](../../evidence/DIAGNOSTIC.md). |
| 27 | [`notebook`](../notebook/README.md) provides notebook discovery, optional embedded |
| 28 | network access (feature `smb`) and local SQLite persistence and reconnect reconciliation for text, insertion and formatting. |
| 29 | Shared native fixtures, specifications, evidence and Python/VM tools stay at the |
| 30 | repository root; `fuzz/` remains an independent cargo-fuzz workspace. |
| 31 | |
| 32 | Run Cargo commands from the root. Select `-p onestore` when working only on the |
| 33 | library, or `--workspace` for checks across all crates. Example binary paths |
| 34 | remain `target/debug/examples/…` for the native verification tools. The collaboration |
| 35 | harness also accepts `--client-profile release`. |
| 36 | |
| 37 | ## Supported surface |
| 38 | |
| 39 | | API | Contract | |
| 40 | | --- | --- | |
| 41 | | `Store`, `RevisionIndex`, `ResolvedRevision` | Parse storage, resolve revisions and reference graphs, expose roots and objects | |
| 42 | | `PropertySets`, `Object::references` | Decode properties and ID streams while retaining raw values | |
| 43 | | `Object::file_reference`, `Store::file_data` | Identify internal/external payloads and read internal payload bytes | |
| 44 | | `document::Document`, `Revision::text_runs` | Interpret document objects and inherited text formatting while retaining unknown properties and revision identities | |
| 45 | | `Document::active`, `Document::pages_in`, `Revision::parents`, `RevisionIndex::active` | Resolve the active revision, the pages of a space and parent links without repeating the lookups | |
| 46 | | `Page::copy` | The page's content under fresh identities for copying into another section, definitions and payloads included; indent levels only an outline group carries refuse | |
| 47 | | `page::link::internal_link`, `page::link::parse_internal_link` | Build and read the `onenote:#…` URLs OneNote stores for links to sections, pages and paragraphs, by identity | |
| 48 | | `page::Recording`, `page::MediaIndex`, `PageOp::Media` | An audio or video file's recording identity, kind and length, and a paragraph's link to a moment in recordings, as OneNote stores them; the page lists its recordings as they come and go | |
| 49 | | `page::Page`, `page::Paragraph`, `page::Ink`, `page::Math` | Build an editable page model (title, outlines, paragraphs with coalesced text spans, tables, images, attachments in paragraphs or on the page, ink drawings and handwriting decoded to stroke polylines in page points with each point's pen pressure, a moved drawing offset by its position, a highlighter's raster operation, a drawn shape's kind and anchors) with stored identities; equations parse from their linear text and run data into a tree that renders the MathML OneNote exports; content outside the model is retained as `Unsupported` | |
| 50 | | `protected::Key`, `Section::unlock` | Open a password-protected section's key with its password (`Key::open`), or make one for a new password with OneNote 2010's encryption data (`Key::new`); keep the section open under it, decoding every object and payload, and seal edits under it as OneNote does (a fresh IV per object) | |
| 51 | | `protected::rekey` | Set, change or remove a section's password as OneNote 2010 does: the section written anew under fresh file, space and payload identities, each space keeping its labelled revisions as checkpoints | |
| 52 | | `protected::UnlockedSection` | A protected section's labelled revisions decoded into a `Document` for inspection and export, by password or `Key`; cleared on drop, while strings and exports made from it are the caller's | |
| 53 | | `create_section` | Create one page containing one plain-text paragraph and an author, including Unicode | |
| 54 | | `PageCreation`, `SectionOp::Create` | Add an empty top-level page and its section entry atomically, under identities the intent retains across retries | |
| 55 | | `PageEdit`, `SectionOp::Pages` | Publish explicitly selected page moves and indentation changes together, preserving page content and historical revisions | |
| 56 | | `SectionOp::Delete` | Remove explicit pages and their section references atomically while retaining stored revisions | |
| 57 | | `create_table_of_contents` | Create ordered section entries from filenames and file identities | |
| 58 | | `TocEdit`, `edit_table_of_contents` | Add, rename, order and remove a table of contents' section and group entries, and colour the notebook, as one revision's `Transaction`; a section's colour is `SectionOp::Color` in its own metadata | |
| 59 | | `place`, `place_file` | Name a file for its notebook as OneNote does on adoption (parent TOC identity and name CRC in the header), so OneNote keeps its identity, on any `CommitIo` or under the filesystem adapter | |
| 60 | | `place_image`, `reidentify` | The same on an image held whole, or outside any notebook; and a new file identity and version, as OneNote's Save As copies and Unpack Notebook give (`corpus/notebook-package`) | |
| 61 | | `TextAttribute`, `PageOp::Format` | Change character formatting over a UTF-16 range while sharing immutable styles; preserve unselected runs | |
| 62 | | `PageOp::Style`, `PageOp::Restyle`, `op::restyle` | Give a paragraph a paragraph style (with its NextStyle, as OneNote 2010 writes its headings), or move every paragraph of a style to a new style object of the same name; style objects are read-only, and run or paragraph values equal to what the old style gave are cleared so they follow the new one. `op::restyle(page, sheet)` brings a page's named styles to a sheet of definitions, joining styles that share a name | |
| 63 | | `OutlineEdit`, `PageOp::Outline` | Change ordinary outline position/width, the position of a file or ink drawing on the page, or a paragraph's saved expansion default, preserving identities and content | |
| 64 | | `Transaction`, `Stamp` | The bytes a commit writes (appended data, in-place list-tail and log patches, header) and the header and length it requires unchanged; `commit` under caller-held exclusion, `commit_file` under the conservative filesystem adapter, `apply` to the base image in memory; serializable for queues | |
| 65 | | `Arena`, `Section` | Keep a section parsed across edits: `open` validates an image once; `seal` appends one revision per changed space as a `Transaction` on `stamp`, checking only what it appends; `replay` applies a queued transaction; `image` and `page` read the result | |
| 66 | | `op::{Edit, Op, PageOp, TableEdit, SectionOp}`, `Section::apply` | Object-level edits (text, formatting, links, equations, paragraph insertion/split/join/move/deletion/levels, outlines, lists, tags, styles, paragraph formatting, tables, pictures, attachments, ink, page creation/import/moves/removal, conflict pages) applied whole or not to a kept-open section with emitter-chosen identities, UTF-16 ranges and the edit's time; refusals name the target, identity or structure at fault | |
| 67 | | `SectionOp::Conflict`, `Section::conflicts`, `ConflictPage` | Keep the version of a page a merge could not take as OneNote 2010 does: a read-only conflict page (`jcidConflictPageMetaData`, `IsConflictPage`, conflict objects marked) under the page's manifest, which says it has conflict pages; list each page's, newest first, and delete one (`SectionOp::Delete`) as OneNote's Delete Conflict Page does | |
| 68 | | `Section::versions`, `Section::version`, `PageVersion`, `SectionOp::RestoreVersion`, `SectionOp::DeleteVersions` | A page's versions as OneNote 2010 keeps them: earlier revisions of the page's own space, each current under its own context and listed by the page's version history, newest first with its author and time; read one as a page (O(section)); restore one as Restore Version does (the page's next revision builds on it and the page as it stood becomes the newest version) or unlist some as Delete Version does, their revisions staying stored ([corpus](../../corpus/page-versions/README.md)) | |
| 69 | | `Section::page_at`, `Section::open_at`, `Section::seal_as` | Read a page as any revision the file stores holds it, or the section as chosen revisions of its spaces leave it (a merge's common ancestor); seal revisions under chosen identities so a later merge recognises them | |
| 70 | | `op::lower`, `op::lower_page` | The ops turning a range of paragraphs, or a whole page model, into another, computed from the models alone, for editors and imports | |
| 71 | | `op::predict` | The page an op leaves, as the section stores it and reading it back shows it | |
| 72 | | `read_file` | Read a snapshot under whole-file exclusion | |
| 73 | | `read_snapshot` | Read a validated snapshot through fresh positioned I/O while the caller excludes maintenance | |
| 74 | | `CommitIo`, `confirm`, `confirm_file`, `Stamp::check` | Supply another storage backend with equivalent exclusion and ordered durability; check that a stamp still holds | |
| 75 | | `supersede_file` | Put a file written anew (`protected::rekey`) in place of a section under the exclusion `commit_file` takes, while its stamp holds; on Windows the old file goes aside first, as OneNote's maintenance does | |
| 76 | |
| 77 | Edits enter a kept-open `Section` as ops and leave as one appended revision per |
| 78 | changed space; only section and page creation, imports and opening files handle |
| 79 | whole images. Text edits maintain run boundaries, inherit the insertion run's formatting, |
| 80 | and promote legacy text to Unicode when needed. Explicit and body-derived |
| 81 | navigation titles update in the same transaction; unsupported fields, |
| 82 | protected objects and split surrogate pairs are |
| 83 | rejected before writing. Appended snapshots cap revision dependency depth at 512 |
| 84 | while retaining historical revisions. TOC snapshots can remap encoded CompactIDs |
| 85 | without changing their resolved references. A password-protected section opens with |
| 86 | `Section::unlock` under the `Key` its password opens, and every revision it seals names |
| 87 | the key (MS-ONESTORE 2.5.19 asks that of each revision; OneNote names it only in those |
| 88 | without a dependency, and reads both). `Section::open` refuses one. Incorrect |
| 89 | passwords, unsupported protection profiles and work-limit failures remain distinct |
| 90 | (`protected::Error`). |
| 91 | `PageCreation::new` appends, or inserts before the first page space of an existing |
| 92 | series. `Some("")` creates an empty title field; `None` omits the title node. |
| 93 | `dated(date, time)` gives the title OneNote 2010's date and time fields showing that |
| 94 | text, as OneNote titles a new page; `in_space(guid)` creates the page in the space `{guid},1` with a series named after it, as a merge that must make the same page every time does; `keeping(identity, created)` keeps another page's |
| 95 | identity and creation time, as a page moved to the recycle bin keeps them. The page has |
| 96 | no body outlines or applied template; `create_empty_section` makes a section for such |
| 97 | pages, and `PageOp::Color` sets or clears a page's colour (`0x14001d2a` on the page node, |
| 98 | absent for "No color"), and `PageOp::RuleLines` its rule lines (six properties on the page |
| 99 | node, absent for None; [rule-lines](../../corpus/rule-lines/README.md)). |
| 100 | Retain the intent to preserve its page, title and space identities; existing |
| 101 | identities require reconciliation before retry. Body insertion and title edits |
| 102 | use those identities through page ops. [Native page-creation fixtures](../../corpus/page-lifecycle/creation/README.md) |
| 103 | cover duplicate Unicode titles, native edits and Rust follow-up edits. |
| 104 | `PageEdit::set_level` changes one page's indentation in place. `PageEdit::move_to` |
| 105 | moves before an existing page space, or appends for `None`, and sets its level. |
| 106 | `SectionOp::Pages` applies moves in slice order and publishes the final order, |
| 107 | series membership and metadata levels in one transaction. Each page occurs once; |
| 108 | levels are 1–3 and the final first page must have level 1. A following deeper-level |
| 109 | page remains in place unless explicitly selected. To move a group, supply all its |
| 110 | pages in order. Retain the intents across retries so newly formed series keep |
| 111 | their identities; `reposition(PagePosition, level)` revises their placement while |
| 112 | preserving those identities. [Native page-edit fixtures](../../corpus/page-lifecycle/page-edits/README.md) |
| 113 | cover individual tabs, selected and collapsed groups, nesting and promotion. |
| 114 | `SectionOp::Delete` removes exactly the supplied page spaces, |
| 115 | including subpages only when selected explicitly. The first remaining page becomes |
| 116 | top-level; other page levels and surviving content are retained. The operation |
| 117 | creates no recycle-bin copies and preserves prior revisions, so it is not secure |
| 118 | erasure. Duplicate, missing or non-page identities reject the entire batch; |
| 119 | an empty selection leaves the file unchanged. |
| 120 | `PageOp::Insert` and `PageOp::Add` update child references, reference counts, |
| 121 | modification times and automatic titles atomically. Paragraphs can be nested or |
| 122 | inserted into table cells; outline coordinates use points. Inserted text keeps its |
| 123 | spans' formats; `PageOp::Format` over an empty text's `0..0` sets its insertion style. |
| 124 | New objects carry the emitter's identities, so a queued edit replays onto another |
| 125 | image unchanged; an identity already on the page refuses the edit. Formatting accepts explicit attributes, preserves inherited values, |
| 126 | and gives retired immutable styles zero current references while retaining history. |
| 127 | Outline layout edits use points. Width is at least 36 points; an explicit user width |
| 128 | and an automatic maximum-width hint remain distinct. Native layout generates the |
| 129 | rendered height. Saved paragraph collapse defaults can be overridden by the native |
| 130 | client's cached view. [Native layout captures](../../corpus/outline-edit/README.md) |
| 131 | verify these edits through a fresh OneNote cache, including fields and nested content. |
| 132 | `PageOp::Move` takes an existing parent and an optional direct sibling to insert |
| 133 | before; `None` appends. Paragraphs retain their descendants and explicit list styles. |
| 134 | Outlines remain page children, so reordering changes their stacking order while |
| 135 | retaining coordinates. `PageOp::Delete` removes the selected subtree from the active |
| 136 | graph and preserves historical objects. Empty outlines/groups are removed; surviving |
| 137 | group indentation is normalized without shifting other paragraphs. An edit that |
| 138 | leaves a table cell without a paragraph is refused; insert its replacement in the |
| 139 | same edit. |
| 140 | Title/protected content, ambiguous ancestry, cycles, and incompatible destinations |
| 141 | reject before publication. Modification times, move attribution and automatic titles |
| 142 | publish with the tree change. These explicit destinations differ from keyboard list |
| 143 | indentation, which can also substitute list markers. |
| 144 | Paragraph splits preserve character formatting, retain tags on the left, and clone |
| 145 | mutable list objects without restarting numbering. `PageOp::Split` names |
| 146 | the new right paragraph/text identities. Its publication includes |
| 147 | the complete child graph and title metadata; repeating an existing identity requires |
| 148 | reconciliation. Title containers, generated fields, recording-linked text and |
| 149 | associated run metadata are rejected before I/O. Native split controls and subsequent |
| 150 | typing checks reside in [the paragraph corpus](../../corpus/paragraph-edit/README.md). |
| 151 | Joins retain the left paragraph. Nonempty left text keeps its identity; empty left |
| 152 | text adopts the right text identity. **The left tags win: right-side tags are removed |
| 153 | from active text even when the left text is empty.** History retains the original |
| 154 | objects. Select the preceding leaf text; where that leaf is deeper than the right |
| 155 | paragraph, right children move to its ancestor at the right paragraph's level. |
| 156 | Ambiguous ancestry, unsupported indentation transitions and unknown implicit |
| 157 | font/language inheritance reject before I/O. This is a logical join, so keyboard |
| 158 | actions that only change list or indentation state remain separate operations. |
| 159 | Generated fields, protected targets and unsupported run-data boundary changes are |
| 160 | rejected before publication. The [notebook crate](../notebook/README.md) |
| 161 | documents durable local operations and reconciliation. The |
| 162 | [document-writer acceptance](../../evidence/MILESTONE9.md#document-writer-and-offline-acceptance) |
| 163 | includes twelve mixed native/Rust clients, outages, lost replies and native revision retirement. |
| 164 | |
| 165 | External `.onebin` references identify payloads for the caller to obtain. Cloud |
| 166 | FSSHTTP synchronization and a C ABI are outside the implemented surface. |
| 167 | |
| 168 | ## Try it |
| 169 | |
| 170 | Requires Rust 1.97 or later for the verified build. Examples create new destinations |
| 171 | and refuse to overwrite them. The Python tools require Python 3.10 or later and Pillow. |
| 172 | |
| 173 | ```sh |
| 174 | cargo run --example create_notebook -- /tmp/one-demo 'Hello from Rust.' 'Example Author' |
| 175 | cargo run --example inventory -- /tmp/one-demo/synthetic.one |
| 176 | cargo run --example inspect -- /tmp/one-demo/synthetic.one |
| 177 | cargo run --example document -- /tmp/one-demo/synthetic.one /tmp/one-model |
| 178 | cargo build -p notebook --bin onestore-diagnostic |
| 179 | python3 tools/notebook_report.py /tmp/one-demo /tmp/one-report --timezone America/Los_Angeles |
| 180 | ``` |
| 181 | |
| 182 | A seeded text edit on a disposable copy records its page, UTF-16 range, |
| 183 | replacement and outcome as JSON: |
| 184 | |
| 185 | ```sh |
| 186 | cargo run --example random_edit -- /tmp/one-demo/synthetic.one /tmp/edited.one 42 |
| 187 | ``` |
| 188 | |
| 189 | The report contains readable pages, document JSON, assets, source identities and |
| 190 | coordinates. It preserves paragraph nesting, lists, tables, links and tags. |
| 191 | Historical contexts, recycle-bin pages and default templates are represented |
| 192 | separately. Native ink is decoded to strokes and equations to MathML; both |
| 193 | retain their source data. The report is a reading view; its native |
| 194 | PDF references supply the original canvas layout. |
| 195 | |
| 196 | Read and validate a snapshot before interpreting its graph: |
| 197 | |
| 198 | ```rust,no_run |
| 199 | use onestore::{read_file, RevisionIndex, Store}; |
| 200 | |
| 201 | fn main() -> Result<(), Box<dyn std::error::Error>> { |
| 202 | let bytes = read_file("notebook/synthetic.one")?; |
| 203 | let store = Store::parse(&bytes)?; |
| 204 | if !store.checksum_mismatches.is_empty() { |
| 205 | return Err("Transaction checksum damage".into()); |
| 206 | } |
| 207 | let index = RevisionIndex::parse(&store)?; |
| 208 | index.validate_current()?; |
| 209 | Ok(()) |
| 210 | } |
| 211 | ``` |
| 212 | |
| 213 | `Store::parse` exposes checksum mismatches for diagnostic readers; the writer |
| 214 | rejects them. The resolved graph borrows the snapshot. Open it as a `Section`, |
| 215 | apply ops naming objects from that graph, seal, and commit the `Transaction`. The |
| 216 | `edit_property` example demonstrates selection by JCID/property/expected bytes, |
| 217 | with optional explicit IDs when conflict copies contain identical text. |
| 218 | |
| 219 | ## Commit behavior |
| 220 | |
| 221 | ```text |
| 222 | exclusive lock → header and length check |
| 223 | → append data → flush |
| 224 | → prepare header metadata → flush |
| 225 | → publish transaction counter → flush |
| 226 | → finish counter rollover → flush |
| 227 | → notify cached readers → flush → unlock |
| 228 | ``` |
| 229 | |
| 230 | Stale snapshots fail before writing: every committed transaction and placement rewrites |
| 231 | the header (MS-ONESTORE 2.3.1), so the body is never compared. Live readers must use |
| 232 | equivalent exclusion. |
| 233 | Native conflict creation can still expose cross-space references before their |
| 234 | targets are saved; such snapshots must be rejected and reread while synchronization |
| 235 | proceeds. `read_file` and `Transaction::commit_file` serialize within the process because |
| 236 | macOS SMB locks can be reentrant. The lock is nonblocking across processes; |
| 237 | contention requires a fresh read and a later retry. |
| 238 | |
| 239 | | Error state | Meaning and caller action | |
| 240 | | --- | --- | |
| 241 | | `NotCommitted` | This edit was not published. Preparation bytes may exist. Reread before retrying. | |
| 242 | | `Unknown` | Publication may have persisted despite the error. Reread and resolve the intended edit before retrying. | |
| 243 | | `Committed` | Publication was durably acknowledged; counter cleanup or lock release failed. Reopen the committed result instead of replaying the edit. | |
| 244 | |
| 245 | At counter rollover, empty transactions make intermediate published counts valid. |
| 246 | The highest changed byte is flushed before lower bytes are cleaned up. The |
| 247 | 255→256 and 65535→65536 boundaries and interrupted cleanup states have independent |
| 248 | native acceptance captures. A no-op still flushes; a failed flush has an unknown |
| 249 | durability outcome. |
| 250 | |
| 251 | The filesystem adapter uses whole-file locking. On macOS it acquires the lock |
| 252 | as part of opening the file (`O_EXLOCK` to commit, `O_SHLOCK` to read, with |
| 253 | `O_NONBLOCK`): separate open and `flock` calls allowed overlapping exclusive holders |
| 254 | and stranded server locks under multi-process SMB contention. |
| 255 | `tools/smb_lock_race.py` reproduces that failure without notebook parsing or writing, |
| 256 | and with `--shared-reads` checks that shared readers never overlap a writer. On the |
| 257 | tested macOS SMB mount, POSIX byte-range locks returned `ENOTSUP`, and `flock` of |
| 258 | either kind became an exclusive lock over the whole file, which fails OneNote's reads. |
| 259 | smbfs sends `O_SHLOCK` and `O_EXLOCK` as share modes: a read excludes writers, |
| 260 | OneNote's included, but not OneNote's readers; a commit excludes everyone. It does not |
| 261 | reproduce native reader/writer concurrency. The |
| 262 | [locking audit](../../evidence/LOCKING.md) records native coordination bytes, write-open share |
| 263 | modes, and a working macOS SMB-specific byte-range lock probe. `sync_all` falls |
| 264 | back to `fsync` on macOS only when `F_FULLFSYNC` is unsupported. Successful SMB FLUSH replies were observed on the |
| 265 | wire. Correctness requires the backend to honor exclusion and ordered flushes. |
| 266 | The evidence covers transport failures, not physical server power loss or every |
| 267 | filesystem's lock implementation. |
| 268 | |
| 269 | For shared network notebooks, use the optional |
| 270 | [`notebook::smb`](../notebook/README.md) module. It uses native share modes, |
| 271 | shared reader guards, writer exclusion and fresh pathname identity checks without |
| 272 | an OS-mounted share. Its [coordination acceptance](../../evidence/MILESTONE9.md) covers native |
| 273 | maintenance, mixed readers/writers, reconnects and uncertain publication. The |
| 274 | filesystem adapter retains its conservative locking; mounted-path freshness |
| 275 | across native replacement is not established by that exclusion. |
| 276 | |
| 277 | ## Verification |
| 278 | |
| 279 | ```sh |
| 280 | cargo test --all-targets |
| 281 | cargo clippy --all-targets -- -D warnings |
| 282 | python3 tools/verify-corpus.py |
| 283 | python3 tools/verify-reader.py # requires Pillow and the local private corpus |
| 284 | python3 tools/verify-writer.py |
| 285 | python3 tools/verify-collaboration.py |
| 286 | python3 tools/verify-document.py /path/to/copied/notebook /path/to/native/read |
| 287 | ``` |
| 288 | |
| 289 | The frozen private corpus is excluded from version control. Its verifier checks 26 pages, |
| 290 | 581 text objects, hyperlink targets, exact image bytes, and native image conversions. |
| 291 | Synthetic corpus manifests bind binary fixtures to independent native XML and |
| 292 | attachment captures. `verify-corpus.py` also requires that private corpus. |
| 293 | |
| 294 | Storage tests exercise malformed references, deep properties, historical revision |
| 295 | preservation, short I/O, stale snapshots, counter tears, and every injected I/O |
| 296 | failure point at ordinary and rollover commits. The crash model persists arbitrary |
| 297 | subsets of unflushed bytes and is shared with the stateful commit fuzzer. |
| 298 | |
| 299 | ```sh |
| 300 | cargo +nightly fuzz run revisions -- -max_total_time=120 -max_len=262144 -rss_limit_mb=2048 |
| 301 | cargo +nightly fuzz run commit -- -max_total_time=300 -max_len=4096 -rss_limit_mb=2048 |
| 302 | cargo +nightly fuzz run paragraph -- -max_total_time=120 -max_len=160 -rss_limit_mb=2048 |
| 303 | ``` |
| 304 | |
| 305 | Fuzz targets cover storage, properties, revisions, scalar edits, creation, and |
| 306 | multi-edit interrupted commits. Seed the revision target with native `.one` files |
| 307 | using symlinks under `fuzz/corpus/revisions`; empty seed directories mostly exercise |
| 308 | header rejection. Bounded run counts and native findings live in `PROGRESS.md`. |
| 309 | |
| 310 | The document fuzzer mutates native property streams, repairs their checksums, and |
| 311 | traverses every retained revision and resolved text run. Its public seeds live in |
| 312 | the source target; private seeds are supplied only at runtime. Native edit-history |
| 313 | tests compare independently generated operations, OneNote XML and the Rust model; |
| 314 | failed histories can be replayed and shrunk in fresh disposable clones. The |
| 315 | document feature matrix is in [FEATURES.md](../../evidence/FEATURES.md), and the milestone's |
| 316 | acceptance contract is in [MILESTONE6.md](../../evidence/MILESTONE6.md). |
| 317 | |
| 318 | The stage-5 gate uses OneNote 2010 build 14.0.7015.1000 on Windows 7, a macOS SMB |
| 319 | mount, and Samba on zenith. [The collaboration corpus](../../corpus/collaboration/round-01) |
| 320 | captures native lock contention, different-paragraph merging, same-paragraph |
| 321 | conflicts, offline editing/reconnection, and lost successful FLUSH replies at |
| 322 | preparation, publication, and counter cleanup. Each final notebook was reopened |
| 323 | from a fresh native cache. Competing text survives as native conflict pages; |
| 324 | OneNote's COM hierarchy omits those pages, so the offline case also includes a |
| 325 | native UI capture. Full-page COM updates produced an extra conflict copy during |
| 326 | the disjoint case and two recorded geometry changes; the verifier checks those |
| 327 | exact changes as well as retained image, ink, table, and attachment content. |
| 328 | |
| 329 | `tools/native/profile.ps1` parks/restores the personal native profile and cache; |
| 330 | `cold.ps1`, `read.ps1`, and `collaborate.ps1` operate on disposable test roots. |
| 331 | `tools/zenith-locks` observes server locks. `tools/smb-proxy.py` traces and interrupts |
| 332 | a dedicated loopback test session; its control JSON selects the successful response |
| 333 | and occurrence to withhold. The captured trace and result files are the regression |
| 334 | oracle; replaying the native experiments requires the supplied Windows/share setup. |
| 335 | |
| 336 | ## Password-protected sections |
| 337 | |
| 338 | OneNote 2010 wraps an AES-128 key in Office's Agile password encryption (MS-OFFCRYPTO: |
| 339 | SHA-1 of a 16-byte salt and the UTF-16LE password, then 100,000 rounds of SHA-1 over the |
| 340 | round number and the hash; three block keys decrypt, with the salt as IV, the verifier |
| 341 | input, its SHA-1 zero-padded to 32 bytes, and the key). The XML sits after the words |
| 342 | `3, length, 16, length - 16` and the version `4.4` with flags `0x40`, inside the |
| 343 | encryption-data container every revision names. Each property object is stored as its |
| 344 | reference streams, a length, a random IV and the CBC encryption of a padding count, the |
| 345 | property bytes and random padding; read-only objects hash the clear bytes zero-padded to |
| 346 | 8 bytes. Payloads are their length and bytes, randomly padded, under CBC with the IV |
| 347 | SHA-1(key data salt, block 0). Setting, changing or removing a password writes the section |
| 348 | anew, as OneNote does, under fresh identities so that no cache confuses the old file's |
| 349 | objects or payloads with the new ones (`corpus/protected-sections`). |
| 350 | |
| 351 | ```rust,no_run |
| 352 | use onestore::{Arena, Section, protected::{Key, rekey}}; |
| 353 | # fn example(image: Vec<u8>, password: &str) -> Result<(), Box<dyn std::error::Error>> { |
| 354 | let key = Key::open(&image, password)?; |
| 355 | let arena = Arena::default(); |
| 356 | let mut section = Section::unlock(&arena, image.clone(), &key)?; |
| 357 | assert!(!section.pages()?.is_empty()); |
| 358 | let changed = rekey(&image, Some(&key), Some(&Key::new("another password")?))?; |
| 359 | # drop(changed); |
| 360 | # Ok(()) } |
| 361 | ``` |
| 362 | |
| 363 | A `Key` holds no password; its key is cleared when its last clone drops. A `Section` |
| 364 | decodes into its `Arena`, which is not cleared; drop both when the section locks. |
| 365 | `UnlockedSection` owns its decoded buffers and clears them on drop; the views it lends |
| 366 | cannot outlive it, while copies of parsed strings, serialized models and exports are |
| 367 | plaintext with lifetimes of their own. CBC has no general |
| 368 | ciphertext-authentication guarantee; native read-only hashes and model validation |
| 369 | check the corresponding structure. Internal payloads are decoded; external payload |
| 370 | references remain references, and their protected decoding is not implemented. |
| 371 | |
| 372 | For a deliberate plaintext diagnostic export, run the notebook exporter |
| 373 | (`examples/document`) with `--password-file PATH` after the source and optional |
| 374 | new output directory. The file contains exact UTF-8 password bytes; no newline is |
| 375 | removed or Unicode normalization applied. The exporter creates protected exports |
| 376 | under an owner-only directory on Unix and reports protected external payloads as |
| 377 | unsupported. It never rewrites the encrypted source. |