1# The file format and `onestore`
2
3Everything Snowbound promises (opening the notebooks you already have, editing
4them beside OneNote, collaborating without a server) comes down to one thing:
5reading and writing OneNote 2010's files exactly as OneNote does. `onestore` is
6the crate that owns that. It knows nothing about SQLite, networks or pixels. It
7parses files, turns them into a model an editor can work with, and turns edits
8back into bytes OneNote will accept.
9
10## A revision store
11
12A notebook is a folder. Each section is a `.one` file. The folder's
13`Open Notebook.onetoc2` lists the sections and section groups in order, and a
14section group is a subfolder with its own `.onetoc2`. Both kinds of file share
15one container format, the *revision store* described in Microsoft's
16MS-ONESTORE specification. The content inside them follows MS-ONE.
17
18A revision store is closer to a small database than to a document. Logically
19it looks like this (physically, list fragments and revision data interleave as
20the file grows):
21
22```text
23┌──────────────────────┐
24│ header (1024 bytes) │ commit point: transaction count, file version GUID, list roots
25├──────────────────────┤
26│ file node lists │ chains of fragments; each fragment ends by pointing at the next
27│ ├ root list │ object spaces in the file
28│ ├ per object space │ its revisions, in order
29│ ├ transaction log │ how many list entries each committed transaction added
30│ └ file data store │ embedded pictures and attachments
31├──────────────────────┤
32│ revision 1 │ objects the revision declares, in object groups
33│ revision 2 ─dep─► 1 │ only what changed, plus the ancestors that point at it
34│ revision 3 ─dep─► 2 │
35│ … (appended) │
36└──────────────────────┘
37```
38
39- **Object spaces** partition the content. A section has a root space for
40 section-wide metadata and the page series, plus one space per page. A
41 conflict page, when there is one, gets a space of its own too.
42- **Revisions** are per space. A revision declares the objects it adds or
43 changes and depends on the revision before it. OneNote itself typically
44 appends a small dependent revision for each edit: the changed objects plus
45 the chain of containers above them, whose modification times moved.
46- **Contexts** label revisions besides the current one. A page's *versions* are earlier
47 revisions of its own space kept current under contexts of their own, and its version
48 history is one more revision, under a fixed context, listing them. Restoring a version
49 forks the page's chain from it; deleting one only unlists it.
50- **Objects** have a type (a JCID) and a property set. They reference each
51 other by *ExtendedGUID*: a GUID plus a small integer. Inside a revision these
52 are compressed to compact IDs through a global ID table.
53- **The header** is the commit point. Bytes appended past the old end of the
54 file mean nothing until the header's transaction count says a transaction
55 holding them has committed.
56
57The format lets payloads (pictures, attachments, recordings) live beside the
58section as `.onebin` files in a `_onefiles` folder, referred to by name. OneNote
592010 never writes them: every attachment it stores, 300 MiB ones included and
60on a share too, goes into the section's own file data store, with a 32-pixel
61PNG of the file's icon beside it. Snowbound writes the same. OneNote also
62stores identical bytes once per section, so two attachments of one file (or
63one icon) share a payload; Snowbound stores each anew, which OneNote reads the
64same.
65
66## What "append one revision" means
67
68Every edit Snowbound makes ends as a `Transaction`: bytes appended at the old
69end of the file, a few small patches inside it (the tail of each list gains a
70link to its new fragment, and the transaction log gains an entry), and a new
71header. A transaction is written for a particular base, named by its `Stamp`:
72the base's header and its length.
73
74```text
75exclusive lock ─► stamp still matches? no ─► NotCommitted, reread and rebase
76 │ yes
77 ▼
78 append new fragments, patch list tails ─► flush
79 header fields that describe the append ─► flush
80 transaction count (the commit) ─► flush
81 file version GUID (wakes cached readers)─► flush ─► unlock
82```
83
84The stamp works because every committed transaction rewrites the header,
85including the file version GUID. So equal stamps mean the same committed image,
86and a commit never has to compare the file's body. The flushes are what make a
87torn write recoverable. At any cut point, the file is either the old image with
88some ignored bytes past its end, or the new image. The version GUID goes last
89because OneNote's cached readers watch it, not the transaction count.
90Publishing it any earlier was once a real race with native readers.
91
92A commit ends in one of three states. Callers must honour them:
93
94| State | Meaning |
95| --- | --- |
96| `NotCommitted` | Nothing was published. Reread before retrying. |
97| `Unknown` | The reply was lost after the point of no return, so it may have landed. Reread and find out before doing anything else. |
98| `Committed` | Durable. Only cleanup failed. Never replay it. |
99
100The only operations that handle a whole image are creating a section or page,
101and opening a file. Everything else is an appended revision. The project holds
102this as a rule rather than an optimisation. A writer that regenerates a page
103from a model does page-sized work (parsing, diffing, rereading the file) for
104every keystroke. On a share, that work happens inside the writer lock every
105other client is waiting on.
106
107## Kept open: `Section`
108
109`Section` is a section file parsed once and kept in memory across edits. Ops
110apply to its spaces in memory. `seal` then turns everything that changed into
111one `Transaction` that appends one revision per changed space. A seal checks
112only what it appends: every fragment is linked from its list's old tail, every
113declared object parses and resolves, reference counts are the incremental
114counts, and the log entries have the CRC the state predicts. The full-file
115validator runs when a file opens and throughout the tests. The section borrows
116its bytes from an arena that lives beside it, so there is no self-reference and
117no `unsafe` (the crate forbids it).
118
119The writer caps revision dependency chains with a checkpoint revision, because
120native cold opens fail on very long chains. It also keeps a small reservation
121after a transaction-log fragment that ends the file, because OneNote does.
122
123## The page model
124
125`onestore::page` is the editable view of a page: title, outlines, paragraphs
126of text with formatting spans, lists, tags, tables, pictures, attachments, ink,
127and equations. Every node carries its stored identity. Content outside the
128model isn't dropped. It becomes `Unsupported`, which keeps its identity, type
129and layout, and its bytes stay untouched in the file. The canvas edits this
130model and the notebook crate stores it. Neither needs to know how it is encoded.
131
132## Ops
133
134`onestore::op` is how an edit is expressed: what the editor emits, what the
135queue stores and what `Section::apply` writes. An `Edit` is one user action (a
136list of ops and a timestamp), and it applies entirely or not at all. The ops are
137object-level, for example "replace this UTF-16 range of this text object",
138"split this paragraph here, naming the new paragraph", "move this subtree
139before that sibling" or "add these table rows". A refusal names the target,
140identity or structure at fault.
141
142Three properties make ops work as a sync currency:
143
144- **The emitter chooses new identities.** Replaying the same op on another copy
145 of the section creates the same objects, so an edit keeps its meaning across
146 retries, rebases and devices. The identity scheme copies OneNote's own
147 (`{page guid},n`, counting from 1). An earlier scheme that used 0 was accepted
148 by OneNote's integrity check, but OneNote then silently dropped elements in
149 roughly one build in five. Only a large native gate caught that.
150- **Ranges are UTF-16 code units.** That is how the file stores text, and it is
151 what UIKit's text input speaks. Ranges that would split a surrogate pair or a
152 hidden field are refused.
153- **Ops are plain data.** They serialize, they carry no UI types, and payload
154 bytes travel beside them by hash.
155
156`op::model` interprets ops on the page model without touching bytes. The
157lowering code (`op::lower`, which turns a model change into ops) uses it to
158predict what each op leaves, and the tests use it as an oracle.
159
160## Documented, observed, and reverse-engineered
161
162MS-ONESTORE and MS-ONE are good, but they are not the whole truth:
163
164- Some things OneNote writes aren't in the spec, such as the author initials it
165 stores beside every author name. Snowbound writes them too.
166- Some things the spec requires aren't needed by OneNote to open a file.
167 Snowbound writes them anyway, because other readers exist.
168- Ink and equations are absent from MS-ONE entirely, and so is the link from a
169 note to a moment in a recording. Ink and math were reverse-engineered, each
170 against an independent oracle. For ink, the stroke extents from OneNote's own
171 export, and for the Draw tab's pens, highlighters and shapes, the objects
172 OneNote stored while drawing them in the lab (`corpus/ink-tools`), and for pressure,
173 its PDF export of strokes it took in from ISF (`corpus/ink-pressure`). For math, the MathML it exports, matched byte for byte. Recordings
174 and their links follow what OneNote stored while recording in the lab, read
175 back through its XML export (`corpus/recording`); OneNote lists attached
176 .avi, .mpg and .wmv files as video recordings but not .mp4 or .mov, so video
177 is stored as Motion JPEG AVI, which it plays. Embedded objects are read
178 and kept, but never authored.
179
180Readers stay tolerant (files in the wild are older, odder, or written by other
181tools), while the writer stays strict.
182
183## Protected sections
184
185A password-protected section keeps its structure in the clear and encrypts every
186object and payload under one AES-128 key, which Office's Agile password encryption
187wraps (SHA-1, 100,000 rounds). `protected::Key` opens it with the password, and
188`Section::unlock` keeps the section open under it: objects decode into the arena as it
189opens, and each seal encrypts what it appends with a fresh IV and names the key in every
190revision. Edits are ops like any other's; only the bytes differ.
191
192Setting, changing or removing a password is one of the few whole-image writes, as it is
193in OneNote 2010, which writes the section anew under a new file identity, new object
194space identities and new payload identities, and leaves its TOC to follow. Snowbound does
195the same (`protected::rekey`): keeping any identity would let a cache, OneNote's or a
196replica, mistake the old file's revisions or payloads for the new file's. The evidence,
197with the crypto, is in `corpus/protected-sections`.
198
199## Looking inside
200
201The crate's examples are the fastest way to get a feel for a file.
202`inventory`, `inspect` and `document` dump the lists, revisions and document
203model. `tools/notebook_report.py` renders a readable report of a whole
204notebook, and the `onestore-diagnostic` binary in `notebook` is a small HTML
205editor for poking at one. The [crate README](../crates/onestore/README.md) is
206the reference for the public surface.