clover's git

snowbound

1	# The data layer and sync
2
3	OneNote 2010 got multi-user editing without a server. A notebook is a folder
4	on a file share. Every client edits the section files in place, and the
5	format's append-only revisions plus some careful file locking let clients
6	merge each other's work. Snowbound joins that arrangement as one more peer. It
7	has no service, no account and no sync protocol of its own. The share is the
8	source of truth, and whatever OneNote can do to a file while Snowbound is
9	using it, Snowbound has to handle.
10
11	This essay follows an edit from the keyboard to the share, then covers what
12	happens when the share is busy, changed or gone.
13
14	## Three threads
15
16	```text
17	UI thread (winit, UIKit) section thread sync thread
18	──────────────────────── ────────────────────────────── ──────────────────────────────
19	editor emits Edit (ops) ───► apply to the parsed Section read the stamp (header + length)
20	and returns at once write each burst to SQLite unchanged: ask for a seal,
21	page reads ◄──────────────── answered from the Section publish it
22	events ◄──────────────────── Changed, Rejected, published changed: read once, ask
23	seal ◄────────────────────────── for a rebase
24	rebase onto the new image ◄───── acknowledge: base += transaction
25	```
26
27	- **The UI thread only emits ops.** The editor records the ops each change
28	lowers to as it makes the change. Submitting them to the session returns
29	immediately. Parsing, revision building and SQLite never run on a frame.
30	- **The section thread** (`notebook::working`) is the only owner of the parsed
31	`onestore::Section`. It applies edits, writes each burst of them to the
32	replica in one durable commit, and answers page reads. A rebase runs on a
33	fresh thread that builds its own section while the old one keeps answering
34	reads, so opening a page never waits on the network.
35	- **The sync thread** (`notebook::worker`, stepping `notebook::sync`) does all
36	network I/O. It reads the stamp when a local edit waits or the notebook's watch
37	reports the file (every two seconds where nothing watches), then publishes or
38	rereads. Idle, it reads nothing.
39
40	`notebook::session` wraps this up for apps. `Notebook` covers discovery and
41	structure, `Section` covers one section's pages, edits, events and conflict
42	pages. Both desktop and iOS use exactly this surface.
43
44	## The replica
45
46	Each open section has a replica: a SQLite database in the app's cache, named by
47	where the notebook is and the section's logical identity, so the same file
48	reopens the same queue after a relaunch. It holds:
49
50	- **the base**: the last remote image the queue applies to, stored in chunks so
51	that acknowledging a publication rewrites only the chunks the transaction
52	touched;
53	- **batches of edits**: each edit's ops, serialized, grouped into the batch
54	that will publish together;
55	- **payloads**: picture and attachment bytes, stored once each, keyed by hash.
56
57	The working state is never stored. It is the base with each sealed batch
58	replayed and the open batch applied, rebuilt on open.
59
60	A password-protected section's replica holds nothing in the clear. Its base and sealed
61	transactions are the file's own ciphertext, and its edits and payloads are sealed with
62	AES-256-GCM under a key derived from the section's, so the queue opens, offline or after a
63	relaunch, only once the section is unlocked. A locked section is never opened in the
64	background; its edits wait until it is unlocked again. A replica of a section another device
65	then protects is deleted the next time the notebook is read, unless edits wait in it; those
66	wait for the key, then come back as copies of the pages they changed. The database runs in WAL
67	mode with full synchronous commits, and every setting is read back to check it
68	took.
69
70	The location matters because OneNote lets a notebook folder be copied whole, a
71	Finder duplicate or an iCloud `Name 2` conflict copy, and every section in the
72	copy keeps its identity. Keyed by identity alone, the copy would open the
73	original's queue and publish its edits into the wrong file. So replicas live in
74	`cache/replicas/<hash of location>/<identity>.sqlite` (`notebook::location`),
75	where a location is a canonical local path or `smb://server/share/root`, and a
76	`location` file in each folder names it. A section renamed or moved inside its
77	notebook keeps its replica; a copy of a section inside one notebook (`Cross
78	2.one` beside `Cross.one`, which OneNote opens as a section of its own) is keyed
79	by its own path instead, the original being the one the folder's TOC lists.
80
81	When a notebook moves, its queue follows by one of three roads:
82
83	- **the app moved it** (iOS Rename, or Notebook Properties' "Rename the folder
84	too"): `location::moved` moves the folder's replicas to the new location, the
85	latter through `Notebook::rename_folder` once no other writer holds a file;
86	- **something else moved it** (Finder): on open, a section with no replica takes
87	one from the folder of a local location that no longer exists, if the file
88	stands as that replica's base or has moved on from it (a higher header
89	generation). A replica whose base is newer than the file, or a different state
90	of the same generation, belongs to another copy and stays;
91	- **an older install** named replicas by identity alone in the cache's top or
92	its `smb` folder: the first location to open the section takes it.
93
94	An attached file is a payload like a picture: its bytes ride in the edit that
95	inserts it and publish inside that edit's revision. No other file is written,
96	so nothing needs to reach the share before the revision that names it. That
97	ordering would matter only for `_onefiles` payloads, which OneNote 2010 never
98	writes ([file format](file-format.md)). A large file makes a publication as
99	large as itself.
100
101	A keystroke that publishes immediately costs three commits: the edit, the seal
102	(which also records the publication attempt), and the receipt. Each one is
103	ordered against something outside the cache. The edit must be durable before
104	`apply` answers. The attempt must be durable before any byte reaches the
105	share, because an attempt that might have landed is never replayed blindly.
106	The receipt follows the file's own commit. Merging any two of them would open a
107	window where a crash forgets or duplicates work.
108
109	## A sync step
110
111	```text
112	stamp = remote.stamp() one small, uncoordinated read
113	if stamp == base.stamp:
114	publish the sealed batch (if any) Ok ─► acknowledge
115	NotCommitted ─► retry later
116	Unknown ─► attempted; confirm before anything else
117	else:
118	image = remote.read() coordinated snapshot, only now
119	rebase the queue onto image merge, maybe conflict pages
120	base := image
121	```
122
123	A batch stays open while the sync thread is busy, so keystrokes that arrive
124	during one round trip publish together in the next. A check costs a
125	kilobyte-sized read. A publication costs one appended revision, not the
126	section.
127
128	Uncertainty is handled explicitly. A publication attempt is recorded before
129	the first network byte. If the reply is lost, the attempt confirms only when
130	the remote holds its revisions (or every page it changed, as it changed them).
131	Otherwise it waits as `AwaitingConfirmation`, and `release` resolves it
132	explicitly: it exports a recovery archive first, then republishes or abandons.
133
134	## Merging
135
136	When the stamp has moved, someone else committed. The queue replays on the new
137	image one op at a time (`notebook::merge`):
138
139	- an op whose objects the remote left alone applies as it is;
140	- a text op on text the remote also changed shifts past the remote's changes,
141	provided its range stays clear of them;
142	- a move, deletion or setting the remote already made is dropped as done;
143	- anything else conflicts.
144
145	Conflicts follow OneNote 2010 exactly, because the other clients are OneNote.
146	The conflicting op is dropped, along with every later op that names what it
147	named. The remote's version stays the page. The local version becomes a
148	read-only **conflict page** under it, with the conflicting objects marked and
149	the page labelled with its author, so OneNote and Snowbound both show the
150	same bar, the same highlighted paragraphs and the same Delete Conflict Page. A
151	conflict never blocks the queue: it becomes one more queued edit. Page-list
152	edits merge the way OneNote merges page series (a page someone moved keeps
153	their placement, and a page deleted remotely but edited locally comes back as a
154	copy). Each rule was checked against a capture of two real OneNote clients
155	doing it first (`corpus/conflict-page`).
156
157	## Why an embedded SMB client
158
159	OneNote doesn't use lock files. It coordinates through the section file
160	itself, with share modes on open and one-byte locks far past the end of the
161	data:
162
163	| Role | Open | Byte locks |
164	| --- | --- | --- |
165	| reader | read, sharing read/write/delete | shared lock on the reader byte, held for the read |
166	| writer | read/write, denying other writers | the reader byte shared, plus an exclusive writer byte |
167	| maintenance (Optimize) | as a writer | an exclusive range that covers the reader byte |
168
169	Maintenance rewrites the file under a new name and renames it into place, so a
170	handle to the old file goes stale even though its contents still parse.
171
172	Snowbound has to take exactly these locks, or OneNote will either trample it
173	or be locked out. macOS's `smbfs` can't express them. POSIX byte-range locks
174	are unsupported. `flock` of either kind becomes an exclusive lock over the
175	whole file, which fails OneNote's reads. The only exclusive-byte primitive is
176	private and has no shared form. On top of that, `smbfs` serves reads from its
177	own lease-backed cache, defers closes, writes whole cached pages back on
178	`fsync`, and leaves AppleDouble `._` files beside sections it writes. On iOS
179	an app sees a share only through Files, with no control over locks.
180
181	So `notebook::smb` speaks SMB2 itself (on the `smb2` crate's message layer)
182	and takes OneNote's opens and bytes exactly, without leases. It never denies
183	OneNote a read. The leases matter because a Windows client holding a lease
184	keeps its byte locks local and only reveals them when another open breaks the
185	lease. So a peer must open before it locks, and never hold handles between
186	operations. A commit is a handful of compound requests: open; take the locks,
187	check identity and stamp; write appended bytes and patches, then flush; write
188	the header, the counter and the version, each flushed in order; close.
189
190	A notebook on a share that macOS has mounted is opened through the embedded
191	client with the account the system keeps for that mount. It falls back to the
192	mount only when it can't sign in that way.
193
194	A notebook can also be opened from its server's address, with no mount at all (File ▸
195	Open Notebook from Server), which is how Mac OS X 10.6 reaches a server that no longer
196	speaks SMB1, the only version its Finder has. The notebook is kept by that address
197	(`smb://[domain;]user@server/share/folder`), never with a password; the password lives in
198	the keychain (the Secret Service on Linux) when the user asks to remember it, and
199	otherwise the sign-in asks again at the next launch.
200
201	A file's identity is its root object space, not its server file ID, which
202	changes when maintenance replaces the file. A check opens by path each time
203	for the same reason.
204
205	## Offline
206
207	Offline is just a sync step that fails. Edits keep landing in the replica, and
208	the sync thread retries with backoff and wakes on new local edits. A notebook's
209	last listing lets it open while the server is unreachable, and a section opens
210	from its replica. On reconnect the queue publishes, or rebases and publishes,
211	through the same path as always. A failed directory read keeps the previous
212	catalog, so an unreachable share never looks like an emptied notebook.
213	Working offline on purpose, as OneNote's Work Offline does, is the same state
214	chosen: the sync thread stops stepping until Sync Now or until the user works
215	online again.
216
217	## Sections that aren't open
218
219	OneNote keeps every section of an open notebook in sync and on this computer, not just
220	the one on screen. Watched in the lab with a notebook of 200 sections on Samba, OneNote
221	2010:
222
223	- reads every section file once when it first opens the notebook (all 200 within 15
224	seconds), so a section never shown before opens with the share gone; opening it again
225	with its cache warm, it only lists the folders and reads the files listed otherwise;
226	- arms one CHANGE_NOTIFY on the notebook's folder, with WATCH_TREE and a filter of names,
227	attributes, size and last write, and otherwise sends nothing: 22 idle minutes put no SMB
228	request on the wire, not even an echo, and the section on screen waits on the same watch;
229	- lists the folder once when the server reports a change, and reads only the section that
230	changed, some seconds later;
231	- tries a section it could not reach again about every 31 seconds.
232
233	Snowbound does the same. Opening a notebook lists its folders and reads only the files whose
234	listed size or last write time changed since it last read them, keeping what it took from
235	each, with the file's stamp, in the cache (`discover::Cache`). On a share, a changed section
236	whose replica already holds it as it stands, as after its own edits published, costs just a
237	stamp read. Each section it did read is handed on with its image, which makes the offline copy
238	or rebases the replica while the file's stamp is still the image's, so a launch reads each file
239	once, as OneNote does. Dot files (macOS's `.DS_Store` and AppleDouble `._` companions, the
240	`.snowbound` folder) and Office's `~$` files are not the notebook's and are skipped.
241	`session::Background` then keeps the sections in sync, one thread per notebook:
242
243	```text
244	connect: arm the watch, list every folder
245	listed as before: compare the known stamp with the replica, locally
246	listed otherwise: check, one section every 100 ms
247	share ─ CHANGE_NOTIFY (tree) ─► the file it names is due in 1 s ─► stamp ─► moved: rebase
248	└► a folder it names is listed in 1 s ─► the files listed otherwise
249	failing: again in 31 s otherwise: again in an hour, against a lost notification
250	```
251
252	A check costs one stamp read. Only a section whose replica has edits waiting, or whose file
253	moved past the replica's base, has its replica opened for the usual sync steps, and it is
254	closed again afterwards; a section without one gets its offline copy. The section open on
255	screen is left to its session, whose worker no longer polls: the watch wakes it, and once the
256	session lets go of the replica the background checks the file at once. A connection that drops
257	takes its watch with it, so the next one lists every folder again, and so does working online
258	again.
259
260	A notebook in a folder on this computer has no copies and learns of changes from FSEvents or
261	inotify. One on a network volume the system mounted keeps copies. A Mac's SMB mount reports
262	another client's change to FSEvents only for a folder watched in its own right, and names just
263	that folder, so every folder of the notebook is watched and a report lists it. A Linux SMB
264	mount reports none of another client's changes to inotify, so there each section's stamp is
265	read every 15 seconds. On iOS, a folder on the device (Snowbound's own, or On My iPhone) has no
266	copies and learns of other apps' writes through file coordination (`NSFilePresenter`); one a
267	file provider keeps elsewhere, as iCloud Drive does, keeps copies and is checked every 15
268	seconds. Closing a notebook deletes its copies, except any with edits still waiting.
269
270	## iCloud Drive
271
272	A cloud drive syncs whole files with no lock and no compare-and-swap. The stamp check still runs
273	before each append, but only against this device's copy; when two devices append to the same
274	base, iCloud keeps one as the file and the other beside it as a **conflict version**
275	(`NSFileVersion`), on every device. A conflict version is a stamp check that failed after the
276	fact, and each sync step merges the versions it finds before anything else:
277
278	```text
279	for each version V the remote lists (Remote::versions)
280	C := the file, read coordinated
281	per space of V: a := its newest revision C holds, or holds merged
282	ancestor := V opened at those revisions (Section::open_at)
283	ops := lower_page(ancestor[S], V[S]) per page V changed, V's new pages imported under their
284	identities, its deletions, moves and conflict pages
285	replay ops on C through merge.rs (conflict pages where they clash), seal_as the named
286	revisions, publish on C's stamp; then retire V (resolved, removed)
287	```
288
289	Each revision the merge writes is named after the version's revision it merged (a hash of it),
290	so "holds merged" is a lookup by revision identity. A version merged once, by this device or
291	another, merges as nothing the next time; two devices merging one version at once write the
292	same names, and the merge of their two results finds everything held. Any device merges any
293	version at once; nothing waits for the device that wrote it. A version of another section, or
294	one that cannot be read, is kept beside the file as `Name (Device).one` instead.
295
296	On a cloud drive every read and write of a section file runs under `NSFileCoordinator`, so the
297	iCloud daemon never swaps a file between the stamp check and the append. A file iCloud evicted
298	(a `.Name.icloud` placeholder, or on macOS 14 and later a dataless file under its own name) lists
299	as last read, or as downloading; the host asks for it, and reads the notebook again as files
300	arrive. The sync thread never waits for one: the replica serves it meanwhile. Opening a section
301	with no replica yet waits for its file. macOS and iOS offer no public way to keep a file
302	downloaded, as the Finder's Keep Downloaded does; evicted files are asked for again whenever
303	seen. Each publication is an upload and a
304	chance to conflict, so edits wait for a 3 second pause in typing (at most 30 seconds,
305	`Section::set_pause`), and publish at once when the app leaves the foreground, the screen locks,
306	the Mac's window loses focus or the app quits. A folder presenter reports other devices'
307	changes and conflict versions, which change no file, and every section is checked every 15
308	seconds besides. Signing out of iCloud closes its notebooks; replicas holding edits stay.
309
310	`crates/notebook/tests/cloud.rs` runs devices against a fake iCloud (uploads without
311	compare-and-swap, either side of a race winning, late deliveries, offline spans) and checks that
312	every device ends on the same file with each typed string once. OneNote 2010 cold-opens a
313	merged section with its conflict page as its own (`corpus/icloud-merge`).
314
315	## Live Share
316
317	Live Share (`notebook::live::share`, beta) opens a notebook one Snowbound holds on others,
318	with no service holding the notebook. The host serves its notebook's storage verbs to the peers
319	in the share's room; a guest runs the replica, queue and merge it runs on an SMB share against
320	them (`Notebook::open_hosted`, `Section::resume_hosted`, `Background::hosted`), so offline
321	queueing, rebases and conflict pages behave as on a share, and the host's files are only ever
322	written by its own storage, OneNote's locks included. A guest meets the host first in the
323	room of a short code (`7KQ-4MZ-9XR`: Crockford base32, its room's number, 30 bits of
324	secret and a check symbol; SPAKE2 on its secret and any password) through
325	Snowbound's relay or by mDNS, and is welcomed with the share's room and its random secret;
326	stopping or restarting a share retires the secret. A guest's commit is checked on the host's
327	image before it is committed, and a guest can name nothing outside the notebook. Large reads
328	and uploads travel 128 KiB at a time, at most 512 KiB unanswered, so a relay never holds much
329	for a slow peer. A host that goes away leaves its guests working offline: the notebook opens
330	from its last listing and edits publish when it is back. The relay is
331	`crates/relay`.
332
333	Live presence, on by default, shows who else has a notebook others reach too (on a server, in
334	iCloud Drive, in another computer's folder, or by Live Share): their avatars, page and caret.
335	Its room's secret is a random one in `.snowbound/live.json`, which Live Share never serves.
336
337	## Notebook structure
338
339	Sections, groups and the notebook's own colour live in the `.onetoc2` files,
340	which are revision stores edited through the same transaction path. Moving or
341	renaming a file also rewrites two header fields OneNote checks on open (the
342	parent TOC's identity and a CRC of the file's name). Without them OneNote
343	treats the file as a stranger and re-identifies it. Deleting sends sections and
344	pages to `OneNote_RecycleBin`, as OneNote does, and opening a notebook empties
345	what the bin has held unchanged for 60 days, as OneNote prunes it: judged by the
346	content's last change, the pages in one revision, binned section files deleted
347	with their TOC entries left (`corpus/recycle-purge`).
348
349	What only Snowbound reads lives in the notebook's `.snowbound` folder, which carries the
350	Windows hidden attribute so that OneNote never makes a section group of it. Its files are
351	plain files, not revision stores: pictures named by their content, and small JSON
352	mappings that every writer merges before replacing. The first is tag art: a tag keeps
353	OneNote's definition and a fallback symbol on the page, and `tags.json` maps its name and
354	symbol to a picture that Snowbound draws in the symbol's place. `live.json` holds the
355	notebook's presence room secret.
356
357	## Recovery and migration
358
359	`export_recovery` snapshots a replica into a single read-only archive: base and
360	remote images, queue, attempts, receipts and media. It is the evidence behind
361	any decision that could lose work. Cache schema conversions check their own
362	output: every converted page must equal the page the old cache held, or the
363	conversion rolls back and names the archive it made first.
364
365	The [notebook README](../crates/notebook/README.md) documents the API in detail.