authorgravatar for git@paperclover.netclover caruso <git@paperclover.net> 2026-09-08 20:51:37-07:00
committergravatar for git@paperclover.netclover caruso <git@paperclover.net> 2026-09-08 20:51:37-07:00
logcffb05d4803d38e6fbc9acef1aff140f067a5ede
treef9ca4542f1fd8285c787093f9ae0b861f9cbb350
parent50cc916d18c0cf6e5bf920d93cbfc83a8d827dec
signature Signed by SSH key SHA256:52mNGHRsVFBDED9IAX5pe+LRWUefqTbxEReunq21QvU

feat: publish revisions across existing object spaces atomically

Assemble revision manifests for multiple spaces behind one transaction-count update while retaining per-space histories, reference counts and checkpoints. Keep the public API unchanged. Verify a three-space page nesting against native OneNote and a fresh cold reopen. Cover every interrupted I/O, 514 paired revisions across the counter carry and both checkpoints, 130 core tests, 99 offline tests, 148 Python tests, Clippy, both iOS targets and 3,687 sanitizer executions. Fix property copying stopping early on multi-reference arrays; cover all three streams with direct and nested regression cases. Assisted-by: gpt-6-astra

18 files changed, 882 insertions(+), 354 deletions(-)

corpus/page-lifecycle/atomic-nesting/README.md created+21
......@@ -0,0 +1,21 @@
1# Atomic page nesting
2
3The candidate nests `child` and `grandchild` from `../03-renamed` beneath the
4`Same title` page, at levels 2 and 3. Section series order, section metadata
5copies and the two page metadata revisions publish in one transaction. Page
6bodies and prior revisions retain their identities and contents.
7
8`../04-nested` is the independent native control. `cold` is a fresh OneNote
9reopen of the Rust candidate; all active page and section objects compare exactly.
10The files differ in 48 header bytes, including file identity and version metadata;
11all bytes after the header are identical. Both images are retained. The capture's
12owned clone was removed; `provenance.json` records its source and teardown.
13
14Set `ONESTORE_PAGE_BATCH_OUTPUT` to a new directory and run the core unit test
15`write::tests::multi::nesting_publishes_section_order_and_page_levels_in_one_transaction`
16to export another candidate. Capture it with `tools/native_runner.py INPUT OUTPUT
17--expected-pages 9 --collect-notebook`. Run `tools.test_page_lifecycle` after
18building workspace examples to compare native navigation and document content.
19
20This fixture verifies the internal transaction writer. It does not define a
21public page operation, offline conflict policy or implicit subpage movement.
corpus/page-lifecycle/atomic-nesting/candidate/Lifecycle.one created
Binary files /dev/null and b/corpus/page-lifecycle/atomic-nesting/candidate/Lifecycle.one differ
corpus/page-lifecycle/atomic-nesting/cold/notebook/Lifecycle.one created
Binary files /dev/null and b/corpus/page-lifecycle/atomic-nesting/cold/notebook/Lifecycle.one differ
corpus/page-lifecycle/atomic-nesting/cold/read/hierarchy.xml created+2
......@@ -0,0 +1,2 @@
1<?xml version="1.0"?>
2<one:Notebook xmlns:one="http://schemas.microsoft.com/office/onenote/2010/onenote" name="notebook" nickname="notebook" ID="{F5B1F840-D34A-416F-B1FA-1B8E5493F8A1}{1}{B0}" path="C:\one-tests\runs\capture\notebook" lastModifiedTime="2026-09-09T03:56:15.000Z" color="#B49EDE"><one:Section name="Lifecycle" ID="{4A1FE443-51FD-0EA6-0E24-D52F8F7A58E7}{1}{B0}" path="C:\one-tests\runs\capture\notebook\Lifecycle.one" lastModifiedTime="2026-09-09T03:56:15.000Z" color="#91BAAE"><one:Page ID="{AA25BDB6-1E7D-06A1-1B39-45F756307A77}{1}{B0}" name="Untitled page" dateTime="2026-09-09T03:09:19.000Z" lastModifiedTime="2026-09-09T03:56:15.000Z" pageLevel="1"/><one:Page ID="{E048C445-8ED6-0F33-0F7A-45D6BBA0B159}{1}{B0}" name="Untitled page" dateTime="2026-09-09T03:09:19.000Z" lastModifiedTime="2026-09-09T03:56:15.000Z" pageLevel="1"/><one:Page ID="{C0470576-870E-0811-1FE7-83224012993C}{1}{B0}" name="Untitled page" dateTime="2026-09-09T03:09:19.000Z" lastModifiedTime="2026-09-09T03:56:15.000Z" pageLevel="1"/><one:Page ID="{D48B4C0F-81CB-4D47-A2E4-CB9EF846A6E4}{1}{B0}" name="Same title" dateTime="2026-09-09T03:09:27.000Z" lastModifiedTime="2026-09-09T03:56:15.000Z" pageLevel="1"/><one:Page ID="{8907679C-078D-4741-9C09-F34ABC6C3215}{1}{B0}" name="child" dateTime="2026-09-09T03:09:29.000Z" lastModifiedTime="2026-09-09T03:56:15.000Z" pageLevel="2"/><one:Page ID="{D4DBA55F-9D47-491E-802F-2BE696DE6EC3}{1}{B0}" name="grandchild" dateTime="2026-09-09T03:09:29.000Z" lastModifiedTime="2026-09-09T03:56:15.000Z" pageLevel="3"/><one:Page ID="{07185772-4ECB-4A13-BC88-87DC41A9838F}{1}{B0}" name="Renamed 🦋 é" dateTime="2026-09-09T03:09:30.000Z" lastModifiedTime="2026-09-09T03:56:15.000Z" pageLevel="1"/><one:Page ID="{6CFDADD2-7837-4022-8328-8587D6901795}{1}{B0}" name="Body trailing 🦀 é preserved" dateTime="2026-09-09T03:09:30.000Z" lastModifiedTime="2026-09-09T03:56:15.000Z" pageLevel="1"/><one:Page ID="{17B4C931-2D3E-4740-A892-ED86D2593AA4}{1}{B0}" name="Body automatic 🦀 é preserved" dateTime="2026-09-09T03:09:30.000Z" lastModifiedTime="2026-09-09T03:56:15.000Z" pageLevel="1"/></one:Section></one:Notebook>
corpus/page-lifecycle/atomic-nesting/cold/read/page-000.xml created+4
......@@ -0,0 +1,4 @@
1<?xml version="1.0"?>
2<one:Page xmlns:one="http://schemas.microsoft.com/office/onenote/2010/onenote" ID="{07185772-4ECB-4A13-BC88-87DC41A9838F}{1}{B0}" name="Renamed 🦋 é" dateTime="2026-09-09T03:09:30.000Z" lastModifiedTime="2026-09-09T03:56:15.000Z" pageLevel="1" lang="en-US"><one:QuickStyleDef index="0" name="p" fontColor="automatic" highlightColor="automatic" font="Calibri" fontSize="11.0" spaceBefore="0.0" spaceAfter="0.0"/><one:PageSettings RTL="false" color="automatic"><one:PageSize><one:Automatic/></one:PageSize><one:RuleLines visible="false"/></one:PageSettings><one:Title lang="en-US"><one:OE author="virtual" authorInitials="V" lastModifiedBy="virtual" lastModifiedByInitials="V" creationTime="2026-09-09T03:09:34.000Z" lastModifiedTime="2026-09-09T03:09:34.000Z" objectID="{2AE0B6C6-A870-4D63-AAF6-679E97FF06F0}{16}{B0}" alignment="left" quickStyleIndex="0" style="font-family:Calibri;font-size:17.0pt"><one:T><![CDATA[Renamed &#129419; é]]></one:T></one:OE></one:Title><one:Outline author="virtual" authorInitials="V" lastModifiedBy="virtual" lastModifiedByInitials="V" lastModifiedTime="2026-09-09T03:09:30.000Z" objectID="{61979D6C-C04D-41F9-AFB6-F8FE349FB7CE}{39}{B0}"><one:Position x="36.0" y="86.4000015258789" z="0"/><one:Size width="170.2942504882812" height="13.42771339416504"/><one:OEChildren><one:OE creationTime="2026-09-09T03:09:30.000Z" lastModifiedTime="2026-09-09T03:09:30.000Z" objectID="{61979D6C-C04D-41F9-AFB6-F8FE349FB7CE}{40}{B0}" alignment="left" quickStyleIndex="0"><one:T><![CDATA[<span
3style='font-weight:bold' lang=en-US>Body duplicate &#129408; é</span><span
4lang=en-US> </span><span style='font-style:italic' lang=x-none>preserved</span>]]></one:T></one:OE></one:OEChildren></one:Outline></one:Page>
corpus/page-lifecycle/atomic-nesting/cold/read/page-001.xml created+4
......@@ -0,0 +1,4 @@
1<?xml version="1.0"?>
2<one:Page xmlns:one="http://schemas.microsoft.com/office/onenote/2010/onenote" ID="{17B4C931-2D3E-4740-A892-ED86D2593AA4}{1}{B0}" name="Body automatic 🦀 é preserved" dateTime="2026-09-09T03:09:30.000Z" lastModifiedTime="2026-09-09T03:56:15.000Z" pageLevel="1" lang="en-US"><one:QuickStyleDef index="0" name="p" fontColor="automatic" highlightColor="automatic" font="Calibri" fontSize="11.0" spaceBefore="0.0" spaceAfter="0.0"/><one:PageSettings RTL="false" color="automatic"><one:PageSize><one:Automatic/></one:PageSize><one:RuleLines visible="false"/></one:PageSettings><one:Outline author="virtual" authorInitials="V" lastModifiedBy="virtual" lastModifiedByInitials="V" lastModifiedTime="2026-09-09T03:09:30.000Z" objectID="{A0CAB379-8594-48A1-AA50-36B8F10799B7}{14}{B0}"><one:Position x="36.0" y="14.40000057220459" z="0"/><one:Size width="174.3547973632812" height="13.42771625518799"/><one:OEChildren><one:OE creationTime="2026-09-09T03:09:30.000Z" lastModifiedTime="2026-09-09T03:09:30.000Z" objectID="{A0CAB379-8594-48A1-AA50-36B8F10799B7}{15}{B0}" alignment="left" quickStyleIndex="0"><one:T><![CDATA[<span
3style='font-weight:bold' lang=en-US>Body automatic &#129408; é</span><span
4lang=en-US> </span><span style='font-style:italic' lang=x-none>preserved</span>]]></one:T></one:OE></one:OEChildren></one:Outline></one:Page>
corpus/page-lifecycle/atomic-nesting/cold/read/page-002.xml created+4
......@@ -0,0 +1,4 @@
1<?xml version="1.0"?>
2<one:Page xmlns:one="http://schemas.microsoft.com/office/onenote/2010/onenote" ID="{6CFDADD2-7837-4022-8328-8587D6901795}{1}{B0}" name="Body trailing 🦀 é preserved" dateTime="2026-09-09T03:09:30.000Z" lastModifiedTime="2026-09-09T03:56:15.000Z" pageLevel="1" lang="en-US"><one:QuickStyleDef index="0" name="p" fontColor="automatic" highlightColor="automatic" font="Calibri" fontSize="11.0" spaceBefore="0.0" spaceAfter="0.0"/><one:PageSettings RTL="false" color="automatic"><one:PageSize><one:Automatic/></one:PageSize><one:RuleLines visible="false"/></one:PageSettings><one:Title lang="en-US"><one:OE author="virtual" authorInitials="V" lastModifiedBy="virtual" lastModifiedByInitials="V" creationTime="2026-09-09T03:09:34.000Z" lastModifiedTime="2026-09-09T03:09:34.000Z" objectID="{2973BC5F-757A-4590-96F9-39879A8C0725}{16}{B0}" alignment="left" quickStyleIndex="0" style="font-family:Calibri;font-size:17.0pt"><one:T><![CDATA[]]></one:T></one:OE></one:Title><one:Outline author="virtual" authorInitials="V" lastModifiedBy="virtual" lastModifiedByInitials="V" lastModifiedTime="2026-09-09T03:09:30.000Z" objectID="{FDCC8DAF-57A7-4A12-B17D-E763F017FDD9}{39}{B0}"><one:Position x="36.0" y="86.4000015258789" z="0"/><one:Size width="160.1752014160156" height="13.42771339416504"/><one:OEChildren><one:OE creationTime="2026-09-09T03:09:30.000Z" lastModifiedTime="2026-09-09T03:09:30.000Z" objectID="{FDCC8DAF-57A7-4A12-B17D-E763F017FDD9}{40}{B0}" alignment="left" quickStyleIndex="0"><one:T><![CDATA[<span
3style='font-weight:bold' lang=en-US>Body trailing &#129408; é</span><span
4lang=en-US> </span><span style='font-style:italic' lang=x-none>preserved</span>]]></one:T></one:OE></one:OEChildren></one:Outline></one:Page>
corpus/page-lifecycle/atomic-nesting/cold/read/page-003.xml created+4
......@@ -0,0 +1,4 @@
1<?xml version="1.0"?>
2<one:Page xmlns:one="http://schemas.microsoft.com/office/onenote/2010/onenote" ID="{8907679C-078D-4741-9C09-F34ABC6C3215}{1}{B0}" name="child" dateTime="2026-09-09T03:09:29.000Z" lastModifiedTime="2026-09-09T03:56:15.000Z" pageLevel="2" lang="en-US"><one:QuickStyleDef index="0" name="p" fontColor="automatic" highlightColor="automatic" font="Calibri" fontSize="11.0" spaceBefore="0.0" spaceAfter="0.0"/><one:PageSettings RTL="false" color="automatic"><one:PageSize><one:Automatic/></one:PageSize><one:RuleLines visible="false"/></one:PageSettings><one:Title lang="en-US"><one:OE author="virtual" authorInitials="V" lastModifiedBy="virtual" lastModifiedByInitials="V" creationTime="2026-09-09T03:09:29.000Z" lastModifiedTime="2026-09-09T03:09:29.000Z" objectID="{035312EF-E5A0-43E9-B44E-EC1807D7EBB1}{28}{B0}" alignment="left" quickStyleIndex="0" style="font-family:Calibri;font-size:17.0pt"><one:T><![CDATA[child]]></one:T></one:OE></one:Title><one:Outline author="virtual" authorInitials="V" lastModifiedBy="virtual" lastModifiedByInitials="V" lastModifiedTime="2026-09-09T03:09:30.000Z" objectID="{035312EF-E5A0-43E9-B44E-EC1807D7EBB1}{35}{B0}"><one:Position x="36.0" y="86.4000015258789" z="0"/><one:Size width="149.6102294921875" height="13.42771339416504"/><one:OEChildren><one:OE creationTime="2026-09-09T03:09:29.000Z" lastModifiedTime="2026-09-09T03:09:29.000Z" objectID="{035312EF-E5A0-43E9-B44E-EC1807D7EBB1}{36}{B0}" alignment="left" quickStyleIndex="0"><one:T><![CDATA[<span
3style='font-weight:bold' lang=en-US>Body child &#129408; é</span><span
4lang=en-US> </span><span style='font-style:italic' lang=x-none>preserved</span>]]></one:T></one:OE></one:OEChildren></one:Outline></one:Page>
corpus/page-lifecycle/atomic-nesting/cold/read/page-004.xml created+2
......@@ -0,0 +1,2 @@
1<?xml version="1.0"?>
2<one:Page xmlns:one="http://schemas.microsoft.com/office/onenote/2010/onenote" ID="{AA25BDB6-1E7D-06A1-1B39-45F756307A77}{1}{B0}" name="Untitled page" dateTime="2026-09-09T03:09:19.000Z" lastModifiedTime="2026-09-09T03:56:15.000Z" pageLevel="1"><one:QuickStyleDef index="0" name="PageTitle" fontColor="automatic" highlightColor="automatic" font="Calibri" fontSize="17.0" spaceBefore="0.0" spaceAfter="0.0"/><one:PageSettings RTL="false" color="automatic"><one:PageSize><one:Automatic/></one:PageSize><one:RuleLines visible="false"/></one:PageSettings><one:Title lang="en-US"><one:OE author="virtual" authorInitials="V" lastModifiedBy="virtual" lastModifiedByInitials="V" creationTime="2026-09-09T03:09:19.000Z" lastModifiedTime="2026-09-09T03:09:19.000Z" objectID="{6450CCAD-22C8-0EC6-30BE-1EC7CC43F688}{15}{B0}" alignment="left" quickStyleIndex="0"><one:T><![CDATA[]]></one:T></one:OE></one:Title></one:Page>
corpus/page-lifecycle/atomic-nesting/cold/read/page-005.xml created+2
......@@ -0,0 +1,2 @@
1<?xml version="1.0"?>
2<one:Page xmlns:one="http://schemas.microsoft.com/office/onenote/2010/onenote" ID="{C0470576-870E-0811-1FE7-83224012993C}{1}{B0}" name="Untitled page" dateTime="2026-09-09T03:09:19.000Z" lastModifiedTime="2026-09-09T03:56:15.000Z" pageLevel="1"><one:PageSettings RTL="false" color="automatic"><one:PageSize><one:Automatic/></one:PageSize><one:RuleLines visible="false"/></one:PageSettings></one:Page>
corpus/page-lifecycle/atomic-nesting/cold/read/page-006.xml created+4
......@@ -0,0 +1,4 @@
1<?xml version="1.0"?>
2<one:Page xmlns:one="http://schemas.microsoft.com/office/onenote/2010/onenote" ID="{D48B4C0F-81CB-4D47-A2E4-CB9EF846A6E4}{1}{B0}" name="Same title" dateTime="2026-09-09T03:09:27.000Z" lastModifiedTime="2026-09-09T03:56:15.000Z" pageLevel="1" lang="en-US"><one:QuickStyleDef index="0" name="p" fontColor="automatic" highlightColor="automatic" font="Calibri" fontSize="11.0" spaceBefore="0.0" spaceAfter="0.0"/><one:PageSettings RTL="false" color="automatic"><one:PageSize><one:Automatic/></one:PageSize><one:RuleLines visible="false"/></one:PageSettings><one:Title lang="en-US"><one:OE author="virtual" authorInitials="V" lastModifiedBy="virtual" lastModifiedByInitials="V" creationTime="2026-09-09T03:09:27.000Z" lastModifiedTime="2026-09-09T03:09:27.000Z" objectID="{D690D575-D56D-47D2-90DB-0191D6BDB859}{32}{B0}" alignment="left" quickStyleIndex="0" style="font-family:Calibri;font-size:17.0pt"><one:T><![CDATA[Same title]]></one:T></one:OE></one:Title><one:Outline author="virtual" authorInitials="V" lastModifiedBy="virtual" lastModifiedByInitials="V" lastModifiedTime="2026-09-09T03:09:29.000Z" objectID="{D690D575-D56D-47D2-90DB-0191D6BDB859}{39}{B0}"><one:Position x="36.0" y="86.4000015258789" z="0"/><one:Size width="158.2952728271484" height="13.42771339416504"/><one:OEChildren><one:OE creationTime="2026-09-09T03:09:27.000Z" lastModifiedTime="2026-09-09T03:09:29.000Z" objectID="{D690D575-D56D-47D2-90DB-0191D6BDB859}{40}{B0}" alignment="left" quickStyleIndex="0"><one:T><![CDATA[<span
3style='font-weight:bold' lang=en-US>Body parent &#129408; é</span><span
4lang=en-US> </span><span style='font-style:italic' lang=x-none>preserved</span>]]></one:T></one:OE></one:OEChildren></one:Outline></one:Page>
corpus/page-lifecycle/atomic-nesting/cold/read/page-007.xml created+4
......@@ -0,0 +1,4 @@
1<?xml version="1.0"?>
2<one:Page xmlns:one="http://schemas.microsoft.com/office/onenote/2010/onenote" ID="{D4DBA55F-9D47-491E-802F-2BE696DE6EC3}{1}{B0}" name="grandchild" dateTime="2026-09-09T03:09:29.000Z" lastModifiedTime="2026-09-09T03:56:15.000Z" pageLevel="3" lang="en-US"><one:QuickStyleDef index="0" name="p" fontColor="automatic" highlightColor="automatic" font="Calibri" fontSize="11.0" spaceBefore="0.0" spaceAfter="0.0"/><one:PageSettings RTL="false" color="automatic"><one:PageSize><one:Automatic/></one:PageSize><one:RuleLines visible="false"/></one:PageSettings><one:Title lang="en-US"><one:OE author="virtual" authorInitials="V" lastModifiedBy="virtual" lastModifiedByInitials="V" creationTime="2026-09-09T03:09:30.000Z" lastModifiedTime="2026-09-09T03:09:30.000Z" objectID="{440E8EEC-9C7D-4AB1-BC2A-9B6EBF560D82}{32}{B0}" alignment="left" quickStyleIndex="0" style="font-family:Calibri;font-size:17.0pt"><one:T><![CDATA[grandchild]]></one:T></one:OE></one:Title><one:Outline author="virtual" authorInitials="V" lastModifiedBy="virtual" lastModifiedByInitials="V" lastModifiedTime="2026-09-09T03:09:30.000Z" objectID="{440E8EEC-9C7D-4AB1-BC2A-9B6EBF560D82}{39}{B0}"><one:Position x="36.0" y="86.4000015258789" z="0"/><one:Size width="175.9715728759766" height="13.42771339416504"/><one:OEChildren><one:OE creationTime="2026-09-09T03:09:30.000Z" lastModifiedTime="2026-09-09T03:09:30.000Z" objectID="{440E8EEC-9C7D-4AB1-BC2A-9B6EBF560D82}{40}{B0}" alignment="left" quickStyleIndex="0"><one:T><![CDATA[<span
3style='font-weight:bold' lang=en-US>Body grandchild &#129408; é</span><span
4lang=en-US> </span><span style='font-style:italic' lang=x-none>preserved</span>]]></one:T></one:OE></one:OEChildren></one:Outline></one:Page>
corpus/page-lifecycle/atomic-nesting/cold/read/page-008.xml created+2
......@@ -0,0 +1,2 @@
1<?xml version="1.0"?>
2<one:Page xmlns:one="http://schemas.microsoft.com/office/onenote/2010/onenote" ID="{E048C445-8ED6-0F33-0F7A-45D6BBA0B159}{1}{B0}" name="Untitled page" dateTime="2026-09-09T03:09:19.000Z" lastModifiedTime="2026-09-09T03:56:15.000Z" pageLevel="1"><one:QuickStyleDef index="0" name="PageTitle" fontColor="automatic" highlightColor="automatic" font="Calibri" fontSize="17.0" spaceBefore="0.0" spaceAfter="0.0"/><one:PageSettings RTL="false" color="automatic"><one:PageSize><one:Automatic/></one:PageSize><one:RuleLines visible="false"/></one:PageSettings><one:Title lang="en-US"><one:OE author="virtual" authorInitials="V" lastModifiedBy="virtual" lastModifiedByInitials="V" creationTime="2026-09-09T03:09:19.000Z" lastModifiedTime="2026-09-09T03:09:19.000Z" objectID="{CFAC019E-2183-032A-2410-3D6897C66FDB}{15}{B0}" alignment="left" quickStyleIndex="0"><one:T><![CDATA[]]></one:T></one:OE></one:Title></one:Page>
corpus/page-lifecycle/atomic-nesting/provenance.json created+33
......@@ -0,0 +1,33 @@
1{
2 "source": "../03-renamed/notebook/Lifecycle.one",
3 "native_control": "../04-nested",
4 "generator": "write::tests::multi::nesting_publishes_section_order_and_page_levels_in_one_transaction",
5 "capture": "evidence/m10/page-batch-cold",
6 "run": {
7 "notebook": "/Users/clo/dev/one/evidence/m10/page-batch-candidate",
8 "expected_pages": 9,
9 "author": null,
10 "author_timeout_seconds": 600,
11 "inspect": false,
12 "collect_notebook": true,
13 "base": {
14 "file": "win7-office-base.qcow2",
15 "format": "qcow2",
16 "sha256": "a1a4f8fab782ee14885ff801ca2f6347c208fdcfc3513637096f314315c89346",
17 "virtual_size": 68719476736
18 },
19 "scripts": {
20 "cold.ps1": "c177fc72ae6c2634d186f5671a880de20b09f9716532c37c69cb13aa15c7e331",
21 "read.ps1": "04013bfcccee40a17e2a225f9b9e96f40a3a8daad9e350609654f8eda3ccbb41"
22 }
23 },
24 "machine": {
25 "name": "m6-3d4af21a",
26 "hostname": "ONE-M6-3D4AF21A"
27 },
28 "teardown": {
29 "absent": true
30 },
31 "candidate_sha256": "02fc858fb8854dbd14c1d3c58b62fdd653e51216872afc0cd8b8a4b94b651b9d",
32 "cold_sha256": "f7a0c74f1a10d5933b386a7da7fee1ead860190b8794dc0ad3788e36c50f14eb"
33}
crates/onestore/src/write.rs+385-354
......@@ -511,7 +511,7 @@ impl PropertyObject {
511511 bytes: compact_ids,
512512 offset: 0,
513513 };
514 while cursor.offset < cursor.bytes.len() {
514 while !cursor.bytes.is_empty() {
515515 let id = cursor.compact(&source.global_ids)?;
516516 added[index].extend_from_slice(&target.reference(id)?);
517517 }
......@@ -586,6 +586,24 @@ pub(crate) fn write_revision(
586586 source: &[u8],
587587 space: ExGuid,
588588 edit: impl FnOnce(&crate::ResolvedRevision<'_>) -> Result<BTreeMap<ExGuid, PropertyObject>>,
589) -> Result<Vec<u8>> {
590 write_revisions(source, |index| {
591 let rid = *index
592 .spaces
593 .get(&space)
594 .and_then(|space| space.labels.get(&(ExGuid::default(), 1)))
595 .ok_or(Error {
596 offset: 0,
597 message: "Object space has no active default revision",
598 })?;
599 let revision = index.resolve(space, rid)?;
600 Ok(BTreeMap::from([(space, edit(&revision)?)]))
601 })
602}
603
604pub(crate) fn write_revisions(
605 source: &[u8],
606 edit: impl FnOnce(&RevisionIndex<'_>) -> Result<BTreeMap<ExGuid, BTreeMap<ExGuid, PropertyObject>>>,
589607) -> Result<Vec<u8>> {
590608 let store = Store::parse(source)?;
591609 let is_section = store.header.file_type == FileType::Section;
......@@ -597,399 +615,415 @@ pub(crate) fn write_revision(
597615 }
598616 let index = RevisionIndex::parse(&store)?;
599617 index.validate_current()?;
600 let rid = *index
601 .spaces
602 .get(&space)
603 .and_then(|space| space.labels.get(&(ExGuid::default(), 1)))
604 .ok_or(Error {
605 offset: 0,
606 message: "Object space has no active default revision",
607 })?;
608 let mut revision = index.resolve(space, rid)?;
609 let reachable = revision.reachable()?;
610 let mut replacements = edit(&revision)?;
611 for (id, replacement) in &replacements {
612 if let Some(object) = revision.objects.get(id) {
613 if !reachable.contains(id) {
618 let changes = edit(&index)?;
619 let mut output = source.to_vec();
620 let maximum = store
621 .transaction_fragments
622 .iter()
623 .flat_map(|fragment| fragment.entries.chunks_exact(8))
624 .map(|entry| u32::from_le_bytes(entry[..4].try_into().unwrap()))
625 .max()
626 .unwrap();
627 let mut counts = Vec::new();
628 for (space, mut replacements) in changes {
629 let rid = *index
630 .spaces
631 .get(&space)
632 .and_then(|space| space.labels.get(&(ExGuid::default(), 1)))
633 .ok_or(Error {
634 offset: 0,
635 message: "Object space has no active default revision",
636 })?;
637 let mut revision = index.resolve(space, rid)?;
638 let reachable = revision.reachable()?;
639 for (id, replacement) in &replacements {
640 if let Some(object) = revision.objects.get(id) {
641 if !reachable.contains(id) {
642 return Err(Error {
643 offset: 0,
644 message: "Object is not reachable in the active revision",
645 });
646 }
647 if object.jcid & 0x100000 != 0 {
648 return Err(Error {
649 offset: 0,
650 message: "Read-only object requires a new identity",
651 });
652 }
653 if replacement.jcid != object.jcid
654 || !matches!(object.data, ObjectData::Properties(_))
655 {
656 return Err(Error {
657 offset: 0,
658 message: "An existing object's type cannot be changed",
659 });
660 }
661 } else if !is_section {
614662 return Err(Error {
615663 offset: 0,
616 message: "Object is not reachable in the active revision",
664 message: "New objects require a section file",
617665 });
618666 }
619 if object.jcid & 0x100000 != 0 {
667 if replacement.jcid & 0x20000 == 0
668 || replacement.global_ids.keys().any(|i| *i > 0xffffff)
669 {
620670 return Err(Error {
621671 offset: 0,
622 message: "Read-only object requires a new identity",
672 message: "Invalid property object declaration",
623673 });
624674 }
625 if replacement.jcid != object.jcid || !matches!(object.data, ObjectData::Properties(_))
675 compact(*id, &replacement.global_ids)?;
676 PropertySets::parse(&replacement.bytes)?;
677 }
678 // Native coalescing of duplicate readonly styles can leave dangling references.
679 let mut aliases = BTreeMap::new();
680 for (id, replacement) in &replacements {
681 if revision.objects.contains_key(id)
682 || replacement.jcid & 0x100000 == 0
683 || PropertySets::parse(&replacement.bytes)?
684 .sets
685 .iter()
686 .flatten()
687 .any(|p| matches!(p.value, Value::References { .. }))
626688 {
627 return Err(Error {
628 offset: 0,
629 message: "An existing object's type cannot be changed",
630 });
689 continue;
631690 }
632 } else if !is_section {
633 return Err(Error {
634 offset: 0,
635 message: "New objects require a section file",
691 let existing = revision.objects.iter().find_map(|(other, object)| {
692 (reachable.contains(other)
693 && object.jcid == replacement.jcid
694 && object.data == ObjectData::Properties(&replacement.bytes))
695 .then_some(*other)
636696 });
637 }
638 if replacement.jcid & 0x20000 == 0 || replacement.global_ids.keys().any(|i| *i > 0xffffff) {
639 return Err(Error {
640 offset: 0,
641 message: "Invalid property object declaration",
697 let existing = existing.or_else(|| {
698 replacements.range(..id).find_map(|(other, object)| {
699 (object.jcid == replacement.jcid && object.bytes == replacement.bytes)
700 .then_some(*aliases.get(other).unwrap_or(other))
701 })
642702 });
703 if let Some(existing) = existing {
704 aliases.insert(*id, existing);
705 }
643706 }
644 compact(*id, &replacement.global_ids)?;
645 PropertySets::parse(&replacement.bytes)?;
646 }
647 // Native coalescing of duplicate readonly styles can leave dangling references.
648 let mut aliases = BTreeMap::new();
649 for (id, replacement) in &replacements {
650 if revision.objects.contains_key(id)
651 || replacement.jcid & 0x100000 == 0
652 || PropertySets::parse(&replacement.bytes)?
653 .sets
654 .iter()
655 .flatten()
656 .any(|p| matches!(p.value, Value::References { .. }))
657 {
658 continue;
659 }
660 let existing = revision.objects.iter().find_map(|(other, object)| {
661 (reachable.contains(other)
662 && object.jcid == replacement.jcid
663 && object.data == ObjectData::Properties(&replacement.bytes))
664 .then_some(*other)
665 });
666 let existing = existing.or_else(|| {
667 replacements.range(..id).find_map(|(other, object)| {
668 (object.jcid == replacement.jcid && object.bytes == replacement.bytes)
669 .then_some(*aliases.get(other).unwrap_or(other))
670 })
671 });
672 if let Some(existing) = existing {
673 aliases.insert(*id, existing);
707 for id in aliases.keys() {
708 replacements.remove(id);
674709 }
675 }
676 for id in aliases.keys() {
677 replacements.remove(id);
678 }
679 for object in replacements.values_mut() {
680 let mut remapped = Vec::new();
681 for property in PropertySets::parse(&object.bytes)?.sets.iter().flatten() {
682 if let Value::References {
683 stream: crate::IdStream::Objects,
684 compact_ids,
685 } = property.value
686 {
687 for bytes in compact_ids.chunks_exact(4) {
688 let offset = bytes.as_ptr().addr() - object.bytes.as_ptr().addr();
689 let id = crate::bytes::Cursor { bytes, offset }.compact(&object.global_ids)?;
690 if let Some(existing) = aliases.get(&id) {
691 remapped.push((offset, *existing));
710 for object in replacements.values_mut() {
711 let mut remapped = Vec::new();
712 for property in PropertySets::parse(&object.bytes)?.sets.iter().flatten() {
713 if let Value::References {
714 stream: crate::IdStream::Objects,
715 compact_ids,
716 } = property.value
717 {
718 for bytes in compact_ids.chunks_exact(4) {
719 let offset = bytes.as_ptr().addr() - object.bytes.as_ptr().addr();
720 let id =
721 crate::bytes::Cursor { bytes, offset }.compact(&object.global_ids)?;
722 if let Some(existing) = aliases.get(&id) {
723 remapped.push((offset, *existing));
724 }
692725 }
693726 }
694727 }
728 for (offset, id) in remapped {
729 let reference = object.reference(id)?;
730 object.bytes[offset..offset + 4].copy_from_slice(&reference);
731 }
695732 }
696 for (offset, id) in remapped {
697 let reference = object.reference(id)?;
698 object.bytes[offset..offset + 4].copy_from_slice(&reference);
699 }
700 }
701 replacements.retain(|id, replacement| {
702 !revision.objects.get(id).is_some_and(|object| {
703 object.data == ObjectData::Properties(&replacement.bytes)
704 && object.global_ids == replacement.global_ids
705 })
706 });
707 if replacements.is_empty() {
708 return Ok(source.to_vec());
709 }
710 let mut changed: BTreeSet<_> = replacements.keys().copied().collect();
711 for (id, replacement) in &replacements {
712 revision.objects.insert(
713 *id,
714 crate::Object {
715 jcid: replacement.jcid,
716 reference_count: 0,
717 data: ObjectData::Properties(&replacement.bytes),
718 global_ids: Arc::clone(&replacement.global_ids),
719 },
720 );
721 }
722 let incoming = revision.reference_counts()?;
723 if replacements.keys().any(|id| !incoming.contains_key(id)) {
724 return Err(Error {
725 offset: 0,
726 message: "Edited object is not reachable in the resulting revision",
733 replacements.retain(|id, replacement| {
734 !revision.objects.get(id).is_some_and(|object| {
735 object.data == ObjectData::Properties(&replacement.bytes)
736 && object.global_ids == replacement.global_ids
737 })
727738 });
728 }
729 for (id, object) in &mut revision.objects {
730 if reachable.contains(id) || incoming.contains_key(id) {
731 let count = incoming.get(id).copied().unwrap_or(0);
732 if object.reference_count != count {
733 object.reference_count = count;
734 changed.insert(*id);
735 }
739 if replacements.is_empty() {
740 continue;
736741 }
737 }
738
739 // Native cold-open fails on long dependency chains; cap their depth at 512.
740 let checkpoint = std::iter::successors(Some(rid), |id| {
741 index.spaces[&space].revisions[id].dependency
742 })
743 .nth(511)
744 .is_some();
745 let selected: Vec<_> = revision
746 .objects
747 .iter()
748 .filter(|(id, _)| checkpoint || changed.contains(id))
749 .collect();
750 let toc_table = if checkpoint && !is_section {
751 if selected.iter().any(|(_, object)| {
752 object.jcid != 0x20001 || !matches!(object.data, ObjectData::Properties(_))
753 }) {
742 let mut changed: BTreeSet<_> = replacements.keys().copied().collect();
743 for (id, replacement) in &replacements {
744 revision.objects.insert(
745 *id,
746 crate::Object {
747 jcid: replacement.jcid,
748 reference_count: 0,
749 data: ObjectData::Properties(&replacement.bytes),
750 global_ids: Arc::clone(&replacement.global_ids),
751 },
752 );
753 }
754 let incoming = revision.reference_counts()?;
755 if replacements.keys().any(|id| !incoming.contains_key(id)) {
754756 return Err(Error {
755757 offset: 0,
756 message: "TOC checkpoint requires table-of-contents property objects",
758 message: "Edited object is not reachable in the resulting revision",
757759 });
758760 }
759 let guids: BTreeSet<_> = selected
761 for (id, object) in &mut revision.objects {
762 if reachable.contains(id) || incoming.contains_key(id) {
763 let count = incoming.get(id).copied().unwrap_or(0);
764 if object.reference_count != count {
765 object.reference_count = count;
766 changed.insert(*id);
767 }
768 }
769 }
770
771 // Native cold-open fails on long dependency chains; cap their depth at 512.
772 let checkpoint = std::iter::successors(Some(rid), |id| {
773 index.spaces[&space].revisions[id].dependency
774 })
775 .nth(511)
776 .is_some();
777 let selected: Vec<_> = revision
778 .objects
760779 .iter()
761 .flat_map(|(_, object)| object.global_ids.values().copied())
780 .filter(|(id, _)| checkpoint || changed.contains(id))
762781 .collect();
763 if guids.len() > 0x1000000 {
764 return Err(Error {
765 offset: 0,
766 message: "Global ID table exceeds CompactID capacity",
767 });
768 }
769 Some(
770 guids
771 .into_iter()
772 .enumerate()
773 .map(|(i, guid)| (u32::try_from(i).unwrap(), guid))
774 .collect::<BTreeMap<_, _>>(),
775 )
776 } else {
777 None
778 };
779 let mut groups = BTreeMap::<_, Vec<_>>::new();
780 for (id, object) in selected {
781 groups
782 .entry(toc_table.as_ref().unwrap_or(&object.global_ids))
783 .or_default()
784 .push((*id, object));
785 }
786 let mut output = source.to_vec();
787 let maximum = store
788 .transaction_fragments
789 .iter()
790 .flat_map(|fragment| fragment.entries.chunks_exact(8))
791 .map(|entry| u32::from_le_bytes(entry[..4].try_into().unwrap()))
792 .max()
793 .unwrap();
794 let new_rid = ExGuid {
795 guid: fresh_guid()?,
796 n: 1,
797 };
798 let mut start = Vec::new();
799 new_rid.encode(&mut start);
800 if checkpoint { ExGuid::default() } else { rid }.encode(&mut start);
801 if !is_section {
802 start.extend_from_slice(&0_u64.to_le_bytes());
803 }
804 start.extend_from_slice(&1_u32.to_le_bytes());
805 start.extend_from_slice(&0_u16.to_le_bytes());
806 let mut manifest = vec![node(if is_section { 0x1e } else { 0x1b }, None, &start)?];
807 let mut group_counts = Vec::new();
808 for (table, objects) in groups {
809 let mut payload = Vec::new();
810 let mut group = if is_section {
811 ExGuid {
812 guid: fresh_guid()?,
813 n: 1,
782 let toc_table = if checkpoint && !is_section {
783 if selected.iter().any(|(_, object)| {
784 object.jcid != 0x20001 || !matches!(object.data, ObjectData::Properties(_))
785 }) {
786 return Err(Error {
787 offset: 0,
788 message: "TOC checkpoint requires table-of-contents property objects",
789 });
790 }
791 let guids: BTreeSet<_> = selected
792 .iter()
793 .flat_map(|(_, object)| object.global_ids.values().copied())
794 .collect();
795 if guids.len() > 0x1000000 {
796 return Err(Error {
797 offset: 0,
798 message: "Global ID table exceeds CompactID capacity",
799 });
814800 }
815 .encode(&mut payload);
816 vec![node(0xb4, None, &payload)?, node(0x22, None, &[])?]
801 Some(
802 guids
803 .into_iter()
804 .enumerate()
805 .map(|(i, guid)| (u32::try_from(i).unwrap(), guid))
806 .collect::<BTreeMap<_, _>>(),
807 )
817808 } else {
818 vec![node(0x21, None, &[0])?]
809 None
819810 };
820 for (id, guid) in table {
821 let mut entry = id.to_le_bytes().to_vec();
822 entry.extend_from_slice(guid);
823 group.push(node(0x24, None, &entry)?);
811 let mut groups = BTreeMap::<_, Vec<_>>::new();
812 for (id, object) in selected {
813 groups
814 .entry(toc_table.as_ref().unwrap_or(&object.global_ids))
815 .or_default()
816 .push((*id, object));
824817 }
825 group.push(node(0x28, None, &[])?);
826 let mut override_crc = u32::MAX;
827 for (id, object) in objects {
828 let mut declaration = compact(id, table)?.to_vec();
829 override_crc = crc(
830 override_crc,
831 &object.reference_count.to_le_bytes(),
832 FileType::Section,
833 );
834 match object.data {
835 ObjectData::File {
836 reference,
837 extension,
838 } => {
839 declaration.extend_from_slice(&object.jcid.to_le_bytes());
840 declaration.extend_from_slice(&object.reference_count.to_le_bytes());
841 for bytes in [reference, extension] {
842 declaration.extend_from_slice(
843 &u32::try_from(bytes.len() / 2).unwrap().to_le_bytes(),
844 );
845 declaration.extend_from_slice(bytes);
846 }
847 group.push(node(0x73, None, &declaration)?);
818 let new_rid = ExGuid {
819 guid: fresh_guid()?,
820 n: 1,
821 };
822 let mut start = Vec::new();
823 new_rid.encode(&mut start);
824 if checkpoint { ExGuid::default() } else { rid }.encode(&mut start);
825 if !is_section {
826 start.extend_from_slice(&0_u64.to_le_bytes());
827 }
828 start.extend_from_slice(&1_u32.to_le_bytes());
829 start.extend_from_slice(&0_u16.to_le_bytes());
830 let mut manifest = vec![node(if is_section { 0x1e } else { 0x1b }, None, &start)?];
831 for (table, objects) in groups {
832 let mut payload = Vec::new();
833 let mut group = if is_section {
834 ExGuid {
835 guid: fresh_guid()?,
836 n: 1,
848837 }
849 ObjectData::Properties(bytes) => {
850 let references = object.references()?;
851 let flags = u8::from(!references.objects.is_empty())
852 | (u8::from(
853 !references.object_spaces.is_empty() || !references.contexts.is_empty(),
854 ) << 1);
855 let data = if toc_table.is_some() {
856 let mut mapped = bytes.to_vec();
857 for property in PropertySets::parse(bytes)?.sets.iter().flatten() {
858 if let Value::References { compact_ids, .. } = property.value {
859 let offset = compact_ids.as_ptr().addr() - bytes.as_ptr().addr();
860 for (i, value) in compact_ids.chunks_exact(4).enumerate() {
861 let value = u32::from_le_bytes(value.try_into().unwrap());
862 let target = ExGuid {
863 guid: object.global_ids[&(value >> 8)],
864 n: value & 255,
865 };
866 mapped[offset + i * 4..offset + i * 4 + 4]
867 .copy_from_slice(&compact(target, table)?);
838 .encode(&mut payload);
839 vec![node(0xb4, None, &payload)?, node(0x22, None, &[])?]
840 } else {
841 vec![node(0x21, None, &[0])?]
842 };
843 for (id, guid) in table {
844 let mut entry = id.to_le_bytes().to_vec();
845 entry.extend_from_slice(guid);
846 group.push(node(0x24, None, &entry)?);
847 }
848 group.push(node(0x28, None, &[])?);
849 let mut override_crc = u32::MAX;
850 for (id, object) in objects {
851 let mut declaration = compact(id, table)?.to_vec();
852 override_crc = crc(
853 override_crc,
854 &object.reference_count.to_le_bytes(),
855 FileType::Section,
856 );
857 match object.data {
858 ObjectData::File {
859 reference,
860 extension,
861 } => {
862 declaration.extend_from_slice(&object.jcid.to_le_bytes());
863 declaration.extend_from_slice(&object.reference_count.to_le_bytes());
864 for bytes in [reference, extension] {
865 declaration.extend_from_slice(
866 &u32::try_from(bytes.len() / 2).unwrap().to_le_bytes(),
867 );
868 declaration.extend_from_slice(bytes);
869 }
870 group.push(node(0x73, None, &declaration)?);
871 }
872 ObjectData::Properties(bytes) => {
873 let references = object.references()?;
874 let flags = u8::from(!references.objects.is_empty())
875 | (u8::from(
876 !references.object_spaces.is_empty()
877 || !references.contexts.is_empty(),
878 ) << 1);
879 let data = if toc_table.is_some() {
880 let mut mapped = bytes.to_vec();
881 for property in PropertySets::parse(bytes)?.sets.iter().flatten() {
882 if let Value::References { compact_ids, .. } = property.value {
883 let offset =
884 compact_ids.as_ptr().addr() - bytes.as_ptr().addr();
885 for (i, value) in compact_ids.chunks_exact(4).enumerate() {
886 let value = u32::from_le_bytes(value.try_into().unwrap());
887 let target = ExGuid {
888 guid: object.global_ids[&(value >> 8)],
889 n: value & 255,
890 };
891 mapped[offset + i * 4..offset + i * 4 + 4]
892 .copy_from_slice(&compact(target, table)?);
893 }
868894 }
869895 }
870 }
871 append(&mut output, &mapped)?
872 } else if replacements.contains_key(&id) {
873 append(&mut output, bytes)?
874 } else {
875 Chunk {
876 offset: u64::try_from(bytes.as_ptr().addr() - source.as_ptr().addr())
896 append(&mut output, &mapped)?
897 } else if replacements.contains_key(&id) {
898 append(&mut output, bytes)?
899 } else {
900 Chunk {
901 offset: u64::try_from(
902 bytes.as_ptr().addr() - source.as_ptr().addr(),
903 )
877904 .unwrap(),
878 length: u64::try_from(bytes.len()).unwrap(),
879 }
880 };
881 if is_section {
882 declaration.extend_from_slice(&object.jcid.to_le_bytes());
883 declaration.push(flags);
884 } else if checkpoint {
885 let body = 1_u64 | (u64::from(flags & 1) << 16);
886 declaration.extend_from_slice(&body.to_le_bytes()[..6]);
887 } else {
888 declaration.extend_from_slice(&u32::from(flags).to_le_bytes());
889 }
890 declaration.extend_from_slice(&object.reference_count.to_le_bytes());
891 let readonly = object.jcid & 0x100000 != 0;
892 if readonly {
893 declaration.extend_from_slice(&md5::compute(bytes).0);
894 }
895 group.push(node(
905 length: u64::try_from(bytes.len()).unwrap(),
906 }
907 };
896908 if is_section {
897 if readonly { 0xc5 } else { 0xa5 }
909 declaration.extend_from_slice(&object.jcid.to_le_bytes());
910 declaration.push(flags);
898911 } else if checkpoint {
899 0x2e
912 let body = 1_u64 | (u64::from(flags & 1) << 16);
913 declaration.extend_from_slice(&body.to_le_bytes()[..6]);
900914 } else {
901 0x42
902 },
903 Some(Reference::Data(data)),
904 &declaration,
905 )?);
906 }
907 ObjectData::Encrypted(_) => {
908 return Err(Error {
909 offset: 0,
910 message: "Encrypted objects cannot be checkpointed",
911 });
915 declaration.extend_from_slice(&u32::from(flags).to_le_bytes());
916 }
917 declaration.extend_from_slice(&object.reference_count.to_le_bytes());
918 let readonly = object.jcid & 0x100000 != 0;
919 if readonly {
920 declaration.extend_from_slice(&md5::compute(bytes).0);
921 }
922 group.push(node(
923 if is_section {
924 if readonly { 0xc5 } else { 0xa5 }
925 } else if checkpoint {
926 0x2e
927 } else {
928 0x42
929 },
930 Some(Reference::Data(data)),
931 &declaration,
932 )?);
933 }
934 ObjectData::Encrypted(_) => {
935 return Err(Error {
936 offset: 0,
937 message: "Encrypted objects cannot be checkpointed",
938 });
939 }
912940 }
913941 }
914 }
915 if is_section {
916 group.push(node(0xb8, None, &[])?);
917 let group_id = u32::try_from(group_counts.len())
918 .ok()
919 .and_then(|n| n.checked_add(1))
920 .and_then(|n| maximum.checked_add(n))
921 .ok_or(Error {
922 offset: 0,
923 message: "File-node list identities are exhausted",
924 })?;
925 let chunk = append_list(&mut output, group_id, &group)?;
926 group_counts.push((group_id, group.len()));
927 manifest.push(node(0xb0, Some(Reference::NodeList(chunk)), &payload)?);
928 let mut overrides = vec![0; 8];
929 overrides.extend_from_slice(&(!override_crc).to_le_bytes());
930 manifest.push(node(
931 0x84,
932 Some(Reference::Data(Chunk {
933 offset: u64::MAX,
934 length: 0,
935 })),
936 &overrides,
937 )?);
938 } else {
939 manifest.extend(group);
940 }
941 }
942 if checkpoint {
943 for (role, id) in &revision.roots {
944 let mut payload = Vec::new();
945942 if is_section {
946 id.encode(&mut payload);
943 group.push(node(0xb8, None, &[])?);
944 let group_id = u32::try_from(counts.len())
945 .ok()
946 .and_then(|n| n.checked_add(1))
947 .and_then(|n| maximum.checked_add(n))
948 .ok_or(Error {
949 offset: 0,
950 message: "File-node list identities are exhausted",
951 })?;
952 let chunk = append_list(&mut output, group_id, &group)?;
953 counts.push((group_id, group.len()));
954 manifest.push(node(0xb0, Some(Reference::NodeList(chunk)), &payload)?);
955 let mut overrides = vec![0; 8];
956 overrides.extend_from_slice(&(!override_crc).to_le_bytes());
957 manifest.push(node(
958 0x84,
959 Some(Reference::Data(Chunk {
960 offset: u64::MAX,
961 length: 0,
962 })),
963 &overrides,
964 )?);
947965 } else {
948 payload.extend_from_slice(&compact(*id, toc_table.as_ref().unwrap())?);
966 manifest.extend(group);
949967 }
950 payload.extend_from_slice(&role.to_le_bytes());
951 manifest.push(node(if is_section { 0x5a } else { 0x59 }, None, &payload)?);
952968 }
953 }
954 manifest.push(node(0x1c, None, &[])?);
955 let root = store.list(store.header.root)?;
956 let space_node = root
957 .nodes
958 .iter()
959 .find(|node| node.id == 8 && node.fields(&store).exguid() == Ok(space))
960 .ok_or(Error {
961 offset: 0,
962 message: "Object space is absent from the root list",
969 if checkpoint {
970 for (role, id) in &revision.roots {
971 let mut payload = Vec::new();
972 if is_section {
973 id.encode(&mut payload);
974 } else {
975 payload.extend_from_slice(&compact(*id, toc_table.as_ref().unwrap())?);
976 }
977 payload.extend_from_slice(&role.to_le_bytes());
978 manifest.push(node(if is_section { 0x5a } else { 0x59 }, None, &payload)?);
979 }
980 }
981 manifest.push(node(0x1c, None, &[])?);
982 let root = store.list(store.header.root)?;
983 let space_node = root
984 .nodes
985 .iter()
986 .find(|node| node.id == 8 && node.fields(&store).exguid() == Ok(space))
987 .ok_or(Error {
988 offset: 0,
989 message: "Object space is absent from the root list",
990 })?;
991 let space_list = space_node.referenced_list(&store)?;
992 let revision_node = space_list.iter().rfind(|node| node.id == 0x10).unwrap();
993 let Some(Reference::NodeList(manifest_reference)) = revision_node.reference else {
994 unreachable!()
995 };
996 let revision_list = store.list(manifest_reference)?;
997 let last_fragment = *revision_list.fragments.last().unwrap();
998 let list_start = usize::try_from(last_fragment.offset).unwrap();
999 let list_id =
1000 u32::from_le_bytes(source[list_start + 8..list_start + 12].try_into().unwrap());
1001 let manifest_chunk = append_list(&mut output, list_id, &manifest)?;
1002 let manifest_start = usize::try_from(manifest_chunk.offset).unwrap();
1003 let sequence = u32::try_from(revision_list.fragments.len()).map_err(|_| Error {
1004 offset: list_start,
1005 message: "File-node fragment sequences are exhausted",
9631006 })?;
964 let space_list = space_node.referenced_list(&store)?;
965 let revision_node = space_list.iter().rfind(|node| node.id == 0x10).unwrap();
966 let Some(Reference::NodeList(manifest_reference)) = revision_node.reference else {
967 unreachable!()
968 };
969 let revision_list = store.list(manifest_reference)?;
970 let last_fragment = *revision_list.fragments.last().unwrap();
971 let list_start = usize::try_from(last_fragment.offset).unwrap();
972 let list_id = u32::from_le_bytes(source[list_start + 8..list_start + 12].try_into().unwrap());
973 let manifest_chunk = append_list(&mut output, list_id, &manifest)?;
974 let manifest_start = usize::try_from(manifest_chunk.offset).unwrap();
975 let sequence = u32::try_from(revision_list.fragments.len()).map_err(|_| Error {
976 offset: list_start,
977 message: "File-node fragment sequences are exhausted",
978 })?;
979 output[manifest_start + 12..manifest_start + 16].copy_from_slice(&sequence.to_le_bytes());
980 let last_node = revision_list.nodes.last().unwrap();
981 let node_header = u32::from_le_bytes(
982 source[last_node.offset..last_node.offset + 4]
983 .try_into()
984 .unwrap(),
985 );
986 let nodes_end = last_node.offset + usize::try_from((node_header >> 10) & 0x1fff).unwrap();
987 let tail = list_start + usize::try_from(last_fragment.length).unwrap() - 20;
988 if tail - nodes_end >= 4 {
989 output[nodes_end..nodes_end + 4].copy_from_slice(&node(0xff, None, &[])?);
1007 output[manifest_start + 12..manifest_start + 16].copy_from_slice(&sequence.to_le_bytes());
1008 let last_node = revision_list.nodes.last().unwrap();
1009 let node_header = u32::from_le_bytes(
1010 source[last_node.offset..last_node.offset + 4]
1011 .try_into()
1012 .unwrap(),
1013 );
1014 let nodes_end = last_node.offset + usize::try_from((node_header >> 10) & 0x1fff).unwrap();
1015 let tail = list_start + usize::try_from(last_fragment.length).unwrap() - 20;
1016 if tail - nodes_end >= 4 {
1017 output[nodes_end..nodes_end + 4].copy_from_slice(&node(0xff, None, &[])?);
1018 }
1019 output[tail..tail + 8].copy_from_slice(&manifest_chunk.offset.to_le_bytes());
1020 output[tail + 8..tail + 12].copy_from_slice(&(manifest_chunk.length as u32).to_le_bytes());
1021
1022 counts.push((list_id, revision_list.nodes.len() + manifest.len()));
1023 }
1024 if counts.is_empty() {
1025 return Ok(source.to_vec());
9901026 }
991 output[tail..tail + 8].copy_from_slice(&manifest_chunk.offset.to_le_bytes());
992 output[tail + 8..tail + 12].copy_from_slice(&(manifest_chunk.length as u32).to_le_bytes());
9931027
9941028 let transactions = store.header.transaction_count.checked_add(1).ok_or(Error {
9951029 offset: 96,
......@@ -999,10 +1033,7 @@ pub(crate) fn write_revision(
9991033 let commit_byte = (31 - changed_bytes.leading_zeros()) / 8;
10001034 let ceiling = transactions | ((1_u32 << (commit_byte * 8)) - 1);
10011035 let mut entries = Vec::new();
1002 for (id, count) in group_counts
1003 .into_iter()
1004 .chain([(list_id, revision_list.nodes.len() + manifest.len())])
1005 {
1036 for (id, count) in counts {
10061037 entries.extend_from_slice(&id.to_le_bytes());
10071038 entries.extend_from_slice(
10081039 &u32::try_from(count)
......@@ -1088,6 +1119,6 @@ pub(crate) fn write_revision(
10881119 output[228..236].copy_from_slice(&generation.to_le_bytes());
10891120 let written = Store::parse(&output)?;
10901121 let written_index = RevisionIndex::parse(&written)?;
1091 written_index.resolve(space, new_rid)?.reachable()?;
1122 written_index.validate_current()?;
10921123 Ok(output)
10931124}
crates/onestore/src/write/tests.rs+68
......@@ -1,6 +1,74 @@
11use super::patch_properties;
22use crate::{PropertySets, Value, create::properties};
33
4mod multi;
5
6#[test]
7fn copying_reference_arrays_remaps_every_entry_in_each_stream() {
8 use super::PropertyObject;
9 use std::{collections::BTreeMap, sync::Arc};
10 for kind in [9_u32, 11, 13] {
11 for count in [0_u32, 1, 2, 3, 4, 8, 33] {
12 for nested in [false, true] {
13 let property = (kind << 26) | 1;
14 let references: Vec<_> = (0..count)
15 .flat_map(|i| (((i % 3) << 8) | i).to_le_bytes())
16 .collect();
17 let mut bytes = properties(&[(property, references)]).unwrap();
18 let selected = if nested {
19 let parsed = PropertySets::parse(&bytes).unwrap();
20 let offset = parsed.root_ids.as_ptr().addr() - bytes.as_ptr().addr() - 2;
21 bytes.truncate(bytes.len() - parsed.padding.len());
22 bytes.splice(offset..offset, [1, 0, 2, 0, 0, 0x44]);
23 bytes.resize(bytes.len().next_multiple_of(8), 0);
24 0x44000002
25 } else {
26 property
27 };
28 let source = PropertyObject {
29 jcid: 0x6000e,
30 bytes,
31 global_ids: Arc::new(BTreeMap::from([
32 (0, [1; 16]),
33 (1, [2; 16]),
34 (2, [3; 16]),
35 ])),
36 };
37 let mut target = PropertyObject {
38 jcid: 0x6000e,
39 bytes: properties(&[(0x14000003, 47_u32.to_le_bytes().to_vec())]).unwrap(),
40 global_ids: Arc::new(BTreeMap::from([
41 (0, [3; 16]),
42 (1, [9; 16]),
43 (2, [1; 16]),
44 ])),
45 };
46 target.copy_property(&source, selected).unwrap();
47 let parsed = PropertySets::parse(&target.bytes).unwrap();
48 assert_eq!(parsed.sets[0][0].value, Value::Bytes(&47_u32.to_le_bytes()));
49 let copied = parsed
50 .sets
51 .iter()
52 .flatten()
53 .find(|p| p.id == property)
54 .unwrap();
55 let Value::References { compact_ids, .. } = copied.value else {
56 panic!()
57 };
58 assert_eq!(compact_ids.len(), count as usize * 4);
59 for (i, compact) in compact_ids.chunks_exact(4).enumerate() {
60 let compact = u32::from_le_bytes(compact.try_into().unwrap());
61 assert_eq!(compact & 255, i as u32);
62 assert_eq!(target.global_ids[&(compact >> 8)], [(i % 3 + 1) as u8; 16]);
63 }
64 let before = target.bytes.clone();
65 target.copy_property(&source, selected).unwrap();
66 assert_eq!(target.bytes, before);
67 }
68 }
69 }
70}
71
472#[test]
573fn document_insertions_and_formatting_respect_readonly_ancestors() {
674 use super::{PropertyObject, write_revision};
crates/onestore/src/write/tests/multi.rs created+302
......@@ -0,0 +1,302 @@
1use crate::{
2 ExGuid, RevisionIndex, Store,
3 document::{Document, Kind},
4 write::{PropertyObject, write_revisions},
5};
6use std::collections::BTreeMap;
7
8mod disk {
9 use crate as onestore;
10 include!("../../../tests/support/disk.rs");
11}
12
13mod current {
14 use crate as onestore;
15 include!("../../../tests/support/current.rs");
16}
17
18const SOURCE: &[u8] =
19 include_bytes!("../../../../../corpus/page-lifecycle/03-renamed/notebook/Lifecycle.one");
20
21fn nest(source: &[u8]) -> Vec<u8> {
22 write_revisions(source, |index| {
23 let document = Document::parse(index)?;
24 let pages = document.pages()?;
25 assert_eq!(pages.len(), 9);
26 let section = index.resolve(
27 index.root,
28 index.spaces[&index.root].labels[&(ExGuid::default(), 1)],
29 )?;
30 let view = &document.spaces[&index.root];
31 let view = &view.revisions[&view.contexts[&ExGuid::default()]];
32 let root = section.roots[&1];
33 let parent = *view.nodes[&root]
34 .children
35 .iter()
36 .find(|id| view.nodes[id].spaces.contains(&pages[3].0))
37 .unwrap();
38 let mut series = PropertyObject::from_object(&section.objects[&parent])?;
39 let mut spaces = Vec::new();
40 let mut metadata = Vec::new();
41 let mut replacements = BTreeMap::new();
42 let mut changes = BTreeMap::new();
43 for (ordinal, (sid, page)) in pages[3..6].iter().enumerate() {
44 spaces.extend_from_slice(&series.reference(*sid)?);
45 let raw = index.resolve(*sid, index.spaces[sid].labels[&(ExGuid::default(), 1)])?;
46 let id = raw.roots[&2];
47 let mut object = PropertyObject::from_object(&raw.objects[&id])?;
48 object.set(&[(0x14001dff, &((ordinal + 1) as u32).to_le_bytes())])?;
49 changes.insert(*sid, BTreeMap::from([(id, object)]));
50 let mut copy = PropertyObject::from_object(&raw.objects[&id])?;
51 copy.set(&[(0x14001dff, &((ordinal + 1) as u32).to_le_bytes())])?;
52 let mut copy_id = ExGuid {
53 guid: [
54 0x31, 0xc0, 0xa8, 0x22, 0, 0x36, 0xee, 0x42, 0xb7, 0x14, 0xd7, 0xac, 0xda,
55 0x24, 0x35, 0xe8,
56 ],
57 n: 1,
58 };
59 for (value, salt) in copy_id.guid.iter_mut().zip(page.guid) {
60 *value ^= salt;
61 }
62 copy.reference(copy_id)?;
63 metadata.extend_from_slice(&series.reference(copy_id)?);
64 replacements.insert(copy_id, copy);
65 }
66 let links = PropertyObject {
67 jcid: series.jcid,
68 bytes: crate::create::properties(&[(0x2c001d63, spaces)])?,
69 global_ids: std::sync::Arc::clone(&series.global_ids),
70 };
71 series
72 .copy_property(&links, 0x2c001d63)
73 .expect("copy spaces");
74 series
75 .set(&[(0x24003442, &metadata)])
76 .expect("set metadata links");
77 replacements.insert(parent, series);
78 let mut object = PropertyObject::from_object(&section.objects[&root])?;
79 let mut children = Vec::new();
80 for id in &view.nodes[&root].children {
81 if *id == parent
82 || !view.nodes[id]
83 .spaces
84 .iter()
85 .any(|sid| [pages[4].0, pages[5].0].contains(sid))
86 {
87 children.extend_from_slice(&object.reference(*id)?);
88 }
89 }
90 object.set(&[(0x24001c20, &children)])?;
91 replacements.insert(root, object);
92 changes.insert(index.root, replacements);
93 Ok(changes)
94 })
95 .unwrap()
96}
97
98#[test]
99fn nesting_publishes_section_order_and_page_levels_in_one_transaction() {
100 let written = nest(SOURCE);
101 let before = Store::parse(SOURCE).unwrap();
102 let after = Store::parse(&written).unwrap();
103 assert_eq!(
104 after.header.transaction_count,
105 before.header.transaction_count + 1
106 );
107 assert_eq!(after.header.generation, before.header.generation + 1);
108 let old = RevisionIndex::parse(&before).unwrap();
109 let new = RevisionIndex::parse(&after).unwrap();
110 let document = Document::parse(&new).unwrap();
111 let pages = document.pages().unwrap();
112 assert_eq!(pages, Document::parse(&old).unwrap().pages().unwrap());
113 let mut changed_spaces = 0;
114 for (sid, space) in &old.spaces {
115 changed_spaces += usize::from(space.labels != new.spaces[sid].labels);
116 for rid in space.revisions.keys() {
117 assert_eq!(
118 format!("{:?}", old.resolve(*sid, *rid).unwrap()),
119 format!("{:?}", new.resolve(*sid, *rid).unwrap())
120 );
121 }
122 }
123 assert_eq!(changed_spaces, 3);
124 for (ordinal, (sid, _)) in pages.iter().enumerate() {
125 let view = &document.spaces[sid];
126 let view = &view.revisions[&view.contexts[&ExGuid::default()]];
127 let Kind::Metadata { level, .. } = view.nodes[&view.roots[&2]].kind else {
128 panic!()
129 };
130 assert_eq!(
131 level,
132 Some(match ordinal {
133 4 => 2,
134 5 => 3,
135 _ => 1,
136 })
137 );
138 let old_view = old
139 .resolve(*sid, old.spaces[sid].labels[&(ExGuid::default(), 1)])
140 .unwrap();
141 let new_view = new
142 .resolve(*sid, new.spaces[sid].labels[&(ExGuid::default(), 1)])
143 .unwrap();
144 for (id, object) in &old_view.objects {
145 if *id != old_view.roots[&2] {
146 assert_eq!(format!("{object:?}"), format!("{:?}", new_view.objects[id]));
147 }
148 }
149 }
150 let root = &document.spaces[&document.root];
151 let root = &root.revisions[&root.contexts[&ExGuid::default()]];
152 assert_eq!(
153 root.nodes[&root.roots[&1]]
154 .children
155 .iter()
156 .map(|id| root.nodes[id].spaces.len())
157 .collect::<Vec<_>>(),
158 [1, 1, 1, 3, 1, 1, 1]
159 );
160 assert_eq!(nest(&written), written);
161 if let Some(output) = std::env::var_os("ONESTORE_PAGE_BATCH_OUTPUT") {
162 std::fs::create_dir(&output).unwrap();
163 std::fs::write(
164 std::path::Path::new(&output).join("Lifecycle.one"),
165 &written,
166 )
167 .unwrap();
168 }
169}
170
171#[test]
172fn interrupted_multi_space_publication_never_exposes_a_partial_nesting() {
173 let written = nest(SOURCE);
174 let old = current::current(SOURCE);
175 let new = current::current(&written);
176 assert_ne!(old, new);
177 for write_limit in [17, 4096] {
178 let mut complete = disk::Disk {
179 visible: SOURCE.to_vec(),
180 durable: SOURCE.to_vec(),
181 operation: 0,
182 fail_at: None,
183 write_limit,
184 random: 1951,
185 };
186 crate::commit::commit_bytes(&mut complete, SOURCE, &written).unwrap();
187 assert_eq!(complete.durable, written);
188 for fail_at in 1..=complete.operation {
189 let mut interrupted = disk::Disk {
190 visible: SOURCE.to_vec(),
191 durable: SOURCE.to_vec(),
192 operation: 0,
193 fail_at: Some(fail_at),
194 write_limit,
195 random: 1951 + fail_at as u64,
196 };
197 crate::commit::commit_bytes(&mut interrupted, SOURCE, &written).unwrap_err();
198 let observed = current::current(&interrupted.durable);
199 assert!(
200 observed == old || observed == new,
201 "{write_limit}:{fail_at}"
202 );
203 }
204 }
205}
206
207#[test]
208fn empty_batches_and_invalid_spaces_do_not_produce_an_edit() {
209 assert_eq!(
210 write_revisions(SOURCE, |_| Ok(BTreeMap::new())).unwrap(),
211 SOURCE
212 );
213 assert!(
214 write_revisions(SOURCE, |_| Ok(BTreeMap::from([(
215 ExGuid::default(),
216 BTreeMap::new()
217 )])))
218 .is_err()
219 );
220}
221
222#[test]
223fn repeated_multi_space_edits_cross_counter_carries_and_checkpoint_each_space() {
224 let mut source = crate::create_section("batch.one", "Preserved text", "Author").unwrap();
225 let initial = source.clone();
226 let initial_store = Store::parse(&initial).unwrap();
227 let initial_index = RevisionIndex::parse(&initial_store).unwrap();
228 let mut checkpoints = BTreeMap::new();
229 for step in 1_u32..=514 {
230 let written = write_revisions(&source, |index| {
231 let mut changes = BTreeMap::new();
232 for (sid, space) in &index.spaces {
233 let raw = index.resolve(*sid, space.labels[&(ExGuid::default(), 1)])?;
234 let (id, property) = if *sid == index.root {
235 (raw.roots[&2], 0x14001cbe)
236 } else {
237 (
238 *raw.objects
239 .iter()
240 .find(|(_, object)| object.jcid == 0x6000b)
241 .unwrap()
242 .0,
243 0x14001d7a,
244 )
245 };
246 let mut object = PropertyObject::from_object(&raw.objects[&id])?;
247 object.set(&[(property, &step.to_le_bytes())])?;
248 changes.insert(*sid, BTreeMap::from([(id, object)]));
249 }
250 Ok(changes)
251 })
252 .unwrap();
253 let store = Store::parse(&written).unwrap();
254 assert_eq!(
255 store.header.transaction_count,
256 initial_store.header.transaction_count + step
257 );
258 assert!(store.checksum_mismatches.is_empty());
259 let index = RevisionIndex::parse(&store).unwrap();
260 for (sid, space) in &index.spaces {
261 let rid = space.labels[&(ExGuid::default(), 1)];
262 if space.revisions[&rid].dependency.is_none() {
263 *checkpoints.entry(*sid).or_insert(0) += 1;
264 }
265 for old_rid in initial_index.spaces[sid].revisions.keys() {
266 assert_eq!(
267 format!("{:?}", initial_index.resolve(*sid, *old_rid).unwrap()),
268 format!("{:?}", index.resolve(*sid, *old_rid).unwrap())
269 );
270 }
271 }
272 if step == 1 || store.header.transaction_count == 256 || step == 512 {
273 let old = current::current(&source);
274 let new = current::current(&written);
275 let mut complete = disk::Disk {
276 visible: source.clone(),
277 durable: source.clone(),
278 operation: 0,
279 fail_at: None,
280 write_limit: 4096,
281 random: 1952,
282 };
283 crate::commit::commit_bytes(&mut complete, &source, &written).unwrap();
284 for fail_at in 1..=complete.operation {
285 let mut interrupted = disk::Disk {
286 visible: source.clone(),
287 durable: source.clone(),
288 operation: 0,
289 fail_at: Some(fail_at),
290 write_limit: 4096,
291 random: 1952 + fail_at as u64,
292 };
293 crate::commit::commit_bytes(&mut interrupted, &source, &written).unwrap_err();
294 let observed = current::current(&interrupted.durable);
295 assert!(observed == old || observed == new, "{step}:{fail_at}");
296 }
297 }
298 source = written;
299 }
300 assert_eq!(checkpoints.len(), 2);
301 assert!(checkpoints.values().all(|count| *count == 1));
302}
tools/test_page_lifecycle.py+41
......@@ -56,6 +56,47 @@ class PageLifecycleTest(unittest.TestCase):
5656 for oid in section['nodes'][section['roots']['1']]['children']])
5757 self.assertEqual(groups[0], groups[1])
5858
59 def test_atomic_rust_nesting_matches_native_and_survives_cold_reopen(self):
60 fixture = FIXTURE / 'atomic-nesting'
61 self.assertEqual((fixture / 'candidate/Lifecycle.one').read_bytes()[1024:],
62 (fixture / 'cold/notebook/Lifecycle.one').read_bytes()[1024:])
63 with TemporaryDirectory() as temporary:
64 native_read = Path(temporary) / 'native/read'
65 cold_read = Path(temporary) / 'cold/read'
66 shutil.copytree(FIXTURE / '04-nested/read', native_read)
67 shutil.copytree(fixture / 'cold/read', cold_read)
68 models = []
69 for ordinal, source in enumerate((fixture / 'candidate', fixture / 'cold/notebook')):
70 output = Path(temporary) / str(ordinal)
71 subprocess.run([EXPORTER, source / 'Lifecycle.one', output], check=True)
72 models.append(json.loads((output / 'document.json').read_text()))
73 compare(source, native_read)
74 compare(source, cold_read)
75 pages = [list(ordered_pages(model)) for model in models]
76 expected = list(ordered_pages(self.models['04-nested', '']))
77 original = list(ordered_pages(self.models['03-renamed', '']))
78 self.assertEqual(len(pages[0]), 9)
79 for before, native, candidate, cold in zip(original, expected, *pages, strict=True):
80 sid, _, revision, page = candidate
81 self.assertEqual((sid, page), (before[0], before[3]))
82 self.assertEqual((sid, page), (cold[0], cold[3]))
83 self.assertEqual(dict(walk(revision, page)), dict(walk(before[2], before[3])))
84 self.assertEqual(revision['nodes'], cold[2]['nodes'])
85 self.assertEqual(revision['roots'], cold[2]['roots'])
86 metadata = revision['nodes'][revision['roots']['2']]['kind']
87 self.assertEqual(metadata, native[2]['nodes'][native[2]['roots']['2']]['kind'])
88 groups = []
89 for model in (self.models['04-nested', ''], *models):
90 _, revision = view(model, model['root'])
91 groups.append([revision['nodes'][oid]['spaces']
92 for oid in revision['nodes'][revision['roots']['1']]['children']])
93 self.assertEqual(groups[0], groups[1])
94 self.assertEqual(groups[1], groups[2])
95 _, candidate = view(models[0], models[0]['root'])
96 _, cold = view(models[1], models[1]['root'])
97 self.assertEqual(candidate['nodes'], cold['nodes'])
98 self.assertEqual(candidate['roots'], cold['roots'])
99
59100 def test_native_page_styles_titles_order_and_parent_deletion(self):
60101 phases = {phase: list(ordered_pages(self.models[phase, ''])) for phase in PHASES}
61102 initial = [sid for sid, _, _, _ in phases['02-authored']]