authorgravatar for git@paperclover.netclover caruso <git@paperclover.net> 2026-09-08 00:02:13-07:00
committergravatar for git@paperclover.netclover caruso <git@paperclover.net> 2026-09-08 00:02:13-07:00
log93784cadb44ec833ca6d7f3063c5d87255c1d6af
treef302efb1f9d5fdea8f4672603c3c3ab3821651da
parent49bc9b2d95d93ceb9f25de3e238e89ce6fbfb21e
signature Signed by SSH key SHA256:52mNGHRsVFBDED9IAX5pe+LRWUefqTbxEReunq21QvU

feat: replace text across ordinary formatting runs

Preserve surviving style references and terminal typing style while replacing scalar-aligned UTF-16 ranges. Reject affected fields and preserved run data before publication; reuse existing offline text intents after rebasing onto remote styles. Validate exhaustive ranges, history preservation, interrupted commits and lost replies. Retain six independent native fixtures and extend the twelve-client native/Rust workload to insertion, formatting and cross-run replacement. Checks: 180 Rust tests, eight doctests, 126 Python tests, Clippy and formatting; 7,987 ASan executions; 1,280 mixed editing intents through an outage with independent cold OneNote verification. Assisted-by: gpt-6-astra

82 files changed, 1018 insertions(+), 131 deletions(-)

corpus/cross-run-edit/README.md created+20
...@@ -0,0 +1,20 @@
1# Cross-run text replacements
2
3Six edits of the public formatted-insertion notebooks, independently opened by
4OneNote 2010 in disposable cold-cache machines. Cases cover partial and complete
5replacement, a native table cell, deletion, typing after deletion, and insertion
6at a style boundary. Each directory is a separate notebook because the source
7section identities can be shared.
8
9The inserted characters inherit the style at the start of the replaced range;
10surviving characters retain their styles. Empty final runs retain their typing
11style. These are library edit semantics; the native captures check that OneNote
12reads the resulting content and formatting correctly.
13
14Generate candidates with the ignored `export_native_cross_run_edits` test and
15`ONESTORE_CROSS_RUN_OUTPUT` set to a new absolute directory. Capture each candidate
16with `tools/native_runner.py --expected-pages 1 --collect-notebook` and compare
17with `tools/verify-document.py`. The manifest records source paths, target IDs,
18UTF-16 ranges and file hashes. The attachment export links to its canonical copy.
19`tools/test_cross_run_edit.py` checks the retained
20native oracles without starting a VM.
corpus/cross-run-edit/boundary/candidate/synthetic.one created
Binary files /dev/null and b/corpus/cross-run-edit/boundary/candidate/synthetic.one differ
corpus/cross-run-edit/boundary/cold/notebook/Open Notebook.onetoc2 created
Binary files /dev/null and b/corpus/cross-run-edit/boundary/cold/notebook/Open Notebook.onetoc2 differ
corpus/cross-run-edit/boundary/cold/notebook/synthetic.one created
Binary files /dev/null and b/corpus/cross-run-edit/boundary/cold/notebook/synthetic.one differ
corpus/cross-run-edit/boundary/cold/read/environment.json created+7
...@@ -0,0 +1,7 @@
1{
2 "powershell": "5.1.14409.1005",
3 "schema": "xs2010",
4 "hostname": "ONE-M6-107F0FF0",
5 "cold": true,
6 "onenote": "14.0.4763.1000"
7}
corpus/cross-run-edit/boundary/cold/read/hierarchy.xml created+2
...@@ -0,0 +1,2 @@
1<?xml version="1.0"?>
2<one:Notebook xmlns:one="http://schemas.microsoft.com/office/onenote/2010/onenote" name="notebook" nickname="notebook" ID="{2055F8D1-7921-41C1-9C97-7A20D9C33AB3}{1}{B0}" path="C:\one-tests\runs\capture\notebook" lastModifiedTime="2026-09-08T07:07:16.000Z" color="#B49EDE"><one:Section name="synthetic" ID="{22E2CA5A-AB75-0C2F-27EE-1002E7557596}{1}{B0}" path="C:\one-tests\runs\capture\notebook\synthetic.one" lastModifiedTime="2026-09-08T07:07:16.000Z" color="#8AA8E4"><one:Page ID="{0EAB39DC-22BE-4F3E-A089-B8931B6D6989}{1}{B0}" name="BoldMiddleitalic é color 東京" dateTime="2026-09-05T05:06:54.000Z" lastModifiedTime="2026-09-08T07:07:16.000Z" pageLevel="1"/></one:Section></one:Notebook>
corpus/cross-run-edit/boundary/cold/read/page-000.xml created+12
...@@ -0,0 +1,12 @@
1<?xml version="1.0"?>
2<one:Page xmlns:one="http://schemas.microsoft.com/office/onenote/2010/onenote" ID="{0EAB39DC-22BE-4F3E-A089-B8931B6D6989}{1}{B0}" name="BoldMiddleitalic é color 東京" dateTime="2026-09-05T05:06:54.000Z" lastModifiedTime="2026-09-08T07:07:16.000Z" pageLevel="1" lang="en-US"><one:QuickStyleDef index="0" name="p" fontColor="automatic" highlightColor="automatic" font="Calibri" fontSize="11.0" spaceBefore="0.0" spaceAfter="0.0"/><one:PageSettings RTL="false" color="automatic"><one:PageSize><one:Automatic/></one:PageSize><one:RuleLines visible="false"/></one:PageSettings><one:Outline author="Rich author" lastModifiedBy="Rich author" lastModifiedTime="2026-09-08T07:02:08.000Z" objectID="{69679160-69CB-4C88-AF7B-3448E4F18CE8}{10}{B0}"><one:Position x="36.0" y="14.40000057220459" z="0"/><one:Size width="127.5136947631836" height="41.15543365478516"/><one:OEChildren><one:OE creationTime="2026-09-08T04:47:24.000Z" lastModifiedTime="2026-09-08T07:02:08.000Z" objectID="{6E023997-9AB6-0782-1F2F-F7422F9AFE1E}{1}{B0}" alignment="left" style="font-family:Calibri;font-size:11.0pt"><one:T><![CDATA[<span
3style='font-weight:bold;text-decoration:underline'>Bold</span>Middle<span
4style='font-style:italic;text-decoration:line-through'>italic</span> <span
5style='vertical-align:super'>é</span> <span style='color:#0C2238;background:
6yellow;mso-highlight:yellow'>color</span> <span style='vertical-align:sub'>東京</span><br>
7End]]></one:T></one:OE><one:OE author="snow" authorInitials="S" lastModifiedBy="snow" lastModifiedByInitials="S" creationTime="2026-09-05T05:06:55.000Z" lastModifiedTime="2026-09-05T05:06:56.000Z" objectID="{69679160-69CB-4C88-AF7B-3448E4F18CE8}{11}{B0}" alignment="left" quickStyleIndex="0" style="font-size:11.0pt;color:#1F4E79"><one:T><![CDATA[<span
8style='font-weight:bold;font-family:Calibri' lang=en-US>Fictitious: café, </span><span
9style='font-weight:bold;font-family:SimSun' lang=en-US>東京</span><span
10style='font-weight:bold;font-family:Calibri' lang=en-US>, </span><span
11style='font-weight:bold;font-family:Arial;direction:rtl;unicode-bidi:embed'
12lang=ar-SA>مرحبا</span>]]></one:T></one:OE></one:OEChildren></one:Outline></one:Page>
corpus/cross-run-edit/boundary/cold/read/payloads.json created+3
...@@ -0,0 +1,3 @@
1[
2
3]
\ No newline at end of file
corpus/cross-run-edit/boundary/cold/run.json created+18
...@@ -0,0 +1,18 @@
1{
2 "notebook": "/Users/clo/dev/one/evidence/m10/cross-run-native/boundary/candidate",
3 "expected_pages": 1,
4 "author": null,
5 "author_timeout_seconds": 600,
6 "inspect": false,
7 "collect_notebook": true,
8 "base": {
9 "file": "win7-office-base.qcow2",
10 "format": "qcow2",
11 "sha256": "a1a4f8fab782ee14885ff801ca2f6347c208fdcfc3513637096f314315c89346",
12 "virtual_size": 68719476736
13 },
14 "scripts": {
15 "cold.ps1": "c177fc72ae6c2634d186f5671a880de20b09f9716532c37c69cb13aa15c7e331",
16 "read.ps1": "04013bfcccee40a17e2a225f9b9e96f40a3a8daad9e350609654f8eda3ccbb41"
17 }
18}
corpus/cross-run-edit/boundary/cold/source.json created+8
...@@ -0,0 +1,8 @@
1[
2 {
3 "path": "synthetic.one",
4 "bytes": 20840,
5 "sha256": "a3fac402a10648e99d4bdb4b37b41c825e2b8b34f12d8b1c234e28bd2ec8ac0b",
6 "mtime_ns": 1788850928468116429
7 }
8]
corpus/cross-run-edit/boundary/cold/teardown.json created+1
...@@ -0,0 +1 @@
1{"absent": true}
corpus/cross-run-edit/clear/candidate/synthetic.one created
Binary files /dev/null and b/corpus/cross-run-edit/clear/candidate/synthetic.one differ
corpus/cross-run-edit/clear/cold/notebook/Open Notebook.onetoc2 created
Binary files /dev/null and b/corpus/cross-run-edit/clear/cold/notebook/Open Notebook.onetoc2 differ
corpus/cross-run-edit/clear/cold/notebook/synthetic.one created
Binary files /dev/null and b/corpus/cross-run-edit/clear/cold/notebook/synthetic.one differ
corpus/cross-run-edit/clear/cold/read/environment.json created+7
...@@ -0,0 +1,7 @@
1{
2 "powershell": "5.1.14409.1005",
3 "schema": "xs2010",
4 "hostname": "ONE-M6-8787CA35",
5 "cold": true,
6 "onenote": "14.0.4763.1000"
7}
corpus/cross-run-edit/clear/cold/read/hierarchy.xml created+2
...@@ -0,0 +1,2 @@
1<?xml version="1.0"?>
2<one:Notebook xmlns:one="http://schemas.microsoft.com/office/onenote/2010/onenote" name="notebook" nickname="notebook" ID="{DB8815C1-7E0F-4425-AF09-7274689810EC}{1}{B0}" path="C:\one-tests\runs\capture\notebook" lastModifiedTime="2026-09-08T07:07:14.000Z" color="#B49EDE"><one:Section name="synthetic" ID="{0858E4C2-01A9-068A-13A5-8233A106496B}{1}{B0}" path="C:\one-tests\runs\capture\notebook\synthetic.one" lastModifiedTime="2026-09-08T07:07:14.000Z" color="#8AA8E4"><one:Page ID="{24111744-8862-459B-94C2-2AA25D3E5574}{1}{B0}" name="Fictitious: café, 東京, مرحبا" dateTime="2026-09-05T05:06:54.000Z" lastModifiedTime="2026-09-08T07:07:14.000Z" pageLevel="1"/></one:Section></one:Notebook>
corpus/cross-run-edit/clear/cold/read/page-000.xml created+7
...@@ -0,0 +1,7 @@
1<?xml version="1.0"?>
2<one:Page xmlns:one="http://schemas.microsoft.com/office/onenote/2010/onenote" ID="{24111744-8862-459B-94C2-2AA25D3E5574}{1}{B0}" name="Fictitious: café, 東京, مرحبا" dateTime="2026-09-05T05:06:54.000Z" lastModifiedTime="2026-09-08T07:07:14.000Z" pageLevel="1" lang="en-US"><one:QuickStyleDef index="0" name="p" fontColor="automatic" highlightColor="automatic" font="Calibri" fontSize="11.0" spaceBefore="0.0" spaceAfter="0.0"/><one:PageSettings RTL="false" color="automatic"><one:PageSize><one:Automatic/></one:PageSize><one:RuleLines visible="false"/></one:PageSettings><one:Outline author="Rich author" lastModifiedBy="Rich author" lastModifiedTime="2026-09-08T07:02:08.000Z" objectID="{43DDBFF8-C317-462D-9B30-A679A2A2B015}{10}{B0}"><one:Position x="36.0" y="14.40000057220459" z="0"/><one:Size width="127.5136947631836" height="27.72771453857422"/><one:OEChildren><one:OE creationTime="2026-09-08T04:47:24.000Z" lastModifiedTime="2026-09-08T07:02:08.000Z" objectID="{44B8170F-306A-0D27-2B64-657369C9C2E3}{1}{B0}" alignment="left" style="font-family:Calibri;font-size:11.0pt"><one:T><![CDATA[]]></one:T></one:OE><one:OE author="snow" authorInitials="S" lastModifiedBy="snow" lastModifiedByInitials="S" creationTime="2026-09-05T05:06:55.000Z" lastModifiedTime="2026-09-05T05:06:56.000Z" objectID="{43DDBFF8-C317-462D-9B30-A679A2A2B015}{11}{B0}" alignment="left" quickStyleIndex="0" style="font-size:11.0pt;color:#1F4E79"><one:T><![CDATA[<span
3style='font-weight:bold;font-family:Calibri' lang=en-US>Fictitious: café, </span><span
4style='font-weight:bold;font-family:SimSun' lang=en-US>東京</span><span
5style='font-weight:bold;font-family:Calibri' lang=en-US>, </span><span
6style='font-weight:bold;font-family:Arial;direction:rtl;unicode-bidi:embed'
7lang=ar-SA>مرحبا</span>]]></one:T></one:OE></one:OEChildren></one:Outline></one:Page>
corpus/cross-run-edit/clear/cold/read/payloads.json created+3
...@@ -0,0 +1,3 @@
1[
2
3]
\ No newline at end of file
corpus/cross-run-edit/clear/cold/run.json created+18
...@@ -0,0 +1,18 @@
1{
2 "notebook": "/Users/clo/dev/one/evidence/m10/cross-run-native/clear/candidate",
3 "expected_pages": 1,
4 "author": null,
5 "author_timeout_seconds": 600,
6 "inspect": false,
7 "collect_notebook": true,
8 "base": {
9 "file": "win7-office-base.qcow2",
10 "format": "qcow2",
11 "sha256": "a1a4f8fab782ee14885ff801ca2f6347c208fdcfc3513637096f314315c89346",
12 "virtual_size": 68719476736
13 },
14 "scripts": {
15 "cold.ps1": "c177fc72ae6c2634d186f5671a880de20b09f9716532c37c69cb13aa15c7e331",
16 "read.ps1": "04013bfcccee40a17e2a225f9b9e96f40a3a8daad9e350609654f8eda3ccbb41"
17 }
18}
corpus/cross-run-edit/clear/cold/source.json created+8
...@@ -0,0 +1,8 @@
1[
2 {
3 "path": "synthetic.one",
4 "bytes": 21736,
5 "sha256": "d6d8c775b4b87e3b8379757773c313a0a720da52dd5e76b4799f2b8e16bb5320",
6 "mtime_ns": 1788850928454071185
7 }
8]
corpus/cross-run-edit/clear/cold/teardown.json created+1
...@@ -0,0 +1 @@
1{"absent": true}
corpus/cross-run-edit/manifest.json created+89
...@@ -0,0 +1,89 @@
1{
2 "generator": "crates/onestore/tests/edit.rs:export_native_cross_run_edits",
3 "cases": [
4 {
5 "name": "partial",
6 "object": "{04B13EBF-F611-4402-829D-C72E23221C8B},2",
7 "range": {
8 "end": 16,
9 "start": 2
10 },
11 "replacement": "\u4e2d\ud83e\udd80",
12 "retype": false,
13 "source": "formatted-insertion/native-paragraph/candidate/synthetic.one",
14 "space": "{64183EF4-4E19-0CBE-3D3B-88FF17D58B1C},1",
15 "candidate_sha256": "0a93fdb8c8378989a6522b5de803f073109b8972088e61afdf97639eeaea6a02",
16 "native_sha256": "85e90850f7f0582cd9e4e8f93bb8f77cd773d883cd993b24280b57bd70078a4f"
17 },
18 {
19 "name": "table",
20 "object": "{6928B2F5-7D36-44AF-BE3B-36CEBACE8657},2",
21 "range": {
22 "end": 24,
23 "start": 0
24 },
25 "replacement": "Across \ud83d\udc08 ",
26 "retype": false,
27 "source": "formatted-insertion/native-cell/candidate/synthetic.one",
28 "space": "{64183EF4-4E19-0CBE-3D3B-88FF17D58B1C},1",
29 "candidate_sha256": "89ce5a50f050c05ecbfb92c8c8eb0538bd041d0596290a5b974a9878e171e6e9",
30 "native_sha256": "561222f0ecb1ccc17592c9fd44f08f6f4e0b0bfc6120f3b099a902f46c3e8ed2"
31 },
32 {
33 "name": "clear",
34 "object": "{04B13EBF-F611-4402-829D-C72E23221C8B},2",
35 "range": {
36 "end": 30,
37 "start": 0
38 },
39 "replacement": "",
40 "retype": false,
41 "source": "formatted-insertion/native-paragraph/candidate/synthetic.one",
42 "space": "{64183EF4-4E19-0CBE-3D3B-88FF17D58B1C},1",
43 "candidate_sha256": "d6d8c775b4b87e3b8379757773c313a0a720da52dd5e76b4799f2b8e16bb5320",
44 "native_sha256": "30dfd26ad6912225f3e637f529f5135f67f45de0c1a0b705ad70bf97866a554e"
45 },
46 {
47 "name": "replace",
48 "object": "{04B13EBF-F611-4402-829D-C72E23221C8B},2",
49 "range": {
50 "end": 30,
51 "start": 0
52 },
53 "replacement": "Replacement e\u0301\ud83e\udd80",
54 "retype": false,
55 "source": "formatted-insertion/native-paragraph/candidate/synthetic.one",
56 "space": "{64183EF4-4E19-0CBE-3D3B-88FF17D58B1C},1",
57 "candidate_sha256": "4b0ae05431ade4777541592cd8040f7ff18dbfe4ada35a686cda963299988e81",
58 "native_sha256": "f0cca32ec56e15b7d19aa581dbe7ae3d59bd0cc498450a0576ae9d168365c377"
59 },
60 {
61 "name": "retype",
62 "object": "{04B13EBF-F611-4402-829D-C72E23221C8B},2",
63 "range": {
64 "end": 30,
65 "start": 0
66 },
67 "replacement": "",
68 "retype": true,
69 "source": "formatted-insertion/native-paragraph/candidate/synthetic.one",
70 "space": "{64183EF4-4E19-0CBE-3D3B-88FF17D58B1C},1",
71 "candidate_sha256": "0d37a5a762e9b08f40ea47b2fad48591fcb5f782d1a130f5363f5116b1175fb3",
72 "native_sha256": "7b70f0ebe27f09c31a623f27b37ffac4362f9513dd26b2d1a69d1e2c0a38c15a"
73 },
74 {
75 "name": "boundary",
76 "object": "{04B13EBF-F611-4402-829D-C72E23221C8B},2",
77 "range": {
78 "end": 8,
79 "start": 4
80 },
81 "replacement": "Middle",
82 "retype": false,
83 "source": "formatted-insertion/native-paragraph/candidate/synthetic.one",
84 "space": "{64183EF4-4E19-0CBE-3D3B-88FF17D58B1C},1",
85 "candidate_sha256": "a3fac402a10648e99d4bdb4b37b41c825e2b8b34f12d8b1c234e28bd2ec8ac0b",
86 "native_sha256": "e9852bed7545600f4fec537be8e614a1cdb4ec8d7ee40c7933a609bd51a7ce8b"
87 }
88 ]
89}
corpus/cross-run-edit/partial/candidate/synthetic.one created
Binary files /dev/null and b/corpus/cross-run-edit/partial/candidate/synthetic.one differ
corpus/cross-run-edit/partial/cold/notebook/Open Notebook.onetoc2 created
Binary files /dev/null and b/corpus/cross-run-edit/partial/cold/notebook/Open Notebook.onetoc2 differ
corpus/cross-run-edit/partial/cold/notebook/synthetic.one created
Binary files /dev/null and b/corpus/cross-run-edit/partial/cold/notebook/synthetic.one differ
corpus/cross-run-edit/partial/cold/read/environment.json created+7
...@@ -0,0 +1,7 @@
1{
2 "powershell": "5.1.14409.1005",
3 "schema": "xs2010",
4 "hostname": "ONE-M6-A348FDB7",
5 "cold": true,
6 "onenote": "14.0.4763.1000"
7}
corpus/cross-run-edit/partial/cold/read/hierarchy.xml created+2
...@@ -0,0 +1,2 @@
1<?xml version="1.0"?>
2<one:Notebook xmlns:one="http://schemas.microsoft.com/office/onenote/2010/onenote" name="notebook" nickname="notebook" ID="{93F1F719-446A-4165-8228-120BD9DCB466}{1}{B0}" path="C:\one-tests\runs\capture\notebook" lastModifiedTime="2026-09-08T07:04:03.000Z" color="#B49EDE"><one:Section name="synthetic" ID="{D1CA9D05-C385-0C6D-079D-1C98BA259D9C}{1}{B0}" path="C:\one-tests\runs\capture\notebook\synthetic.one" lastModifiedTime="2026-09-08T07:04:03.000Z" color="#8AA8E4"><one:Page ID="{FD836E83-4A4E-4F7C-80FA-B409461D8183}{1}{B0}" name="Bo中🦀́ color 東京" dateTime="2026-09-05T05:06:54.000Z" lastModifiedTime="2026-09-08T07:04:03.000Z" pageLevel="1"/></one:Section></one:Notebook>
corpus/cross-run-edit/partial/cold/read/page-000.xml created+11
...@@ -0,0 +1,11 @@
1<?xml version="1.0"?>
2<one:Page xmlns:one="http://schemas.microsoft.com/office/onenote/2010/onenote" ID="{FD836E83-4A4E-4F7C-80FA-B409461D8183}{1}{B0}" name="Bo中🦀́ color 東京" dateTime="2026-09-05T05:06:54.000Z" lastModifiedTime="2026-09-08T07:04:03.000Z" pageLevel="1" lang="en-US"><one:QuickStyleDef index="0" name="p" fontColor="automatic" highlightColor="automatic" font="Calibri" fontSize="11.0" spaceBefore="0.0" spaceAfter="0.0"/><one:PageSettings RTL="false" color="automatic"><one:PageSize><one:Automatic/></one:PageSize><one:RuleLines visible="false"/></one:PageSettings><one:Outline author="Rich author" lastModifiedBy="Rich author" lastModifiedTime="2026-09-08T07:02:08.000Z" objectID="{9A4FC63F-013B-4CCA-8F08-38D2B98164E2}{10}{B0}"><one:Position x="36.0" y="14.40000057220459" z="0"/><one:Size width="127.5136947631836" height="41.15543365478516"/><one:OEChildren><one:OE creationTime="2026-09-08T04:47:24.000Z" lastModifiedTime="2026-09-08T07:02:08.000Z" objectID="{9D2A6EC8-F246-07C0-3F5C-FBD872EA1614}{1}{B0}" alignment="left" style="font-family:Calibri;font-size:11.0pt"><one:T><![CDATA[<span
3style='font-weight:bold;text-decoration:underline'>Bo中&#129408;</span><span
4style='vertical-align:super'>́</span> <span style='color:#0C2238;background:
5yellow;mso-highlight:yellow'>color</span> <span style='vertical-align:sub'>東京</span><br>
6End]]></one:T></one:OE><one:OE author="snow" authorInitials="S" lastModifiedBy="snow" lastModifiedByInitials="S" creationTime="2026-09-05T05:06:55.000Z" lastModifiedTime="2026-09-05T05:06:56.000Z" objectID="{9A4FC63F-013B-4CCA-8F08-38D2B98164E2}{11}{B0}" alignment="left" quickStyleIndex="0" style="font-size:11.0pt;color:#1F4E79"><one:T><![CDATA[<span
7style='font-weight:bold;font-family:Calibri' lang=en-US>Fictitious: café, </span><span
8style='font-weight:bold;font-family:SimSun' lang=en-US>東京</span><span
9style='font-weight:bold;font-family:Calibri' lang=en-US>, </span><span
10style='font-weight:bold;font-family:Arial;direction:rtl;unicode-bidi:embed'
11lang=ar-SA>مرحبا</span>]]></one:T></one:OE></one:OEChildren></one:Outline></one:Page>
corpus/cross-run-edit/partial/cold/read/payloads.json created+3
...@@ -0,0 +1,3 @@
1[
2
3]
\ No newline at end of file
corpus/cross-run-edit/partial/cold/run.json created+18
...@@ -0,0 +1,18 @@
1{
2 "notebook": "/Users/clo/dev/one/evidence/m10/cross-run-native/partial/candidate",
3 "expected_pages": 1,
4 "author": null,
5 "author_timeout_seconds": 600,
6 "inspect": false,
7 "collect_notebook": true,
8 "base": {
9 "file": "win7-office-base.qcow2",
10 "format": "qcow2",
11 "sha256": "a1a4f8fab782ee14885ff801ca2f6347c208fdcfc3513637096f314315c89346",
12 "virtual_size": 68719476736
13 },
14 "scripts": {
15 "cold.ps1": "c177fc72ae6c2634d186f5671a880de20b09f9716532c37c69cb13aa15c7e331",
16 "read.ps1": "04013bfcccee40a17e2a225f9b9e96f40a3a8daad9e350609654f8eda3ccbb41"
17 }
18}
corpus/cross-run-edit/partial/cold/source.json created+8
...@@ -0,0 +1,8 @@
1[
2 {
3 "path": "synthetic.one",
4 "bytes": 20944,
5 "sha256": "0a93fdb8c8378989a6522b5de803f073109b8972088e61afdf97639eeaea6a02",
6 "mtime_ns": 1788850928443537242
7 }
8]
corpus/cross-run-edit/partial/cold/teardown.json created+1
...@@ -0,0 +1 @@
1{"absent": true}
corpus/cross-run-edit/replace/candidate/synthetic.one created
Binary files /dev/null and b/corpus/cross-run-edit/replace/candidate/synthetic.one differ
corpus/cross-run-edit/replace/cold/notebook/Open Notebook.onetoc2 created
Binary files /dev/null and b/corpus/cross-run-edit/replace/cold/notebook/Open Notebook.onetoc2 differ
corpus/cross-run-edit/replace/cold/notebook/synthetic.one created
Binary files /dev/null and b/corpus/cross-run-edit/replace/cold/notebook/synthetic.one differ
corpus/cross-run-edit/replace/cold/read/environment.json created+7
...@@ -0,0 +1,7 @@
1{
2 "powershell": "5.1.14409.1005",
3 "schema": "xs2010",
4 "hostname": "ONE-M6-105B87CC",
5 "cold": true,
6 "onenote": "14.0.4763.1000"
7}
corpus/cross-run-edit/replace/cold/read/hierarchy.xml created+2
...@@ -0,0 +1,2 @@
1<?xml version="1.0"?>
2<one:Notebook xmlns:one="http://schemas.microsoft.com/office/onenote/2010/onenote" name="notebook" nickname="notebook" ID="{0A502D6E-6E90-496E-B55C-03E1AC281CF0}{1}{B0}" path="C:\one-tests\runs\capture\notebook" lastModifiedTime="2026-09-08T07:04:07.000Z" color="#B49EDE"><one:Section name="synthetic" ID="{8FC8AEE6-30D3-0B67-02C3-29ADD51A2A5C}{1}{B0}" path="C:\one-tests\runs\capture\notebook\synthetic.one" lastModifiedTime="2026-09-08T07:04:07.000Z" color="#8AA8E4"><one:Page ID="{A3815D60-B918-4876-85A4-813C29223643}{1}{B0}" name="Replacement é🦀" dateTime="2026-09-05T05:06:54.000Z" lastModifiedTime="2026-09-08T07:04:07.000Z" pageLevel="1"/></one:Section></one:Notebook>
corpus/cross-run-edit/replace/cold/read/page-000.xml created+8
...@@ -0,0 +1,8 @@
1<?xml version="1.0"?>
2<one:Page xmlns:one="http://schemas.microsoft.com/office/onenote/2010/onenote" ID="{A3815D60-B918-4876-85A4-813C29223643}{1}{B0}" name="Replacement é🦀" dateTime="2026-09-05T05:06:54.000Z" lastModifiedTime="2026-09-08T07:04:07.000Z" pageLevel="1" lang="en-US"><one:QuickStyleDef index="0" name="p" fontColor="automatic" highlightColor="automatic" font="Calibri" fontSize="11.0" spaceBefore="0.0" spaceAfter="0.0"/><one:PageSettings RTL="false" color="automatic"><one:PageSize><one:Automatic/></one:PageSize><one:RuleLines visible="false"/></one:PageSettings><one:Outline author="Rich author" lastModifiedBy="Rich author" lastModifiedTime="2026-09-08T07:02:08.000Z" objectID="{C44DF5DC-F26D-4BC0-8A56-0DE7D6BED322}{10}{B0}"><one:Position x="36.0" y="14.40000057220459" z="0"/><one:Size width="127.5136947631836" height="27.72771453857422"/><one:OEChildren><one:OE creationTime="2026-09-08T04:47:24.000Z" lastModifiedTime="2026-09-08T07:02:08.000Z" objectID="{C3285D2B-0110-00CA-3A02-CEED1DD5A1D4}{1}{B0}" alignment="left" style="font-family:Calibri;font-size:11.0pt"><one:T><![CDATA[<span
3style='font-weight:bold;text-decoration:underline'>Replacement é&#129408;</span>]]></one:T></one:OE><one:OE author="snow" authorInitials="S" lastModifiedBy="snow" lastModifiedByInitials="S" creationTime="2026-09-05T05:06:55.000Z" lastModifiedTime="2026-09-05T05:06:56.000Z" objectID="{C44DF5DC-F26D-4BC0-8A56-0DE7D6BED322}{11}{B0}" alignment="left" quickStyleIndex="0" style="font-size:11.0pt;color:#1F4E79"><one:T><![CDATA[<span
4style='font-weight:bold;font-family:Calibri' lang=en-US>Fictitious: café, </span><span
5style='font-weight:bold;font-family:SimSun' lang=en-US>東京</span><span
6style='font-weight:bold;font-family:Calibri' lang=en-US>, </span><span
7style='font-weight:bold;font-family:Arial;direction:rtl;unicode-bidi:embed'
8lang=ar-SA>مرحبا</span>]]></one:T></one:OE></one:OEChildren></one:Outline></one:Page>
corpus/cross-run-edit/replace/cold/read/payloads.json created+3
...@@ -0,0 +1,3 @@
1[
2
3]
\ No newline at end of file
corpus/cross-run-edit/replace/cold/run.json created+18
...@@ -0,0 +1,18 @@
1{
2 "notebook": "/Users/clo/dev/one/evidence/m10/cross-run-native/replace/candidate",
3 "expected_pages": 1,
4 "author": null,
5 "author_timeout_seconds": 600,
6 "inspect": false,
7 "collect_notebook": true,
8 "base": {
9 "file": "win7-office-base.qcow2",
10 "format": "qcow2",
11 "sha256": "a1a4f8fab782ee14885ff801ca2f6347c208fdcfc3513637096f314315c89346",
12 "virtual_size": 68719476736
13 },
14 "scripts": {
15 "cold.ps1": "c177fc72ae6c2634d186f5671a880de20b09f9716532c37c69cb13aa15c7e331",
16 "read.ps1": "04013bfcccee40a17e2a225f9b9e96f40a3a8daad9e350609654f8eda3ccbb41"
17 }
18}
corpus/cross-run-edit/replace/cold/source.json created+8
...@@ -0,0 +1,8 @@
1[
2 {
3 "path": "synthetic.one",
4 "bytes": 21520,
5 "sha256": "4b0ae05431ade4777541592cd8040f7ff18dbfe4ada35a686cda963299988e81",
6 "mtime_ns": 1788850928458083445
7 }
8]
corpus/cross-run-edit/replace/cold/teardown.json created+1
...@@ -0,0 +1 @@
1{"absent": true}
corpus/cross-run-edit/retype/candidate/synthetic.one created
Binary files /dev/null and b/corpus/cross-run-edit/retype/candidate/synthetic.one differ
corpus/cross-run-edit/retype/cold/notebook/Open Notebook.onetoc2 created
Binary files /dev/null and b/corpus/cross-run-edit/retype/cold/notebook/Open Notebook.onetoc2 differ
corpus/cross-run-edit/retype/cold/notebook/synthetic.one created
Binary files /dev/null and b/corpus/cross-run-edit/retype/cold/notebook/synthetic.one differ
corpus/cross-run-edit/retype/cold/read/environment.json created+7
...@@ -0,0 +1,7 @@
1{
2 "powershell": "5.1.14409.1005",
3 "schema": "xs2010",
4 "hostname": "ONE-M6-71FBAE77",
5 "cold": true,
6 "onenote": "14.0.4763.1000"
7}
corpus/cross-run-edit/retype/cold/read/hierarchy.xml created+2
...@@ -0,0 +1,2 @@
1<?xml version="1.0"?>
2<one:Notebook xmlns:one="http://schemas.microsoft.com/office/onenote/2010/onenote" name="notebook" nickname="notebook" ID="{2C5C26FC-21E8-467D-BA08-524B57C107FD}{1}{B0}" path="C:\one-tests\runs\capture\notebook" lastModifiedTime="2026-09-08T07:07:16.000Z" color="#B49EDE"><one:Section name="synthetic" ID="{1F28C682-3F93-018E-3DDE-73E7F6610C3E}{1}{B0}" path="C:\one-tests\runs\capture\notebook\synthetic.one" lastModifiedTime="2026-09-08T07:07:16.000Z" color="#8AA8E4"><one:Page ID="{33613504-B658-429F-BAB9-DB760A591021}{1}{B0}" name="Retyped 🦀" dateTime="2026-09-05T05:06:54.000Z" lastModifiedTime="2026-09-08T07:07:16.000Z" pageLevel="1"/></one:Section></one:Notebook>
corpus/cross-run-edit/retype/cold/read/page-000.xml created+7
...@@ -0,0 +1,7 @@
1<?xml version="1.0"?>
2<one:Page xmlns:one="http://schemas.microsoft.com/office/onenote/2010/onenote" ID="{33613504-B658-429F-BAB9-DB760A591021}{1}{B0}" name="Retyped 🦀" dateTime="2026-09-05T05:06:54.000Z" lastModifiedTime="2026-09-08T07:07:16.000Z" pageLevel="1" lang="en-US"><one:QuickStyleDef index="0" name="p" fontColor="automatic" highlightColor="automatic" font="Calibri" fontSize="11.0" spaceBefore="0.0" spaceAfter="0.0"/><one:PageSettings RTL="false" color="automatic"><one:PageSize><one:Automatic/></one:PageSize><one:RuleLines visible="false"/></one:PageSettings><one:Outline author="Rich author" lastModifiedBy="Rich author" lastModifiedTime="2026-09-08T07:02:08.000Z" objectID="{54AD9DB8-FD2D-4129-B54B-57ADF5C5F540}{10}{B0}"><one:Position x="36.0" y="14.40000057220459" z="0"/><one:Size width="127.5136947631836" height="27.72771453857422"/><one:OEChildren><one:OE creationTime="2026-09-08T04:47:24.000Z" lastModifiedTime="2026-09-08T07:02:08.000Z" objectID="{53C8354F-0E50-0A23-051F-94A73EAE87B6}{1}{B0}" alignment="left" style="font-family:Calibri;font-size:11.0pt"><one:T><![CDATA[Retyped &#129408;]]></one:T></one:OE><one:OE author="snow" authorInitials="S" lastModifiedBy="snow" lastModifiedByInitials="S" creationTime="2026-09-05T05:06:55.000Z" lastModifiedTime="2026-09-05T05:06:56.000Z" objectID="{54AD9DB8-FD2D-4129-B54B-57ADF5C5F540}{11}{B0}" alignment="left" quickStyleIndex="0" style="font-size:11.0pt;color:#1F4E79"><one:T><![CDATA[<span
3style='font-weight:bold;font-family:Calibri' lang=en-US>Fictitious: café, </span><span
4style='font-weight:bold;font-family:SimSun' lang=en-US>東京</span><span
5style='font-weight:bold;font-family:Calibri' lang=en-US>, </span><span
6style='font-weight:bold;font-family:Arial;direction:rtl;unicode-bidi:embed'
7lang=ar-SA>مرحبا</span>]]></one:T></one:OE></one:OEChildren></one:Outline></one:Page>
corpus/cross-run-edit/retype/cold/read/payloads.json created+3
...@@ -0,0 +1,3 @@
1[
2
3]
\ No newline at end of file
corpus/cross-run-edit/retype/cold/run.json created+18
...@@ -0,0 +1,18 @@
1{
2 "notebook": "/Users/clo/dev/one/evidence/m10/cross-run-native/retype/candidate",
3 "expected_pages": 1,
4 "author": null,
5 "author_timeout_seconds": 600,
6 "inspect": false,
7 "collect_notebook": true,
8 "base": {
9 "file": "win7-office-base.qcow2",
10 "format": "qcow2",
11 "sha256": "a1a4f8fab782ee14885ff801ca2f6347c208fdcfc3513637096f314315c89346",
12 "virtual_size": 68719476736
13 },
14 "scripts": {
15 "cold.ps1": "c177fc72ae6c2634d186f5671a880de20b09f9716532c37c69cb13aa15c7e331",
16 "read.ps1": "04013bfcccee40a17e2a225f9b9e96f40a3a8daad9e350609654f8eda3ccbb41"
17 }
18}
corpus/cross-run-edit/retype/cold/source.json created+8
...@@ -0,0 +1,8 @@
1[
2 {
3 "path": "synthetic.one",
4 "bytes": 22832,
5 "sha256": "0d37a5a762e9b08f40ea47b2fad48591fcb5f782d1a130f5363f5116b1175fb3",
6 "mtime_ns": 1788850928464340419
7 }
8]
corpus/cross-run-edit/retype/cold/teardown.json created+1
...@@ -0,0 +1 @@
1{"absent": true}
corpus/cross-run-edit/table/candidate/synthetic.one created
Binary files /dev/null and b/corpus/cross-run-edit/table/candidate/synthetic.one differ
corpus/cross-run-edit/table/cold/notebook/Open Notebook.onetoc2 created
Binary files /dev/null and b/corpus/cross-run-edit/table/cold/notebook/Open Notebook.onetoc2 differ
corpus/cross-run-edit/table/cold/notebook/synthetic.one created
Binary files /dev/null and b/corpus/cross-run-edit/table/cold/notebook/synthetic.one differ
corpus/cross-run-edit/table/cold/read/af1af4b8987109ce1e335e538910fb3860b7c57eb3b6110a06aa2114e06d69a7.attachment created+1
...@@ -0,0 +1 @@
1../../../../append/round-01/native/append-01-complex/read/af1af4b8987109ce1e335e538910fb3860b7c57eb3b6110a06aa2114e06d69a7.attachment
\ No newline at end of file
corpus/cross-run-edit/table/cold/read/environment.json created+7
...@@ -0,0 +1,7 @@
1{
2 "powershell": "5.1.14409.1005",
3 "schema": "xs2010",
4 "hostname": "ONE-M6-46174C24",
5 "cold": true,
6 "onenote": "14.0.4763.1000"
7}
corpus/cross-run-edit/table/cold/read/hierarchy.xml created+2
...@@ -0,0 +1,2 @@
1<?xml version="1.0"?>
2<one:Notebook xmlns:one="http://schemas.microsoft.com/office/onenote/2010/onenote" name="notebook" nickname="notebook" ID="{9F6E2E81-C2FD-4BB7-A300-60EF3CF93913}{1}{B0}" path="C:\one-tests\runs\capture\notebook" lastModifiedTime="2026-09-08T07:04:07.000Z" color="#B49EDE"><one:Section name="synthetic" ID="{756534DD-2F0E-0901-12BA-F33A7AD1A925}{1}{B0}" path="C:\one-tests\runs\capture\notebook\synthetic.one" lastModifiedTime="2026-09-08T07:04:07.000Z" color="#8AA8E4"><one:Page ID="{592CC75B-A6C5-4A10-95DD-5BAB86E9B53A}{1}{B0}" name="Fictitious: café, 東京, مرحبا" dateTime="2026-09-05T05:06:54.000Z" lastModifiedTime="2026-09-08T07:04:07.000Z" pageLevel="1"/></one:Section></one:Notebook>
corpus/cross-run-edit/table/cold/read/page-000.xml created+11
...@@ -0,0 +1,11 @@
1<?xml version="1.0"?>
2<one:Page xmlns:one="http://schemas.microsoft.com/office/onenote/2010/onenote" ID="{592CC75B-A6C5-4A10-95DD-5BAB86E9B53A}{1}{B0}" name="Fictitious: café, 東京, مرحبا" dateTime="2026-09-05T05:06:54.000Z" lastModifiedTime="2026-09-08T07:04:07.000Z" pageLevel="1" lang="en-US"><one:QuickStyleDef index="0" name="p" fontColor="automatic" highlightColor="automatic" font="Calibri" fontSize="11.0" spaceBefore="0.0" spaceAfter="0.0"/><one:PageSettings RTL="false" color="automatic"><one:PageSize><one:Automatic/></one:PageSize><one:RuleLines visible="false"/></one:PageSettings><one:Outline author="snow" authorInitials="S" lastModifiedBy="snow" lastModifiedByInitials="S" lastModifiedTime="2026-09-05T05:06:56.000Z" objectID="{3EE06FE7-EDB0-49A6-9A2F-D7707975505B}{10}{B0}"><one:Position x="36.0" y="14.40000057220459" z="0"/><one:Size width="127.5136947631836" height="14.30000114440918"/><one:OEChildren><one:OE creationTime="2026-09-05T05:06:55.000Z" lastModifiedTime="2026-09-05T05:06:56.000Z" objectID="{3EE06FE7-EDB0-49A6-9A2F-D7707975505B}{11}{B0}" alignment="left" quickStyleIndex="0" style="font-size:11.0pt;color:#1F4E79"><one:T><![CDATA[<span
3style='font-weight:bold;font-family:Calibri' lang=en-US>Fictitious: café, </span><span
4style='font-weight:bold;font-family:SimSun' lang=en-US>東京</span><span
5style='font-weight:bold;font-family:Calibri' lang=en-US>, </span><span
6style='font-weight:bold;font-family:Arial;direction:rtl;unicode-bidi:embed'
7lang=ar-SA>مرحبا</span>]]></one:T></one:OE></one:OEChildren></one:Outline><one:Outline author="snow" authorInitials="S" lastModifiedBy="snow" lastModifiedByInitials="S" lastModifiedTime="2026-09-05T05:06:57.000Z" objectID="{6850B3F2-5EE5-43FB-8099-55F8DD79B412}{10}{B0}"><one:Position x="144.0" y="96.0" z="1"/><one:Size width="167.0449523925781" height="13.42771339416504"/><one:OEChildren><one:OE creationTime="2026-09-05T05:06:57.000Z" lastModifiedTime="2026-09-05T05:06:57.000Z" objectID="{6850B3F2-5EE5-43FB-8099-55F8DD79B412}{11}{B0}" alignment="left" quickStyleIndex="0"><one:T><![CDATA[Fictitious positioned outline.]]></one:T></one:OE></one:OEChildren></one:Outline><one:Outline author="snow" authorInitials="S" lastModifiedBy="snow" lastModifiedByInitials="S" lastModifiedTime="2026-09-05T05:06:58.000Z" objectID="{5579DB7D-B87F-4A55-B6E9-FE0C3114FC9C}{10}{B0}"><one:Position x="144.0" y="192.0" z="2"/><one:Size width="72.0" height="0.750005722045898"/><one:OEChildren><one:OE creationTime="2026-09-05T05:06:58.000Z" lastModifiedTime="2026-09-05T05:06:58.000Z" objectID="{5579DB7D-B87F-4A55-B6E9-FE0C3114FC9C}{11}{B0}" alignment="left"><one:Image format="png" originalPageNumber="0"><one:Data>iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAusB9Y9J1uoA
8AAAASUVORK5CYII=</one:Data></one:Image></one:OE></one:OEChildren></one:Outline><one:Outline author="snow" authorInitials="S" lastModifiedBy="snow" lastModifiedByInitials="S" lastModifiedTime="2026-09-05T05:06:59.000Z" objectID="{D063C31D-7E07-4993-9344-99F205C533AF}{10}{B0}"><one:Position x="264.0" y="192.0" z="3"/><one:Size width="72.00001525878906" height="63.0"/><one:OEChildren><one:OE creationTime="2026-09-05T05:06:59.000Z" lastModifiedTime="2026-09-05T05:06:59.000Z" objectID="{D063C31D-7E07-4993-9344-99F205C533AF}{11}{B0}" alignment="left"><one:InsertedFile pathCache="C:\Users\clover\AppData\Local\Microsoft\OneNote\14.0\OneNoteOfflineCache_Files\458001d8-0f1e-451f-81cb-621828fc0b39.txt" pathSource="C:\one-tests\runs\20260905-05\assets\fictitious-attachment.txt" preferredName="fictitious-attachment.txt"/></one:OE></one:OEChildren></one:Outline><one:Outline author="snow" authorInitials="S" lastModifiedBy="snow" lastModifiedByInitials="S" lastModifiedTime="2026-09-08T07:02:08.000Z" objectID="{FEBCBA4D-1F97-4213-8C4C-7BA72D96C8E9}{10}{B0}"><one:Position x="144.0" y="312.0" z="4"/><one:Size width="125.6063766479492" height="48.80315780639648"/><one:OEChildren><one:OE creationTime="2026-09-05T05:07:00.000Z" lastModifiedTime="2026-09-08T07:02:08.000Z" objectID="{FEBCBA4D-1F97-4213-8C4C-7BA72D96C8E9}{11}{B0}" alignment="left"><one:Table bordersVisible="true" lastModifiedTime="2026-09-08T07:02:08.000Z" objectID="{FEBCBA4D-1F97-4213-8C4C-7BA72D96C8E9}{12}{B0}"><one:Columns><one:Column index="0" width="72.33071136474609"/><one:Column index="1" width="45.14567184448242"/></one:Columns><one:Row objectID="{FEBCBA4D-1F97-4213-8C4C-7BA72D96C8E9}{13}{B0}" lastModifiedTime="2026-09-08T07:02:08.000Z"><one:Cell lastModifiedTime="2026-09-08T07:02:08.000Z" objectID="{FEBCBA4D-1F97-4213-8C4C-7BA72D96C8E9}{14}{B0}" lastModifiedByInitials="S"><one:OEChildren><one:OE creationTime="2026-09-05T05:07:00.000Z" lastModifiedTime="2026-09-05T05:07:00.000Z" objectID="{FEBCBA4D-1F97-4213-8C4C-7BA72D96C8E9}{17}{B0}" alignment="left" quickStyleIndex="0"><one:T><![CDATA[Left cell]]></one:T></one:OE><one:OE author="Rich author" lastModifiedBy="Rich author" creationTime="2026-09-08T04:47:24.000Z" lastModifiedTime="2026-09-08T07:02:08.000Z" objectID="{541C4B5A-95EA-0201-16DD-E59A2BF2B871}{1}{B0}" alignment="left" style="font-family:Calibri;font-size:11.0pt"><one:T><![CDATA[<span
9style='font-weight:bold;text-decoration:underline'>Across &#128008; </span><span
10style='vertical-align:sub'>東京</span><br>
11End]]></one:T></one:OE></one:OEChildren></one:Cell><one:Cell lastModifiedTime="2026-09-05T05:07:00.000Z" objectID="{FEBCBA4D-1F97-4213-8C4C-7BA72D96C8E9}{22}{B0}" lastModifiedByInitials="S"><one:OEChildren><one:OE creationTime="2026-09-05T05:07:00.000Z" lastModifiedTime="2026-09-05T05:07:00.000Z" objectID="{FEBCBA4D-1F97-4213-8C4C-7BA72D96C8E9}{25}{B0}" alignment="left" quickStyleIndex="0"><one:T><![CDATA[Right cell]]></one:T></one:OE></one:OEChildren></one:Cell></one:Row></one:Table></one:OE></one:OEChildren></one:Outline></one:Page>
corpus/cross-run-edit/table/cold/read/payloads.json created+10
...@@ -0,0 +1,10 @@
1[
2 {
3 "sha256": "af1af4b8987109ce1e335e538910fb3860b7c57eb3b6110a06aa2114e06d69a7",
4 "name": "fictitious-attachment.txt",
5 "object": "{D063C31D-7E07-4993-9344-99F205C533AF}{11}{B0}",
6 "kind": "InsertedFile",
7 "page": "{592CC75B-A6C5-4A10-95DD-5BAB86E9B53A}{1}{B0}",
8 "bytes": 43
9 }
10]
\ No newline at end of file
corpus/cross-run-edit/table/cold/run.json created+18
...@@ -0,0 +1,18 @@
1{
2 "notebook": "/Users/clo/dev/one/evidence/m10/cross-run-native/table/candidate",
3 "expected_pages": 1,
4 "author": null,
5 "author_timeout_seconds": 600,
6 "inspect": false,
7 "collect_notebook": true,
8 "base": {
9 "file": "win7-office-base.qcow2",
10 "format": "qcow2",
11 "sha256": "a1a4f8fab782ee14885ff801ca2f6347c208fdcfc3513637096f314315c89346",
12 "virtual_size": 68719476736
13 },
14 "scripts": {
15 "cold.ps1": "c177fc72ae6c2634d186f5671a880de20b09f9716532c37c69cb13aa15c7e331",
16 "read.ps1": "04013bfcccee40a17e2a225f9b9e96f40a3a8daad9e350609654f8eda3ccbb41"
17 }
18}
corpus/cross-run-edit/table/cold/source.json created+8
...@@ -0,0 +1,8 @@
1[
2 {
3 "path": "synthetic.one",
4 "bytes": 30864,
5 "sha256": "89ce5a50f050c05ecbfb92c8c8eb0538bd041d0596290a5b974a9878e171e6e9",
6 "mtime_ns": 1788850928450254092
7 }
8]
corpus/cross-run-edit/table/cold/teardown.json created+1
...@@ -0,0 +1 @@
1{"absent": true}
crates/onestore-offline/examples/smb_offline_client.rs+23-6
...@@ -23,6 +23,8 @@ mod support {...@@ -23,6 +23,8 @@ mod support {
23use concurrent::document_view;23use concurrent::document_view;
24use support::view::view;24use support::view::view;
2525
26const DOCUMENT_OPERATIONS: [&str; 3] = ["insert", "format", "text"];
27
26fn now() -> u128 {28fn now() -> u128 {
27 SystemTime::now()29 SystemTime::now()
28 .duration_since(UNIX_EPOCH)30 .duration_since(UNIX_EPOCH)
...@@ -275,7 +277,7 @@ fn queue_document(...@@ -275,7 +277,7 @@ fn queue_document(
275 operation: usize,277 operation: usize,
276 parent: Option<ExGuid>,278 parent: Option<ExGuid>,
277 deadline: Instant,279 deadline: Instant,
278) -> Result<(ExGuid, [u64; 2]), Box<dyn std::error::Error>> {280) -> Result<(ExGuid, [u64; DOCUMENT_OPERATIONS.len()]), Box<dyn std::error::Error>> {
279 let source = cache.snapshot()?;281 let source = cache.snapshot()?;
280 let store = Store::parse(&source)?;282 let store = Store::parse(&source)?;
281 let index = RevisionIndex::parse(&store)?;283 let index = RevisionIndex::parse(&store)?;
...@@ -320,8 +322,15 @@ fn queue_document(...@@ -320,8 +322,15 @@ fn queue_document(
320 TextAttribute::FontSize(18.0 + (operation % 9) as f32),322 TextAttribute::FontSize(18.0 + (operation % 9) as f32),
321 TextAttribute::Color(Some([0x12, 0x34, 0x56])),323 TextAttribute::Color(Some([0x12, 0x34, 0x56])),
322 ];324 ];
323 let mut ids = [0; 2];325 let mut ids = [0; DOCUMENT_OPERATIONS.len()];
324 for (step, id) in ids.iter_mut().enumerate() {326 for (step, id) in ids.iter_mut().enumerate() {
327 let kind = DOCUMENT_OPERATIONS[step];
328 let range = if step == 2 {
329 let end = u32::try_from(text.encode_utf16().count())?;
330 end - 3..end
331 } else {
332 range.clone()
333 };
325 loop {334 loop {
326 if Instant::now() >= deadline {335 if Instant::now() >= deadline {
327 return Err("Document queue timed out; cache retained".into());336 return Err("Document queue timed out; cache retained".into());
...@@ -330,7 +339,7 @@ fn queue_document(...@@ -330,7 +339,7 @@ fn queue_document(
330 let started = now();339 let started = now();
331 let result = if step == 0 {340 let result = if step == 0 {
332 cache.insert(&source, space, &insertion)341 cache.insert(&source, space, &insertion)
333 } else {342 } else if step == 1 {
334 cache.format(343 cache.format(
335 &source,344 &source,
336 space,345 space,
...@@ -338,13 +347,21 @@ fn queue_document(...@@ -338,13 +347,21 @@ fn queue_document(
338 range.clone(),347 range.clone(),
339 &attributes,348 &attributes,
340 )349 )
350 } else {
351 cache.edit_text(
352 &source,
353 space,
354 insertion.text_object(),
355 range.clone(),
356 " e\u{301}🐈",
357 )
341 };358 };
342 match result {359 match result {
343 Ok(Some(acknowledged)) => {360 Ok(Some(acknowledged)) => {
344 *id = acknowledged;361 *id = acknowledged;
345 println!(362 println!(
346 "{}",363 "{}",
347 json!({"event":"local_document_commit","id":acknowledged,"operation":operation,"kind":if step==0 {"insert"} else {"format"},"space":space.to_string(),"object":insertion.text_object().to_string(),"text":text,"insertion":if step==0 {Some(&insertion)} else {None},"range":[range.start,range.end],"attributes":attributes,"started_us":started,"finished_us":now()})364 json!({"event":"local_document_commit","id":acknowledged,"operation":operation,"kind":kind,"space":space.to_string(),"object":insertion.text_object().to_string(),"text":text,"insertion":if step==0 {Some(&insertion)} else {None},"range":[range.start,range.end],"attributes":attributes,"replacement":if step==2 {Some(" e\u{301}🐈")} else {None},"started_us":started,"finished_us":now()})
348 );365 );
349 break;366 break;
350 }367 }
...@@ -400,7 +417,7 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {...@@ -400,7 +417,7 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
400 let cache = Arc::new(Replica::create(&cache_path, &source)?);417 let cache = Arc::new(Replica::create(&cache_path, &source)?);
401 println!(418 println!(
402 "{}",419 "{}",
403 json!({"event":"ready", "pid":std::process::id(), "actor":args[2], "offline":true, "document_operations":documents})420 json!({"event":"ready", "pid":std::process::id(), "actor":args[2], "offline":true, "document_operations":documents,"document_kinds":if documents {DOCUMENT_OPERATIONS.as_slice()} else {&[]}})
404 );421 );
405 while !Path::new(&args[4]).exists() {422 while !Path::new(&args[4]).exists() {
406 if Instant::now() >= deadline {423 if Instant::now() >= deadline {
...@@ -470,7 +487,7 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {...@@ -470,7 +487,7 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
470 {487 {
471 1488 1
472 } else if documents && outage.is_some() {489 } else if documents && outage.is_some() {
473 24490 8 * (1 + DOCUMENT_OPERATIONS.len())
474 } else {491 } else {
475 8492 8
476 };493 };
crates/onestore-offline/src/lib.rs+1
...@@ -180,6 +180,7 @@ impl Replica {...@@ -180,6 +180,7 @@ impl Replica {
180180
181 /// Atomically records an intent and its resulting local image; returns its durable ID.181 /// Atomically records an intent and its resulting local image; returns its durable ID.
182 /// Unchanged text returns `None`. A stale image returns `Io(ResourceBusy)`.182 /// Unchanged text returns `None`. A stale image returns `Io(ResourceBusy)`.
183 /// On synchronization, replacement text inherits the remote style at the rebased start.
183 /// After a database error, reopen and inspect the cache before retrying the edit.184 /// After a database error, reopen and inspect the cache before retrying the edit.
184 pub fn edit_text(185 pub fn edit_text(
185 &self,186 &self,
crates/onestore-offline/tests/sync.rs+76
...@@ -1888,6 +1888,82 @@ fn uncertain_insertions_reconcile_the_original_revision_without_duplicate_object...@@ -1888,6 +1888,82 @@ fn uncertain_insertions_reconcile_the_original_revision_without_duplicate_object
1888 }1888 }
1889}1889}
18901890
1891#[test]
1892fn cross_run_text_rebases_preserve_remote_styles_and_survive_lost_replies() {
1893 use onestore::TextAttribute as A;
1894 for fault in [Fault::None, Fault::UnknownAfter, Fault::PanicAfter] {
1895 let directory = tempfile::tempdir().unwrap();
1896 let path = directory.path().join("cross-run.sqlite");
1897 let plain = onestore::create_section("cross.one", "ab🦀cde\u{301}fg", "Author").unwrap();
1898 let (sid, oid, _) = text(&plain);
1899 let bold = PreparedEdit::format(&plain, sid, oid, 1..4, &[A::Bold(true)]).unwrap();
1900 let source = PreparedEdit::format(bold.as_bytes(), sid, oid, 4..7, &[A::Italic(true)])
1901 .unwrap()
1902 .as_bytes()
1903 .to_vec();
1904 let cache = Replica::create(&path, &source).unwrap();
1905 let first = cache
1906 .edit_text(&source, sid, oid, 2..7, "日本語")
1907 .unwrap()
1908 .unwrap();
1909 let local = cache.snapshot().unwrap();
1910 let second = cache
1911 .edit_text(&local, sid, oid, 3..4, "🐈")
1912 .unwrap()
1913 .unwrap();
1914 let pending = cache.pending().unwrap();
1915 drop(cache);
1916 let cache = Replica::open(&path).unwrap();
1917 assert_eq!(cache.pending().unwrap(), pending);
1918 let prefix = PreparedEdit::text(&source, sid, oid, 0..0, "Prefix ").unwrap();
1919 let remote =
1920 PreparedEdit::format(prefix.as_bytes(), sid, oid, 9..11, &[A::Underline(true)])
1921 .unwrap();
1922 let mut server = Server::new(remote.as_bytes());
1923 server.fault = fault;
1924 let attempted = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
1925 cache.sync_once(&mut server)
1926 }));
1927 if matches!(fault, Fault::None) {
1928 assert!(
1929 matches!(attempted.unwrap().unwrap(), Some((id, EditStatus::Published { .. })) if id == first)
1930 );
1931 } else {
1932 assert!(matches!(
1933 cache.status(first).unwrap(),
1934 Some(EditStatus::AwaitingConfirmation { .. })
1935 ));
1936 }
1937 drop(cache);
1938 let cache = Replica::open(&path).unwrap();
1939 while let Some((_, status)) = cache.sync_once(&mut server).unwrap() {
1940 assert!(matches!(status, EditStatus::Published { .. }));
1941 }
1942 assert_eq!(server.publications, 2);
1943 assert!(matches!(
1944 cache.status(second).unwrap(),
1945 Some(EditStatus::Published { .. })
1946 ));
1947 assert_eq!(text(&server.durable).2, "Prefix ab日🐈語\u{301}fg");
1948 assert_eq!(cache.snapshot().unwrap(), server.durable);
1949 let store = Store::parse(&server.durable).unwrap();
1950 let index = RevisionIndex::parse(&store).unwrap();
1951 let document = Document::parse(&index).unwrap();
1952 let space = &document.spaces[&sid];
1953 let revision = &space.revisions[&space.contexts[&ExGuid::default()]];
1954 let runs = revision.text_runs(oid).unwrap();
1955 let replacement = runs.iter().find(|run| run.text == "日🐈語").unwrap();
1956 assert_eq!(replacement.format.bold, Some(true));
1957 assert_eq!(replacement.format.underline, Some(true));
1958 assert_ne!(replacement.format.italic, Some(true));
1959 let suffix = runs.last().unwrap();
1960 assert_eq!(suffix.text, "\u{301}fg");
1961 assert_ne!(suffix.format.bold, Some(true));
1962 assert_ne!(suffix.format.underline, Some(true));
1963 assert_ne!(suffix.format.italic, Some(true));
1964 }
1965}
1966
1891#[test]1967#[test]
1892fn offline_formatting_rebases_text_and_merges_independent_attributes() {1968fn offline_formatting_rebases_text_and_merges_independent_attributes() {
1893 use onestore::TextAttribute as A;1969 use onestore::TextAttribute as A;
crates/onestore/README.md+1-1
...@@ -47,7 +47,7 @@ harness also accepts `--client-profile release`....@@ -47,7 +47,7 @@ harness also accepts `--client-profile release`.
47| `create_section` | Create one page containing one plain-text paragraph and an author, including Unicode |47| `create_section` | Create one page containing one plain-text paragraph and an author, including Unicode |
48| `create_table_of_contents` | Create ordered section entries from filenames and file identities |48| `create_table_of_contents` | Create ordered section entries from filenames and file identities |
49| `replace_property_bytes` | Append one scalar-property revision; preserve prior revisions and unrelated property values and references |49| `replace_property_bytes` | Append one scalar-property revision; preserve prior revisions and unrelated property values and references |
50| `replace_text`, `commit_text`, `commit_file_text` | Replace a UTF-16 range within one ordinary text run; publish text, run boundaries and modification time together |50| `replace_text`, `commit_text`, `commit_file_text` | Replace a UTF-16 range across ordinary text runs; publish text, run boundaries and modification time together |
51| `Insertion`, `PreparedEdit::insert` | Insert paragraphs into editable containers or positioned outlines into a page, retaining intent identities across rebases |51| `Insertion`, `PreparedEdit::insert` | Insert paragraphs into editable containers or positioned outlines into a page, retaining intent identities across rebases |
52| `TextAttribute`, `PreparedEdit::format` | Change character formatting over a UTF-16 range while sharing immutable styles; preserve unselected runs |52| `TextAttribute`, `PreparedEdit::format` | Change character formatting over a UTF-16 range while sharing immutable styles; preserve unselected runs |
53| `PreparedEdit::commit`, `PreparedEdit::commit_file` | Publish the exact prepared image under caller-held exclusion or the conservative filesystem adapter |53| `PreparedEdit::commit`, `PreparedEdit::commit_file` | Publish the exact prepared image under caller-held exclusion or the conservative filesystem adapter |
crates/onestore/src/edit.rs+64-29
...@@ -18,9 +18,11 @@ pub(crate) fn automatic_title(text: &str) -> &str {...@@ -18,9 +18,11 @@ pub(crate) fn automatic_title(text: &str) -> &str {
18 line18 line
19}19}
2020
21/// Replaces UTF-16 character positions within one ordinary text run.21/// Replaces UTF-16 character positions across ordinary text runs.
22/// The inserted text inherits that run's formatting; other runs retain their identities.22/// Inserted text inherits the run at the start; surviving text retains its formatting.
23/// Insertion at a run boundary uses the following run, except at the end of text.23/// Insertion at a run boundary uses the following run, except at the end of text.
24/// The final run retains its insertion style even when emptied.
25/// Replacing a range with identical text leaves its existing formatting unchanged.
24/// Returns a complete file image without I/O; use `commit_file_text` to update an existing file.26/// Returns a complete file image without I/O; use `commit_file_text` to update an existing file.
25pub fn replace_text(27pub fn replace_text(
26 source: &[u8],28 source: &[u8],
...@@ -79,22 +81,25 @@ pub fn replace_text(...@@ -79,22 +81,25 @@ pub fn replace_text(
79 }81 }
80 let selected = runs82 let selected = runs
81 .iter()83 .iter()
82 .rposition(|run| run.start <= range.start && range.end <= run.end)84 .rposition(|run| run.start <= range.start && range.start <= run.end)
83 .ok_or_else(|| invalid("The edit must stay within one text run"))?;85 .ok_or_else(|| invalid("The edit range exceeds the text"))?;
84 let resolved = revision.text_runs(object)?;86 let resolved = revision.text_runs(object)?;
85 let format = &resolved[selected].format;87 for (i, run) in runs.iter().enumerate() {
86 if [88 if i == selected || (run.start < range.end && range.start < run.end) {
87 format.hidden,89 let format = &resolved[i].format;
88 format.hyperlink,90 if [
89 format.math,91 format.hidden,
90 format.embedded_object,92 format.hyperlink,
91 ]93 format.math,
92 .contains(&Some(true))94 format.embedded_object,
93 || runs[selected]95 ]
94 .extra_set96 .contains(&Some(true))
95 .is_some_and(|set| !node.extra[set].is_empty())97 || resolved[i].text.contains(['\u{fffc}', '\u{fddf}'])
96 {98 || run.extra_set.is_some_and(|set| !node.extra[set].is_empty())
97 return Err(invalid("This text run contains a field or embedded data"));99 {
100 return Err(invalid("This text run contains a field or embedded data"));
101 }
102 }
98 }103 }
99 let ObjectData::Properties(blob) = raw.objects[&object].data else {104 let ObjectData::Properties(blob) = raw.objects[&object].data else {
100 unreachable!()105 unreachable!()
...@@ -141,20 +146,47 @@ pub fn replace_text(...@@ -141,20 +146,47 @@ pub fn replace_text(
141 .checked_sub(removed)146 .checked_sub(removed)
142 .and_then(|n| n.checked_add(added))147 .and_then(|n| n.checked_add(added))
143 .ok_or_else(|| invalid("Edited text exceeds the UTF-16 offset range"))?;148 .ok_or_else(|| invalid("Edited text exceeds the UTF-16 offset range"))?;
144 let mut boundaries = Vec::new();149 let mut segments = Vec::new();
145 let mut previous = None;150 let mut position = 0;
146 for (i, run) in runs[..runs.len() - 1].iter().enumerate() {151 for (i, run) in runs.iter().enumerate() {
147 let end = if i >= selected {152 let length = run.end.min(range.start).saturating_sub(run.start)
148 run.end - removed + added153 + run.end.saturating_sub(run.start.max(range.end))
149 } else {154 + if i == selected { added } else { 0 };
150 run.end155 let untouched = i != selected && (run.end <= range.start || range.end <= run.start);
151 };156 if length > 0 || i == runs.len() - 1 || untouched {
152 if previous.is_some_and(|p| p >= end) {157 position += length;
153 return Err(invalid("The edit would collapse a formatting boundary"));158 segments.push((i, position));
154 }159 }
155 previous = Some(end);
156 boundaries.extend_from_slice(&end.to_le_bytes());
157 }160 }
161 if segments.len() != runs.len() && properties.sets[0].iter().any(|p| p.id == 0x40003499) {
162 return Err(invalid("Text edits cannot remove preserved run data"));
163 }
164 if segments[..segments.len() - 1]
165 .windows(2)
166 .any(|pair| pair[0].1 >= pair[1].1)
167 {
168 return Err(invalid("Text-run boundaries must be strictly increasing"));
169 }
170 let boundaries: Vec<_> = segments[..segments.len() - 1]
171 .iter()
172 .flat_map(|(_, end)| end.to_le_bytes())
173 .collect();
174 let formats = properties.sets[0]
175 .iter()
176 .find(|p| p.id == 0x24001e13)
177 .map(|p| {
178 let crate::Value::References { compact_ids, .. } = p.value else {
179 unreachable!()
180 };
181 if compact_ids.is_empty() {
182 Vec::new()
183 } else {
184 segments
185 .iter()
186 .flat_map(|(i, _)| compact_ids[i * 4..i * 4 + 4].iter().copied())
187 .collect()
188 }
189 });
158 let mut changed = String::with_capacity(text.len() - (end - start) + replacement.len());190 let mut changed = String::with_capacity(text.len() - (end - start) + replacement.len());
159 changed.push_str(&text[..start]);191 changed.push_str(&text[..start]);
160 changed.push_str(replacement);192 changed.push_str(replacement);
...@@ -183,6 +215,9 @@ pub fn replace_text(...@@ -183,6 +215,9 @@ pub fn replace_text(
183 if properties.sets[0].iter().any(|p| p.id == 0x1c001e12) {215 if properties.sets[0].iter().any(|p| p.id == 0x1c001e12) {
184 updates.push((0x1c001e12, &boundaries));216 updates.push((0x1c001e12, &boundaries));
185 }217 }
218 if let Some(formats) = &formats {
219 updates.push((0x24001e13, formats));
220 }
186 let modified = crate::create::current_timestamps()?.0.to_le_bytes();221 let modified = crate::create::current_timestamps()?.0.to_le_bytes();
187 updates.push((0x14001d7a, &modified));222 updates.push((0x14001d7a, &modified));
188 let mut edits = vec![crate::write::ObjectEdit {223 let mut edits = vec![crate::write::ObjectEdit {
crates/onestore/src/write/tests.rs+50-7
...@@ -589,7 +589,7 @@ fn character_formatting_preserves_inheritance_and_associated_data() {...@@ -589,7 +589,7 @@ fn character_formatting_preserves_inheritance_and_associated_data() {
589 .find_map(|(id, o)| (o.jcid == 0x6000e).then_some((*sid, *id)))589 .find_map(|(id, o)| (o.jcid == 0x6000e).then_some((*sid, *id)))
590 })590 })
591 .unwrap();591 .unwrap();
592 for variant in 0..7 {592 for variant in 0..10 {
593 let fixture = write_revision(&source, sid, |raw| {593 let fixture = write_revision(&source, sid, |raw| {
594 let mut target = PropertyObject::from_object(&raw.objects[&text])?;594 let mut target = PropertyObject::from_object(&raw.objects[&text])?;
595 target.bytes = properties(&[595 target.bytes = properties(&[
...@@ -617,8 +617,17 @@ fn character_formatting_preserves_inheritance_and_associated_data() {...@@ -617,8 +617,17 @@ fn character_formatting_preserves_inheritance_and_associated_data() {
617 let reference = target.reference(author)?;617 let reference = target.reference(author)?;
618 target.set(&[(0x24003458, &reference)])?;618 target.set(&[(0x24003458, &reference)])?;
619 }619 }
620 3 => {620 3 | 7 | 8 | 9 => {
621 // An unknown nested run property must survive a style-only edit byte for byte.621 if variant != 3 {
622 let ends = if variant == 9 { [2_u32, 2] } else { [2, 4] };
623 target.set(&[(
624 0x1c001e12,
625 &ends
626 .into_iter()
627 .flat_map(u32::to_le_bytes)
628 .collect::<Vec<_>>(),
629 )])?;
630 }
622 let parsed = PropertySets::parse(&target.bytes)?;631 let parsed = PropertySets::parse(&target.bytes)?;
623 let at = parsed.root_ids.as_ptr().addr() - target.bytes.as_ptr().addr();632 let at = parsed.root_ids.as_ptr().addr() - target.bytes.as_ptr().addr();
624 let count = parsed.sets[0].len();633 let count = parsed.sets[0].len();
...@@ -626,11 +635,17 @@ fn character_formatting_preserves_inheritance_and_associated_data() {...@@ -626,11 +635,17 @@ fn character_formatting_preserves_inheritance_and_associated_data() {
626 let mut bytes = target.bytes[..end].to_vec();635 let mut bytes = target.bytes[..end].to_vec();
627 bytes[at - 2..at].copy_from_slice(&((count + 1) as u16).to_le_bytes());636 bytes[at - 2..at].copy_from_slice(&((count + 1) as u16).to_le_bytes());
628 bytes.splice(at + count * 4..at + count * 4, 0x40003499_u32.to_le_bytes());637 bytes.splice(at + count * 4..at + count * 4, 0x40003499_u32.to_le_bytes());
629 bytes.extend_from_slice(&1_u32.to_le_bytes());638 let runs: u32 = if variant == 3 { 1 } else { 3 };
639 bytes.extend_from_slice(&runs.to_le_bytes());
630 bytes.extend_from_slice(&0x44001234_u32.to_le_bytes());640 bytes.extend_from_slice(&0x44001234_u32.to_le_bytes());
631 bytes.extend_from_slice(&1_u16.to_le_bytes());641 for run in 0..runs {
632 bytes.extend_from_slice(&0x14001234_u32.to_le_bytes());642 let populated = (variant == 3 || variant == 8) && run == runs - 1;
633 bytes.extend_from_slice(&0xdeadbeef_u32.to_le_bytes());643 bytes.extend_from_slice(&u16::from(populated).to_le_bytes());
644 if populated {
645 bytes.extend_from_slice(&0x14001234_u32.to_le_bytes());
646 bytes.extend_from_slice(&0xdeadbeef_u32.to_le_bytes());
647 }
648 }
634 bytes.resize(bytes.len().next_multiple_of(8), 0);649 bytes.resize(bytes.len().next_multiple_of(8), 0);
635 target.bytes = bytes;650 target.bytes = bytes;
636 }651 }
...@@ -639,6 +654,34 @@ fn character_formatting_preserves_inheritance_and_associated_data() {...@@ -639,6 +654,34 @@ fn character_formatting_preserves_inheritance_and_associated_data() {
639 Ok(BTreeMap::from([(text, target)]))654 Ok(BTreeMap::from([(text, target)]))
640 })655 })
641 .unwrap();656 .unwrap();
657 if variant == 9 {
658 assert!(PreparedEdit::text(&fixture, sid, text, 0..0, "x").is_err());
659 continue;
660 }
661 if variant >= 7 {
662 let edit = PreparedEdit::text(&fixture, sid, text, 1..3, "longer").unwrap();
663 let [before, after] = [&fixture, edit.as_bytes()].map(|bytes| {
664 let store = Store::parse(bytes).unwrap();
665 let index = RevisionIndex::parse(&store).unwrap();
666 let document = Document::parse(&index).unwrap();
667 let space = &document.spaces[&sid];
668 let view = &space.revisions[&space.contexts[&ExGuid::default()]];
669 let Kind::RichText { runs, .. } = &view.nodes[&text].kind else {
670 panic!()
671 };
672 assert_eq!(runs.len(), 3);
673 format!("{:?}", view.nodes[&text].extra)
674 });
675 assert_eq!(before, after);
676 assert!(PreparedEdit::text(&fixture, sid, text, 0..4, "x").is_err());
677 assert_eq!(
678 PreparedEdit::text(&fixture, sid, text, 3..5, "x").is_ok(),
679 variant == 7
680 );
681 continue;
682 }
683 let text_edit = PreparedEdit::text(&fixture, sid, text, 1..5, "x");
684 assert_eq!(text_edit.is_ok(), variant == 0);
642 let edit = PreparedEdit::format(&fixture, sid, text, 0..6, &[TextAttribute::Bold(true)]);685 let edit = PreparedEdit::format(&fixture, sid, text, 0..6, &[TextAttribute::Bold(true)]);
643 if matches!(variant, 1 | 2 | 4 | 5 | 6) {686 if matches!(variant, 1 | 2 | 4 | 5 | 6) {
644 assert!(edit.is_err());687 assert!(edit.is_err());
crates/onestore/tests/edit.rs+177-3
...@@ -368,9 +368,10 @@ fn invalid_text_edits_never_touch_storage() {...@@ -368,9 +368,10 @@ fn invalid_text_edits_never_touch_storage() {
368 for (range, replacement) in [368 for (range, replacement) in [
369 (1..1, "x"),369 (1..1, "x"),
370 (0..999, "x"),370 (0..999, "x"),
371 (0..21, "x"),
372 (0..0, "\n"),371 (0..0, "\n"),
373 (0..0, "\0"),372 (0..0, "\0"),
373 (0..0, "\u{fffc}"),
374 (0..0, "\u{fddf}"),
374 ] {375 ] {
375 let mut disk = Disk {376 let mut disk = Disk {
376 visible: source.clone(),377 visible: source.clone(),
...@@ -388,6 +389,179 @@ fn invalid_text_edits_never_touch_storage() {...@@ -388,6 +389,179 @@ fn invalid_text_edits_never_touch_storage() {
388 }389 }
389}390}
390391
392#[test]
393fn cross_run_edits_reject_native_fields_before_storage_io() {
394 let source = include_bytes!("../../../corpus/m6/native-probes-01/notebook/synthetic.one");
395 let store = Store::parse(source).unwrap();
396 let index = RevisionIndex::parse(&store).unwrap();
397 let document = Document::parse(&index).unwrap();
398 let mut fields = 0;
399 for (sid, space) in &document.spaces {
400 let revision = &space.revisions[&space.contexts[&ExGuid::default()]];
401 for (oid, node) in &revision.nodes {
402 let Kind::RichText { text, runs, .. } = &node.kind else {
403 continue;
404 };
405 let resolved = revision.text_runs(*oid).unwrap();
406 for (run, resolved) in runs.iter().zip(resolved) {
407 if resolved.format.hyperlink != Some(true) {
408 continue;
409 }
410 fields += 1;
411 for range in [0..run.end, run.start..text.encode_utf16().count() as u32] {
412 let mut disk = Disk {
413 visible: source.to_vec(),
414 durable: source.to_vec(),
415 operation: 0,
416 fail_at: None,
417 write_limit: 17,
418 random: 9,
419 };
420 let error = onestore::commit_text(&mut disk, source, *sid, *oid, range, "x")
421 .unwrap_err();
422 assert_eq!(error.state, CommitState::NotCommitted);
423 assert_eq!(disk.operation, 0);
424 assert_eq!(disk.durable, source);
425 }
426 }
427 }
428 }
429 assert!(fields > 0);
430}
431
432#[test]
433fn cross_run_splices_match_a_character_model_and_preserve_history() {
434 let (sid, oid) = target(SOURCE);
435 let characters = |source: &[u8]| {
436 let runs = text_runs(source, sid, oid);
437 let runs = runs.as_array().unwrap();
438 let mut result = Vec::new();
439 for run in runs {
440 result.extend(
441 run["text"]
442 .as_str()
443 .unwrap()
444 .chars()
445 .map(|c| (c, run["format"].clone())),
446 );
447 }
448 result.push(('\0', runs.last().unwrap()["format"].clone()));
449 result
450 };
451 let before = characters(SOURCE);
452 let offsets: Vec<_> = std::iter::once(0)
453 .chain(before[..before.len() - 1].iter().scan(0, |at, (c, _)| {
454 *at += c.len_utf16() as u32;
455 Some(*at)
456 }))
457 .collect();
458 let store = Store::parse(SOURCE).unwrap();
459 let index = RevisionIndex::parse(&store).unwrap();
460 for a in 0..before.len() {
461 for b in a..before.len() {
462 let replacement = ["", "🦀e\u{301}", "日本語"][b % 3];
463 let mut expected = before.clone();
464 expected.splice(a..b, replacement.chars().map(|c| (c, before[a].1.clone())));
465 let edited =
466 onestore::replace_text(SOURCE, sid, oid, offsets[a]..offsets[b], replacement)
467 .unwrap();
468 assert_eq!(characters(&edited), expected, "characters {a}..{b}");
469 let current_store = Store::parse(&edited).unwrap();
470 assert_eq!(
471 current_store.header.transaction_count,
472 store.header.transaction_count
473 + usize::from(a != b || !replacement.is_empty()) as u32
474 );
475 let current = RevisionIndex::parse(&current_store).unwrap();
476 for (space, history) in &index.spaces {
477 for revision in history.revisions.keys() {
478 let old = index.resolve(*space, *revision).unwrap();
479 let retained = current.resolve(*space, *revision).unwrap();
480 assert_eq!(old.roots, retained.roots);
481 for (id, object) in old.objects {
482 assert_eq!(object.data, retained.objects[&id].data);
483 }
484 }
485 }
486 }
487 }
488 let cleared =
489 onestore::replace_text(SOURCE, sid, oid, 0..*offsets.last().unwrap(), "").unwrap();
490 let typed = onestore::replace_text(&cleared, sid, oid, 0..0, "new").unwrap();
491 let format = &before.last().unwrap().1;
492 assert_eq!(
493 characters(&typed),
494 "new\0"
495 .chars()
496 .map(|c| (c, format.clone()))
497 .collect::<Vec<_>>()
498 );
499 let text: String = before[..before.len() - 1].iter().map(|(c, _)| *c).collect();
500 assert_eq!(
501 onestore::replace_text(SOURCE, sid, oid, 0..*offsets.last().unwrap(), &text).unwrap(),
502 SOURCE
503 );
504}
505
506#[test]
507#[ignore = "exports cross-run edits for independent native validation"]
508fn export_native_cross_run_edits() {
509 use std::{fs, path::PathBuf};
510 let output = PathBuf::from(std::env::var_os("ONESTORE_CROSS_RUN_OUTPUT").unwrap());
511 assert!(output.is_absolute());
512 fs::create_dir(&output).unwrap();
513 let mut manifest = Vec::new();
514 for (name, fixture, range, replacement, retype) in [
515 ("partial", "native-paragraph", 2..16, "中🦀", false),
516 ("table", "native-cell", 0..24, "Across 🐈 ", false),
517 ("clear", "native-paragraph", 0..u32::MAX, "", false),
518 (
519 "replace",
520 "native-paragraph",
521 0..u32::MAX,
522 "Replacement e\u{301}🦀",
523 false,
524 ),
525 ("retype", "native-paragraph", 0..u32::MAX, "", true),
526 ("boundary", "native-paragraph", 4..8, "Middle", false),
527 ] {
528 let input = format!("../../corpus/formatted-insertion/{fixture}/candidate/synthetic.one");
529 let source = fs::read(&input).unwrap();
530 let store = Store::parse(&source).unwrap();
531 let index = RevisionIndex::parse(&store).unwrap();
532 let document = Document::parse(&index).unwrap();
533 let (sid, oid, length) = document
534 .spaces
535 .iter()
536 .find_map(|(sid, space)| {
537 let view = &space.revisions[&space.contexts[&ExGuid::default()]];
538 view.nodes.iter().find_map(|(oid, node)| {
539 let Kind::RichText { text, .. } = &node.kind else {
540 return None;
541 };
542 text.starts_with("Bold ")
543 .then(|| (*sid, *oid, text.encode_utf16().count() as u32))
544 })
545 })
546 .unwrap();
547 let range = range.start..range.end.min(length);
548 let mut edited =
549 onestore::replace_text(&source, sid, oid, range.clone(), replacement).unwrap();
550 if retype {
551 edited = onestore::replace_text(&edited, sid, oid, 0..0, "Retyped 🦀").unwrap();
552 }
553 let candidate = output.join(name).join("candidate");
554 fs::create_dir_all(&candidate).unwrap();
555 fs::write(candidate.join("synthetic.one"), &edited).unwrap();
556 manifest.push(serde_json::json!({"name":name,"source":input,"space":sid,"object":oid,"range":range,"replacement":replacement,"retype":retype}));
557 }
558 fs::write(
559 output.join("manifest.json"),
560 serde_json::to_vec_pretty(&manifest).unwrap(),
561 )
562 .unwrap();
563}
564
391#[test]565#[test]
392fn insertion_at_a_style_boundary_uses_the_following_style() {566fn insertion_at_a_style_boundary_uses_the_following_style() {
393 let (sid, oid) = target(SOURCE);567 let (sid, oid) = target(SOURCE);
...@@ -701,14 +875,14 @@ fn interrupted_text_commits_never_publish_mismatched_run_boundaries() {...@@ -701,14 +875,14 @@ fn interrupted_text_commits_never_publish_mismatched_run_boundaries() {
701 };875 };
702 let before = text_runs(source, sid, oid);876 let before = text_runs(source, sid, oid);
703 let mut success = make_disk(None, 1);877 let mut success = make_disk(None, 1);
704 onestore::commit_text(&mut success, source, sid, oid, 0..10, "🐈 mixed edit").unwrap();878 onestore::commit_text(&mut success, source, sid, oid, 0..23, "🐈 mixed edit").unwrap();
705 let after = text_runs(&success.durable, sid, oid);879 let after = text_runs(&success.durable, sid, oid);
706 assert_ne!(before, after);880 assert_ne!(before, after);
707 for at in source.len().div_ceil(193)..=success.operation {881 for at in source.len().div_ceil(193)..=success.operation {
708 for seed in [1, 42] {882 for seed in [1, 42] {
709 let mut disk = make_disk(Some(at), seed);883 let mut disk = make_disk(Some(at), seed);
710 let error =884 let error =
711 onestore::commit_text(&mut disk, source, sid, oid, 0..10, "🐈 mixed edit")885 onestore::commit_text(&mut disk, source, sid, oid, 0..23, "🐈 mixed edit")
712 .unwrap_err();886 .unwrap_err();
713 let observed = text_runs(&disk.durable, sid, oid);887 let observed = text_runs(&disk.durable, sid, oid);
714 match error.state {888 match error.state {
fuzz/fuzz_targets/edit_text.rs+18-2
...@@ -129,7 +129,7 @@ fuzz_target!(|input: &[u8]| {...@@ -129,7 +129,7 @@ fuzz_target!(|input: &[u8]| {
129 let (source, sid, oid) = &CASES[usize::from(input.last().copied().unwrap_or(0)) % CASES.len()];129 let (source, sid, oid) = &CASES[usize::from(input.last().copied().unwrap_or(0)) % CASES.len()];
130 let (sid, oid) = (*sid, *oid);130 let (sid, oid) = (*sid, *oid);
131 let mut persisted_source = source.clone();131 let mut persisted_source = source.clone();
132 let mut caches = std::array::from_fn::<_, 6, _>(|_| source.clone());132 let mut caches = std::array::from_fn::<_, 12, _>(|_| source.clone());
133 for step in input.chunks_exact(8).take(16) {133 for step in input.chunks_exact(8).take(16) {
134 let actor = usize::from(step[6]) % caches.len();134 let actor = usize::from(step[6]) % caches.len();
135 if step[7] % 3 == 0 {135 if step[7] % 3 == 0 {
...@@ -154,7 +154,14 @@ fuzz_target!(|input: &[u8]| {...@@ -154,7 +154,14 @@ fuzz_target!(|input: &[u8]| {
154 let replacement = ["", "a", "🦀e\u{301}", "日本語", "\n", "\0"][usize::from(step[2]) % 6];154 let replacement = ["", "a", "🦀e\u{301}", "日本語", "\n", "\0"][usize::from(step[2]) % 6];
155 let mut expected = before.clone();155 let mut expected = before.clone();
156 let format = before[start].1.clone();156 let format = before[start].1.clone();
157 expected.splice(start..end, replacement.chars().map(|c| (c, format.clone())));157 if before[start..end]
158 .iter()
159 .map(|(c, _)| *c)
160 .collect::<String>()
161 != replacement
162 {
163 expected.splice(start..end, replacement.chars().map(|c| (c, format.clone())));
164 }
158 let mut storage = Disk {165 let mut storage = Disk {
159 visible: persisted_source.clone(),166 visible: persisted_source.clone(),
160 durable: persisted_source.clone(),167 durable: persisted_source.clone(),
...@@ -173,6 +180,15 @@ fuzz_target!(|input: &[u8]| {...@@ -173,6 +180,15 @@ fuzz_target!(|input: &[u8]| {
173 replacement,180 replacement,
174 );181 );
175 let persisted = characters(&storage.durable, sid, oid);182 let persisted = characters(&storage.durable, sid, oid);
183 if !replacement.contains(['\n', '\0']) {
184 assert!(
185 result
186 .as_ref()
187 .err()
188 .is_none_or(|error| error.error.kind() != std::io::ErrorKind::InvalidData),
189 "Valid ordinary text edit was rejected: {result:?}"
190 );
191 }
176 match result {192 match result {
177 Ok(()) => assert_eq!(persisted, expected),193 Ok(()) => assert_eq!(persisted, expected),
178 Err(error) => match error.state {194 Err(error) => match error.state {
tools/native_collaboration.py+8
...@@ -54,6 +54,14 @@ def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1,...@@ -54,6 +54,14 @@ def replay(output, server, stress_clients=0, stress_operations=30, sync_every=1,
54 raise ValueError('Choose a new Linux VM name; existing machines are not owned by this run.')54 raise ValueError('Choose a new Linux VM name; existing machines are not owned by this run.')
55 output = output.resolve()55 output = output.resolve()
56 output.mkdir(parents=True, exist_ok=False)56 output.mkdir(parents=True, exist_ok=False)
57 for profile in (('debug', 'release') if client_profile == 'release' else ('debug',)):
58 build = ['cargo', 'build', '--locked', '--workspace', '--all-features', '--examples']
59 if profile == 'release': build.append('--release')
60 with (output / f'build-{profile}.log').open('w') as log:
61 result = subprocess.run(build, cwd=ROOT, env={**os.environ, 'CARGO_TARGET_DIR': str(ROOT / 'target')},
62 stdout=log, stderr=subprocess.STDOUT)
63 (output / f'build-{profile}.json').write_text(json.dumps({'command': build, 'exit': result.returncode}, indent=2))
64 result.check_returncode()
57 mount = output / 'mount'65 mount = output / 'mount'
58 mount.mkdir()66 mount.mkdir()
59 scripts = output / 'scripts'67 scripts = output / 'scripts'
tools/native_stress.py+5-5
...@@ -96,7 +96,7 @@ def verify_capture(output, capture):...@@ -96,7 +96,7 @@ def verify_capture(output, capture):
96 if config.get('document_operations'):96 if config.get('document_operations'):
97 from offline_document_history import document_history97 from offline_document_history import document_history
98 documents = document_history(logs, operations)98 documents = document_history(logs, operations)
99 expected.extend(document['text'] for document in documents.values())99 expected.extend(''.join(char for char, *_ in list(document['states'].values())[-1]['characters']) for document in documents.values())
100 page_file, = capture.glob('page-*.xml')100 page_file, = capture.glob('page-*.xml')
101 page = ET.parse(page_file).getroot()101 page = ET.parse(page_file).getroot()
102 paragraphs = native_characters(page, page.findall('one:Outline', ns))102 paragraphs = native_characters(page, page.findall('one:Outline', ns))
...@@ -111,8 +111,8 @@ def verify_capture(output, capture):...@@ -111,8 +111,8 @@ def verify_capture(output, capture):
111 checks += 1111 checks += 1
112 if documents:112 if documents:
113 from offline_document_history import verify_native113 from offline_document_history import verify_native
114 checks += verify_native(paragraphs, documents)114 checks += verify_native(paragraphs, (list(document['states'].values())[-1]['characters'] for document in documents.values()))
115 return {'rust_intents': len(commits), 'document_intents': len(documents)*2, 'native_intents': sum(map(len, native)),115 return {'rust_intents': len(commits), 'document_intents': sum(len(document['states']) for document in documents.values()), 'native_intents': sum(map(len, native)),
116 'exact_paragraphs': len(expected), 'native_intended_format_checks': checks}116 'exact_paragraphs': len(expected), 'native_intended_format_checks': checks}
117117
118118
...@@ -231,7 +231,7 @@ def exercise(output, shared, clients, action, wait_action, wait_text, checkpoint...@@ -231,7 +231,7 @@ def exercise(output, shared, clients, action, wait_action, wait_text, checkpoint
231 if config.get('document_operations'):231 if config.get('document_operations'):
232 from offline_document_history import document_history232 from offline_document_history import document_history
233 documents = document_history(rust, operations)233 documents = document_history(rust, operations)
234 expected.extend(document['text'] for document in documents.values())234 expected.extend(''.join(char for char, *_ in list(document['states'].values())[-1]['characters']) for document in documents.values())
235 for client in clients: wait_text(client, expected)235 for client in clients: wait_text(client, expected)
236 checkpoint('stress-final')236 checkpoint('stress-final')
237 model = json.loads((output / 'stress-final/model/document.json').read_text())237 model = json.loads((output / 'stress-final/model/document.json').read_text())
...@@ -262,7 +262,7 @@ def exercise(output, shared, clients, action, wait_action, wait_text, checkpoint...@@ -262,7 +262,7 @@ def exercise(output, shared, clients, action, wait_action, wait_text, checkpoint
262 earliest_end = (native['updated_ticks'] - 621355968000000000) // 10 - high262 earliest_end = (native['updated_ticks'] - 621355968000000000) // 10 - high
263 overlap += sum(max(latest_start, rust['started_us']) < min(earliest_end, rust['finished_us']) for rust in commits)263 overlap += sum(max(latest_start, rust['started_us']) < min(earliest_end, rust['finished_us']) for rust in commits)
264 result = {'native_writers': len(clients), 'rust_writers': rust_writers, 'rust_readers': rust_readers,264 result = {'native_writers': len(clients), 'rust_writers': rust_writers, 'rust_readers': rust_readers,
265 'sync_every': sync_every, 'edit': edit, 'seed': seed, 'offline': config.get('offline', False), 'native_edits': operations * len(clients), 'rust_commits': len(commits), 'document_commits': len(documents)*2, 'rust_reads': len(reads),265 'sync_every': sync_every, 'edit': edit, 'seed': seed, 'offline': config.get('offline', False), 'native_edits': operations * len(clients), 'rust_commits': len(commits), 'document_commits': sum(len(document['states']) for document in documents.values()), 'rust_reads': len(reads),
266 'clock_bounded_native_rust_call_overlaps': overlap, 'converged_paragraphs': len(expected)}266 'clock_bounded_native_rust_call_overlaps': overlap, 'converged_paragraphs': len(expected)}
267 (output / 'result.json').write_text(json.dumps(result, indent=2))267 (output / 'result.json').write_text(json.dumps(result, indent=2))
268 assert overlap, 'No native/Rust call overlap established within clock uncertainty'268 assert overlap, 'No native/Rust call overlap established within clock uncertainty'
tools/offline_document_history.py+42-20
...@@ -15,14 +15,21 @@ def characters(observed):...@@ -15,14 +15,21 @@ def characters(observed):
15 return actual15 return actual
1616
1717
18def operation_kinds(events):
19 kinds = tuple(events[0].get('document_kinds', ('insert', 'format')))
20 assert kinds in (('insert', 'format'), ('insert', 'format', 'text')), 'Unknown document workload'
21 return kinds
22
23
18def document_history(logs, operations):24def document_history(logs, operations):
19 documents = {}25 documents = {}
20 for actor, events in logs.items():26 for actor, events in logs.items():
21 if not actor.startswith('w'): continue27 if not actor.startswith('w'): continue
22 assert events[0].get('document_operations') is True, 'Writer omitted document operations'28 assert events[0].get('document_operations') is True, 'Writer omitted document operations'
29 kinds = operation_kinds(events)
23 edits = [row for row in events if row['event'] == 'local_document_commit']30 edits = [row for row in events if row['event'] == 'local_document_commit']
24 assert [(row['operation'], row['kind']) for row in edits] == [31 assert [(row['operation'], row['kind']) for row in edits] == [
25 (i, kind) for i in range(operations) for kind in ('insert', 'format')], 'Missing or duplicate document intent'32 (i, kind) for i in range(operations) for kind in kinds], 'Missing or duplicate document intent'
26 ids = [row['id'] for row in edits]33 ids = [row['id'] for row in edits]
27 assert ids == sorted(set(ids)), 'Document intent IDs are duplicated or unordered'34 assert ids == sorted(set(ids)), 'Document intent IDs are duplicated or unordered'
28 assert not set(ids) & {row['id'] for row in events if row['event'] == 'local_commit'}, 'Text and document intents share an ID'35 assert not set(ids) & {row['id'] for row in events if row['event'] == 'local_commit'}, 'Text and document intents share an ID'
...@@ -64,7 +71,8 @@ def document_history(logs, operations):...@@ -64,7 +71,8 @@ def document_history(logs, operations):
64 else:71 else:
65 assert receipt['revision'] == attempt['revision']72 assert receipt['revision'] == attempt['revision']
66 linked[intent['id']] = {**attempt, 'acknowledged_us': receipt['at_us'], 'receipt_revision': receipt['revision']}73 linked[intent['id']] = {**attempt, 'acknowledged_us': receipt['at_us'], 'receipt_revision': receipt['revision']}
67 for inserted, formatted in zip(edits[::2], edits[1::2], strict=True):74 for at in range(0, len(edits), len(kinds)):
75 inserted, formatted, *replaced = edits[at:at + len(kinds)]
68 number = inserted['operation']76 number = inserted['operation']
69 text = f'Document {actor}:{number} 🦀'77 text = f'Document {actor}:{number} 🦀'
70 insertion = inserted['insertion']78 insertion = inserted['insertion']
...@@ -76,7 +84,7 @@ def document_history(logs, operations):...@@ -76,7 +84,7 @@ def document_history(logs, operations):
76 assert insertion['placement'] == {'Outline': {'x': 144 + int(actor[1:]) * 240, 'y': 144 + number * 72}}, 'Outline placement differs from intent'84 assert insertion['placement'] == {'Outline': {'x': 144 + int(actor[1:]) * 240, 'y': 144 + number * 72}}, 'Outline placement differs from intent'
77 else:85 else:
78 assert insertion['placement'] == {'Paragraph': {'before': None}}86 assert insertion['placement'] == {'Paragraph': {'before': None}}
79 assert insertion['parent'] == identity(edits[(number-1)*2]['insertion'], 1), 'Paragraph lost its outline parent'87 assert insertion['parent'] == identity(edits[(number-1)*len(kinds)]['insertion'], 1), 'Paragraph lost its outline parent'
80 assert formatted['range'] == [1, len(text.encode('utf-16-le')) // 2 - 2]88 assert formatted['range'] == [1, len(text.encode('utf-16-le')) // 2 - 2]
81 assert formatted['attributes'] == [{'Bold': True}, {'FontSize': 18 + number % 9}, {'Color': [18, 52, 86]}]89 assert formatted['attributes'] == [{'Bold': True}, {'FontSize': 18 + number % 9}, {'Color': [18, 52, 86]}]
82 old = [(char, False, 11, 0xff000000) for char in text]90 old = [(char, False, 11, 0xff000000) for char in text]
...@@ -91,8 +99,20 @@ def document_history(logs, operations):...@@ -91,8 +99,20 @@ def document_history(logs, operations):
91 assert created['finished_us'] <= changed['started_us'], 'Formatting preceded its insertion'99 assert created['finished_us'] <= changed['started_us'], 'Formatting preceded its insertion'
92 assert characters(created['documents'][target]) == old, 'Insertion publication differs from its local intent'100 assert characters(created['documents'][target]) == old, 'Insertion publication differs from its local intent'
93 assert characters(changed['documents'][target]) == new, 'Formatting publication differs from its local intent'101 assert characters(changed['documents'][target]) == new, 'Formatting publication differs from its local intent'
94 documents[target] = {'text': text, 'insertion': insertion, 'space': inserted['space'],102 states = {'insert': {'characters': old, 'attempt': created},
95 'old': old, 'new': new, 'insert': created, 'format': changed}103 'format': {'characters': new, 'attempt': changed}}
104 if replaced:
105 replacement, = replaced
106 end = len(text.encode('utf-16-le')) // 2
107 assert replacement['object'] == target and replacement['space'] == inserted['space'], 'Text edit addresses another object'
108 assert replacement['text'] == text and replacement['range'] == [end-3, end], 'Cross-run text range differs from workload'
109 assert replacement['replacement'] == ' e\u0301🐈', 'Cross-run replacement differs from workload'
110 final = new[:-2] + [(char, *new[-2][1:]) for char in replacement['replacement']]
111 attempt = linked[replacement['id']]
112 assert changed['finished_us'] <= attempt['started_us'], 'Text replacement preceded its formatting'
113 assert characters(attempt['documents'][target]) == final, 'Text publication differs from its local intent'
114 states['text'] = {'characters': final, 'attempt': attempt}
115 documents[target] = {'insertion': insertion, 'space': inserted['space'], 'states': states}
96 assert documents, 'No document operations were recorded'116 assert documents, 'No document operations were recorded'
97 for actor, events in logs.items():117 for actor, events in logs.items():
98 previous = {}118 previous = {}
...@@ -103,19 +123,20 @@ def document_history(logs, operations):...@@ -103,19 +123,20 @@ def document_history(logs, operations):
103 observed = read['documents']123 observed = read['documents']
104 assert set(previous) <= set(observed) <= set(documents), 'Reader lost an object or observed an unrecorded insertion'124 assert set(previous) <= set(observed) <= set(documents), 'Reader lost an object or observed an unrecorded insertion'
105 for target, document in documents.items():125 for target, document in documents.items():
106 if read['started_us'] > document['insert']['acknowledged_us']:126 states = list(document['states'].values())
127 if read['started_us'] > states[0]['attempt']['acknowledged_us']:
107 assert target in observed, 'Reader missed an acknowledged insertion'128 assert target in observed, 'Reader missed an acknowledged insertion'
108 if target not in observed: continue129 if target not in observed: continue
109 assert read['finished_us'] >= document['insert']['started_us'], 'Reader observed a future insertion'
110 actual = characters(observed[target])130 actual = characters(observed[target])
111 assert actual in (document['old'], document['new']), 'Reader observed partial or invented formatting'131 matches = [i for i, state in enumerate(states) if actual == state['characters']]
112 formatted = actual == document['new']132 assert len(matches) == 1, 'Reader observed partial or invented document content'
113 assert not previous.get(target, False) or formatted, 'Reader reverted acknowledged formatting'133 current, = matches
114 if read['started_us'] > document['format']['acknowledged_us']:134 assert current >= previous.get(target, 0), 'Reader reverted document content'
115 assert formatted, 'Reader missed acknowledged formatting'135 assert read['finished_us'] >= states[current]['attempt']['started_us'], 'Reader observed future document content'
116 if formatted:136 for i, state in enumerate(states):
117 assert read['finished_us'] >= document['format']['started_us'], 'Reader observed future formatting'137 if read['started_us'] > state['attempt']['acknowledged_us']:
118 previous[target] = formatted138 assert current >= i, 'Reader missed acknowledged document content'
139 previous[target] = current
119 return documents140 return documents
120141
121142
...@@ -138,7 +159,8 @@ def verify_model(model, documents):...@@ -138,7 +159,8 @@ def verify_model(model, documents):
138 found.add(target)159 found.add(target)
139 expected = documents[target]160 expected = documents[target]
140 insertion = expected['insertion']161 insertion = expected['insertion']
141 assert sid == expected['space'] and node['kind']['text'] == expected['text']162 final = list(expected['states'].values())[-1]['characters']
163 assert sid == expected['space'] and node['kind']['text'] == ''.join(char for char, *_ in final)
142 object_id = identity(insertion, 1)164 object_id = identity(insertion, 1)
143 assert object_id in nodes[insertion['parent']]['children'], 'Insertion lost its parent'165 assert object_id in nodes[insertion['parent']]['children'], 'Insertion lost its parent'
144 paragraph = identity(insertion, 3) if 'Outline' in insertion['placement'] else object_id166 paragraph = identity(insertion, 3) if 'Outline' in insertion['placement'] else object_id
...@@ -150,13 +172,13 @@ def verify_model(model, documents):...@@ -150,13 +172,13 @@ def verify_model(model, documents):
150 assert found == set(documents), 'Final model omitted an inserted object'172 assert found == set(documents), 'Final model omitted an inserted object'
151173
152174
153def verify_native(paragraphs, documents):175def verify_native(paragraphs, expected):
154 from PIL import ImageColor176 from PIL import ImageColor
155 by_text = {''.join(char for char, _ in paragraph): paragraph for paragraph in paragraphs}177 by_text = {''.join(char for char, _ in paragraph): paragraph for paragraph in paragraphs}
156 checks = 0178 checks = 0
157 for document in documents.values():179 for final in expected:
158 actual = by_text[document['text']]180 actual = by_text[''.join(char for char, *_ in final)]
159 for (char, style), (wanted, bold, size, color) in zip(actual, document['new'], strict=True):181 for (char, style), (wanted, bold, size, color) in zip(actual, final, strict=True):
160 assert char == wanted and bool(style.get('bold')) == bold, 'Native text or bold differs from intent'182 assert char == wanted and bool(style.get('bold')) == bold, 'Native text or bold differs from intent'
161 assert style.get('font_size', 11) == size, 'Native font size differs from intent'183 assert style.get('font_size', 11) == size, 'Native font size differs from intent'
162 native_color = style.get('color', 'automatic')184 native_color = style.get('color', 'automatic')
tools/offline_outage.py+9-7
...@@ -8,6 +8,7 @@ import time...@@ -8,6 +8,7 @@ import time
8from native_runner import windows8from native_runner import windows
9import linux_vm9import linux_vm
10from verify_smb_overlap import verify10from verify_smb_overlap import verify
11from offline_document_history import operation_kinds
1112
1213
13def interrupt(output, clients, sequences, processes):14def interrupt(output, clients, sequences, processes):
...@@ -104,7 +105,7 @@ def interrupt(output, clients, sequences, processes):...@@ -104,7 +105,7 @@ def interrupt(output, clients, sequences, processes):
104 return105 return
105 wait_for(lambda: all((folder / f'offline-paused-{actor}').exists() for actor in writers)106 wait_for(lambda: all((folder / f'offline-paused-{actor}').exists() for actor in writers)
106 and (not config.get('document_operations') or all(107 and (not config.get('document_operations') or all(
107 sum(row['event'] == 'local_document_commit' for row in logs()[actor]) == 2 for actor in writers))108 sum(row['event'] == 'local_document_commit' for row in logs()[actor]) == len(operation_kinds(logs()[actor])) for actor in writers))
108 and all(any(row['event'] == 'read' for row in logs()[actor]) for actor in readers),109 and all(any(row['event'] == 'read' for row in logs()[actor]) for actor in readers),
109 'Clients did not reach the pre-publication outage barrier')110 'Clients did not reach the pre-publication outage barrier')
110 samples['native_before'] = native_counts('before')111 samples['native_before'] = native_counts('before')
...@@ -115,7 +116,7 @@ def interrupt(output, clients, sequences, processes):...@@ -115,7 +116,7 @@ def interrupt(output, clients, sequences, processes):
115 (folder / 'offline-outage-down').touch()116 (folder / 'offline-outage-down').touch()
116 wait_for(lambda: all(sum(row['event'] == 'local_commit' for row in events) == 8 for actor, events in logs().items() if actor in writers)117 wait_for(lambda: all(sum(row['event'] == 'local_commit' for row in events) == 8 for actor, events in logs().items() if actor in writers)
117 and (not config.get('document_operations') or all(118 and (not config.get('document_operations') or all(
118 sum(row['event'] == 'local_document_commit' for row in logs()[actor]) == 16 for actor in writers))119 sum(row['event'] == 'local_document_commit' for row in logs()[actor]) == 8 * len(operation_kinds(logs()[actor])) for actor in writers))
119 and all(sum(row['event'] == 'transport_read_error' for row in logs()[actor]) > samples['reader_errors_before'][actor] for actor in readers),120 and all(sum(row['event'] == 'transport_read_error' for row in logs()[actor]) > samples['reader_errors_before'][actor] for actor in readers),
120 'Local queues or disconnected readers failed to progress during the outage')121 'Local queues or disconnected readers failed to progress during the outage')
121 time.sleep(3)122 time.sleep(3)
...@@ -185,16 +186,17 @@ def verify_outage(output):...@@ -185,16 +186,17 @@ def verify_outage(output):
185 receipts = [row for row in events if row['event'] == 'remote_receipt']186 receipts = [row for row in events if row['event'] == 'remote_receipt']
186 assert len(receipts) == config['stress_operations'] and all(row['at_us'] > up for row in receipts)187 assert len(receipts) == config['stress_operations'] and all(row['at_us'] > up for row in receipts)
187 if config.get('document_operations'):188 if config.get('document_operations'):
189 kinds = operation_kinds(events)
188 edits = [row for row in events if row['event'] == 'local_document_commit']190 edits = [row for row in events if row['event'] == 'local_document_commit']
189 assert [(row['operation'], row['kind']) for row in edits] == [191 assert [(row['operation'], row['kind']) for row in edits] == [
190 (operation, kind) for operation in range(config['stress_operations']) for kind in ('insert', 'format')]192 (operation, kind) for operation in range(config['stress_operations']) for kind in kinds]
191 assert all(row['finished_us'] < down for row in edits[:2]), 'Initial document edits missed the outage barrier'193 assert all(row['finished_us'] < down for row in edits[:len(kinds)]), 'Initial document edits missed the outage barrier'
192 queued = [row for row in edits if down < row['started_us'] <= row['finished_us'] < up]194 queued = [row for row in edits if down < row['started_us'] <= row['finished_us'] < up]
193 assert [(row['operation'], row['kind']) for row in queued] == [195 assert [(row['operation'], row['kind']) for row in queued] == [
194 (operation, kind) for operation in range(1, 8) for kind in ('insert', 'format')], 'Document edits did not persist during the outage'196 (operation, kind) for operation in range(1, 8) for kind in kinds], 'Document edits did not persist during the outage'
195 queues[actor] += len(queued)197 queues[actor] += len(queued)
196 receipts = [row for row in events if row['event'] == 'document_receipt']198 receipts = [row for row in events if row['event'] == 'document_receipt']
197 assert len(receipts) == config['stress_operations'] * 2 and all(row['at_us'] > up for row in receipts)199 assert len(receipts) == config['stress_operations'] * len(kinds) and all(row['at_us'] > up for row in receipts)
198 trace = [json.loads(line) for line in (output / 'smb-trace.jsonl').read_text().splitlines()]200 trace = [json.loads(line) for line in (output / 'smb-trace.jsonl').read_text().splitlines()]
199 controls = [row['control'] for row in trace if row.get('control', {}).get('phase', '').startswith('offline-')]201 controls = [row['control'] for row in trace if row.get('control', {}).get('phase', '').startswith('offline-')]
200 assert controls == [{'phase': 'offline-down', 'mode': 'down'}, {'phase': 'offline-reconnected'}], 'Unexpected outage control sequence'202 assert controls == [{'phase': 'offline-down', 'mode': 'down'}, {'phase': 'offline-reconnected'}], 'Unexpected outage control sequence'
...@@ -233,7 +235,7 @@ def verify_lost_reply(output):...@@ -233,7 +235,7 @@ def verify_lost_reply(output):
233 receipts = [row for row in logs[actor] if row['event'] in ('remote_receipt', 'document_receipt') and row['revision'] == attempt['revision']]235 receipts = [row for row in logs[actor] if row['event'] in ('remote_receipt', 'document_receipt') and row['revision'] == attempt['revision']]
234 if not receipts and documents:236 if not receipts and documents:
235 target, = attempt['document_changes']237 target, = attempt['document_changes']
236 confirmed_revision = documents[target]['format']['receipt_revision']238 confirmed_revision = documents[target]['states']['format']['attempt']['receipt_revision']
237 receipts = [row for row in logs[actor] if row['event'] == 'document_receipt' and row['revision'] == confirmed_revision]239 receipts = [row for row in logs[actor] if row['event'] == 'document_receipt' and row['revision'] == confirmed_revision]
238 assert len(receipts) == 1 and receipts[0]['at_us'] > sample['up_started_us']240 assert len(receipts) == 1 and receipts[0]['at_us'] > sample['up_started_us']
239 peer_progress = {}241 peer_progress = {}
tools/test_cross_run_edit.py created+25
...@@ -0,0 +1,25 @@
1import json
2from pathlib import Path
3import runpy
4import shutil
5from tempfile import TemporaryDirectory
6import unittest
7
8ROOT = Path(__file__).resolve().parent.parent
9FIXTURE = ROOT / 'corpus/cross-run-edit'
10compare = runpy.run_path(str(ROOT / 'tools/verify-document.py'))['compare']
11
12
13class CrossRunEditTest(unittest.TestCase):
14 def test_cold_native_text_styles_and_structure(self):
15 for case in json.loads((FIXTURE / 'manifest.json').read_text())['cases']:
16 with self.subTest(case=case['name']), TemporaryDirectory() as temporary:
17 fixture = FIXTURE / case['name']
18 native = Path(temporary) / 'read'
19 shutil.copytree(fixture / 'cold/read', native)
20 compare(fixture / 'candidate', native)
21 compare(fixture / 'cold/notebook', native)
22
23
24if __name__ == '__main__':
25 unittest.main()
tools/test_native_runner.py+21
...@@ -1,4 +1,5 @@...@@ -1,4 +1,5 @@
1import json1import json
2import subprocess
2from pathlib import Path3from pathlib import Path
3import tempfile4import tempfile
4import unittest5import unittest
...@@ -8,6 +9,26 @@ import native_runner as runner...@@ -8,6 +9,26 @@ import native_runner as runner
89
910
10class NativeRunnerTest(unittest.TestCase):11class NativeRunnerTest(unittest.TestCase):
12 def test_failed_client_build_cannot_start_a_lab_with_stale_binaries(self):
13 import native_collaboration as collaboration
14 with tempfile.TemporaryDirectory() as temporary:
15 root = Path(temporary)
16 for profile, failed in (('debug', 'debug'), ('release', 'debug'), ('release', 'release')):
17 output = root / (profile + '-' + failed)
18 results = [subprocess.CompletedProcess(['cargo'], 1)]
19 if failed == 'release': results.insert(0, subprocess.CompletedProcess(['cargo'], 0))
20 with patch.object(collaboration, 'linux_vm') as linux, \
21 patch.object(collaboration.subprocess, 'run', side_effect=results) as build:
22 linux.instance_path.return_value = root / 'absent'
23 with self.assertRaises(subprocess.CalledProcessError):
24 collaboration.replay(output, 'fixture', client_profile=profile)
25 self.assertEqual(build.call_count, len(results))
26 linux.create_instance.assert_not_called()
27 linux.launch.assert_not_called()
28 receipt = json.loads((output / f'build-{failed}.json').read_text())
29 self.assertEqual(receipt['exit'], 1)
30 self.assertEqual('--release' in receipt['command'], failed == 'release')
31
11 def test_shutdown_timeout_still_removes_the_owned_clone(self):32 def test_shutdown_timeout_still_removes_the_owned_clone(self):
12 with tempfile.TemporaryDirectory() as temporary:33 with tempfile.TemporaryDirectory() as temporary:
13 output = Path(temporary)34 output = Path(temporary)
tools/test_offline_confirmation.py+2-2
...@@ -62,14 +62,14 @@ class ConfirmationOracle(unittest.TestCase):...@@ -62,14 +62,14 @@ class ConfirmationOracle(unittest.TestCase):
62 markup = '<one:OE><one:T><![CDATA[<span style="font-weight:bold;font-size:18pt;color:#123456">x</span>]]></one:T></one:OE>'62 markup = '<one:OE><one:T><![CDATA[<span style="font-weight:bold;font-size:18pt;color:#123456">x</span>]]></one:T></one:OE>'
63 content = xml('Concurrent edits: [w0:0]', 'Native 0:').replace('</one:OEChildren>', markup + '</one:OEChildren>')63 content = xml('Concurrent edits: [w0:0]', 'Native 0:').replace('</one:OEChildren>', markup + '</one:OEChildren>')
64 page.write_text(content)64 page.write_text(content)
65 with patch('offline_document_history.document_history', return_value={'target': {'text': 'x', 'format': {'receipt_revision': 'revision'}}}) as documents:65 with patch('offline_document_history.document_history', return_value={'target': {'states': {'format': {'attempt': {'receipt_revision': 'revision'}}}}}) as documents:
66 result = verify(root, cold)66 result = verify(root, cold)
67 self.assertEqual(result['validated_paragraphs'], 3)67 self.assertEqual(result['validated_paragraphs'], 3)
68 self.assertEqual(result['native_intended_format_checks'], 3)68 self.assertEqual(result['native_intended_format_checks'], 3)
69 documents.assert_called_once_with({'w0': rows}, 1)69 documents.assert_called_once_with({'w0': rows}, 1)
70 rows[3]['revisions'] = {'space': ['current']}70 rows[3]['revisions'] = {'space': ['current']}
71 rows[3]['current_revisions'] = {'space': 'current'}71 rows[3]['current_revisions'] = {'space': 'current'}
72 documents.return_value['target']['format']['receipt_revision'] = 'current'72 documents.return_value['target']['states']['format']['attempt']['receipt_revision'] = 'current'
73 (root / 'rust/w0.jsonl').write_text('\n'.join(map(json.dumps, rows)))73 (root / 'rust/w0.jsonl').write_text('\n'.join(map(json.dumps, rows)))
74 self.assertEqual(verify(root, cold)['confirmed_revision'], 'current')74 self.assertEqual(verify(root, cold)['confirmed_revision'], 'current')
75 rows[3]['current_revisions'] = {'space': 'unrelated'}75 rows[3]['current_revisions'] = {'space': 'unrelated'}
tools/test_offline_document_history.py+83-42
...@@ -6,52 +6,92 @@ from unittest.mock import patch...@@ -6,52 +6,92 @@ from unittest.mock import patch
6from offline_document_history import document_history, identity, verify_model, verify_native6from offline_document_history import document_history, identity, verify_model, verify_native
77
88
9def history(text_edits=False):
10 logs = {'w0': [{'event': 'ready', 'document_operations': True, 'document_kinds': ['insert', 'format', 'text'] if text_edits else ['insert', 'format']}]}
11 events = logs['w0']
12 observed = {}
13 previous = None
14 for operation in range(2):
15 insertion = {'guid': list(uuid.UUID(int=operation+1).bytes_le), 'text': f'Document w0:{operation} 🦀',
16 'parent': 'page' if operation == 0 else identity(previous, 1), 'author': 'Offline document writer',
17 'placement': {'Outline': {'x': 144, 'y': 144}} if operation == 0 else {'Paragraph': {'before': None}}}
18 previous = insertion
19 target = identity(insertion, 2)
20 for step, kind in enumerate(('insert', 'format', 'text') if text_edits else ('insert', 'format')):
21 timestamp = 10 + operation*30 + step*10
22 local_id = operation*3 + step + 2
23 event = {'event': 'local_document_commit', 'id': local_id, 'operation': operation, 'kind': kind,
24 'space': 'space', 'object': target, 'text': insertion['text'], 'insertion': insertion if step == 0 else None,
25 'range': [1, len(insertion['text'].encode('utf-16-le'))//2-2],
26 'attributes': [{'Bold': True}, {'FontSize': 18+operation}, {'Color': [18, 52, 86]}],
27 'started_us': timestamp-2, 'finished_us': timestamp-1}
28 if kind == 'text':
29 end = len(insertion['text'].encode('utf-16-le'))//2
30 event.update(range=[end-3, end], replacement=' e\u0301🐈')
31 events.append(event)
32 value = insertion['text'].replace(' 🦀', ' e\u0301🐈') if kind == 'text' else insertion['text']
33 runs = []
34 for index, char in enumerate(value):
35 selected = index > 0 and (kind == 'text' or (kind == 'format' and index < len(value)-1))
36 runs.append({'text': char, 'bold': selected, 'size': 18+operation if selected else 11,
37 'color': 0x563412 if selected else 0xff000000})
38 observed[target] = {'text': value, 'runs': runs}
39 revision = f'revision-{local_id}'
40 events.append({'event': 'remote_attempt', 'revision': revision, 'state': 'Committed',
41 'document_changes': {target: copy.deepcopy(observed[target])},
42 'documents': copy.deepcopy(observed), 'started_us': timestamp, 'finished_us': timestamp+1})
43 events.append({'event': 'document_receipt', 'id': local_id, 'revision': revision, 'at_us': timestamp+2})
44 if text_edits:
45 events.append({'event': 'read', 'started_us': timestamp+3, 'finished_us': timestamp+4, 'documents': copy.deepcopy(observed)})
46 events.extend({'event': 'reopened_document_receipt', 'id': row['id'], 'revision': row['revision']}
47 for row in list(events) if row['event'] == 'document_receipt')
48 read = {'event': 'read', 'started_us': 100, 'finished_us': 101, 'documents': observed}
49 events.extend([read, {'event': 'done'}])
50 logs['r0'] = [{'event': 'ready'}, copy.deepcopy(read), {'event': 'done'}]
51 return logs
52
53
9class DocumentHistoryTests(unittest.TestCase):54class DocumentHistoryTests(unittest.TestCase):
10 def setUp(self):55 def setUp(self):
11 self.logs = {'w0': [{'event': 'ready', 'document_operations': True}]}56 self.logs = history()
12 events = self.logs['w0']
13 observed = {}
14 previous = None
15 for operation in range(2):
16 insertion = {'guid': list(uuid.UUID(int=operation+1).bytes_le), 'text': f'Document w0:{operation} 🦀',
17 'parent': 'page' if operation == 0 else identity(previous, 1), 'author': 'Offline document writer',
18 'placement': {'Outline': {'x': 144, 'y': 144}} if operation == 0 else {'Paragraph': {'before': None}}}
19 previous = insertion
20 target = identity(insertion, 2)
21 for step, kind in enumerate(('insert', 'format')):
22 timestamp = 10 + operation*30 + step*10
23 local_id = operation*3 + step + 2
24 event = {'event': 'local_document_commit', 'id': local_id, 'operation': operation, 'kind': kind,
25 'space': 'space', 'object': target, 'text': insertion['text'], 'insertion': insertion if step == 0 else None,
26 'range': [1, len(insertion['text'].encode('utf-16-le'))//2-2],
27 'attributes': [{'Bold': True}, {'FontSize': 18+operation}, {'Color': [18, 52, 86]}],
28 'started_us': timestamp-2, 'finished_us': timestamp-1}
29 events.append(event)
30 runs = []
31 for index, char in enumerate(insertion['text']):
32 selected = kind == 'format' and 0 < index < len(insertion['text'])-1
33 runs.append({'text': char, 'bold': selected, 'size': 18+operation if selected else 11,
34 'color': 0x563412 if selected else 0xff000000})
35 observed[target] = {'text': insertion['text'], 'runs': runs}
36 revision = f'revision-{local_id}'
37 events.append({'event': 'remote_attempt', 'revision': revision, 'state': 'Committed',
38 'document_changes': {target: copy.deepcopy(observed[target])},
39 'documents': copy.deepcopy(observed), 'started_us': timestamp, 'finished_us': timestamp+1})
40 events.append({'event': 'document_receipt', 'id': local_id, 'revision': revision, 'at_us': timestamp+2})
41 events.extend({'event': 'reopened_document_receipt', 'id': row['id'], 'revision': row['revision']}
42 for row in list(events) if row['event'] == 'document_receipt')
43 read = {'event': 'read', 'started_us': 100, 'finished_us': 101, 'documents': observed}
44 events.extend([read, {'event': 'done'}])
45 self.logs['r0'] = [{'event': 'ready'}, copy.deepcopy(read), {'event': 'done'}]
4657
47 def test_document_receipts_and_reader_states_match_the_intents(self):58 def test_document_receipts_and_reader_states_match_the_intents(self):
48 documents = document_history(self.logs, 2)59 documents = document_history(self.logs, 2)
49 self.assertEqual(len(documents), 2)60 self.assertEqual(len(documents), 2)
61 expected = [list(row['states'].values())[-1]['characters'] for row in documents.values()]
50 paragraphs = [[(char, {'bold': bold, 'font_size': size, 'color': 'automatic' if color == 0xff000000 else '#123456'})62 paragraphs = [[(char, {'bold': bold, 'font_size': size, 'color': 'automatic' if color == 0xff000000 else '#123456'})
51 for char, bold, size, color in row['new']] for row in documents.values()]63 for char, bold, size, color in row] for row in expected]
52 self.assertEqual(verify_native(paragraphs, documents), sum(len(row['new'])*3 for row in documents.values()))64 self.assertEqual(verify_native(paragraphs, expected), sum(len(row)*3 for row in expected))
53 paragraphs[0][1][1]['font_size'] = 1965 paragraphs[0][1][1]['font_size'] = 19
54 with self.assertRaisesRegex(AssertionError, 'font size'): verify_native(paragraphs, documents)66 with self.assertRaisesRegex(AssertionError, 'font size'): verify_native(paragraphs, expected)
67
68 def test_cross_run_text_requires_complete_ordered_states_and_exact_receipts(self):
69 logs = history(True)
70 documents = document_history(logs, 2)
71 for document in documents.values():
72 self.assertEqual(list(document['states']), ['insert', 'format', 'text'])
73 final = document['states']['text']['characters']
74 self.assertEqual(''.join(char for char, *_ in final), document['insertion']['text'].replace(' 🦀', ' e\u0301🐈'))
75 for event, field, value in [('local_document_commit', 'replacement', 'wrong'),
76 ('local_document_commit', 'range', [0, 1]),
77 ('document_receipt', 'revision', 'wrong')]:
78 changed = copy.deepcopy(logs)
79 row = next(row for row in changed['w0'] if row['event'] == event and
80 (row.get('kind') == 'text' or row.get('id') == 4))
81 row[field] = value
82 with self.subTest(event=event, field=field), self.assertRaises(AssertionError):
83 document_history(changed, 2)
84 changed = copy.deepcopy(logs)
85 read = changed['r0'][1]
86 target = next(iter(read['documents']))
87 read['documents'][target]['runs'][-1]['bold'] = False
88 with self.assertRaisesRegex(AssertionError, 'partial or invented'):
89 document_history(changed, 2)
90 prior = next(row for row in logs['w0'] if row['event'] == 'read' and row['started_us'] == 23)
91 changed = copy.deepcopy(logs)
92 changed['r0'][1]['documents'][target] = copy.deepcopy(prior['documents'][target])
93 with self.assertRaisesRegex(AssertionError, 'missed acknowledged'):
94 document_history(changed, 2)
5595
56 def test_missing_intents_receipts_or_reopen_records_are_rejected(self):96 def test_missing_intents_receipts_or_reopen_records_are_rejected(self):
57 for name in ('local_document_commit', 'document_receipt', 'reopened_document_receipt', 'remote_attempt'):97 for name in ('local_document_commit', 'document_receipt', 'reopened_document_receipt', 'remote_attempt'):
...@@ -75,10 +115,11 @@ class DocumentHistoryTests(unittest.TestCase):...@@ -75,10 +115,11 @@ class DocumentHistoryTests(unittest.TestCase):
75 run['size'] = 13.5115 run['size'] = 13.5
76 run['color'] = 0x665544116 run['color'] = 0x665544
77 documents = document_history(self.logs, 2)117 documents = document_history(self.logs, 2)
118 expected = [list(row['states'].values())[-1]['characters'] for row in documents.values()]
78 paragraphs = [[(char, {'bold': bold, 'font_size': size,119 paragraphs = [[(char, {'bold': bold, 'font_size': size,
79 'color': '#123456' if color == 0x563412 else '#445566'})120 'color': '#123456' if color == 0x563412 else '#445566'})
80 for char, bold, size, color in row['new']] for row in documents.values()]121 for char, bold, size, color in row] for row in expected]
81 self.assertEqual(verify_native(paragraphs, documents), sum(len(row['new'])*3 for row in documents.values()))122 self.assertEqual(verify_native(paragraphs, expected), sum(len(row)*3 for row in expected))
82 baseline = copy.deepcopy(self.logs)123 baseline = copy.deepcopy(self.logs)
83 for event, key in [('remote_attempt', 'document_changes'), ('read', 'documents')]:124 for event, key in [('remote_attempt', 'document_changes'), ('read', 'documents')]:
84 for field, value in [('size', 11), ('color', 0xff000000)]:125 for field, value in [('size', 11), ('color', 0xff000000)]:
...@@ -102,7 +143,7 @@ class DocumentHistoryTests(unittest.TestCase):...@@ -102,7 +143,7 @@ class DocumentHistoryTests(unittest.TestCase):
102 events[index:index] = [read, confirmation]143 events[index:index] = [read, confirmation]
103 result = document_history(self.logs, 2)144 result = document_history(self.logs, 2)
104 target, = attempt['document_changes']145 target, = attempt['document_changes']
105 self.assertEqual(result[target]['format']['receipt_revision'], 'current-revision')146 self.assertEqual(result[target]['states']['format']['attempt']['receipt_revision'], 'current-revision')
106 for field, value in [('current_revisions', {'space': 'unrelated'}),147 for field, value in [('current_revisions', {'space': 'unrelated'}),
107 ('revisions', {'space': ['revision-3', 'current-revision']}),148 ('revisions', {'space': ['revision-3', 'current-revision']}),
108 ('state', 'NotCommitted')]:149 ('state', 'NotCommitted')]:
...@@ -123,7 +164,7 @@ class DocumentHistoryTests(unittest.TestCase):...@@ -123,7 +164,7 @@ class DocumentHistoryTests(unittest.TestCase):
123 nodes[outline].update(children=[paragraph, appended], layout={'x': 144, 'y': 144})164 nodes[outline].update(children=[paragraph, appended], layout={'x': 144, 'y': 144})
124 for parent, (target, document) in zip([paragraph, appended], documents.items(), strict=True):165 for parent, (target, document) in zip([paragraph, appended], documents.items(), strict=True):
125 nodes[parent]['content'] = [target]166 nodes[parent]['content'] = [target]
126 nodes[target]['kind'] = {'text': document['text']}167 nodes[target]['kind'] = {'text': ''.join(char for char, *_ in list(document['states'].values())[-1]['characters'])}
127 revision = {'nodes': nodes}168 revision = {'nodes': nodes}
128 with patch('offline_document_history.ordered_pages', return_value=[('space', 'revision', revision, 'page')]):169 with patch('offline_document_history.ordered_pages', return_value=[('space', 'revision', revision, 'page')]):
129 verify_model({}, documents)170 verify_model({}, documents)
tools/test_offline_outage.py+1-1
...@@ -85,7 +85,7 @@ class OutageOracle(unittest.TestCase):...@@ -85,7 +85,7 @@ class OutageOracle(unittest.TestCase):
85 (self.root / 'rust' / f'{actor}.jsonl').write_text('\n'.join(map(json.dumps, rows)))85 (self.root / 'rust' / f'{actor}.jsonl').write_text('\n'.join(map(json.dumps, rows)))
86 with patch('offline_history.publication_links') as ledger, patch('offline_document_history.document_history') as documents, patch('offline_outage.verify', return_value={'guarded_pairs': 1}) as overlap:86 with patch('offline_history.publication_links') as ledger, patch('offline_document_history.document_history') as documents, patch('offline_outage.verify', return_value={'guarded_pairs': 1}) as overlap:
87 if self.config.get('offline_client_reply'):87 if self.config.get('offline_client_reply'):
88 documents.return_value = {'target': {'format': {'receipt_revision': 'effect-revision'}}}88 documents.return_value = {'target': {'states': {'format': {'attempt': {'receipt_revision': 'effect-revision'}}}}}
89 result = verify_lost_reply(self.root)89 result = verify_lost_reply(self.root)
90 ledger.assert_called_once_with(self.logs, self.config['stress_operations'])90 ledger.assert_called_once_with(self.logs, self.config['stress_operations'])
91 if self.config.get('document_operations'):91 if self.config.get('document_operations'):
tools/verify_offline.py+2-2
...@@ -32,7 +32,7 @@ def verify(output, max_gap=120):...@@ -32,7 +32,7 @@ def verify(output, max_gap=120):
32 edits = {event['id']: event for event in events if event['event'] in ('local_commit', 'local_document_commit')}32 edits = {event['id']: event for event in events if event['event'] in ('local_commit', 'local_document_commit')}
33 receipts = {event['id']: event for event in events if event['event'] in ('remote_receipt', 'document_receipt')}33 receipts = {event['id']: event for event in events if event['event'] in ('remote_receipt', 'document_receipt')}
34 attempts = {event['revision']: event for event in events if event['event'] == 'remote_attempt'}34 attempts = {event['revision']: event for event in events if event['event'] == 'remote_attempt'}
35 publication_starts = {id: documents[edit['object']][edit['kind']]['started_us'] if edit['event'] == 'local_document_commit'35 publication_starts = {id: documents[edit['object']]['states'][edit['kind']]['attempt']['started_us'] if edit['event'] == 'local_document_commit'
36 else attempts[receipts[id]['revision']]['started_us'] for id, edit in edits.items()}36 else attempts[receipts[id]['revision']]['started_us'] for id, edit in edits.items()}
37 queues[actor] = max(sum(edit['finished_us'] <= at < publication_starts[id] for id, edit in edits.items()) for at in [edit['finished_us'] for edit in edits.values()])37 queues[actor] = max(sum(edit['finished_us'] <= at < publication_starts[id] for id, edit in edits.items()) for at in [edit['finished_us'] for edit in edits.values()])
38 assert queues[actor] >= 2, 'Writer did not establish a durable local queue before publication'38 assert queues[actor] >= 2, 'Writer did not establish a durable local queue before publication'
...@@ -57,7 +57,7 @@ def verify(output, max_gap=120):...@@ -57,7 +57,7 @@ def verify(output, max_gap=120):
57 assert times == sorted(times)57 assert times == sorted(times)
58 progress[f'n{i}'] = max((b-a for a, b in zip(times, times[1:])), default=0) / 10_000_00058 progress[f'n{i}'] = max((b-a for a, b in zip(times, times[1:])), default=0) / 10_000_000
59 assert all(gap <= max_gap for gap in progress.values()), f'Client progress exceeded {max_gap}s: {progress}'59 assert all(gap <= max_gap for gap in progress.values()), f'Client progress exceeded {max_gap}s: {progress}'
60 return {'remote_publications': len(commits), 'document_publications': len(documents)*2, 'remote_text_sha256': hashlib.sha256(text.encode()).hexdigest(), 'maximum_progress_gap_seconds': progress,60 return {'remote_publications': len(commits), 'document_publications': sum(len(document['states']) for document in documents.values()), 'remote_text_sha256': hashlib.sha256(text.encode()).hexdigest(), 'maximum_progress_gap_seconds': progress,
61 'queued_before_publication_lower_bound': queues, 'reviewed_placements': sum(event['event'] == 'reviewed_append' for events in logs.values() for event in events), 'caches': caches}61 'queued_before_publication_lower_bound': queues, 'reviewed_placements': sum(event['event'] == 'reviewed_append' for events in logs.values() for event in events), 'caches': caches}
6262
6363
tools/verify_offline_confirmation.py+3-4
...@@ -68,9 +68,8 @@ def verify(output, cold, *, partial=False):...@@ -68,9 +68,8 @@ def verify(output, cold, *, partial=False):
68 paragraphs.remove(expected)68 paragraphs.remove(expected)
69 if observed:69 if observed:
70 from offline_document_history import characters, verify_native70 from offline_document_history import characters, verify_native
71 expected_documents = {target: {'text': documents[target]['text'], 'new': characters(value)} for target, value in observed.items()}71 format_checks += verify_native(formatted, (characters(value) for value in observed.values()))
72 format_checks += verify_native(formatted, expected_documents)72 for document in observed.values():
73 for document in expected_documents.values():
74 assert paragraphs.count(document['text']) == 1, 'Confirmation duplicated a document paragraph'73 assert paragraphs.count(document['text']) == 1, 'Confirmation duplicated a document paragraph'
75 paragraphs.remove(document['text'])74 paragraphs.remove(document['text'])
76 for index, versions in enumerate(native):75 for index, versions in enumerate(native):
...@@ -78,7 +77,7 @@ def verify(output, cold, *, partial=False):...@@ -78,7 +77,7 @@ def verify(output, cold, *, partial=False):
78 assert len(selected) == 1 and selected[0] in versions, 'Native confirmation image contains an unrecorded edit or loses a prefix'77 assert len(selected) == 1 and selected[0] in versions, 'Native confirmation image contains an unrecorded edit or loses a prefix'
79 checks += len(native) + 1 + len(observed)78 checks += len(native) + 1 + len(observed)
80 assert json.loads((cold / 'teardown.json').read_text()) == {'absent': True}79 assert json.loads((cold / 'teardown.json').read_text()) == {'absent': True}
81 confirmed_revision = documents[next(iter(attempt['document_changes']))]['format']['receipt_revision'] if documents else attempt['revision']80 confirmed_revision = documents[next(iter(attempt['document_changes']))]['states']['format']['attempt']['receipt_revision'] if documents else attempt['revision']
82 return {'complete_workload': not partial, 'attempted_revision': attempt['revision'], 'confirmed_revision': confirmed_revision, 'native_images': len(results), 'exact_rust_paragraphs': len(results),81 return {'complete_workload': not partial, 'attempted_revision': attempt['revision'], 'confirmed_revision': confirmed_revision, 'native_images': len(results), 'exact_rust_paragraphs': len(results),
83 'validated_paragraphs': checks, 'native_intended_format_checks': format_checks,82 'validated_paragraphs': checks, 'native_intended_format_checks': format_checks,
84 'maximum_native_export_seconds': max(row['seconds'] for row in results)}83 'maximum_native_export_seconds': max(row['seconds'] for row in results)}