authorgravatar for git@paperclover.netclover caruso <git@paperclover.net> 2026-09-08 14:28:35-07:00
committergravatar for git@paperclover.netclover caruso <git@paperclover.net> 2026-09-08 14:28:35-07:00
log8bcca78a05a23695cd0b16382164810da6d0c96e
treeced4fcdfe203aa93f0e6fe55540cd647d4d274e9
parentfa4842b0710f130a915f5ae0cea755b17503825f
signature Signed by SSH key SHA256:52mNGHRsVFBDED9IAX5pe+LRWUefqTbxEReunq21QvU

fix: reconcile native subtree normalization in offline deletion

Compare immutable referenced records by content and normalize absent versus empty child lists without weakening mutable identity or unknown-property checks. Preserve version-eight deletion observations until explicit conflict review. Extend the shared native/Rust workload to ten document phases with independent graph and receipt verification. Retain the native regression and migration controls; offline tests, mixed twelve-client outage run, cold OneNote comparison, ASan schedules, Clippy and iOS checks pass. Assisted-by: gpt-6-astra

13 files changed, 530 insertions(+), 56 deletions(-)

corpus/outline-edit/README.md+7
...@@ -1,5 +1,12 @@...@@ -1,5 +1,12 @@
1# Native outline editing controls1# Native outline editing controls
22
3`empty-children` retains a mixed-client deletion regression: `before.one` exposes
4transaction 22 of the retained local image; `remote.one` was observed after native
5OneNote removed an empty child-list property and coalesced equivalent immutable
6styles. The retained intent addresses the same subtree in both files. The offline
7test requires deletion and its dependent edit to publish after reopening the
8cache, while an actual remote text change retains a content conflict.
9
3OneNote 2010 authored fourteen cases plus an unchanged source page. `before`,10OneNote 2010 authored fourteen cases plus an unchanged source page. `before`,
4`after`, and `cold` retain notebook bytes and independent native XML. The final11`after`, and `cold` retain notebook bytes and independent native XML. The final
5phase uses a fresh application cache. Cases preserve Unicode, hyperlinks, tags,12phase uses a fresh application cache. Cases preserve Unicode, hyperlinks, tags,
corpus/outline-edit/empty-children/before.one created
Binary files /dev/null and b/corpus/outline-edit/empty-children/before.one differ
corpus/outline-edit/empty-children/intent.json created+27
...@@ -0,0 +1,27 @@
1[
2 "{C9F8B62B-D016-44A6-BD6D-AD427C226A53},1",
3 {
4 "guid": [
5 247,
6 43,
7 64,
8 79,
9 30,
10 47,
11 235,
12 73,
13 132,
14 188,
15 117,
16 194,
17 60,
18 28,
19 252,
20 6
21 ],
22 "object": "{91881B6E-F0D0-424E-917E-9D3BEF7D35C7},1",
23 "placement": "Delete",
24 "author": "Offline document writer",
25 "created": 1473370703
26 }
27]
corpus/outline-edit/empty-children/remote.one created
Binary files /dev/null and b/corpus/outline-edit/empty-children/remote.one differ
crates/onestore-offline/README.md+7-4
...@@ -107,16 +107,19 @@ already satisfied move still requires guarded confirmation before acknowledgemen...@@ -107,16 +107,19 @@ already satisfied move still requires guarded confirmation before acknowledgemen
107Deletion compares the selected raw property graph, including referenced styles,107Deletion compares the selected raw property graph, including referenced styles,
108tags, unknown fields and internal attachments. Changed content produces108tags, unknown fields and internal attachments. Changed content produces
109`ContentChanged`; property order, CompactID numbering and modification timestamps109`ContentChanged`; property order, CompactID numbering and modification timestamps
110do not affect that comparison. Missing targets retain `TargetUnavailable`.110do not affect that comparison. Immutable records compare by content, and empty
111child lists compare equally to absent child lists. Mutable object identities
112remain significant. Missing targets retain `TargetUnavailable`.
111`rebase_tree_conflict(id, local, remote)` reviews the original move/deletion against113`rebase_tree_conflict(id, local, remote)` reviews the original move/deletion against
112both current images. An emptied cell's replacement paragraph/text identities must114both current images. An emptied cell's replacement paragraph/text identities must
113remain the same before replay or review; later queued edits keep their targets.115remain the same before replay or review; later queued edits keep their targets.
114Uncertain tree attempts require their original revision for confirmation, even116Uncertain tree attempts require their original revision for confirmation, even
115when an independent move or deletion has the same visible effect.117when an independent move or deletion has the same visible effect.
116118
117Recognized earlier caches migrate transactionally to version eight, which adds119Recognized earlier caches migrate transactionally to version nine. Version-eight
118tree intents and content conflicts. The migration retains120deletion observations retain their original identity-sensitive preconditions;
119images, local IDs, publication attempts, conflicts,121explicit conflict review upgrades them to the immutable-content comparison.
122The migration retains images, local IDs, publication attempts, conflicts,
120receipts and the autoincrement sequence; it does not reuse acknowledged IDs when123receipts and the autoincrement sequence; it does not reuse acknowledged IDs when
121the pending queue is empty.124the pending queue is empty.
122125
crates/onestore-offline/examples/smb_offline_client.rs+69-6
...@@ -3,7 +3,7 @@ mod concurrent;...@@ -3,7 +3,7 @@ mod concurrent;
33
4use onestore::{4use onestore::{
5 CommitError, CommitState, ExGuid, Insertion, ParagraphJoin, ParagraphSplit, PreparedEdit,5 CommitError, CommitState, ExGuid, Insertion, ParagraphJoin, ParagraphSplit, PreparedEdit,
6 RevisionIndex, Store, TextAttribute, document::Document,6 RevisionIndex, Store, TextAttribute, TreeEdit, document::Document,
7};7};
8use onestore_offline::{EditStatus, Error, Remote, Replica, SmbRemote};8use onestore_offline::{EditStatus, Error, Remote, Replica, SmbRemote};
9use onestore_smb::{Client, Credentials};9use onestore_smb::{Client, Credentials};
...@@ -41,7 +41,18 @@ impl DocumentView {...@@ -41,7 +41,18 @@ impl DocumentView {
41 }41 }
42}42}
4343
44const DOCUMENT_OPERATIONS: [&str; 6] = ["insert", "format", "text", "split", "right_text", "join"];44const DOCUMENT_OPERATIONS: [&str; 10] = [
45 "insert",
46 "format",
47 "text",
48 "split",
49 "right_text",
50 "nest",
51 "unnest",
52 "join",
53 "tail_split",
54 "delete",
55];
4556
46fn now() -> u128 {57fn now() -> u128 {
47 SystemTime::now()58 SystemTime::now()
...@@ -331,6 +342,8 @@ fn queue_document(...@@ -331,6 +342,8 @@ fn queue_document(
331 };342 };
332 let end = u32::try_from(text.encode_utf16().count())?;343 let end = u32::try_from(text.encode_utf16().count())?;
333 let split = ParagraphSplit::new(insertion.text_object(), end - 2, "Offline document writer")?;344 let split = ParagraphSplit::new(insertion.text_object(), end - 2, "Offline document writer")?;
345 let tail_split =
346 ParagraphSplit::new(insertion.text_object(), end + 1, "Offline document writer")?;
334 let join = ParagraphJoin::new(347 let join = ParagraphJoin::new(
335 insertion.text_object(),348 insertion.text_object(),
336 split.text_object(),349 split.text_object(),
...@@ -344,14 +357,50 @@ fn queue_document(...@@ -344,14 +357,50 @@ fn queue_document(
344 let mut ids = [0; DOCUMENT_OPERATIONS.len()];357 let mut ids = [0; DOCUMENT_OPERATIONS.len()];
345 for (step, id) in ids.iter_mut().enumerate() {358 for (step, id) in ids.iter_mut().enumerate() {
346 let kind = DOCUMENT_OPERATIONS[step];359 let kind = DOCUMENT_OPERATIONS[step];
360 let tree = match kind {
361 "nest" => {
362 let source = cache.snapshot()?;
363 let store = Store::parse(&source)?;
364 let index = RevisionIndex::parse(&store)?;
365 let document = Document::parse(&index)?;
366 let section = &document.spaces[&space];
367 let view = &section.revisions[&section.contexts[&ExGuid::default()]];
368 let paragraph = view
369 .nodes
370 .iter()
371 .find(|(_, node)| node.content == [insertion.text_object()])
372 .map(|(id, _)| *id)
373 .ok_or("Missing inserted paragraph")?;
374 Some(TreeEdit::move_to(
375 split.object(),
376 paragraph,
377 None,
378 "Offline document writer",
379 )?)
380 }
381 "unnest" => Some(TreeEdit::move_to(
382 split.object(),
383 parent,
384 None,
385 "Offline document writer",
386 )?),
387 "delete" => Some(TreeEdit::delete(
388 tail_split.object(),
389 "Offline document writer",
390 )?),
391 _ => None,
392 };
347 let range = match kind {393 let range = match kind {
348 "text" => end - 3..end,394 "text" => end - 3..end,
349 "split" => end - 2..end - 2,395 "split" => end - 2..end - 2,
396 "tail_split" => end + 1..end + 1,
350 "right_text" => 0..2,397 "right_text" => 0..2,
351 _ => 1..end - 2,398 _ => 1..end - 2,
352 };399 };
353 let target = if kind == "right_text" {400 let target = if kind == "right_text" {
354 split.text_object()401 split.text_object()
402 } else if kind == "delete" {
403 tail_split.text_object()
355 } else {404 } else {
356 insertion.text_object()405 insertion.text_object()
357 };406 };
...@@ -373,7 +422,9 @@ fn queue_document(...@@ -373,7 +422,9 @@ fn queue_document(
373 cache.edit_text(&source, space, target, range.clone(), replacement.unwrap())422 cache.edit_text(&source, space, target, range.clone(), replacement.unwrap())
374 }423 }
375 "split" => cache.split(&source, space, &split),424 "split" => cache.split(&source, space, &split),
425 "tail_split" => cache.split(&source, space, &tail_split),
376 "join" => cache.join(&source, space, &join),426 "join" => cache.join(&source, space, &join),
427 "nest" | "unnest" | "delete" => cache.tree(&source, space, tree.as_ref().unwrap()),
377 _ => unreachable!(),428 _ => unreachable!(),
378 };429 };
379 match result {430 match result {
...@@ -384,7 +435,8 @@ fn queue_document(...@@ -384,7 +435,8 @@ fn queue_document(
384 json!({"event":"local_document_commit","id":acknowledged,"operation":operation,"kind":kind,435 json!({"event":"local_document_commit","id":acknowledged,"operation":operation,"kind":kind,
385 "space":space.to_string(),"object":target.to_string(),"document":insertion.text_object().to_string(),436 "space":space.to_string(),"object":target.to_string(),"document":insertion.text_object().to_string(),
386 "text":text,"insertion":if kind=="insert" {Some(&insertion)} else {None},437 "text":text,"insertion":if kind=="insert" {Some(&insertion)} else {None},
387 "split":if kind=="split" {Some(&split)} else {None},438 "split":match kind { "split" => Some(&split), "tail_split" => Some(&tail_split), _ => None },
439 "tree":tree,
388 "joined":if kind=="join" {Some(join.texts().map(|id| id.to_string()))} else {None},440 "joined":if kind=="join" {Some(join.texts().map(|id| id.to_string()))} else {None},
389 "range":[range.start,range.end],"attributes":attributes,"replacement":replacement,441 "range":[range.start,range.end],"attributes":attributes,"replacement":replacement,
390 "started_us":started,"finished_us":now()})442 "started_us":started,"finished_us":now()})
...@@ -555,7 +607,12 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {...@@ -555,7 +607,12 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
555 let remote = cache.remote_snapshot()?;607 let remote = cache.remote_snapshot()?;
556 let current = view(&remote)?;608 let current = view(&remote)?;
557 let onestore_offline::Operation::Text(edit) = &intent.operation else {609 let onestore_offline::Operation::Text(edit) = &intent.operation else {
558 return Err("Expected text probe intents".into());610 return Err(format!(
611 "Document intent {} requires review: {:?}",
612 intent.id,
613 cache.status(intent.id)?
614 )
615 .into());
559 };616 };
560 let at = append_position(&edit.before, &current.text, &edit.replacement)617 let at = append_position(&edit.before, &current.text, &edit.replacement)
561 .ok_or("Append model disagrees with retained history")?;618 .ok_or("Append model disagrees with retained history")?;
...@@ -713,9 +770,15 @@ mod tests {...@@ -713,9 +770,15 @@ mod tests {
713 cache = Replica::open(&path).unwrap();770 cache = Replica::open(&path).unwrap();
714 }771 }
715 let local = cache.snapshot().unwrap();772 let local = cache.snapshot().unwrap();
716 assert_eq!(cache.pending().unwrap().len(), 12);773 assert_eq!(
774 cache.pending().unwrap().len(),
775 2 * DOCUMENT_OPERATIONS.len()
776 );
717 for (step, id) in ids.iter().enumerate() {777 for (step, id) in ids.iter().enumerate() {
718 remote.remote.lost_reply = matches!(step % 6, 3 | 5);778 remote.remote.lost_reply = matches!(
779 DOCUMENT_OPERATIONS[step % DOCUMENT_OPERATIONS.len()],
780 "split" | "nest" | "unnest" | "join" | "tail_split" | "delete"
781 );
719 let result = cache.sync_once(&mut remote);782 let result = cache.sync_once(&mut remote);
720 let state = if result.is_err() {783 let state = if result.is_err() {
721 assert!(matches!(784 assert!(matches!(
crates/onestore-offline/src/lib.rs+1-1
...@@ -51,7 +51,7 @@ pub enum Error {...@@ -51,7 +51,7 @@ pub enum Error {
51type Result<T> = std::result::Result<T, Error>;51type Result<T> = std::result::Result<T, Error>;
5252
53const APPLICATION_ID: u32 = 0x4f4e454f;53const APPLICATION_ID: u32 = 0x4f4e454f;
54const SCHEMA_VERSION: u32 = 8;54const SCHEMA_VERSION: u32 = 9;
5555
56/// Text and its observed precondition, retained across cache reopen and rebasing.56/// Text and its observed precondition, retained across cache reopen and rebasing.
57#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]57#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
crates/onestore-offline/src/tree.rs+199-21
...@@ -4,13 +4,20 @@ use serde::{Deserialize, Serialize};...@@ -4,13 +4,20 @@ use serde::{Deserialize, Serialize};
4use sha2::{Digest, Sha256};4use sha2::{Digest, Sha256};
5use std::collections::{BTreeMap, BTreeSet};5use std::collections::{BTreeMap, BTreeSet};
66
7#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
8#[serde(untagged, deny_unknown_fields)]
9enum Content {
10 Legacy([u8; 32]),
11 Semantic { sha256: [u8; 32] },
12}
13
7#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]14#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
8#[serde(deny_unknown_fields)]15#[serde(deny_unknown_fields)]
9struct Observed {16struct Observed {
10 path: Vec<(ExGuid, u8)>,17 path: Vec<(ExGuid, u8)>,
11 siblings: BTreeMap<ExGuid, bool>,18 siblings: BTreeMap<ExGuid, bool>,
12 destination: Vec<(ExGuid, u8)>,19 destination: Vec<(ExGuid, u8)>,
13 content: Option<[u8; 32]>,20 content: Option<Content>,
14}21}
1522
16/// A subtree intent with observed placement, deletion content and replacement identities.23/// A subtree intent with observed placement, deletion content and replacement identities.
...@@ -22,23 +29,41 @@ pub struct TreeEdit {...@@ -22,23 +29,41 @@ pub struct TreeEdit {
22 created: BTreeSet<ExGuid>,29 created: BTreeSet<ExGuid>,
23}30}
2431
25fn fingerprint(store: &Store<'_>, raw: &ResolvedRevision<'_>, root: ExGuid) -> Result<[u8; 32]> {32fn fingerprint(
33 store: &Store<'_>,
34 raw: &ResolvedRevision<'_>,
35 root: ExGuid,
36 legacy: bool,
37) -> Result<[u8; 32]> {
26 let mut objects = BTreeMap::new();38 let mut objects = BTreeMap::new();
27 let mut pending = vec![root];39 let mut visiting = BTreeSet::new();
28 while let Some(id) = pending.pop() {40 let mut pending = vec![(root, false)];
41 while let Some((id, complete)) = pending.pop() {
29 if objects.contains_key(&id) {42 if objects.contains_key(&id) {
30 continue;43 continue;
31 }44 }
32 let object = &raw.objects[&id];45 let object = &raw.objects[&id];
33 let references = object.references()?;46 let references = object.references()?;
34 pending.extend(&references.objects);47 if !complete {
48 if !visiting.insert(id) {
49 return Err(io::Error::new(
50 io::ErrorKind::InvalidData,
51 "Subtree property references form a cycle",
52 )
53 .into());
54 }
55 pending.push((id, true));
56 pending.extend(references.objects.iter().map(|id| (*id, false)));
57 continue;
58 }
59 visiting.remove(&id);
35 let mut hash = Sha256::new();60 let mut hash = Sha256::new();
36 hash.update(object.jcid.to_le_bytes());61 hash.update(object.jcid.to_le_bytes());
37 match object.data {62 match object.data {
38 ObjectData::Properties(bytes) => {63 ObjectData::Properties(bytes) => {
39 hash.update([0]);64 hash.update([0]);
40 let properties = PropertySets::parse(bytes)?;65 let properties = PropertySets::parse(bytes)?;
41 let mut objects = references.objects.into_iter();66 let mut object_ids = references.objects.into_iter();
42 let mut spaces = references.object_spaces.into_iter();67 let mut spaces = references.object_spaces.into_iter();
43 let mut contexts = references.contexts.into_iter();68 let mut contexts = references.contexts.into_iter();
44 let mut fields = Vec::new();69 let mut fields = Vec::new();
...@@ -54,20 +79,36 @@ fn fingerprint(store: &Store<'_>, raw: &ResolvedRevision<'_>, root: ExGuid) -> R...@@ -54,20 +79,36 @@ fn fingerprint(store: &Store<'_>, raw: &ResolvedRevision<'_>, root: ExGuid) -> R
54 compact_ids,79 compact_ids,
55 } => {80 } => {
56 let references = match stream {81 let references = match stream {
57 IdStream::Objects => &mut objects,82 IdStream::Objects => &mut object_ids,
58 IdStream::ObjectSpaces => &mut spaces,83 IdStream::ObjectSpaces => &mut spaces,
59 IdStream::Contexts => &mut contexts,84 IdStream::Contexts => &mut contexts,
60 };85 };
61 for reference in references.take(compact_ids.len() / 4) {86 for reference in references.take(compact_ids.len() / 4) {
62 value.update(reference.guid);87 if !legacy && *stream == IdStream::Objects {
63 value.update(reference.n.to_le_bytes());88 let immutable =
89 raw.objects[&reference].jcid & 0x100000 != 0;
90 value.update([u8::from(immutable)]);
91 if !immutable {
92 value.update(reference.guid);
93 value.update(reference.n.to_le_bytes());
94 }
95 value.update(objects[&reference]);
96 } else {
97 value.update(reference.guid);
98 value.update(reference.n.to_le_bytes());
99 }
64 }100 }
65 }101 }
66 Value::Sets(_) => {}102 Value::Sets(_) => {}
67 }103 }
68 if property.id != 0x14001d7a {104 if property.id == 0x14001d7a
69 values.insert(property.id, value);105 || (!legacy
106 && property.id == 0x24001c20
107 && matches!(&property.value, Value::References { compact_ids, .. } if compact_ids.is_empty()))
108 {
109 continue;
70 }110 }
111 values.insert(property.id, value);
71 }112 }
72 fields.push(values);113 fields.push(values);
73 }114 }
...@@ -106,6 +147,9 @@ fn fingerprint(store: &Store<'_>, raw: &ResolvedRevision<'_>, root: ExGuid) -> R...@@ -106,6 +147,9 @@ fn fingerprint(store: &Store<'_>, raw: &ResolvedRevision<'_>, root: ExGuid) -> R
106 }147 }
107 objects.insert(id, <[u8; 32]>::from(hash.finalize()));148 objects.insert(id, <[u8; 32]>::from(hash.finalize()));
108 }149 }
150 if !legacy {
151 return Ok(objects[&root]);
152 }
109 let mut hash = Sha256::new();153 let mut hash = Sha256::new();
110 for (id, value) in objects {154 for (id, value) in objects {
111 hash.update(id.guid);155 hash.update(id.guid);
...@@ -115,7 +159,12 @@ fn fingerprint(store: &Store<'_>, raw: &ResolvedRevision<'_>, root: ExGuid) -> R...@@ -115,7 +159,12 @@ fn fingerprint(store: &Store<'_>, raw: &ResolvedRevision<'_>, root: ExGuid) -> R
115 Ok(hash.finalize().into())159 Ok(hash.finalize().into())
116}160}
117161
118fn observe(source: &[u8], space: ExGuid, intent: &onestore::TreeEdit) -> Result<Option<Observed>> {162fn observe(
163 source: &[u8],
164 space: ExGuid,
165 intent: &onestore::TreeEdit,
166 legacy: bool,
167) -> Result<Option<Observed>> {
119 let store = Store::parse(source)?;168 let store = Store::parse(source)?;
120 let index = RevisionIndex::parse(&store)?;169 let index = RevisionIndex::parse(&store)?;
121 let document = Document::parse(&index)?;170 let document = Document::parse(&index)?;
...@@ -168,7 +217,12 @@ fn observe(source: &[u8], space: ExGuid, intent: &onestore::TreeEdit) -> Result<...@@ -168,7 +217,12 @@ fn observe(source: &[u8], space: ExGuid, intent: &onestore::TreeEdit) -> Result<
168 .collect(),217 .collect(),
169 destination,218 destination,
170 content: if intent.destination().is_none() {219 content: if intent.destination().is_none() {
171 Some(fingerprint(&store, &index.resolve(space, *rid)?, object)?)220 let sha256 = fingerprint(&store, &index.resolve(space, *rid)?, object, legacy)?;
221 Some(if legacy {
222 Content::Legacy(sha256)
223 } else {
224 Content::Semantic { sha256 }
225 })
172 } else {226 } else {
173 None227 None
174 },228 },
...@@ -206,7 +260,7 @@ impl TreeEdit {...@@ -206,7 +260,7 @@ impl TreeEdit {
206 intent: &onestore::TreeEdit,260 intent: &onestore::TreeEdit,
207 ) -> Result<(Self, PreparedEdit<'a>)> {261 ) -> Result<(Self, PreparedEdit<'a>)> {
208 let prepared = PreparedEdit::tree(source, space, intent)?;262 let prepared = PreparedEdit::tree(source, space, intent)?;
209 let observed = observe(source, space, intent)?.ok_or_else(|| {263 let observed = observe(source, space, intent, false)?.ok_or_else(|| {
210 io::Error::new(264 io::Error::new(
211 io::ErrorKind::InvalidData,265 io::ErrorKind::InvalidData,
212 "Prepared subtree edit has no active target",266 "Prepared subtree edit has no active target",
...@@ -229,7 +283,13 @@ impl TreeEdit {...@@ -229,7 +283,13 @@ impl TreeEdit {
229 source: &'a [u8],283 source: &'a [u8],
230 space: ExGuid,284 space: ExGuid,
231 ) -> Result<std::result::Result<PreparedEdit<'a>, ConflictKind>> {285 ) -> Result<std::result::Result<PreparedEdit<'a>, ConflictKind>> {
232 let Some(current) = observe(source, space, &self.intent)? else {286 let Some(current) = observe(
287 source,
288 space,
289 &self.intent,
290 matches!(self.observed.content, Some(Content::Legacy(_))),
291 )?
292 else {
233 return Ok(Err(ConflictKind::TargetUnavailable));293 return Ok(Err(ConflictKind::TargetUnavailable));
234 };294 };
235 let prepared = match PreparedEdit::tree(source, space, &self.intent) {295 let prepared = match PreparedEdit::tree(source, space, &self.intent) {
...@@ -304,7 +364,7 @@ mod tests {...@@ -304,7 +364,7 @@ mod tests {
304 guid: [2; 16],364 guid: [2; 16],
305 n: 7,365 n: 7,
306 };366 };
307 let hash = |reverse: bool, global: u32, timestamp: u32, change: usize| {367 let hash = |reverse: bool, global: u32, timestamp: u32, change: usize, immutable: bool| {
308 let mut fields = vec![368 let mut fields = vec![
309 (0x14001d7a, timestamp.to_le_bytes().to_vec()),369 (0x14001d7a, timestamp.to_le_bytes().to_vec()),
310 (0x20000001, vec![]),370 (0x20000001, vec![]),
...@@ -317,6 +377,12 @@ mod tests {...@@ -317,6 +377,12 @@ mod tests {
317 set(&[(if change == 2 { 0x08000005 } else { 0x88000005 }, vec![])]),377 set(&[(if change == 2 { 0x08000005 } else { 0x88000005 }, vec![])]),
318 ),378 ),
319 ];379 ];
380 if change != 5 {
381 fields.push((0x24001c20, 0_u32.to_le_bytes().to_vec()));
382 }
383 if change == 6 {
384 fields.push((0x24000007, 0_u32.to_le_bytes().to_vec()));
385 }
320 if reverse {386 if reverse {
321 fields.reverse();387 fields.reverse();
322 }388 }
...@@ -351,7 +417,7 @@ mod tests {...@@ -351,7 +417,7 @@ mod tests {
351 (417 (
352 target,418 target,
353 Object {419 Object {
354 jcid: 0x6000e,420 jcid: if immutable { 0x12004d } else { 0x6000e },
355 reference_count: 1,421 reference_count: 1,
356 data: ObjectData::Properties(&child_bytes),422 data: ObjectData::Properties(&child_bytes),
357 global_ids: ids,423 global_ids: ids,
...@@ -359,12 +425,124 @@ mod tests {...@@ -359,12 +425,124 @@ mod tests {
359 ),425 ),
360 ]),426 ]),
361 };427 };
362 fingerprint(&store, &raw, root).unwrap()428 fingerprint(&store, &raw, root, false).unwrap()
363 };429 };
364 let original = hash(false, 0, 1, 0);430 let original = hash(false, 0, 1, 0, false);
365 assert_eq!(original, hash(true, 137, 2, 0));431 assert_eq!(original, hash(true, 137, 2, 0, false));
432 assert_eq!(original, hash(true, 137, 2, 5, false));
433 assert_ne!(original, hash(true, 137, 2, 6, false));
366 for change in 1..=4 {434 for change in 1..=4 {
367 assert_ne!(original, hash(true, 137, 2, change));435 assert_ne!(original, hash(true, 137, 2, change, false));
436 }
437 let immutable = hash(false, 0, 1, 0, true);
438 assert_eq!(immutable, hash(true, 137, 2, 4, true));
439 for change in 1..=3 {
440 assert_ne!(immutable, hash(true, 137, 2, change, true));
441 }
442 }
443
444 #[test]
445 fn version_eight_deletion_observations_survive_migration_and_upgrade_only_after_review() {
446 struct Server(Vec<u8>);
447 impl Remote for Server {
448 fn read(&mut self) -> io::Result<Vec<u8>> {
449 Ok(self.0.clone())
450 }
451 fn publish(
452 &mut self,
453 edit: &PreparedEdit<'_>,
454 ) -> std::result::Result<(), onestore::CommitError> {
455 self.0 = edit.as_bytes().to_vec();
456 Ok(())
457 }
458 fn confirm(
459 &mut self,
460 snapshot: &[u8],
461 ) -> std::result::Result<(), onestore::CommitError> {
462 assert!(self.0 == snapshot);
463 Ok(())
464 }
465 }
466 let source = onestore::create_section("legacy.one", "AlphaOmega", "Author").unwrap();
467 let store = Store::parse(&source).unwrap();
468 let index = RevisionIndex::parse(&store).unwrap();
469 let document = Document::parse(&index).unwrap();
470 let (sid, _) = document.pages().unwrap()[0];
471 let space = &document.spaces[&sid];
472 let view = &space.revisions[&space.contexts[&ExGuid::default()]];
473 let text = *view.nodes.iter().find(|(_, node)| matches!(&node.kind, Kind::RichText { text, .. } if text == "AlphaOmega")).unwrap().0;
474 let directory = tempfile::tempdir().unwrap();
475 let path = directory.path().join("legacy.sqlite");
476 let cache = Replica::create(&path, &source).unwrap();
477 cache
478 .format(
479 &source,
480 sid,
481 text,
482 0..10,
483 &[onestore::TextAttribute::Bold(true)],
484 )
485 .unwrap();
486 let split = onestore::ParagraphSplit::new(text, 5, "Author").unwrap();
487 cache
488 .split(&cache.snapshot().unwrap(), sid, &split)
489 .unwrap();
490 let before = cache.snapshot().unwrap();
491 let intent = onestore::TreeEdit::delete(split.object(), "Author").unwrap();
492 let deletion = cache.tree(&before, sid, &intent).unwrap().unwrap();
493 let dependent = cache
494 .edit_text(&cache.snapshot().unwrap(), sid, text, 0..0, "Local ")
495 .unwrap()
496 .unwrap();
497 let mut queue = cache.pending().unwrap();
498 let Operation::Tree(edit) = &mut queue[2].operation else {
499 panic!()
500 };
501 edit.observed = observe(&before, sid, &intent, true).unwrap().unwrap();
502 let serialized = serde_json::to_string(&queue[2].operation).unwrap();
503 let local = cache.snapshot().unwrap();
504 drop(cache);
505 let db = Connection::open(&path).unwrap();
506 db.execute(
507 "UPDATE edits SET operation=?1 WHERE id=?2",
508 params![serialized, i64::try_from(deletion).unwrap()],
509 )
510 .unwrap();
511 db.pragma_update(None, "user_version", 8).unwrap();
512 drop(db);
513 let cache = Replica::open(&path).unwrap();
514 assert_eq!(cache.pending().unwrap(), queue);
515 assert!(cache.snapshot().unwrap() == local);
516 let mut server = Server(source);
517 for _ in 0..2 {
518 assert!(matches!(
519 cache.sync_once(&mut server).unwrap(),
520 Some((_, EditStatus::Published { .. }))
521 ));
522 }
523 assert_eq!(
524 cache.sync_once(&mut server).unwrap(),
525 Some((deletion, EditStatus::Conflict(ConflictKind::ContentChanged)))
526 );
527 cache
528 .rebase_tree_conflict(deletion, &local, &server.0)
529 .unwrap();
530 let queue = cache.pending().unwrap();
531 let Operation::Tree(edit) = &queue[0].operation else {
532 panic!()
533 };
534 assert!(matches!(
535 edit.observed.content,
536 Some(Content::Semantic { .. })
537 ));
538 for id in [deletion, dependent] {
539 assert!(
540 matches!(cache.sync_once(&mut server).unwrap(), Some((actual, EditStatus::Published { .. })) if actual == id)
541 );
368 }542 }
543 assert_eq!(
544 paragraph(&server.0, sid, text).unwrap().as_deref(),
545 Some("Local Alpha")
546 );
369 }547 }
370}548}
crates/onestore-offline/tests/cache.rs+1-1
...@@ -797,7 +797,7 @@ fn unrecognized_persisted_operations_are_rejected_without_dropping_fields() {...@@ -797,7 +797,7 @@ fn unrecognized_persisted_operations_are_rejected_without_dropping_fields() {
797797
798#[test]798#[test]
799fn prior_schema_migrations_retain_queue_evidence_assets_and_enable_content_conflicts() {799fn prior_schema_migrations_retain_queue_evidence_assets_and_enable_content_conflicts() {
800 for (version, ceiling) in [(5, 3), (6, 4), (7, 5)] {800 for (version, ceiling) in [(5, 3), (6, 4), (7, 5), (8, 6)] {
801 use onestore_offline::{ConflictKind, EditStatus, Recovery};801 use onestore_offline::{ConflictKind, EditStatus, Recovery};
802 use sha2::{Digest, Sha256};802 use sha2::{Digest, Sha256};
803 let directory = tempfile::tempdir().unwrap();803 let directory = tempfile::tempdir().unwrap();
crates/onestore-offline/tests/support/tree_schedule.rs+26-1
...@@ -105,6 +105,20 @@ impl Remote for Session<'_> {...@@ -105,6 +105,20 @@ impl Remote for Session<'_> {
105 .unwrap();105 .unwrap();
106 text.replace_range(0..1, &edit.replacement);106 text.replace_range(0..1, &edit.replacement);
107 }107 }
108 Operation::Format(format) => {
109 let [onestore::TextAttribute::FontSize(size)] = format.attributes.as_slice() else {
110 panic!()
111 };
112 let store = Store::parse(edit.as_bytes()).unwrap();
113 let index = RevisionIndex::parse(&store).unwrap();
114 let document = Document::parse(&index).unwrap();
115 let space = &document.spaces[&SOURCE.1];
116 let view = &space.revisions[&space.contexts[&ExGuid::default()]];
117 assert_eq!(
118 view.text_runs(format.object).unwrap()[0].format.font_size,
119 Some(*size)
120 );
121 }
108 _ => panic!(),122 _ => panic!(),
109 }123 }
110 assert_eq!(rows(edit.as_bytes()), expected);124 assert_eq!(rows(edit.as_bytes()), expected);
...@@ -163,7 +177,7 @@ pub fn run(input: &[u8]) {...@@ -163,7 +177,7 @@ pub fn run(input: &[u8]) {
163 &TreeEdit::delete(local[target].0, "Offline").unwrap(),177 &TreeEdit::delete(local[target].0, "Offline").unwrap(),
164 )178 )
165 .unwrap(),179 .unwrap(),
166 _ => cache180 _ if step[3] & 1 == 0 => cache
167 .edit_text(181 .edit_text(
168 &snapshot,182 &snapshot,
169 *sid,183 *sid,
...@@ -172,6 +186,17 @@ pub fn run(input: &[u8]) {...@@ -172,6 +186,17 @@ pub fn run(input: &[u8]) {
172 &char::from(b'A' + step[3] % 26).to_string(),186 &char::from(b'A' + step[3] % 26).to_string(),
173 )187 )
174 .unwrap(),188 .unwrap(),
189 _ => cache
190 .format(
191 &snapshot,
192 *sid,
193 local[target].1,
194 0..1,
195 &[onestore::TextAttribute::FontSize(
196 12.0 + f32::from(step[3] % 20),
197 )],
198 )
199 .unwrap(),
175 };200 };
176 let next = cache.pending().unwrap();201 let next = cache.pending().unwrap();
177 assert_eq!(next[..pending.len()], pending);202 assert_eq!(next[..pending.len()], pending);
crates/onestore-offline/tests/sync/tree.rs+105
...@@ -100,6 +100,111 @@ fn move_preserves_remote_content_and_dependent_edits_through_reopen() {...@@ -100,6 +100,111 @@ fn move_preserves_remote_content_and_dependent_edits_through_reopen() {
100 assert_eq!(server.publications, 2);100 assert_eq!(server.publications, 2);
101}101}
102102
103#[test]
104fn queued_format_split_and_delete_reconcile_equivalent_immutable_style_identities() {
105 let (source, sid, _, _, texts) = fixture();
106 let directory = tempfile::tempdir().unwrap();
107 let path = directory.path().join("cache.sqlite");
108 let cache = Replica::create(&path, &source).unwrap();
109 cache
110 .format(
111 &source,
112 sid,
113 texts[0],
114 0..8,
115 &[onestore::TextAttribute::Bold(true)],
116 )
117 .unwrap();
118 let split = onestore::ParagraphSplit::new(texts[0], 4, "Author").unwrap();
119 cache
120 .split(&cache.snapshot().unwrap(), sid, &split)
121 .unwrap();
122 cache
123 .tree(
124 &cache.snapshot().unwrap(),
125 sid,
126 &TreeEdit::delete(split.object(), "Author").unwrap(),
127 )
128 .unwrap();
129 cache
130 .edit_text(&cache.snapshot().unwrap(), sid, texts[0], 0..0, "Local ")
131 .unwrap();
132 let queue = cache.pending().unwrap();
133 drop(cache);
134 let mut server = Server::new(&source);
135 for intent in queue {
136 let cache = Replica::open(&path).unwrap();
137 assert!(
138 matches!(cache.sync_once(&mut server).unwrap(), Some((id, EditStatus::Published { .. })) if id == intent.id)
139 );
140 }
141 assert_eq!(
142 super::outline::node(&server.durable, sid, texts[0])["kind"]["text"],
143 "Local Orig"
144 );
145}
146
147#[test]
148fn native_empty_child_list_normalization_preserves_deletion_and_its_dependents() {
149 let source = include_bytes!("../../../../corpus/outline-edit/empty-children/before.one");
150 let remote = include_bytes!("../../../../corpus/outline-edit/empty-children/remote.one");
151 let (sid, intent): (ExGuid, TreeEdit) = serde_json::from_str(include_str!(
152 "../../../../corpus/outline-edit/empty-children/intent.json"
153 ))
154 .unwrap();
155 let store = Store::parse(source).unwrap();
156 let index = RevisionIndex::parse(&store).unwrap();
157 let document = Document::parse(&index).unwrap();
158 let space = &document.spaces[&sid];
159 let view = &space.revisions[&space.contexts[&ExGuid::default()]];
160 let tail = view.nodes[&intent.object()].content[0];
161 let (outline_id, outline) = view
162 .nodes
163 .iter()
164 .find(|(_, node)| node.children.contains(&intent.object()))
165 .unwrap();
166 let left = view.nodes[&outline.children[0]].content[0];
167 for changed in [false, true] {
168 let directory = tempfile::tempdir().unwrap();
169 let path = directory.path().join("cache.sqlite");
170 let cache = Replica::create(&path, source).unwrap();
171 let deletion = cache.tree(source, sid, &intent).unwrap().unwrap();
172 let dependent = cache
173 .edit_text(&cache.snapshot().unwrap(), sid, left, 0..0, "Local ")
174 .unwrap()
175 .unwrap();
176 let local = cache.snapshot().unwrap();
177 drop(cache);
178 let cache = Replica::open(&path).unwrap();
179 let mut server = if changed {
180 let edited = PreparedEdit::text(remote, sid, tail, 0..2, "Changed").unwrap();
181 Server::new(edited.as_bytes())
182 } else {
183 Server::new(remote)
184 };
185 if changed {
186 assert_eq!(
187 cache.sync_once(&mut server).unwrap(),
188 Some((deletion, EditStatus::Conflict(ConflictKind::ContentChanged)))
189 );
190 assert_eq!(cache.snapshot().unwrap(), local);
191 assert_eq!(cache.pending().unwrap().len(), 2);
192 assert_eq!(server.publications, 0);
193 } else {
194 for expected in [deletion, dependent] {
195 assert!(
196 matches!(cache.sync_once(&mut server).unwrap(), Some((id, EditStatus::Published { .. })) if id == expected)
197 );
198 }
199 assert!(cache.pending().unwrap().is_empty());
200 assert_eq!(server.publications, 2);
201 assert!(!children(&server.durable, sid, *outline_id).contains(&intent.object()));
202 let node = super::outline::node(&server.durable, sid, left);
203 assert!(node["kind"]["text"].as_str().unwrap().starts_with("Local "));
204 }
205 }
206}
207
103#[test]208#[test]
104fn deletion_requires_review_of_remote_content_and_preserves_later_work() {209fn deletion_requires_review_of_remote_content_and_preserves_later_work() {
105 let (source, sid, outline, paragraphs, texts) = fixture();210 let (source, sid, outline, paragraphs, texts) = fixture();
tools/offline_document_history.py+46-19
...@@ -18,7 +18,8 @@ def characters(observed):...@@ -18,7 +18,8 @@ def characters(observed):
18def operation_kinds(events):18def operation_kinds(events):
19 kinds = tuple(events[0].get('document_kinds', ('insert', 'format')))19 kinds = tuple(events[0].get('document_kinds', ('insert', 'format')))
20 assert kinds in (('insert', 'format'), ('insert', 'format', 'text'),20 assert kinds in (('insert', 'format'), ('insert', 'format', 'text'),
21 ('insert', 'format', 'text', 'split', 'right_text', 'join')), 'Unknown document workload'21 ('insert', 'format', 'text', 'split', 'right_text', 'join'),
22 ('insert', 'format', 'text', 'split', 'right_text', 'nest', 'unnest', 'join', 'tail_split', 'delete')), 'Unknown document workload'
22 return kinds23 return kinds
2324
2425
...@@ -41,8 +42,9 @@ def document_history(logs, operations):...@@ -41,8 +42,9 @@ def document_history(logs, operations):
41 linked = {}42 linked = {}
42 for intent, receipt in zip(edits, receipts, strict=True):43 for intent, receipt in zip(edits, receipts, strict=True):
43 changed_targets = {intent['object']}44 changed_targets = {intent['object']}
44 if intent['kind'] == 'split': changed_targets.add(identity(intent['split'], 2))45 if intent['kind'] in ('split', 'tail_split'): changed_targets.add(identity(intent['split'], 2))
45 if intent['kind'] == 'join': changed_targets = set(intent['joined'])46 if intent['kind'] == 'join': changed_targets = set(intent['joined'])
47 if intent['kind'] in ('nest', 'unnest'): changed_targets = set()
46 attempts = [row for row in events if row['event'] == 'remote_attempt' and row['revision'] == receipt['revision']48 attempts = [row for row in events if row['event'] == 'remote_attempt' and row['revision'] == receipt['revision']
47 and set(row.get('document_changes') or {}) == changed_targets]49 and set(row.get('document_changes') or {}) == changed_targets]
48 if not attempts and intent['kind'] == 'format':50 if not attempts and intent['kind'] == 'format':
...@@ -51,10 +53,12 @@ def document_history(logs, operations):...@@ -51,10 +53,12 @@ def document_history(logs, operations):
51 assert len(attempts) == 1, 'Document receipt lacks one publication attempt'53 assert len(attempts) == 1, 'Document receipt lacks one publication attempt'
52 attempt, = attempts54 attempt, = attempts
53 assert attempt['state'] in ('Committed', 'Unknown'), 'Receipt identifies an unpublished document operation'55 assert attempt['state'] in ('Committed', 'Unknown'), 'Receipt identifies an unpublished document operation'
54 assert intent['object'] in attempt['documents'] and attempt['document_changes'] == {target: attempt['documents'].get(target) for target in changed_targets}, 'Document publication changed another target'56 assert (intent['object'] in attempt['documents']) == (intent['kind'] != 'delete'), 'Document publication retained or lost its target'
57 assert attempt['document_changes'] == {target: attempt['documents'].get(target) for target in changed_targets}, 'Document publication changed another target'
55 successful = [row for row in events if row['event'] == 'remote_attempt'58 successful = [row for row in events if row['event'] == 'remote_attempt'
56 and row['state'] in ('Committed', 'Unknown')59 and row['state'] in ('Committed', 'Unknown')
57 and row.get('document_changes') == attempt['document_changes']]60 and row.get('document_changes') == attempt['document_changes']
61 and row.get('document_graph_changes') == attempt.get('document_graph_changes')]
58 assert successful == [attempt], 'Document intent was published or attempted uncertainly more than once'62 assert successful == [attempt], 'Document intent was published or attempted uncertainly more than once'
59 assert intent['started_us'] <= attempt['started_us'] <= attempt['finished_us'] <= receipt['at_us'] and intent['started_us'] <= intent['finished_us'] <= receipt['at_us'], 'Document acknowledgement order is invalid'63 assert intent['started_us'] <= attempt['started_us'] <= attempt['finished_us'] <= receipt['at_us'] and intent['started_us'] <= intent['finished_us'] <= receipt['at_us'], 'Document acknowledgement order is invalid'
60 if attempt['state'] == 'Unknown':64 if attempt['state'] == 'Unknown':
...@@ -77,7 +81,7 @@ def document_history(logs, operations):...@@ -77,7 +81,7 @@ def document_history(logs, operations):
77 linked[intent['id']] = {**attempt, 'acknowledged_us': receipt['at_us'], 'receipt_revision': receipt['revision']}81 linked[intent['id']] = {**attempt, 'acknowledged_us': receipt['at_us'], 'receipt_revision': receipt['revision']}
78 assert {(row['revision'], row['started_us']) for row in events82 assert {(row['revision'], row['started_us']) for row in events
79 if row['event'] == 'remote_attempt' and row['state'] in ('Committed', 'Unknown')83 if row['event'] == 'remote_attempt' and row['state'] in ('Committed', 'Unknown')
80 and row.get('document_changes')} == {84 and (row.get('document_changes') or row.get('document_graph_changes'))} == {
81 (attempt['revision'], attempt['started_us']) for attempt in linked.values()85 (attempt['revision'], attempt['started_us']) for attempt in linked.values()
82 }, 'A document publication lacks its recorded intent and receipt'86 }, 'A document publication lacks its recorded intent and receipt'
83 for at in range(0, len(edits), len(kinds)):87 for at in range(0, len(edits), len(kinds)):
...@@ -125,7 +129,8 @@ def document_history(logs, operations):...@@ -125,7 +129,8 @@ def document_history(logs, operations):
125 for state in states.values():129 for state in states.values():
126 state['parts'] = [(paragraph, target, 0, len(state['characters']))]130 state['parts'] = [(paragraph, target, 0, len(state['characters']))]
127 if replaced and boundaries:131 if replaced and boundaries:
128 split, right_edit, join = boundaries132 split, right_edit, *following = boundaries
133 join = next(row for row in following if row['kind'] == 'join')
129 assert events[0].get('document_graph') is True, 'Boundary workload requires structural observations'134 assert events[0].get('document_graph') is True, 'Boundary workload requires structural observations'
130 assert all(row.get('document') == target and row['space'] == inserted['space']135 assert all(row.get('document') == target and row['space'] == inserted['space']
131 for row in edits[at:at + len(kinds)]), 'Boundary operation lost its owning document'136 for row in edits[at:at + len(kinds)]), 'Boundary operation lost its owning document'
...@@ -137,17 +142,37 @@ def document_history(logs, operations):...@@ -137,17 +142,37 @@ def document_history(logs, operations):
137 assert right_edit['object'] == right and right_edit['range'] == [0, 2] and right_edit['replacement'] == 'B🦋', 'Dependent right edit differs from workload'142 assert right_edit['object'] == right and right_edit['range'] == [0, 2] and right_edit['replacement'] == 'B🦋', 'Dependent right edit differs from workload'
138 assert join['object'] == target and join['joined'] == [target, right], 'Join does not retain its original targets'143 assert join['object'] == target and join['joined'] == [target, right], 'Join does not retain its original targets'
139 updated = final[:boundary] + [(char, *final[boundary][1:]) for char in 'B🦋'] + final[boundary+2:]144 updated = final[:boundary] + [(char, *final[boundary][1:]) for char in 'B🦋'] + final[boundary+2:]
140 for row, value, parts in [145 split_parts = [(paragraph, target, 0, boundary), (right_paragraph, right, boundary, len(final))]
141 (split, final, [(paragraph, target, 0, boundary), (right_paragraph, right, boundary, len(final))]),146 edited_parts = [(paragraph, target, 0, boundary), (right_paragraph, right, boundary, len(updated))]
142 (right_edit, updated, [(paragraph, target, 0, boundary), (right_paragraph, right, boundary, len(updated))]),147 stages = [(split, final, split_parts, {}), (right_edit, updated, edited_parts, {})]
143 (join, updated, [(paragraph, target, 0, len(updated))]),148 if len(following) > 1:
144 ]:149 nest, unnest, _, tail_split, deleted = following
150 outline = identity(insertion, 1) if 'Outline' in insertion['placement'] else insertion['parent']
151 for row, parent in [(nest, paragraph), (unnest, outline)]:
152 tree = row['tree']
153 assert row['object'] == target and tree['object'] == right_paragraph and tree['author'] == insertion['author'], 'Tree move addresses another subtree or author'
154 assert tree['placement'] == {'Move': {'parent': parent, 'before': None}}, 'Tree destination differs from workload'
155 stages += [(nest, updated, edited_parts, {right_paragraph: paragraph}),
156 (unnest, updated, edited_parts, {})]
157 stages.append((join, updated, [(paragraph, target, 0, len(updated))], {}))
158 if len(following) > 1:
159 tail = tail_split['split']
160 tail_text, tail_paragraph = identity(tail, 2), identity(tail, 1)
161 assert tail_split['object'] == tail['text'] == target and tail['author'] == insertion['author'], 'Tail split addresses another text or author'
162 assert tail['offset'] == end+1 and tail_split['range'] == [end+1, end+1], 'Tail split boundary differs from workload'
163 tree = deleted['tree']
164 assert deleted['object'] == tail_text and tree['object'] == tail_paragraph and tree['author'] == insertion['author'], 'Deletion addresses another subtree or author'
165 assert tree['placement'] == 'Delete', 'Deletion became a move'
166 stages += [(tail_split, updated, [(paragraph, target, 0, len(updated)-1), (tail_paragraph, tail_text, len(updated)-1, len(updated))], {}),
167 (deleted, updated[:-1], [(paragraph, target, 0, len(updated)-1)], {})]
168 known_texts = {oid for _, _, pieces, _ in stages for _, oid, _, _ in pieces}
169 for row, value, parts, parents in stages:
145 attempt = linked[row['id']]170 attempt = linked[row['id']]
146 assert list(states.values())[-1]['attempt']['finished_us'] <= attempt['started_us'], 'Boundary publication preceded its dependency'171 assert list(states.values())[-1]['attempt']['finished_us'] <= attempt['started_us'], 'Boundary publication preceded its dependency'
147 assert set(attempt['documents']) & {target, right} == {oid for _, oid, _, _ in parts}, 'Boundary publication omitted or resurrected a text object'172 assert set(attempt['documents']) & known_texts == {oid for _, oid, _, _ in parts}, 'Boundary publication omitted or resurrected a text object'
148 for _, oid, start, stop in parts:173 for _, oid, start, stop in parts:
149 assert characters(attempt['documents'][oid]) == value[start:stop], 'Boundary publication differs from its local intent'174 assert characters(attempt['documents'][oid]) == value[start:stop], 'Boundary publication differs from its local intent'
150 states[row['kind']] = {'characters': value, 'parts': parts, 'attempt': attempt}175 states[row['kind']] = {'characters': value, 'parts': parts, 'parents': parents, 'attempt': attempt}
151 allocated = {oid for state in states.values() for paragraph, text, _, _ in state['parts'] for oid in (paragraph, text)}176 allocated = {oid for state in states.values() for paragraph, text, _, _ in state['parts'] for oid in (paragraph, text)}
152 existing = {oid for document in documents.values() for state in document['states'].values()177 existing = {oid for document in documents.values() for state in document['states'].values()
153 for paragraph, text, _, _ in state['parts'] for oid in (paragraph, text)}178 for paragraph, text, _, _ in state['parts'] for oid in (paragraph, text)}
...@@ -177,6 +202,8 @@ def document_history(logs, operations):...@@ -177,6 +202,8 @@ def document_history(logs, operations):
177 states = list(document['states'].values())202 states = list(document['states'].values())
178 known_texts = {oid for state in states for _, oid, _, _ in state['parts']}203 known_texts = {oid for state in states for _, oid, _, _ in state['parts']}
179 known_paragraphs = {oid for state in states for oid, _, _, _ in state['parts']}204 known_paragraphs = {oid for state in states for oid, _, _, _ in state['parts']}
205 insertion = document['insertion']
206 outline = identity(insertion, 1) if 'Outline' in insertion['placement'] else insertion['parent']
180 if is_read and row['started_us'] > states[0]['attempt']['acknowledged_us']:207 if is_read and row['started_us'] > states[0]['attempt']['acknowledged_us']:
181 assert target in observed, 'Reader missed an acknowledged insertion'208 assert target in observed, 'Reader missed an acknowledged insertion'
182 if target not in observed:209 if target not in observed:
...@@ -185,7 +212,9 @@ def document_history(logs, operations):...@@ -185,7 +212,9 @@ def document_history(logs, operations):
185 actual = {oid: characters(observed[oid]) for oid in known_texts & observed.keys()}212 actual = {oid: characters(observed[oid]) for oid in known_texts & observed.keys()}
186 matches = [i for i, state in enumerate(states)213 matches = [i for i, state in enumerate(states)
187 if actual == {oid: state['characters'][start:stop] for _, oid, start, stop in state['parts']}214 if actual == {oid: state['characters'][start:stop] for _, oid, start, stop in state['parts']}
188 and (not structural or known_paragraphs & graph.keys() == {oid for oid, _, _, _ in state['parts']})]215 and (not structural or (known_paragraphs & graph.keys() == {oid for oid, _, _, _ in state['parts']}
216 and all(graph[p]['parent'] == state.get('parents', {}).get(p, outline)
217 for p, _, _, _ in state['parts'])))]
189 assert matches, 'Reader observed partial or invented document content'218 assert matches, 'Reader observed partial or invented document content'
190 if is_read:219 if is_read:
191 matches = [i for i in matches if i >= previous.get(target, 0)]220 matches = [i for i in matches if i >= previous.get(target, 0)]
...@@ -197,16 +226,14 @@ def document_history(logs, operations):...@@ -197,16 +226,14 @@ def document_history(logs, operations):
197 assert matches, 'Reader missed acknowledged document content'226 assert matches, 'Reader missed acknowledged document content'
198 previous[target] = min(matches)227 previous[target] = min(matches)
199 if structural:228 if structural:
200 insertion = document['insertion']
201 outline = insertion['parent']
202 if 'Outline' in insertion['placement']:229 if 'Outline' in insertion['placement']:
203 outline = identity(insertion, 1)
204 expected_graph[outline] = {'parent': insertion['parent'], 'children': [], 'content': [],230 expected_graph[outline] = {'parent': insertion['parent'], 'children': [], 'content': [],
205 'child_level': 1, 'position': insertion['placement']['Outline']}231 'child_level': 1, 'position': insertion['placement']['Outline']}
206 assert outline in expected_graph, 'Snapshot omitted the inserted outline'232 assert outline in expected_graph, 'Snapshot omitted the inserted outline'
207 for paragraph, oid, _, _ in states[min(matches)]['parts']:233 for paragraph, oid, _, _ in states[min(matches)]['parts']:
208 expected_graph[outline]['children'].append(paragraph)234 parent = states[min(matches)].get('parents', {}).get(paragraph, outline)
209 expected_graph[paragraph] = {'parent': outline, 'children': [], 'content': [oid],235 expected_graph[parent]['children'].append(paragraph)
236 expected_graph[paragraph] = {'parent': parent, 'children': [], 'content': [oid],
210 'child_level': 1, 'position': None}237 'child_level': 1, 'position': None}
211 if structural:238 if structural:
212 assert graph == expected_graph, 'Snapshot contains partial, reordered or invented paragraph structure'239 assert graph == expected_graph, 'Snapshot contains partial, reordered or invented paragraph structure'
tools/test_offline_document_history.py+42-3
...@@ -149,9 +149,48 @@ class DocumentHistoryTests(unittest.TestCase):...@@ -149,9 +149,48 @@ class DocumentHistoryTests(unittest.TestCase):
149 self.assertEqual(result.returncode, 0, result.stdout + result.stderr)149 self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
150 events = [json.loads(line) for line in result.stdout.splitlines() if line.startswith('{')]150 events = [json.loads(line) for line in result.stdout.splitlines() if line.startswith('{')]
151 documents = document_history({'w0': events}, 2)151 documents = document_history({'w0': events}, 2)
152 self.assertEqual(sum(len(document['states']) for document in documents.values()), 12)152 self.assertEqual(sum(len(document['states']) for document in documents.values()), 20)
153 self.assertEqual(sum(row['event'] == 'remote_attempt' and row['state'] == 'Unknown' for row in events), 4)153 self.assertEqual(sum(row['event'] == 'remote_attempt' and row['state'] == 'Unknown' for row in events), 12)
154 self.assertEqual(sum(row['event'] == 'remote_confirm' and row['state'] == 'Committed' for row in events), 4)154 self.assertEqual(sum(row['event'] == 'remote_confirm' and row['state'] == 'Committed' for row in events), 12)
155 document = next(iter(documents.values()))
156 states = document['states']
157 for phase, next_phase, stale in [('nest', 'unnest', 'right_text'), ('delete', None, 'tail_split')]:
158 read = copy.deepcopy(states[stale]['attempt'])
159 started = states[phase]['attempt']['acknowledged_us'] + 1
160 if next_phase:
161 self.assertLess(started + 1, states[next_phase]['attempt']['started_us'])
162 read.update(event='read', started_us=started, finished_us=started + 1)
163 logs = {'w0': events, 'r0': [{'event': 'ready', 'document_graph': True}, read, {'event': 'done'}]}
164 with self.subTest(phase=phase), self.assertRaisesRegex(AssertionError, 'missed acknowledged'):
165 document_history(logs, 2)
166 for mutation in ('nest-intent', 'unnest-intent', 'delete-intent', 'tail-boundary',
167 'nest-children', 'delete-resurrection', 'graph-delta', 'extra-attempt'):
168 changed = copy.deepcopy(events)
169 intents = {row['kind']: row for row in changed if row['event'] == 'local_document_commit' and row['operation'] == 0}
170 attempts = {kind: next(row for row in changed if row['event'] == 'remote_attempt' and row['revision'] == state['attempt']['revision'])
171 for kind, state in states.items()}
172 if mutation == 'nest-intent':
173 intents['nest']['tree']['placement']['Move']['parent'] = intents['unnest']['tree']['placement']['Move']['parent']
174 elif mutation == 'unnest-intent':
175 intents['unnest']['tree']['placement']['Move']['before'] = intents['unnest']['tree']['object']
176 elif mutation == 'delete-intent':
177 intents['delete']['tree']['object'] = intents['nest']['tree']['object']
178 elif mutation == 'tail-boundary':
179 intents['tail_split']['split']['offset'] += 1
180 elif mutation == 'nest-children':
181 parent = intents['nest']['tree']['placement']['Move']['parent']
182 attempts['nest']['document_graph'][parent]['children'] = []
183 elif mutation == 'delete-resurrection':
184 tail = intents['delete']['object']
185 attempts['delete']['documents'][tail] = attempts['tail_split']['documents'][tail]
186 elif mutation == 'graph-delta':
187 attempts['nest']['document_graph_changes'] = {}
188 else:
189 extra = copy.deepcopy(attempts['nest'])
190 extra.update(revision='unrecorded-tree-revision', state='Committed')
191 changed.append(extra)
192 with self.subTest(mutation=mutation), self.assertRaises(AssertionError):
193 document_history({'w0': changed}, 2)
155194
156 def test_boundary_histories_retain_every_intermediate_graph_and_retired_identity(self):195 def test_boundary_histories_retain_every_intermediate_graph_and_retired_identity(self):
157 baseline = history(boundaries=True)196 baseline = history(boundaries=True)