diff --git a/arc/sync.md b/arc/sync.md index 426a30ff8fd5f147df3c0223c536ec0f475935f0..b3676e680cbc3b2a1db55e0cd82f7918be8b9835 100644 --- a/arc/sync.md +++ b/arc/sync.md @@ -176,13 +176,35 @@ online again. ## Sections that aren't open -OneNote keeps every section of an open notebook in sync, not just the one on screen: it -reads a section another client changed within seconds and publishes a closed section's -offline edits the moment the share is back. Snowbound's `session::Background` does the -same with one thread per notebook. Each round reads every section file's stamp. Only a -section whose replica has edits waiting, or whose file moved past the replica's base, -has its replica opened for the usual sync steps, and it is closed again afterwards. The -open section is left to its own session. +OneNote keeps every section of an open notebook in sync and on this computer, not just +the one on screen. Watched in the lab with a notebook of 200 sections on Samba, OneNote +2010: + +- reads every section file once when the notebook opens (all 200 within 15 seconds), so a + section never shown before opens with the share gone; +- arms one CHANGE_NOTIFY on the notebook's folder, with WATCH_TREE and a filter of names, + attributes, size and last write, and otherwise sends nothing: 22 idle minutes put no SMB + request on the wire, not even an echo; +- lists the folder once when the server reports a change, and reads only the section that + changed, some seconds later; +- tries a section it could not reach again about every 31 seconds. + +Snowbound's `session::Background` does the same, with one thread per notebook: + +```text +share ─ CHANGE_NOTIFY (tree) ─► the sections it names are due in 1 s ─► stamp ─► moved: rebase +first pass: one section every 100 ms; a section without a replica gets its offline copy +failing: again in 31 s otherwise: again in an hour, against a lost notification +``` + +A check costs one stamp read. Only a section whose replica has edits waiting, or whose file +moved past the replica's base, has its replica opened for the usual sync steps, and it is +closed again afterwards; the open section is left to its own session. A connection that +drops takes its watch with it, so the next one checks every section again. A notebook in a +folder on this computer has no copies and learns of changes from FSEvents or inotify; one on +a network volume the system mounted, whose server's changes those never report, keeps copies +and is checked every 15 seconds. Closing a notebook deletes its copies, except any with +edits still waiting. ## Notebook structure diff --git a/corpus/custom-tags/README.md b/corpus/custom-tags/README.md index 8710cd7036300fc69bfa957e33a63997f74bc96f..d3ceaf57089254172c25ceaf6a35056b6ea6b412 100644 --- a/corpus/custom-tags/README.md +++ b/corpus/custom-tags/README.md @@ -18,7 +18,9 @@ absolute directory while running the test, then cold-open it with `native/shapes.one` is OneNote 2010's own work in a lab clone: - "Shapes 1-72" and "Shapes 73-143" tag a paragraph with every NoteTagShape, set through - COM (its definitions store NoteTagPropertyStatus 0). + COM (its definitions store NoteTagPropertyStatus 0). `shapes-1-72.png` and + `shapes-73-143.png` are OneNote's render of them at 100%, which + `crates/canvas/src/gpu/mod.rs` pins each shape's art to. - "Custom tags" holds a tag made with New Tag (Snow check, as above), applied at the top of the list (action type 0), then after moving it below To Do (a second definition, action type 1), with To Do applied from both places. The UI stores status 15 and 9. diff --git a/corpus/custom-tags/native/shapes-1-72.png b/corpus/custom-tags/native/shapes-1-72.png new file mode 100644 index 0000000000000000000000000000000000000000..30cd457617deb485161954030de7cb684e434724 Binary files /dev/null and b/corpus/custom-tags/native/shapes-1-72.png differ diff --git a/corpus/custom-tags/native/shapes-73-143.png b/corpus/custom-tags/native/shapes-73-143.png new file mode 100644 index 0000000000000000000000000000000000000000..d04feff39f0c50ba9ce5fc882c22694707208bbf Binary files /dev/null and b/corpus/custom-tags/native/shapes-73-143.png differ diff --git a/corpus/ink-tools/native-ui/notebook/Open Notebook.onetoc2 b/corpus/ink-tools/native-ui/notebook/Open Notebook.onetoc2 new file mode 100644 index 0000000000000000000000000000000000000000..d7172c7eb0b3c8120353bf7e3e13e37e206d1081 Binary files /dev/null and b/corpus/ink-tools/native-ui/notebook/Open Notebook.onetoc2 differ diff --git a/corpus/ink-tools/native-ui/notebook/ink.one b/corpus/ink-tools/native-ui/notebook/ink.one new file mode 100644 index 0000000000000000000000000000000000000000..01939d1854e461748502a18f68771afbe4cbf65b Binary files /dev/null and b/corpus/ink-tools/native-ui/notebook/ink.one differ diff --git a/corpus/task-tags/README.md b/corpus/task-tags/README.md new file mode 100644 index 0000000000000000000000000000000000000000..a1081b7b31d980fd1cb3d62a8ee9917e7715b3ff --- /dev/null +++ b/corpus/task-tags/README.md @@ -0,0 +1,20 @@ +# Outlook task tags through the editor + +`native/` is OneNote 2010 authoring `tools/native/pages.ps1` input (two identical +pages in `Tasks.one`: three paragraphs carrying disabled Outlook tasks, the third +also a To Do tag, and an untagged one). OneNote stores each task as a note tag +without a definition, with ActionItemType 100, NoteTagShape 89 and +NoteTagPropertyStatus 0. + +`candidate/` is the output of `task_tags_survive_tag_removal_and_page_delete_undo` +in `crates/canvas/tests/task_tags.rs`: on "Remove a task" the editor's Remove Tag +clears the first paragraph's task and toggling To Do off the third keeps its +task; "Delete and undo" has all its content deleted and the deletion undone, +each published as its own revision. + +`cold/` is a fresh OneNote 2010 read: the first paragraph has no task, the other +tasks keep their identities, dates and disabled state, and the restored page +reads as OneNote authored it. `tools/test_task_tags.py` checks this without a +VM. Regenerate with `SNOWBOUND_TASK_TAGS_EXPORT` set to a new absolute directory +while running the test, then cold-open it with +`tools/native_runner.py OUTPUT COLD --expected-pages 2 --collect-notebook`. diff --git a/corpus/task-tags/candidate/Open Notebook.onetoc2 b/corpus/task-tags/candidate/Open Notebook.onetoc2 new file mode 100644 index 0000000000000000000000000000000000000000..a79db3efc09ce7ecf9a4e8849013d578d6dfe117 Binary files /dev/null and b/corpus/task-tags/candidate/Open Notebook.onetoc2 differ diff --git a/corpus/task-tags/candidate/Tasks.one b/corpus/task-tags/candidate/Tasks.one new file mode 100644 index 0000000000000000000000000000000000000000..ce9a04f48a37f9932bdc688c51988418fa4bb023 Binary files /dev/null and b/corpus/task-tags/candidate/Tasks.one differ diff --git a/corpus/task-tags/cold/commands.jsonl b/corpus/task-tags/cold/commands.jsonl new file mode 100644 index 0000000000000000000000000000000000000000..276dfefb877170ca7f35ce210d77985ef98ae3ba --- /dev/null +++ b/corpus/task-tags/cold/commands.jsonl @@ -0,0 +1,3 @@ +{"command": "powershell -NoProfile -Command \"Expand-Archive -LiteralPath C:\\one-tests\\transfer.zip -DestinationPath C:\\one-tests\\runs\\capture\\notebook\"", "exit": 0, "stdout": "", "stderr": "", "error": null} +{"command": "powershell -NoProfile -NonInteractive -ExecutionPolicy Bypass -File C:\\one-tests\\read-current.ps1 -Root C:\\one-tests\\runs\\capture -CloneHost ONE-M6-7A22B21E -ExpectedPages 2 -KeepOpen", "exit": 0, "stdout": "Read 1 sections and 2 pages.\r\n", "stderr": "", "error": null} +{"command": "powershell -NoProfile -Command \"Compress-Archive -Force -Path C:\\one-tests\\runs\\capture\\* -DestinationPath C:\\one-tests\\captured.zip\"", "exit": 0, "stdout": "", "stderr": "", "error": null} diff --git a/corpus/task-tags/cold/machine.json b/corpus/task-tags/cold/machine.json new file mode 100644 index 0000000000000000000000000000000000000000..9af02ba40b3bad254d3314ecd4b0f30143b7da03 --- /dev/null +++ b/corpus/task-tags/cold/machine.json @@ -0,0 +1 @@ +{"name": "m6-7a22b21e", "hostname": "ONE-M6-7A22B21E"} diff --git a/corpus/task-tags/cold/notebook/Open Notebook.onetoc2 b/corpus/task-tags/cold/notebook/Open Notebook.onetoc2 new file mode 100644 index 0000000000000000000000000000000000000000..d9d55ade712677c224c1d2cf4f897cf3ae5d2ad2 Binary files /dev/null and b/corpus/task-tags/cold/notebook/Open Notebook.onetoc2 differ diff --git a/corpus/task-tags/cold/notebook/Tasks.one b/corpus/task-tags/cold/notebook/Tasks.one new file mode 100644 index 0000000000000000000000000000000000000000..7ab8175bd5e1b930f1b653a699484034fc1507a7 Binary files /dev/null and b/corpus/task-tags/cold/notebook/Tasks.one differ diff --git a/corpus/task-tags/cold/read/environment.json b/corpus/task-tags/cold/read/environment.json new file mode 100644 index 0000000000000000000000000000000000000000..00ece83ec92016040f0174d6c20f04629f96af5c --- /dev/null +++ b/corpus/task-tags/cold/read/environment.json @@ -0,0 +1,7 @@ +{ + "powershell": "5.1.14409.1005", + "schema": "xs2010", + "hostname": "ONE-M6-7A22B21E", + "cold": true, + "onenote": "14.0.4763.1000" +} diff --git a/corpus/task-tags/cold/read/hierarchy.xml b/corpus/task-tags/cold/read/hierarchy.xml new file mode 100644 index 0000000000000000000000000000000000000000..852fecc0cbf0a42e28d0b2370192d137baa5b10e --- /dev/null +++ b/corpus/task-tags/cold/read/hierarchy.xml @@ -0,0 +1,2 @@ + + diff --git a/corpus/task-tags/cold/read/page-000.navigation.json b/corpus/task-tags/cold/read/page-000.navigation.json new file mode 100644 index 0000000000000000000000000000000000000000..f96bf51a7b355a67f40c9bb4d1d7f1cf04129a47 --- /dev/null +++ b/corpus/task-tags/cold/read/page-000.navigation.json @@ -0,0 +1,13 @@ +{ + "exit": 0, + "stdout": "", + "stderr": "", + "w": 800, + "h": 600, + "error": null, + "win": { + "title": "Delete and undo - Microsoft OneNote", + "class": "Framework::CFrame", + "dialog": false + } +} \ No newline at end of file diff --git a/corpus/task-tags/cold/read/page-000.png b/corpus/task-tags/cold/read/page-000.png new file mode 100644 index 0000000000000000000000000000000000000000..064305ae57a8239de918456eb34bcc60e7e68f80 Binary files /dev/null and b/corpus/task-tags/cold/read/page-000.png differ diff --git a/corpus/task-tags/cold/read/page-000.xml b/corpus/task-tags/cold/read/page-000.xml new file mode 100644 index 0000000000000000000000000000000000000000..0981a4e10a8883abd37fc2260c1e52652100b5e9 --- /dev/null +++ b/corpus/task-tags/cold/read/page-000.xml @@ -0,0 +1,2 @@ + + diff --git a/corpus/task-tags/cold/read/page-001.navigation.json b/corpus/task-tags/cold/read/page-001.navigation.json new file mode 100644 index 0000000000000000000000000000000000000000..e5a5078a1abd65bb9ba64a5a6d5603d374cfc401 --- /dev/null +++ b/corpus/task-tags/cold/read/page-001.navigation.json @@ -0,0 +1,13 @@ +{ + "exit": 0, + "stdout": "", + "stderr": "", + "w": 800, + "h": 600, + "error": null, + "win": { + "title": "Remove a task - Microsoft OneNote", + "class": "Framework::CFrame", + "dialog": false + } +} \ No newline at end of file diff --git a/corpus/task-tags/cold/read/page-001.png b/corpus/task-tags/cold/read/page-001.png new file mode 100644 index 0000000000000000000000000000000000000000..05d3d4e4a616fc3de6837067a64a860720402a4d Binary files /dev/null and b/corpus/task-tags/cold/read/page-001.png differ diff --git a/corpus/task-tags/cold/read/page-001.xml b/corpus/task-tags/cold/read/page-001.xml new file mode 100644 index 0000000000000000000000000000000000000000..c65dcef1ad98e021dc306d4eff810de3113566da --- /dev/null +++ b/corpus/task-tags/cold/read/page-001.xml @@ -0,0 +1,2 @@ + + diff --git a/corpus/task-tags/cold/read/payloads.json b/corpus/task-tags/cold/read/payloads.json new file mode 100644 index 0000000000000000000000000000000000000000..1ce91a413d6fff4d58765187b76756f0e97ae843 --- /dev/null +++ b/corpus/task-tags/cold/read/payloads.json @@ -0,0 +1,3 @@ +[ + +] \ No newline at end of file diff --git a/corpus/task-tags/cold/run.json b/corpus/task-tags/cold/run.json new file mode 100644 index 0000000000000000000000000000000000000000..8e9ec1b77f472065cbd23c61869969409c18f975 --- /dev/null +++ b/corpus/task-tags/cold/run.json @@ -0,0 +1,18 @@ +{ + "notebook": "/private/tmp/tasktags/candidate", + "expected_pages": 2, + "author": null, + "author_timeout_seconds": 600, + "inspect": false, + "collect_notebook": true, + "base": { + "file": "win7-office-base.qcow2", + "format": "qcow2", + "sha256": "a1a4f8fab782ee14885ff801ca2f6347c208fdcfc3513637096f314315c89346", + "virtual_size": 68719476736 + }, + "scripts": { + "cold.ps1": "c177fc72ae6c2634d186f5671a880de20b09f9716532c37c69cb13aa15c7e331", + "read.ps1": "22bc760f955ef59bb5f0ccfd55271f8d868674084127fbe219e1048bd3777f35" + } +} diff --git a/corpus/task-tags/cold/scripts/cold.ps1 b/corpus/task-tags/cold/scripts/cold.ps1 new file mode 100644 index 0000000000000000000000000000000000000000..a0b99e7aeb86f4dc10032537800501a52ad65b4f --- /dev/null +++ b/corpus/task-tags/cold/scripts/cold.ps1 @@ -0,0 +1,27 @@ +param([Parameter(Mandatory=$true)][string]$Root, [string]$CloneHost = '') +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' +$root = [IO.Path]::GetFullPath($Root).TrimEnd('\') +if ([IO.Path]::GetDirectoryName($root) -ne 'C:\one-tests\runs') { + throw 'Choose a run directly below C:\one-tests\runs.' +} +if (Get-Process ONENOTE -ErrorAction SilentlyContinue) { throw 'Close OneNote before resetting its test cache.' } +$key = 'HKCU:\Software\Microsoft\Office\14.0\OneNote' +if ($CloneHost) { + if ($CloneHost -notmatch '^ONE-[A-Z0-9-]+$' -or [Environment]::MachineName -ne $CloneHost) { + throw 'The disposable clone hostname does not match this machine.' + } + New-Item "$key\Options\Paths" -Force | Out-Null + New-ItemProperty "$key\Options\Paths" -Name UnfiledNotesSection -PropertyType ExpandString -Value 'C:\one-tests\Loose.one' -Force | Out-Null +} elseif ((Get-ItemProperty "$key\Options\Paths").UnfiledNotesSection -ne 'C:\one-tests\Loose.one' -or + -not (Test-Path 'C:\one-tests\profile-original-cache')) { + throw 'Park the personal OneNote profile before resetting the test cache.' +} +$cache = Join-Path $env:LOCALAPPDATA 'Microsoft\OneNote\14.0' +$parked = Join-Path 'C:\one-tests\caches' ([IO.Path]::GetFileName($root)) +if (Test-Path $parked) { throw 'Choose a new run; its parked cache already exists.' } +New-Item -ItemType Directory -Path 'C:\one-tests\caches' -Force | Out-Null +if (Test-Path $cache) { Move-Item -LiteralPath $cache -Destination $parked } +if (Test-Path "$key\OpenNotebooks") { Remove-Item "$key\OpenNotebooks" -Recurse } +New-Item "$key\OpenNotebooks" | Out-Null +New-ItemProperty "$key\OpenNotebooks" -Name '1' -PropertyType String -Value "$root\notebook" | Out-Null diff --git a/corpus/task-tags/cold/scripts/read.ps1 b/corpus/task-tags/cold/scripts/read.ps1 new file mode 100644 index 0000000000000000000000000000000000000000..a42366b32e8baaf13dae92e9472cae7a071220cc --- /dev/null +++ b/corpus/task-tags/cold/scripts/read.ps1 @@ -0,0 +1,144 @@ +param( + [Parameter(Mandatory=$true)][string]$Root, + [int]$ExpectedPages = -1, + [switch]$UseCurrentCache, + [switch]$Pdf, + [switch]$KeepOpen, + [string]$CloneHost = '' +) +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' +$root = [IO.Path]::GetFullPath($Root).TrimEnd('\') +if ([IO.Path]::GetDirectoryName($root) -ne 'C:\one-tests\runs') { + throw 'Choose a run directly below C:\one-tests\runs.' +} +$notebook = Join-Path $root 'notebook' +$output = Join-Path $root 'read' +if (Test-Path $output) { throw 'Choose a new read destination.' } +if ($UseCurrentCache) { + if ((Get-ItemProperty 'HKCU:\Software\Microsoft\Office\14.0\OneNote\Options\Paths').UnfiledNotesSection -ne 'C:\one-tests\Loose.one') { + throw 'Park the personal OneNote profile before reading test notebooks.' + } +} else { + & "$PSScriptRoot\cold-current.ps1" -Root $root -CloneHost $CloneHost +} +New-Item -ItemType Directory -Path $output | Out-Null +$app = New-Object -ComObject OneNote.Application +$notebookId = '' +$failure = $null +try { + $app.OpenHierarchy($notebook, '', [ref]$notebookId, 0) + $process = Get-Process ONENOTE + @{ hostname = [Environment]::MachineName; onenote = $process.MainModule.FileVersionInfo.FileVersion; + powershell = $PSVersionTable.PSVersion.ToString(); schema = 'xs2010'; cold = (-not $UseCurrentCache.IsPresent) } | + ConvertTo-Json | Set-Content (Join-Path $output 'environment.json') -Encoding UTF8 + $sections = @() + foreach ($file in @(Get-ChildItem $notebook -Recurse | Where-Object { $_.Extension -eq '.one' })) { + $id = '' + $app.OpenHierarchy($file.FullName, '', [ref]$id, 0) + $sections += $id + } + $deadline = [DateTime]::UtcNow.AddSeconds(300) + $previous = '' + $lastChange = '' + $stableSince = [DateTime]::UtcNow + $settled = $false + do { + $pages = @{} + foreach ($section in $sections) { + $hierarchy = '' + $app.GetHierarchy($section, 4, [ref]$hierarchy, 1) + [xml]$xml = $hierarchy + foreach ($node in $xml.SelectNodes('//*[@path]')) { + if (-not $node.GetAttribute('path').StartsWith("$notebook\", [StringComparison]::OrdinalIgnoreCase)) { + throw 'OneNote opened a section outside the copied notebook.' + } + } + foreach ($node in $xml.SelectNodes('//*[local-name()="Page"]')) { + $id = $node.GetAttribute('ID') + $content = '' + $app.GetPageContent($id, [ref]$content, 1, 1) + $pages[$id] = $content + } + } + $signature = [String]::Join('|', @($pages.Keys | Sort-Object | ForEach-Object { $_ + $pages[$_] })) + if ($signature -ne $previous) { + $lastChange = $previous + $previous = $signature + $stableSince = [DateTime]::UtcNow + } + if ((($ExpectedPages -ge 0 -and $pages.Count -eq $ExpectedPages) -or + ($ExpectedPages -lt 0 -and $pages.Count -gt 0)) -and + ([DateTime]::UtcNow - $stableSince).TotalSeconds -ge 2) { $settled = $true; break } + Start-Sleep -Milliseconds 250 + } while ([DateTime]::UtcNow -lt $deadline) + if (-not $settled -or ($ExpectedPages -ge 0 -and $pages.Count -ne $ExpectedPages) -or ($ExpectedPages -lt 0 -and $pages.Count -eq 0)) { + [IO.File]::WriteAllText((Join-Path $output 'previous-signature.txt'), $lastChange, [Text.Encoding]::UTF8) + $index = 0 + foreach ($id in @($pages.Keys | Sort-Object)) { + [IO.File]::WriteAllText((Join-Path $output ('unsettled-{0:d3}.xml' -f $index)), $pages[$id], [Text.Encoding]::UTF8) + $index++ + } + $hierarchy = '' + $app.GetHierarchy($notebookId, 4, [ref]$hierarchy, 1) + [IO.File]::WriteAllText((Join-Path $output 'unsettled-hierarchy.xml'), $hierarchy, [Text.Encoding]::UTF8) + throw "Expected $ExpectedPages stable pages; OneNote returned $($pages.Count), settled=$settled." + } + $index = 0 + $payloads = @() + foreach ($id in @($pages.Keys | Sort-Object)) { + [IO.File]::WriteAllText((Join-Path $output ('page-{0:d3}.xml' -f $index)), $pages[$id], [Text.Encoding]::UTF8) + if ($Pdf) { + $pdfPath = Join-Path $output ('page-{0:d3}.pdf' -f $index) + $app.NavigateTo($id, '', $false) + $app.Publish($id, $pdfPath, 3, '') + if (-not (Test-Path $pdfPath) -or (Get-Item $pdfPath).Length -eq 0) { throw 'OneNote did not publish the page PDF.' } + } + [xml]$page = $pages[$id] + foreach ($file in $page.SelectNodes('//*[local-name()="InsertedFile" or local-name()="MediaFile"]')) { + $bytes = [IO.File]::ReadAllBytes($file.GetAttribute('pathCache')) + $hash = [BitConverter]::ToString([Security.Cryptography.SHA256]::Create().ComputeHash($bytes)).Replace('-', '').ToLowerInvariant() + [IO.File]::WriteAllBytes((Join-Path $output ($hash + '.attachment')), $bytes) + # A paragraph's file is named by its paragraph, one on the page by itself. + $owner = if ($file.ParentNode.LocalName -eq 'Page') { $file } else { $file.ParentNode } + $payloads += @{ page = $id; object = $owner.GetAttribute('objectID'); + kind = $file.LocalName; name = $file.GetAttribute('preferredName'); + sha256 = $hash; bytes = $bytes.Length } + } + $index++ + } + [IO.File]::WriteAllText((Join-Path $output 'payloads.json'), (ConvertTo-Json -InputObject $payloads -Depth 4), [Text.Encoding]::UTF8) + $all = '' + $app.GetHierarchy($notebookId, 4, [ref]$all, 1) + [xml]$finalTree = $all + $finalIds = @($finalTree.SelectNodes('//*[local-name()="Page"]') | ForEach-Object { $_.GetAttribute('ID') } | Sort-Object -Unique) + if ($finalIds.Count -ne $pages.Count -or @($finalIds | Where-Object { -not $pages.ContainsKey($_) }).Count -ne 0) { + throw 'The notebook hierarchy changed while collecting page evidence; repeat the cold read.' + } + [IO.File]::WriteAllText((Join-Path $output 'hierarchy.xml'), $all, [Text.Encoding]::UTF8) + Write-Output "Read $($sections.Count) sections and $($pages.Count) pages." +} catch { + $failure = $_ + [IO.File]::WriteAllText((Join-Path $output 'failure.txt'), ($_ | Out-String), [Text.Encoding]::UTF8) + throw +} finally { + try { + try { + if ($notebookId -and $CloneHost) { $app.SyncHierarchy($notebookId) } + if ($notebookId -and -not $KeepOpen -and (-not $UseCurrentCache -or $CloneHost)) { + $app.CloseNotebook($notebookId, $false) + } + } catch { + if ($null -eq $failure) { throw } + [IO.File]::WriteAllText((Join-Path $output 'cleanup-failure.txt'), ($_ | Out-String), [Text.Encoding]::UTF8) + } + } finally { + [void][Runtime.InteropServices.Marshal]::FinalReleaseComObject($app) + } + $app = $null + [GC]::Collect() + [GC]::WaitForPendingFinalizers() + if (-not $UseCurrentCache -and -not $CloneHost) { + Get-Process ONENOTE -ErrorAction SilentlyContinue | Wait-Process -Timeout 10 + } +} diff --git a/corpus/task-tags/cold/source.json b/corpus/task-tags/cold/source.json new file mode 100644 index 0000000000000000000000000000000000000000..ac722a07015446c753a4481e06ac17245e3c3451 --- /dev/null +++ b/corpus/task-tags/cold/source.json @@ -0,0 +1,14 @@ +[ + { + "path": "Open Notebook.onetoc2", + "bytes": 1604, + "sha256": "a1609ca18a7e5a25f51bd75d7f7eaeb815b339406cb4c60c7cc14cd8422e6272", + "mtime_ns": 1790725054434635462 + }, + { + "path": "Tasks.one", + "bytes": 20912, + "sha256": "a40370dd47c7591f67162db6d1448bdd306705cccbbc575076daa99080fda3ac", + "mtime_ns": 1790725054434461086 + } +] diff --git a/corpus/task-tags/cold/teardown.json b/corpus/task-tags/cold/teardown.json new file mode 100644 index 0000000000000000000000000000000000000000..05a47793de40f322e745c4c4183727e3590ecf70 --- /dev/null +++ b/corpus/task-tags/cold/teardown.json @@ -0,0 +1 @@ +{"absent": true} diff --git a/corpus/task-tags/native/commands.jsonl b/corpus/task-tags/native/commands.jsonl new file mode 100644 index 0000000000000000000000000000000000000000..d8eb267daaceb2d15beb16e2e90f59141eadda4c --- /dev/null +++ b/corpus/task-tags/native/commands.jsonl @@ -0,0 +1,4 @@ +{"command": "powershell -NoProfile -Command \"Expand-Archive -LiteralPath C:\\one-tests\\transfer.zip -DestinationPath C:\\one-tests\\runs\\capture\\notebook\"", "exit": 0, "stdout": "", "stderr": "", "error": null} +{"command": "powershell -NoProfile -NonInteractive -ExecutionPolicy Bypass -File C:\\one-tests\\author.ps1 -Root C:\\one-tests\\runs\\capture -CloneHost ONE-M6-7E19D637", "exit": 0, "stdout": "", "stderr": "", "error": null} +{"command": "powershell -NoProfile -NonInteractive -ExecutionPolicy Bypass -File C:\\one-tests\\read-current.ps1 -Root C:\\one-tests\\runs\\capture -CloneHost ONE-M6-7E19D637 -ExpectedPages 3 -UseCurrentCache -KeepOpen", "exit": 0, "stdout": "Read 2 sections and 3 pages.\r\n", "stderr": "", "error": null} +{"command": "powershell -NoProfile -Command \"Compress-Archive -Force -Path C:\\one-tests\\runs\\capture\\* -DestinationPath C:\\one-tests\\captured.zip\"", "exit": 0, "stdout": "", "stderr": "", "error": null} diff --git a/corpus/task-tags/native/machine.json b/corpus/task-tags/native/machine.json new file mode 100644 index 0000000000000000000000000000000000000000..5694c3afccb4b8c2eb135352bcfd70833a0a00a3 --- /dev/null +++ b/corpus/task-tags/native/machine.json @@ -0,0 +1 @@ +{"name": "m6-7e19d637", "hostname": "ONE-M6-7E19D637"} diff --git a/corpus/task-tags/native/notebook/Open Notebook.onetoc2 b/corpus/task-tags/native/notebook/Open Notebook.onetoc2 new file mode 100644 index 0000000000000000000000000000000000000000..472d6e7550e214918ba659757f43e019b2cf823d Binary files /dev/null and b/corpus/task-tags/native/notebook/Open Notebook.onetoc2 differ diff --git a/corpus/task-tags/native/notebook/Tasks.one b/corpus/task-tags/native/notebook/Tasks.one new file mode 100644 index 0000000000000000000000000000000000000000..08e153510b758c1c4246131099a0606e8cd5e7d0 Binary files /dev/null and b/corpus/task-tags/native/notebook/Tasks.one differ diff --git a/corpus/task-tags/native/notebook/fixture.json b/corpus/task-tags/native/notebook/fixture.json new file mode 100644 index 0000000000000000000000000000000000000000..744a1a27b2a4bf593cd939baec710332fc024232 --- /dev/null +++ b/corpus/task-tags/native/notebook/fixture.json @@ -0,0 +1,15 @@ +{ + "sections": [ + "Tasks.one" + ], + "pages": [ + { + "section": "Tasks.one", + "xml": "Remove a taskFirst taskSecond taskTagged taskUntagged" + }, + { + "section": "Tasks.one", + "xml": "Delete and undoFirst taskSecond taskTagged taskUntagged" + } + ] +} \ No newline at end of file diff --git a/corpus/task-tags/native/notebook/synthetic.one b/corpus/task-tags/native/notebook/synthetic.one new file mode 100644 index 0000000000000000000000000000000000000000..1ed24299e20474a4366648beba61b958cc609d9d Binary files /dev/null and b/corpus/task-tags/native/notebook/synthetic.one differ diff --git a/corpus/task-tags/native/read/environment.json b/corpus/task-tags/native/read/environment.json new file mode 100644 index 0000000000000000000000000000000000000000..8e54223dba9fed8f8acde2a6ccc065fddcec0607 --- /dev/null +++ b/corpus/task-tags/native/read/environment.json @@ -0,0 +1,7 @@ +{ + "powershell": "5.1.14409.1005", + "schema": "xs2010", + "hostname": "ONE-M6-7E19D637", + "cold": false, + "onenote": "14.0.4763.1000" +} diff --git a/corpus/task-tags/native/read/hierarchy.xml b/corpus/task-tags/native/read/hierarchy.xml new file mode 100644 index 0000000000000000000000000000000000000000..96653181b75da23827728de2327b2175641f5ec8 --- /dev/null +++ b/corpus/task-tags/native/read/hierarchy.xml @@ -0,0 +1,2 @@ + + diff --git a/corpus/task-tags/native/read/page-000.navigation.json b/corpus/task-tags/native/read/page-000.navigation.json new file mode 100644 index 0000000000000000000000000000000000000000..002a1c45a5ba0121a28f35c11466c61b8a2aac13 --- /dev/null +++ b/corpus/task-tags/native/read/page-000.navigation.json @@ -0,0 +1,13 @@ +{ + "exit": 0, + "stdout": "", + "stderr": "", + "w": 800, + "h": 600, + "error": null, + "win": { + "title": "A notebook created in Rust. - Microsoft OneNote", + "class": "Framework::CFrame", + "dialog": false + } +} \ No newline at end of file diff --git a/corpus/task-tags/native/read/page-000.png b/corpus/task-tags/native/read/page-000.png new file mode 100644 index 0000000000000000000000000000000000000000..1d05020740f0a612f768176deb75dd49f24a7188 Binary files /dev/null and b/corpus/task-tags/native/read/page-000.png differ diff --git a/corpus/task-tags/native/read/page-000.xml b/corpus/task-tags/native/read/page-000.xml new file mode 100644 index 0000000000000000000000000000000000000000..60f94b3d923c8f5a7dafc5da13daa2b3e9f31eac --- /dev/null +++ b/corpus/task-tags/native/read/page-000.xml @@ -0,0 +1,2 @@ + + diff --git a/corpus/task-tags/native/read/page-001.navigation.json b/corpus/task-tags/native/read/page-001.navigation.json new file mode 100644 index 0000000000000000000000000000000000000000..f96bf51a7b355a67f40c9bb4d1d7f1cf04129a47 --- /dev/null +++ b/corpus/task-tags/native/read/page-001.navigation.json @@ -0,0 +1,13 @@ +{ + "exit": 0, + "stdout": "", + "stderr": "", + "w": 800, + "h": 600, + "error": null, + "win": { + "title": "Delete and undo - Microsoft OneNote", + "class": "Framework::CFrame", + "dialog": false + } +} \ No newline at end of file diff --git a/corpus/task-tags/native/read/page-001.png b/corpus/task-tags/native/read/page-001.png new file mode 100644 index 0000000000000000000000000000000000000000..2fd9dc14761352197d5a577db5c87b6552193ac3 Binary files /dev/null and b/corpus/task-tags/native/read/page-001.png differ diff --git a/corpus/task-tags/native/read/page-001.xml b/corpus/task-tags/native/read/page-001.xml new file mode 100644 index 0000000000000000000000000000000000000000..6ebf7cb44a3a7e3d59c403dd43712123f95fb2b3 --- /dev/null +++ b/corpus/task-tags/native/read/page-001.xml @@ -0,0 +1,2 @@ + + diff --git a/corpus/task-tags/native/read/page-002.navigation.json b/corpus/task-tags/native/read/page-002.navigation.json new file mode 100644 index 0000000000000000000000000000000000000000..e5a5078a1abd65bb9ba64a5a6d5603d374cfc401 --- /dev/null +++ b/corpus/task-tags/native/read/page-002.navigation.json @@ -0,0 +1,13 @@ +{ + "exit": 0, + "stdout": "", + "stderr": "", + "w": 800, + "h": 600, + "error": null, + "win": { + "title": "Remove a task - Microsoft OneNote", + "class": "Framework::CFrame", + "dialog": false + } +} \ No newline at end of file diff --git a/corpus/task-tags/native/read/page-002.png b/corpus/task-tags/native/read/page-002.png new file mode 100644 index 0000000000000000000000000000000000000000..91e9eae3c8ca4e6a29fa14b1dd69c8ce16b243f8 Binary files /dev/null and b/corpus/task-tags/native/read/page-002.png differ diff --git a/corpus/task-tags/native/read/page-002.xml b/corpus/task-tags/native/read/page-002.xml new file mode 100644 index 0000000000000000000000000000000000000000..9ce6cd2ec2b05abcbd5ac71a37ed2cbda9fbc1cb --- /dev/null +++ b/corpus/task-tags/native/read/page-002.xml @@ -0,0 +1,2 @@ + + diff --git a/corpus/task-tags/native/read/payloads.json b/corpus/task-tags/native/read/payloads.json new file mode 100644 index 0000000000000000000000000000000000000000..1ce91a413d6fff4d58765187b76756f0e97ae843 --- /dev/null +++ b/corpus/task-tags/native/read/payloads.json @@ -0,0 +1,3 @@ +[ + +] \ No newline at end of file diff --git a/corpus/task-tags/native/run.json b/corpus/task-tags/native/run.json new file mode 100644 index 0000000000000000000000000000000000000000..368201b132e9cc2d3ce44f8b52a844e2ad9d1974 --- /dev/null +++ b/corpus/task-tags/native/run.json @@ -0,0 +1,19 @@ +{ + "notebook": "/private/tmp/tasktags/input/notebook", + "expected_pages": 3, + "author": "tools/native/pages.ps1", + "author_timeout_seconds": 600, + "inspect": false, + "collect_notebook": true, + "base": { + "file": "win7-office-base.qcow2", + "format": "qcow2", + "sha256": "a1a4f8fab782ee14885ff801ca2f6347c208fdcfc3513637096f314315c89346", + "virtual_size": 68719476736 + }, + "scripts": { + "author.ps1": "dfbf0e016aa4c8486ecea9f39768c633de05ac0db8214139052ca2e9c43bbd02", + "cold.ps1": "c177fc72ae6c2634d186f5671a880de20b09f9716532c37c69cb13aa15c7e331", + "read.ps1": "22bc760f955ef59bb5f0ccfd55271f8d868674084127fbe219e1048bd3777f35" + } +} diff --git a/corpus/task-tags/native/scripts/author.ps1 b/corpus/task-tags/native/scripts/author.ps1 new file mode 100644 index 0000000000000000000000000000000000000000..8e428618e3452caa35da88687293b520c273b6c6 --- /dev/null +++ b/corpus/task-tags/native/scripts/author.ps1 @@ -0,0 +1,42 @@ +param([Parameter(Mandatory=$true)][string]$Root, [Parameter(Mandatory=$true)][string]$CloneHost) +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' +& "$PSScriptRoot\cold-current.ps1" -Root $Root -CloneHost $CloneHost +$fixture = Get-Content (Join-Path $Root 'notebook\fixture.json') -Raw -Encoding UTF8 | ConvertFrom-Json +$inputs = Join-Path $Root 'inputs' +New-Item -ItemType Directory -Path $inputs | Out-Null +$app = New-Object -ComObject OneNote.Application +$notebookId = '' +try { + $app.OpenHierarchy((Join-Path $Root 'notebook'), '', [ref]$notebookId, 0) + $sections = @{} + foreach ($path in $fixture.sections) { + $parts = $path.Split('/') + if (@($parts | Where-Object { $_ -in @('', '.', '..') -or $_.Contains('\') -or $_.Contains(':') }).Count) { throw 'Invalid fixture section path.' } + $parent = $notebookId + for ($index = 0; $index -lt $parts.Count; $index++) { + $id = '' + $app.OpenHierarchy($parts[$index], $parent, [ref]$id, $(if ($index -eq $parts.Count - 1) {3} else {2})) + $parent = $id + } + $sections[$path] = $id + } + $index = 0 + foreach ($item in $fixture.pages) { + $pageId = '' + $app.CreateNewPage($sections[$item.section], [ref]$pageId, 0) + [xml]$xml = $item.xml.Replace('__ASSETS__', [Security.SecurityElement]::Escape((Join-Path $Root 'notebook\fixture-data'))) + $xml.DocumentElement.SetAttribute('ID', $pageId) + [IO.File]::WriteAllText((Join-Path $inputs ('page-{0:d3}.xml' -f $index)), $xml.OuterXml, [Text.Encoding]::UTF8) + $app.UpdatePageContent($xml.OuterXml, [DateTime]::MinValue, 1, $true) + $index++ + } + $app.SyncHierarchy($notebookId) + $app.CloseNotebook($notebookId, $false) + $notebookId = '' +} finally { + if ($notebookId) { $app.CloseNotebook($notebookId, $false) } + [void][Runtime.InteropServices.Marshal]::FinalReleaseComObject($app) + [GC]::Collect() + [GC]::WaitForPendingFinalizers() +} diff --git a/corpus/task-tags/native/scripts/cold.ps1 b/corpus/task-tags/native/scripts/cold.ps1 new file mode 100644 index 0000000000000000000000000000000000000000..a0b99e7aeb86f4dc10032537800501a52ad65b4f --- /dev/null +++ b/corpus/task-tags/native/scripts/cold.ps1 @@ -0,0 +1,27 @@ +param([Parameter(Mandatory=$true)][string]$Root, [string]$CloneHost = '') +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' +$root = [IO.Path]::GetFullPath($Root).TrimEnd('\') +if ([IO.Path]::GetDirectoryName($root) -ne 'C:\one-tests\runs') { + throw 'Choose a run directly below C:\one-tests\runs.' +} +if (Get-Process ONENOTE -ErrorAction SilentlyContinue) { throw 'Close OneNote before resetting its test cache.' } +$key = 'HKCU:\Software\Microsoft\Office\14.0\OneNote' +if ($CloneHost) { + if ($CloneHost -notmatch '^ONE-[A-Z0-9-]+$' -or [Environment]::MachineName -ne $CloneHost) { + throw 'The disposable clone hostname does not match this machine.' + } + New-Item "$key\Options\Paths" -Force | Out-Null + New-ItemProperty "$key\Options\Paths" -Name UnfiledNotesSection -PropertyType ExpandString -Value 'C:\one-tests\Loose.one' -Force | Out-Null +} elseif ((Get-ItemProperty "$key\Options\Paths").UnfiledNotesSection -ne 'C:\one-tests\Loose.one' -or + -not (Test-Path 'C:\one-tests\profile-original-cache')) { + throw 'Park the personal OneNote profile before resetting the test cache.' +} +$cache = Join-Path $env:LOCALAPPDATA 'Microsoft\OneNote\14.0' +$parked = Join-Path 'C:\one-tests\caches' ([IO.Path]::GetFileName($root)) +if (Test-Path $parked) { throw 'Choose a new run; its parked cache already exists.' } +New-Item -ItemType Directory -Path 'C:\one-tests\caches' -Force | Out-Null +if (Test-Path $cache) { Move-Item -LiteralPath $cache -Destination $parked } +if (Test-Path "$key\OpenNotebooks") { Remove-Item "$key\OpenNotebooks" -Recurse } +New-Item "$key\OpenNotebooks" | Out-Null +New-ItemProperty "$key\OpenNotebooks" -Name '1' -PropertyType String -Value "$root\notebook" | Out-Null diff --git a/corpus/task-tags/native/scripts/read.ps1 b/corpus/task-tags/native/scripts/read.ps1 new file mode 100644 index 0000000000000000000000000000000000000000..a42366b32e8baaf13dae92e9472cae7a071220cc --- /dev/null +++ b/corpus/task-tags/native/scripts/read.ps1 @@ -0,0 +1,144 @@ +param( + [Parameter(Mandatory=$true)][string]$Root, + [int]$ExpectedPages = -1, + [switch]$UseCurrentCache, + [switch]$Pdf, + [switch]$KeepOpen, + [string]$CloneHost = '' +) +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' +$root = [IO.Path]::GetFullPath($Root).TrimEnd('\') +if ([IO.Path]::GetDirectoryName($root) -ne 'C:\one-tests\runs') { + throw 'Choose a run directly below C:\one-tests\runs.' +} +$notebook = Join-Path $root 'notebook' +$output = Join-Path $root 'read' +if (Test-Path $output) { throw 'Choose a new read destination.' } +if ($UseCurrentCache) { + if ((Get-ItemProperty 'HKCU:\Software\Microsoft\Office\14.0\OneNote\Options\Paths').UnfiledNotesSection -ne 'C:\one-tests\Loose.one') { + throw 'Park the personal OneNote profile before reading test notebooks.' + } +} else { + & "$PSScriptRoot\cold-current.ps1" -Root $root -CloneHost $CloneHost +} +New-Item -ItemType Directory -Path $output | Out-Null +$app = New-Object -ComObject OneNote.Application +$notebookId = '' +$failure = $null +try { + $app.OpenHierarchy($notebook, '', [ref]$notebookId, 0) + $process = Get-Process ONENOTE + @{ hostname = [Environment]::MachineName; onenote = $process.MainModule.FileVersionInfo.FileVersion; + powershell = $PSVersionTable.PSVersion.ToString(); schema = 'xs2010'; cold = (-not $UseCurrentCache.IsPresent) } | + ConvertTo-Json | Set-Content (Join-Path $output 'environment.json') -Encoding UTF8 + $sections = @() + foreach ($file in @(Get-ChildItem $notebook -Recurse | Where-Object { $_.Extension -eq '.one' })) { + $id = '' + $app.OpenHierarchy($file.FullName, '', [ref]$id, 0) + $sections += $id + } + $deadline = [DateTime]::UtcNow.AddSeconds(300) + $previous = '' + $lastChange = '' + $stableSince = [DateTime]::UtcNow + $settled = $false + do { + $pages = @{} + foreach ($section in $sections) { + $hierarchy = '' + $app.GetHierarchy($section, 4, [ref]$hierarchy, 1) + [xml]$xml = $hierarchy + foreach ($node in $xml.SelectNodes('//*[@path]')) { + if (-not $node.GetAttribute('path').StartsWith("$notebook\", [StringComparison]::OrdinalIgnoreCase)) { + throw 'OneNote opened a section outside the copied notebook.' + } + } + foreach ($node in $xml.SelectNodes('//*[local-name()="Page"]')) { + $id = $node.GetAttribute('ID') + $content = '' + $app.GetPageContent($id, [ref]$content, 1, 1) + $pages[$id] = $content + } + } + $signature = [String]::Join('|', @($pages.Keys | Sort-Object | ForEach-Object { $_ + $pages[$_] })) + if ($signature -ne $previous) { + $lastChange = $previous + $previous = $signature + $stableSince = [DateTime]::UtcNow + } + if ((($ExpectedPages -ge 0 -and $pages.Count -eq $ExpectedPages) -or + ($ExpectedPages -lt 0 -and $pages.Count -gt 0)) -and + ([DateTime]::UtcNow - $stableSince).TotalSeconds -ge 2) { $settled = $true; break } + Start-Sleep -Milliseconds 250 + } while ([DateTime]::UtcNow -lt $deadline) + if (-not $settled -or ($ExpectedPages -ge 0 -and $pages.Count -ne $ExpectedPages) -or ($ExpectedPages -lt 0 -and $pages.Count -eq 0)) { + [IO.File]::WriteAllText((Join-Path $output 'previous-signature.txt'), $lastChange, [Text.Encoding]::UTF8) + $index = 0 + foreach ($id in @($pages.Keys | Sort-Object)) { + [IO.File]::WriteAllText((Join-Path $output ('unsettled-{0:d3}.xml' -f $index)), $pages[$id], [Text.Encoding]::UTF8) + $index++ + } + $hierarchy = '' + $app.GetHierarchy($notebookId, 4, [ref]$hierarchy, 1) + [IO.File]::WriteAllText((Join-Path $output 'unsettled-hierarchy.xml'), $hierarchy, [Text.Encoding]::UTF8) + throw "Expected $ExpectedPages stable pages; OneNote returned $($pages.Count), settled=$settled." + } + $index = 0 + $payloads = @() + foreach ($id in @($pages.Keys | Sort-Object)) { + [IO.File]::WriteAllText((Join-Path $output ('page-{0:d3}.xml' -f $index)), $pages[$id], [Text.Encoding]::UTF8) + if ($Pdf) { + $pdfPath = Join-Path $output ('page-{0:d3}.pdf' -f $index) + $app.NavigateTo($id, '', $false) + $app.Publish($id, $pdfPath, 3, '') + if (-not (Test-Path $pdfPath) -or (Get-Item $pdfPath).Length -eq 0) { throw 'OneNote did not publish the page PDF.' } + } + [xml]$page = $pages[$id] + foreach ($file in $page.SelectNodes('//*[local-name()="InsertedFile" or local-name()="MediaFile"]')) { + $bytes = [IO.File]::ReadAllBytes($file.GetAttribute('pathCache')) + $hash = [BitConverter]::ToString([Security.Cryptography.SHA256]::Create().ComputeHash($bytes)).Replace('-', '').ToLowerInvariant() + [IO.File]::WriteAllBytes((Join-Path $output ($hash + '.attachment')), $bytes) + # A paragraph's file is named by its paragraph, one on the page by itself. + $owner = if ($file.ParentNode.LocalName -eq 'Page') { $file } else { $file.ParentNode } + $payloads += @{ page = $id; object = $owner.GetAttribute('objectID'); + kind = $file.LocalName; name = $file.GetAttribute('preferredName'); + sha256 = $hash; bytes = $bytes.Length } + } + $index++ + } + [IO.File]::WriteAllText((Join-Path $output 'payloads.json'), (ConvertTo-Json -InputObject $payloads -Depth 4), [Text.Encoding]::UTF8) + $all = '' + $app.GetHierarchy($notebookId, 4, [ref]$all, 1) + [xml]$finalTree = $all + $finalIds = @($finalTree.SelectNodes('//*[local-name()="Page"]') | ForEach-Object { $_.GetAttribute('ID') } | Sort-Object -Unique) + if ($finalIds.Count -ne $pages.Count -or @($finalIds | Where-Object { -not $pages.ContainsKey($_) }).Count -ne 0) { + throw 'The notebook hierarchy changed while collecting page evidence; repeat the cold read.' + } + [IO.File]::WriteAllText((Join-Path $output 'hierarchy.xml'), $all, [Text.Encoding]::UTF8) + Write-Output "Read $($sections.Count) sections and $($pages.Count) pages." +} catch { + $failure = $_ + [IO.File]::WriteAllText((Join-Path $output 'failure.txt'), ($_ | Out-String), [Text.Encoding]::UTF8) + throw +} finally { + try { + try { + if ($notebookId -and $CloneHost) { $app.SyncHierarchy($notebookId) } + if ($notebookId -and -not $KeepOpen -and (-not $UseCurrentCache -or $CloneHost)) { + $app.CloseNotebook($notebookId, $false) + } + } catch { + if ($null -eq $failure) { throw } + [IO.File]::WriteAllText((Join-Path $output 'cleanup-failure.txt'), ($_ | Out-String), [Text.Encoding]::UTF8) + } + } finally { + [void][Runtime.InteropServices.Marshal]::FinalReleaseComObject($app) + } + $app = $null + [GC]::Collect() + [GC]::WaitForPendingFinalizers() + if (-not $UseCurrentCache -and -not $CloneHost) { + Get-Process ONENOTE -ErrorAction SilentlyContinue | Wait-Process -Timeout 10 + } +} diff --git a/corpus/task-tags/native/source.json b/corpus/task-tags/native/source.json new file mode 100644 index 0000000000000000000000000000000000000000..2a633edfa35e5cbd6be7e671e6ad7070024b433f --- /dev/null +++ b/corpus/task-tags/native/source.json @@ -0,0 +1,20 @@ +[ + { + "path": "Open Notebook.onetoc2", + "bytes": 1612, + "sha256": "fafa5dcb3cd827c7edd30c8ea2fbc957a59fb4d2a8b28b667a3c551c8261ab86", + "mtime_ns": 1790724703557857638 + }, + { + "path": "fixture.json", + "bytes": 2596, + "sha256": "7b243286f1ae5147ae8c97d87ae911e82a6ebfd264fc15450c43176067fbf067", + "mtime_ns": 1790724703583723809 + }, + { + "path": "synthetic.one", + "bytes": 3348, + "sha256": "b0662387c3fdc256d8dbe809e3b8f73b3586559696467fb1fff8f589095fa797", + "mtime_ns": 1790724703558075431 + } +] diff --git a/corpus/task-tags/native/teardown.json b/corpus/task-tags/native/teardown.json new file mode 100644 index 0000000000000000000000000000000000000000..05a47793de40f322e745c4c4183727e3590ecf70 --- /dev/null +++ b/corpus/task-tags/native/teardown.json @@ -0,0 +1 @@ +{"absent": true} diff --git a/crates/canvas/assets/tags/box-small.svg b/crates/canvas/assets/tags/box-small.svg index 17ed0f99c5a134e131cf29b3392fb1940423c000..40989796ea848921c5e00e7994b062800153692a 100644 --- a/crates/canvas/assets/tags/box-small.svg +++ b/crates/canvas/assets/tags/box-small.svg @@ -1,11 +1,10 @@ - - - + - - - - + + + + + diff --git a/crates/canvas/assets/tags/checkbox-green.svg b/crates/canvas/assets/tags/checkbox-green.svg index 26a689bc0d346500fb20cddcb55bb7b8cfda1d45..3ec49601240885b8a1aed61da12bdc5b359f22ab 100644 --- a/crates/canvas/assets/tags/checkbox-green.svg +++ b/crates/canvas/assets/tags/checkbox-green.svg @@ -1,24 +1,18 @@ - - - - - + - - - - - - - - - + - - - + + + + + + + + + diff --git a/crates/canvas/assets/tags/checkbox-yellow.svg b/crates/canvas/assets/tags/checkbox-yellow.svg index 20d79b4f4cb3c19d23e417529533d4693455260a..1ff1a3cebf068760af0b5b93e62ef9d34eb52c67 100644 --- a/crates/canvas/assets/tags/checkbox-yellow.svg +++ b/crates/canvas/assets/tags/checkbox-yellow.svg @@ -1,24 +1,18 @@ - - - - - + - - - - - - - - - + - - - + + + + + + + + + diff --git a/crates/canvas/assets/tags/checkbox.svg b/crates/canvas/assets/tags/checkbox.svg index 17d1680279ef1c19bfdce1024d5f83a5bb86d8ea..31341b84e162e3c92e14379bcd23692d39629207 100644 --- a/crates/canvas/assets/tags/checkbox.svg +++ b/crates/canvas/assets/tags/checkbox.svg @@ -1,24 +1,19 @@ - - - - - + - - - - - - - - - + - - - + + + + + + + + + + diff --git a/crates/canvas/assets/tags/flag.svg b/crates/canvas/assets/tags/flag.svg deleted file mode 100644 index 7219fb33baf9f7dd45400dc26be74ead45ac4751..0000000000000000000000000000000000000000 --- a/crates/canvas/assets/tags/flag.svg +++ /dev/null @@ -1,10 +0,0 @@ - - - - - - - - - - diff --git a/crates/canvas/assets/tags/mark-arrow-blue.svg b/crates/canvas/assets/tags/mark-arrow-blue.svg index 97c2c3c44577ef5dad304ca3244098182964baa4..40d398d682e325cdce853ed5e6fc43e82493c533 100644 --- a/crates/canvas/assets/tags/mark-arrow-blue.svg +++ b/crates/canvas/assets/tags/mark-arrow-blue.svg @@ -1,6 +1,42 @@ - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/crates/canvas/assets/tags/mark-arrow-green.svg b/crates/canvas/assets/tags/mark-arrow-green.svg index 512ac40f957d091737b171cb1fc8ca4ea9d435d5..c3a3032b80e6ed03ff9df0b201a96d90764529f4 100644 --- a/crates/canvas/assets/tags/mark-arrow-green.svg +++ b/crates/canvas/assets/tags/mark-arrow-green.svg @@ -1,6 +1,36 @@ - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/crates/canvas/assets/tags/mark-arrow-yellow.svg b/crates/canvas/assets/tags/mark-arrow-yellow.svg index 34389c3c7493b5cc263937bac07d4e3e4a0372cc..8208759cc5a422d5405a30fc3393292fcc46c275 100644 --- a/crates/canvas/assets/tags/mark-arrow-yellow.svg +++ b/crates/canvas/assets/tags/mark-arrow-yellow.svg @@ -1,6 +1,41 @@ - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/crates/canvas/assets/tags/mark-exclamation-blue.svg b/crates/canvas/assets/tags/mark-exclamation-blue.svg index de951416c56df76d3df63727c07414ea48fb5bab..2b2c26355aaaff4494870a3f6d9b35ac0aeabda9 100644 --- a/crates/canvas/assets/tags/mark-exclamation-blue.svg +++ b/crates/canvas/assets/tags/mark-exclamation-blue.svg @@ -1,9 +1,44 @@ - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/crates/canvas/assets/tags/mark-exclamation-green.svg b/crates/canvas/assets/tags/mark-exclamation-green.svg index 67a7f3799ecb6aea7aefa3536d92e6e5c9cba7f9..e1d8f1a9745f83fad3cf2b173275e39d8bf7e51e 100644 --- a/crates/canvas/assets/tags/mark-exclamation-green.svg +++ b/crates/canvas/assets/tags/mark-exclamation-green.svg @@ -1,9 +1,45 @@ - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/crates/canvas/assets/tags/mark-exclamation-yellow.svg b/crates/canvas/assets/tags/mark-exclamation-yellow.svg index b562e9082ec9bf2d0e6f6dad91a46658025f3f36..ee1fc42c7749fa31ce04c95f50e69cb59116b506 100644 --- a/crates/canvas/assets/tags/mark-exclamation-yellow.svg +++ b/crates/canvas/assets/tags/mark-exclamation-yellow.svg @@ -1,9 +1,44 @@ - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/crates/canvas/assets/tags/mark-flag-blue.svg b/crates/canvas/assets/tags/mark-flag-blue.svg index 878943aa2791d75a33791f1ab3d7fe0daf4b7e90..036ed55b31d97694f175e24c1271077ced41a284 100644 --- a/crates/canvas/assets/tags/mark-flag-blue.svg +++ b/crates/canvas/assets/tags/mark-flag-blue.svg @@ -1,7 +1,55 @@ - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/crates/canvas/assets/tags/mark-flag-green.svg b/crates/canvas/assets/tags/mark-flag-green.svg index fa9780a1eb2742cfc2cc73232cfbc3dd75f41235..ea2e17c01c3f844945e14d4f999f4b960f157865 100644 --- a/crates/canvas/assets/tags/mark-flag-green.svg +++ b/crates/canvas/assets/tags/mark-flag-green.svg @@ -1,7 +1,58 @@ - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/crates/canvas/assets/tags/mark-flag-yellow.svg b/crates/canvas/assets/tags/mark-flag-yellow.svg index 7700e494609f2c94eb96cddbfcaa1dcaf960f9b8..2ecc942fbe74de91305e188828c958a0d4c69bc4 100644 --- a/crates/canvas/assets/tags/mark-flag-yellow.svg +++ b/crates/canvas/assets/tags/mark-flag-yellow.svg @@ -1,7 +1,57 @@ - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/crates/canvas/assets/tags/mark-one-blue.svg b/crates/canvas/assets/tags/mark-one-blue.svg index f95d8aa3643bd45b9b71e6c9d4756543526d9c00..a6eb26cdf061ef5170214d25b46cd2d475f512d9 100644 --- a/crates/canvas/assets/tags/mark-one-blue.svg +++ b/crates/canvas/assets/tags/mark-one-blue.svg @@ -1,7 +1,6 @@ - - - - - + + + + diff --git a/crates/canvas/assets/tags/mark-one-green.svg b/crates/canvas/assets/tags/mark-one-green.svg index e0f1a75f255736d27b2f38c20733cb0deabc2997..6a8441d50ce49007d94cd9a19b55eb836ef925e6 100644 --- a/crates/canvas/assets/tags/mark-one-green.svg +++ b/crates/canvas/assets/tags/mark-one-green.svg @@ -1,7 +1,6 @@ - - - - - + + + + diff --git a/crates/canvas/assets/tags/mark-one-yellow.svg b/crates/canvas/assets/tags/mark-one-yellow.svg index 218834e755712a51e2d812bded96a093729f0cbe..07443c8a4bce54b886cd59950624480f791274cc 100644 --- a/crates/canvas/assets/tags/mark-one-yellow.svg +++ b/crates/canvas/assets/tags/mark-one-yellow.svg @@ -1,7 +1,6 @@ - - - - - + + + + diff --git a/crates/canvas/assets/tags/mark-person-blue.svg b/crates/canvas/assets/tags/mark-person-blue.svg index 1f2d6f0affa1495cf724a9825b126c6d175e5a81..e52b0cde393fcf4ff16d845c582c5fb763081d64 100644 --- a/crates/canvas/assets/tags/mark-person-blue.svg +++ b/crates/canvas/assets/tags/mark-person-blue.svg @@ -1,9 +1,64 @@ - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/crates/canvas/assets/tags/mark-person-green.svg b/crates/canvas/assets/tags/mark-person-green.svg index e57d0341831f043771649d26b2d00876f0ae7ada..f42ddb12a4725b372547a3dba0dc4d3e5dddd568 100644 --- a/crates/canvas/assets/tags/mark-person-green.svg +++ b/crates/canvas/assets/tags/mark-person-green.svg @@ -1,9 +1,69 @@ - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/crates/canvas/assets/tags/mark-person-yellow.svg b/crates/canvas/assets/tags/mark-person-yellow.svg index c7b1b3e74d3f5454151a4bd08edcabb2f54a4dee..15f89ca5d82aac6dce087b592c81f527bf6ecc81 100644 --- a/crates/canvas/assets/tags/mark-person-yellow.svg +++ b/crates/canvas/assets/tags/mark-person-yellow.svg @@ -1,9 +1,71 @@ - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/crates/canvas/assets/tags/mark-star-blue.svg b/crates/canvas/assets/tags/mark-star-blue.svg index 64fc936e96b2dca0a40fc90e64909cbccd1b376c..2ea21816e39ffba2157e84beec390025c62cf68d 100644 --- a/crates/canvas/assets/tags/mark-star-blue.svg +++ b/crates/canvas/assets/tags/mark-star-blue.svg @@ -1,7 +1,59 @@ - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/crates/canvas/assets/tags/mark-star-green.svg b/crates/canvas/assets/tags/mark-star-green.svg index ab4796db3423d428c90af8be4b51038af88a5119..a8755c7e4d054f4f903a4f88d6f878fac6e26c0c 100644 --- a/crates/canvas/assets/tags/mark-star-green.svg +++ b/crates/canvas/assets/tags/mark-star-green.svg @@ -1,7 +1,59 @@ - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/crates/canvas/assets/tags/mark-star-yellow.svg b/crates/canvas/assets/tags/mark-star-yellow.svg index bd1f6c89d96aff646618f66d240d73b103f46cf3..900bbf00365a2f747a675bb305276903f1ae6c35 100644 --- a/crates/canvas/assets/tags/mark-star-yellow.svg +++ b/crates/canvas/assets/tags/mark-star-yellow.svg @@ -1,7 +1,57 @@ - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/crates/canvas/assets/tags/mark-three-blue.svg b/crates/canvas/assets/tags/mark-three-blue.svg index 0e8fb6b92842fa4224379bf207198bafbba11154..86ae99bb698e80008b209361982094762296f3a4 100644 --- a/crates/canvas/assets/tags/mark-three-blue.svg +++ b/crates/canvas/assets/tags/mark-three-blue.svg @@ -1,7 +1,6 @@ - - - - - + + + + diff --git a/crates/canvas/assets/tags/mark-three-green.svg b/crates/canvas/assets/tags/mark-three-green.svg index cb2c99e7a2d95c066a73922b046b00f0b6a53861..03df97e3b44f9a8318651c069649bd889654a073 100644 --- a/crates/canvas/assets/tags/mark-three-green.svg +++ b/crates/canvas/assets/tags/mark-three-green.svg @@ -1,7 +1,6 @@ - - - - - + + + + diff --git a/crates/canvas/assets/tags/mark-three-yellow.svg b/crates/canvas/assets/tags/mark-three-yellow.svg index a742086017b7bfab0a90e6fc57b97b065f9e4bab..438143c1514947b8762d99fc747319fe620459a9 100644 --- a/crates/canvas/assets/tags/mark-three-yellow.svg +++ b/crates/canvas/assets/tags/mark-three-yellow.svg @@ -1,7 +1,6 @@ - - - - - + + + + diff --git a/crates/canvas/assets/tags/mark-two-blue.svg b/crates/canvas/assets/tags/mark-two-blue.svg index 40e030cafc219eb29bd7ee96e2208fa36dc01f32..b57326193a2c4b62a47a274c4a62049a8ec21a68 100644 --- a/crates/canvas/assets/tags/mark-two-blue.svg +++ b/crates/canvas/assets/tags/mark-two-blue.svg @@ -1,7 +1,6 @@ - - - - - + + + + diff --git a/crates/canvas/assets/tags/mark-two-green.svg b/crates/canvas/assets/tags/mark-two-green.svg index a7484b31afe5f08b34e55e527546ded2bb5e8620..d9db6b1a87a1705d37b2d26353f3728f9637e8ac 100644 --- a/crates/canvas/assets/tags/mark-two-green.svg +++ b/crates/canvas/assets/tags/mark-two-green.svg @@ -1,7 +1,6 @@ - - - - - + + + + diff --git a/crates/canvas/assets/tags/mark-two-yellow.svg b/crates/canvas/assets/tags/mark-two-yellow.svg index 43c86f79bd98b9127b769d353f8089e1f89d1294..1c620d8c589296a1d40829a30cd15af91a6ebeae 100644 --- a/crates/canvas/assets/tags/mark-two-yellow.svg +++ b/crates/canvas/assets/tags/mark-two-yellow.svg @@ -1,7 +1,6 @@ - - - - - + + + + diff --git a/crates/canvas/assets/tags/star8-green.svg b/crates/canvas/assets/tags/star8-green.svg new file mode 100644 index 0000000000000000000000000000000000000000..7feb0242557aa3451be873965d4037847b2627d2 --- /dev/null +++ b/crates/canvas/assets/tags/star8-green.svg @@ -0,0 +1,4 @@ + + + + diff --git a/crates/canvas/src/document.rs b/crates/canvas/src/document.rs index 1687d88dd6d9b83cc9b77271854c2b65486a8362..b8c418d7570b901ebacc4afed19f0e3825a02955 100644 --- a/crates/canvas/src/document.rs +++ b/crates/canvas/src/document.rs @@ -1578,6 +1578,8 @@ mod tests { nodes[1].tags.push(onestore::document::Tag { definition: Some(new_id().unwrap()), action_type: Some(0), + shape: None, + property_status: None, status: 1, created: Some(123), completed: None, diff --git a/crates/canvas/src/editor/evidence.rs b/crates/canvas/src/editor/evidence.rs index 19e44c28c7f1132dafb9bad94664402c0f54223f..21e92c05ac22102c3a301dcbbde4da240392e514 100644 --- a/crates/canvas/src/editor/evidence.rs +++ b/crates/canvas/src/editor/evidence.rs @@ -97,6 +97,8 @@ fn open(engine: &mut TextEngine, summary: &str) -> CanvasEditor { node.text_mut().unwrap().tags.push(Tag { definition: Some(definition), action_type: None, + shape: None, + property_status: None, status: u16::from(completed), created: Some(1), completed: Some(u32::from(completed)), diff --git a/crates/canvas/src/editor/format.rs b/crates/canvas/src/editor/format.rs index bd5849f029408c8ea71a5bbf31f258ae1ecb0f88..2654ce08cf56997b2bbaea9bbad4b408ccbe006d 100644 --- a/crates/canvas/src/editor/format.rs +++ b/crates/canvas/src/editor/format.rs @@ -769,6 +769,8 @@ impl CanvasEditor { Tag { definition: Some(id), action_type: None, + shape: None, + property_status: None, status: u16::from(!checkable), created, completed: if checkable { Some(0) } else { created }, @@ -1809,6 +1811,8 @@ mod tests { nodes[2].tags.push(Tag { definition: Some(music), action_type: None, + shape: None, + property_status: None, status: 1, created: Some(1), completed: Some(1), @@ -1938,7 +1942,7 @@ mod tests { .iter() .map(|tag| match tag.icon { crate::outline::TagIcon::Symbol { shape, .. } => shape, - crate::outline::TagIcon::Task => unreachable!(), + crate::outline::TagIcon::Task { .. } => unreachable!(), }) .collect::>(); assert_eq!(shapes.len(), defaults.len() - 2); diff --git a/crates/canvas/src/gpu/mod.rs b/crates/canvas/src/gpu/mod.rs index 11694f2fb8d7abeaefe92b7cc1255de7913e54c0..5d73aea37db05a69280742365d21480eabc67bfe 100644 --- a/crates/canvas/src/gpu/mod.rs +++ b/crates/canvas/src/gpu/mod.rs @@ -141,7 +141,6 @@ pub(crate) fn painted_layout(text: &dyn Glyphs) -> &TextLayout { } const CHECKMARK: &str = include_str!("../../assets/tags/checkmark.svg"); -const FLAG: &str = include_str!("../../assets/tags/flag.svg"); const BOX_SMALL: &str = include_str!("../../assets/tags/box-small.svg"); const CHECK_SMALL: &str = include_str!("../../assets/tags/check-small.svg"); const TAG: &str = include_str!("../../assets/tags/tag.svg"); @@ -155,8 +154,7 @@ enum Art { } /// The artwork of symbol `shape` of MS-ONE's NoteTagShape, as OneNote 2010 draws it on the -/// page: it draws 54, Green 8-Point Star, blue there. A shape MS-ONE does not list shows a -/// plain tag. +/// page. A shape MS-ONE does not list shows a plain tag. fn art(shape: u16) -> Art { use Art::{Badged, Box, Plain}; macro_rules! tag { @@ -218,7 +216,7 @@ fn art(shape: u16) -> Art { 51 => Plain(tag!("circle-2-green")), 52 => Badged(tag!("mark-three-green")), 53 => Plain(tag!("circle-3-green")), - 54 => Plain(tag!("star8-blue")), + 54 => Plain(tag!("star8-green")), 55 => Plain(tag!("tick-green")), 56 => Plain(tag!("circle-green")), 57 => Plain(tag!("arrow-down-green")), @@ -318,8 +316,9 @@ const SYMBOLS: u16 = 143; /// The tag's artwork, drawn in order. pub fn tag_sources(icon: TagIcon) -> &'static [&'static str] { static SOURCES: std::sync::OnceLock; 2]>> = std::sync::OnceLock::new(); - let TagIcon::Symbol { shape, checked } = icon else { - return &[FLAG]; + let (shape, checked) = match icon { + TagIcon::Symbol { shape, checked } => (shape, checked), + TagIcon::Task { shape } => (shape, false), }; let sources = SOURCES.get_or_init(|| { (0..=SYMBOLS) @@ -350,6 +349,83 @@ mod tests { use onestore::page::text::Paragraph; use std::time::Duration; + /// OneNote 2010 draws paragraph `shape N` of `corpus/custom-tags/native/shapes.one` with + /// the art `DRAWN`'s Nth word names (its render: `shapes-1-72.png`, `shapes-73-143.png`); + /// each paragraph's tag definition stores the NoteTagShape. + #[test] + fn symbols_draw_the_art_onenote_draws_for_their_number() { + use onestore::page::{PageObject, ParagraphContent}; + use onestore::{RevisionIndex, Store, document::Document, document::Kind, page::Page}; + const DRAWN: &str = "\ + checkbox-green checkbox-yellow checkbox mark-star-green mark-star-yellow \ + mark-star-blue mark-exclamation-green mark-exclamation-yellow mark-exclamation-blue \ + mark-arrow-green mark-arrow-yellow mark-arrow-blue star follow-up question \ + arrow-right-blue exclamation phone calendar clock idea pushpin address blog smiley \ + ribbon key mark-one-blue circle-1-blue mark-two-blue circle-2-blue mark-three-blue \ + circle-3-blue star8-blue tick-blue circle-blue arrow-down-blue arrow-left-blue \ + solid-target-blue star-blue sun-blue target-blue triangle-blue umbrella-blue \ + arrow-up-blue x-dots-blue x-blue mark-one-green circle-1-green mark-two-green \ + circle-2-green mark-three-green circle-3-green star8-green tick-green circle-green \ + arrow-down-green arrow-left-green arrow-right-green solid-target-green star-green \ + sun-green target-green triangle-green umbrella-green arrow-up-green x-dots-green \ + x-green mark-one-yellow circle-1-yellow mark-two-yellow circle-2-yellow \ + mark-three-yellow circle-3-yellow star8-yellow tick-yellow circle-yellow \ + arrow-down-yellow arrow-left-yellow arrow-right-yellow solid-target-yellow \ + sun-yellow target-yellow triangle-yellow umbrella-yellow arrow-up-yellow \ + x-dots-yellow x-yellow flag-today flag-tomorrow flag-this-week flag-next-week \ + flag-no-date mark-person-blue mark-person-yellow mark-person-green mark-flag-blue \ + mark-flag-yellow mark-flag-green red-square yellow-square blue-square green-square \ + orange-square pink-square email envelope envelope-open mobile phone-clock \ + question-balloon paperclip frown im-contact person people bell contact rose date \ + music movie quote globe web laptop plane car binoculars presentation password book \ + notebook paper research highlight dollar coins schedule lightning cloud heart \ + sunflower"; + let bytes = include_bytes!("../../../../corpus/custom-tags/native/shapes.one"); + let store = Store::parse(bytes).unwrap(); + let index = RevisionIndex::parse(&store).unwrap(); + let document = Document::parse(&index).unwrap(); + let files: Vec<&str> = DRAWN.split_whitespace().collect(); + let mut seen = Vec::new(); + for (space, _) in document.pages().unwrap() { + let page = Page::from_space(&document, space).unwrap(); + if !page.title.starts_with("Shapes") { + continue; + } + for object in &page.objects { + let PageObject::Outline(outline) = object else { + continue; + }; + for paragraph in &outline.paragraphs { + let ParagraphContent::Text(text) = ¶graph.content else { + continue; + }; + let label: usize = text.text.text()["shape ".len()..].parse().unwrap(); + let definition = text.tags[0].definition.unwrap(); + let Kind::TagDefinition { + shape: Some(shape), .. + } = page.definitions[&definition].kind + else { + panic!("shape {label} stores no symbol"); + }; + let (Art::Box(drawn) | Art::Badged(drawn) | Art::Plain(drawn)) = art(shape); + let file = files[label - 1]; + let expected = std::fs::read_to_string(format!( + "{}/assets/tags/{file}.svg", + env!("CARGO_MANIFEST_DIR") + )) + .unwrap(); + assert_eq!( + drawn, expected, + "shape {label}, stored {shape}, draws {file}" + ); + seen.push(label); + } + } + } + seen.sort_unstable(); + assert_eq!(seen, (1..=143).collect::>()); + } + /// A page colour paints as OneNote paints it on white paper, and on dark paper as a /// dark paper of its hue; no colour leaves the paper as it was. #[test] @@ -465,7 +541,7 @@ mod tests { ] .into_iter() .map(|(shape, checked)| TagIcon::of(shape, checked).unwrap()) - .chain([TagIcon::Task]) + .chain([TagIcon::Task { shape: 91 }]) .collect(); let text = TextEngine::default() .layout( diff --git a/crates/canvas/src/gpu/page.rs b/crates/canvas/src/gpu/page.rs index 2c4cdd47853e46e6333419516c432bf8d8922d61..2256e577d5f1a9f3c70dfecc4f154d907432ec7e 100644 --- a/crates/canvas/src/gpu/page.rs +++ b/crates/canvas/src/gpu/page.rs @@ -1777,6 +1777,8 @@ mod tests { definition: Some(tag), status: 1, action_type: None, + shape: None, + property_status: None, created: None, completed: None, start: None, diff --git a/crates/canvas/src/interaction/accessibility.rs b/crates/canvas/src/interaction/accessibility.rs index c4b2d33ab987b057760be50f3af52345c5923c28..d96beb8059a2814c3c18484dacaedbc3e2a2eb7c 100644 --- a/crates/canvas/src/interaction/accessibility.rs +++ b/crates/canvas/src/interaction/accessibility.rs @@ -598,7 +598,7 @@ fn runs_of(shaped: &ParagraphLayout, breaks: bool) -> Result let label = match tag.icon { _ if !tag.label.is_empty() => &tag.label, TagIcon::Symbol { shape, .. } => symbol_name(shape).unwrap_or("Tag"), - TagIcon::Task => "Outlook task", + TagIcon::Task { .. } => "Outlook task", }; let state = match tag.icon { TagIcon::Symbol { checked: true, .. } => ", completed", @@ -946,6 +946,8 @@ pub(super) mod tests { paragraphs[index].text_mut().unwrap().tags.push(Tag { definition: Some(id), action_type: None, + shape: None, + property_status: None, status, created: None, completed: None, diff --git a/crates/canvas/src/interaction/mod.rs b/crates/canvas/src/interaction/mod.rs index 3710fef7a6b230d0376a9a22f97a5ed71799f09d..839af8f9900da95d25927e21fc3b43d140004764 100644 --- a/crates/canvas/src/interaction/mod.rs +++ b/crates/canvas/src/interaction/mod.rs @@ -504,6 +504,37 @@ impl PageView { self.hit_test(self.viewport.document_point(position)) } + /// The Outlook task icon under the pointer, in view device pixels, for the host to name. + pub fn task_under_pointer(&self) -> Option<[f32; 4]> { + if !self.pointer_inside || self.drag.is_some() { + return None; + } + let [x, y] = self.viewport.document_point(self.pointer); + let view = |v: f32, axis: usize| v * self.viewport.scale + self.viewport.origin[axis]; + self.editor.visible_outlines().find_map(|outline| { + let offset = match &self.scene { + Some((_, offset)) if self.editor.has_page_outline(outline.id) => *offset, + _ => [0.0; 2], + }; + let shaped = outline.shaped(); + let left = offset[0] + outline.origin()[0] + shaped.tag_column_offset(); + let top = offset[1] + outline.origin()[1]; + shaped.paragraphs.iter().find_map(|paragraph| { + paragraph.tags.iter().find_map(|tag| { + let x0 = left + tag.origin[0]; + let y0 = top + paragraph.origin[1] + tag.origin[1]; + (matches!(tag.icon, crate::outline::TagIcon::Task { .. }) + && (x0..=x0 + tag.size).contains(&x) + && (y0..=y0 + tag.size).contains(&y)) + .then(|| { + let [x1, y1] = [x0 + tag.size, y0 + tag.size]; + [view(x0, 0), view(y0, 1), view(x1, 0), view(y1, 1)] + }) + }) + }) + }) + } + fn set_object_focus(&mut self, focus: Option) { if self.object_focus != focus { self.editor.finish_composition(); diff --git a/crates/canvas/src/outline.rs b/crates/canvas/src/outline.rs index eb80186eabebc1c959cce5a20d6043a1347cc699..f4791b37f49dcbc84876013108dd74bb2146a2c2 100644 --- a/crates/canvas/src/outline.rs +++ b/crates/canvas/src/outline.rs @@ -227,8 +227,8 @@ pub struct ParagraphLayout { pub enum TagIcon { /// Symbol `shape` of MS-ONE's NoteTagShape, `checked` only where it is a check box. Symbol { shape: u16, checked: bool }, - /// An Outlook task, which OneNote 2010 flags. - Task, + /// An Outlook task, drawn with its stored follow-up flag and never checked from the page. + Task { shape: u16 }, } impl TagIcon { @@ -887,8 +887,8 @@ impl ParagraphLayout { return Err(LayoutError::InvalidSpacing); } // Stored newest first; OneNote lists and paints tags oldest first. - // An Outlook task tag carries no definition; a tag whose definition is missing shows - // nothing, as OneNote draws no icon for it. + // An Outlook task tag carries no definition but its own icon; a tag whose definition + // is missing shows nothing, as OneNote draws no icon for it. let mut tag_definitions: Vec<_> = paragraph .tags .iter() @@ -1065,7 +1065,9 @@ impl ParagraphLayout { for tag in &tag_definitions { let (tag, shape, label) = (tag.0, tag.1, tag.2); let icon = match shape { - None => TagIcon::Task, + None => TagIcon::Task { + shape: tag.shape.unwrap_or(0), + }, Some(shape) => match TagIcon::of(shape.unwrap_or(0), tag.status & 1 != 0) { Some(icon) => icon, None => continue, @@ -2241,6 +2243,8 @@ mod tests { definition: Some(id), status: 3, action_type: None, + shape: None, + property_status: None, created: None, completed: None, start: None, @@ -2356,8 +2360,13 @@ mod tests { } ); let mut task = node.clone(); - task.text_mut().unwrap().tags[0].status |= 4; - assert_eq!(shaped(&task, &definitions).tags[0].icon, TagIcon::Task); + let stored = &mut task.text_mut().unwrap().tags[0]; + stored.status |= 4; + stored.shape = Some(89); + assert_eq!( + shaped(&task, &definitions).tags[0].icon, + TagIcon::Task { shape: 89 } + ); let mut untagged = definitions.clone(); untagged.remove(&id); assert!(shaped(&node, &untagged).tags.is_empty()); @@ -2403,6 +2412,8 @@ mod tests { let tag = Tag { definition: Some(id(902)), action_type: None, + shape: None, + property_status: None, status: 0, created: None, completed: None, @@ -2499,6 +2510,8 @@ mod tests { let tag = Tag { definition: Some(id(902)), action_type: None, + shape: None, + property_status: None, status: 0, created: None, completed: None, @@ -2665,6 +2678,8 @@ mod tests { definition: Some(id), status: 1, action_type: None, + shape: None, + property_status: None, created: None, completed: None, start: None, diff --git a/crates/canvas/tests/task_tags.rs b/crates/canvas/tests/task_tags.rs new file mode 100644 index 0000000000000000000000000000000000000000..cb7dac61188519a6ee9a7b89809d7cd7e9f34b9a --- /dev/null +++ b/crates/canvas/tests/task_tags.rs @@ -0,0 +1,145 @@ +//! Outlook task tags stay read-only through the editor: Remove Tag drops one, removing a +//! normal tag beside one keeps it, and deleting a page and undoing restores them as OneNote +//! 2010 stored them (`corpus/task-tags`). + +use canvas::{ + document::TextPosition, + editor::{CanvasEditor, Formatting, NoteTag, Whole}, + layout::TextEngine, +}; +use onestore::{ + Arena, ExGuid, Section, Store, + document::Tag, + op::{Edit, Op}, + page::{Page, PageObject}, +}; + +const TASKS: &[u8] = include_bytes!("../../../corpus/task-tags/native/notebook/Tasks.one"); + +/// Each body paragraph's text and tags, their arena positions cleared. +fn tagged(page: &Page) -> Vec<(String, Vec)> { + page.objects + .iter() + .find_map(|object| match object { + PageObject::Outline(outline) if !outline.title => Some(&outline.paragraphs), + _ => None, + }) + .unwrap() + .iter() + .map(|paragraph| { + let text = paragraph.text().unwrap(); + let tags = paragraph + .tags + .iter() + .chain(&text.tags) + .map(|tag| Tag { + extra_set: 0, + ..tag.clone() + }) + .collect(); + (text.text.text().to_owned(), tags) + }) + .collect() +} + +fn body(editor: &mut CanvasEditor) { + let body = editor + .outlines() + .iter() + .find(|outline| !outline.title) + .unwrap() + .id; + editor.focus_outline(body).unwrap(); +} + +fn at(paragraph: usize) -> canvas::editor::Selection { + [TextPosition { + paragraph, + offset: 0, + }; 2] + .into() +} + +/// Applies the editor's pending ops to the page in `space` as one edit. +fn publish(section: &mut Section<'_>, space: ExGuid, editor: &mut CanvasEditor, at: u64) { + let ops = editor + .take_ops() + .unwrap() + .into_iter() + .map(|op| Op::Page { space, op }) + .collect(); + section.apply("Author", &Edit { at, ops }).unwrap(); +} + +/// `SNOWBOUND_TASK_TAGS_EXPORT` names a new directory receiving the candidate for a cold +/// reopen. +#[test] +fn task_tags_survive_tag_removal_and_page_delete_undo() { + let arena = Arena::default(); + let mut section = Section::open(&arena, TASKS.to_vec()).unwrap(); + let pages = section.pages().unwrap(); + let space = |title: &str| pages.iter().find(|page| page.1 == title).unwrap().0; + let (remove, restore) = (space("Remove a task"), space("Delete and undo")); + let before = tagged(§ion.page(remove).unwrap()); + let shapes: Vec<_> = before + .iter() + .flat_map(|(_, tags)| tags) + .map(|tag| (tag.definition.is_some(), tag.status & 4 != 0, tag.shape)) + .collect(); + assert_eq!( + shapes, + [ + (false, true, Some(89)), + (false, true, Some(89)), + (false, true, Some(89)), + (true, false, None), + ] + ); + let mut engine = TextEngine::default(); + + let mut editor = CanvasEditor::from_page(section.page(remove).unwrap(), &mut engine).unwrap(); + body(&mut editor); + editor.select(at(0)).unwrap(); + editor.format(&mut engine, Formatting::RemoveTags).unwrap(); + editor.select(at(2)).unwrap(); + let to_do = NoteTag::defaults()[0].clone(); + editor + .format(&mut engine, Formatting::Tag(to_do, 0)) + .unwrap(); + publish(&mut section, remove, &mut editor, 134_000_000_000_000_000); + let mut expected = before.clone(); + expected[0].1.clear(); + expected[2].1.retain(|tag| tag.definition.is_none()); + assert_eq!(tagged(§ion.page(remove).unwrap()), expected); + + let original = section.page(restore).unwrap(); + let kept = tagged(&original); + let mut editor = CanvasEditor::from_page(original, &mut engine).unwrap(); + body(&mut editor); + while editor.whole() != Some(Whole::Page) { + editor.widen_selection().unwrap(); + } + editor.delete(&mut engine, false).unwrap(); + publish(&mut section, restore, &mut editor, 134_000_000_010_000_000); + assert!(editor.undo(&mut engine).unwrap()); + publish(&mut section, restore, &mut editor, 134_000_000_020_000_000); + assert_eq!(tagged(§ion.page(restore).unwrap()), kept); + + section.seal().unwrap(); + let written = section.image(); + let reread = Section::open(&arena, written.clone()).unwrap(); + assert_eq!(tagged(&reread.page(remove).unwrap()), expected); + assert_eq!(tagged(&reread.page(restore).unwrap()), kept); + if let Some(directory) = std::env::var_os("SNOWBOUND_TASK_TAGS_EXPORT") { + let directory = std::path::PathBuf::from(directory); + std::fs::create_dir(&directory).unwrap(); + std::fs::write(directory.join("Tasks.one"), &written).unwrap(); + let file_id = Store::parse(&written).unwrap().header.file_id; + std::fs::write( + directory.join("Open Notebook.onetoc2"), + onestore::create_table_of_contents("Open Notebook.onetoc2", &[("Tasks.one", file_id)]) + .unwrap(), + ) + .unwrap(); + } +} diff --git a/crates/draw/src/render.rs b/crates/draw/src/render.rs index 48cca150a12f8ad15a514a659e2c42f0448e1684..32fc7f447c79f8086da71519b2c7b4b086697e68 100644 --- a/crates/draw/src/render.rs +++ b/crates/draw/src/render.rs @@ -224,7 +224,6 @@ enum AtlasKey { Icon { sources: &'static [&'static str], size: u32, - phase: [u8; 2], /// The bits of the colour `currentColor` paints. ink: [u32; 3], palette: [Option<[u32; 3]>; 3], @@ -1104,27 +1103,24 @@ impl Renderer { tint: [f32; 4], palette: &Palette, ) -> Result<(), RenderError> { - let size = size * space.scale; - let [x, y] = space.origin.map(|v| (v * 4.0).round() * 0.25); + // Whole device pixels keep a sprite's pixel-art edges crisp at every scale. + let size = (size * space.scale).round().max(1.0); + let [x, y] = space.origin.map(f32::round); if !size.is_finite() || [x, y].iter().any(|v| !v.is_finite()) { return Err(RenderError::InvalidPrimitive); } - if x + size + 1.0 < 0.0 - || y + size + 1.0 < 0.0 - || x - 1.0 > space.size[0] as f32 - || y - 1.0 > space.size[1] as f32 + if x + size < 0.0 || y + size < 0.0 || x > space.size[0] as f32 || y > space.size[1] as f32 { return Ok(()); } - if size + 3.0 > self.atlas_side() as f32 { + if size + 2.0 > self.atlas_side() as f32 { return Err(RenderError::AtlasFull); } - let phase = [((x - x.floor()) * 4.0) as u8, ((y - y.floor()) * 4.0) as u8]; + let size = size as u32; let ink = [tint[0], tint[1], tint[2]]; let key = AtlasKey::Icon { sources, - size: size.to_bits(), - phase, + size, ink: ink.map(f32::to_bits), palette: palette.bits(), }; @@ -1134,20 +1130,13 @@ impl Renderer { if self.glyphs.len() >= self.glyph_limit() { return Err(RenderError::AtlasFull); } - let glyph = self.upload(icon::rasterize(sources, size, phase, ink, palette))?; + let glyph = self.upload(icon::rasterize(sources, size, ink, palette))?; self.glyphs.insert(key, Some(glyph)); glyph }; - let left = x.floor(); - let top = y.floor(); self.quad( space, - [ - left, - top, - left + glyph.width as f32, - top + glyph.height as f32, - ], + [x, y, x + glyph.width as f32, y + glyph.height as f32], self.uv(glyph), [1.0, 1.0, 1.0, tint[3]], ) diff --git a/crates/draw/src/render/icon.rs b/crates/draw/src/render/icon.rs index 77b71307eaa064fb42ca867fd11e1ae57cb44c58..b9a97c18d0bfab5511beb37060b12e6ab2b602ad 100644 --- a/crates/draw/src/render/icon.rs +++ b/crates/draw/src/render/icon.rs @@ -1,9 +1,6 @@ use swash::{ scale::image::{Content, Image}, - zeno::{ - self, Cap, Fill, Join, Mask, PathBuilder, Placement, Point, Stroke, Style, Transform, - Vector, - }, + zeno::{self, Cap, Fill, Join, Mask, PathBuilder, Placement, Point, Stroke, Style, Transform}, }; /// Colours for an icon's slots, linear RGB: paths of `class="accent"`, `"highlight"` or @@ -32,24 +29,17 @@ impl Palette { } } -/// Rasterizes 16×16 SVG sources over each other at `size` device pixels, offset by -/// `phase` quarter pixels. Paths are filled and stroked with `#rrggbb`, `currentColor`, +/// Rasterizes 16×16 SVG sources over each other at a whole `size` of device pixels. Paths are filled and stroked with `#rrggbb`, `currentColor`, /// which paints the linear `ink`, or two-stop `linearGradient`s with SVG's defaults /// (`objectBoundingBox` unless `userSpaceOnUse`, `x1`..`y2`, stop offsets, pad spread; no /// `gradientTransform`). `fill-opacity` and `stroke-opacity` apply, and strokes have round /// caps and joins. A slot's paths keep the shading among their colours: the hue turn, /// saturation scale and lightness shift that carry the mean of the slot's colours in a /// source onto the palette's colour apply to each of them. -pub(crate) fn rasterize( - sources: &[&str], - size: f32, - phase: [u8; 2], - ink: [f32; 3], - palette: &Palette, -) -> Image { - let side = (size + 1.0).ceil() as u32; +pub(crate) fn rasterize(sources: &[&str], size: u32, ink: [f32; 3], palette: &Palette) -> Image { + let side = size; + let size = size as f32; let mut pixels = vec![[0.0_f32; 4]; (side * side) as usize]; - let offset = phase.map(|quarters| f32::from(quarters) * 0.25); for source in sources { let svg = roxmltree::Document::parse(source).expect("Bundled icon SVG must be valid"); assert_eq!(svg.root_element().attribute("viewBox"), Some("0 0 16 16")); @@ -135,7 +125,6 @@ pub(crate) fn rasterize( let (mask, _) = Mask::new(data) .style(style) .transform(Some(Transform::scale(size / 16.0, size / 16.0))) - .render_offset(Vector::new(offset[0], offset[1])) .size(side, side) .render(); for (index, (pixel, coverage)) in pixels.iter_mut().zip(mask).enumerate() { @@ -147,7 +136,7 @@ pub(crate) fn rasterize( let [x, y] = [index % side as usize, index / side as usize] .map(|device| device as f32 + 0.5) .map(|device| device * 16.0 / size); - let color = color([x - offset[0] * 16.0 / size, y - offset[1] * 16.0 / size]); + let color = color([x, y]); for channel in 0..3 { pixel[channel] = color[channel] * alpha + pixel[channel] * (1.0 - alpha); } @@ -468,7 +457,7 @@ mod tests { /// The grey painted at a pixel of `source` rasterized at 16 pixels, and the grey a /// black-to-white ramp paints at fraction `t`. fn grey(source: &str, x: usize, y: usize) -> u8 { - rasterize(&[source], 16.0, [0, 0], [1.0; 3], &Palette::default()).data[(y * 17 + x) * 4] + rasterize(&[source], 16, [1.0; 3], &Palette::default()).data[(y * 16 + x) * 4] } fn ramp(t: f32) -> u8 { @@ -516,8 +505,8 @@ mod tests { // Two blues of one hue in the accent slot, a red outside it. const ART: &str = r##""##; let pixel = |palette: &Palette, y: usize| { - let image = rasterize(&[ART], 16.0, [0, 0], [0.0; 3], palette); - let at = (y * 17 + 8) * 4; + let image = rasterize(&[ART], 16, [0.0; 3], palette); + let at = (y * 16 + 8) * 4; let [r, g, b] = [0, 1, 2].map(|channel| f32::from(image.data[at + channel]) / 255.0); hsl([r, g, b]) }; @@ -542,14 +531,8 @@ mod tests { fn current_color_paints_the_ink_and_fixed_colours_keep_theirs() { const LINE: &str = r##""##; let ink = super::super::srgb(0x20, 0x60, 0xa0); - let image = rasterize( - &[LINE], - 16.0, - [0, 0], - [ink[0], ink[1], ink[2]], - &Palette::default(), - ); - let pixel = |x: usize, y: usize| &image.data[(y * 17 + x) * 4..][..4]; + let image = rasterize(&[LINE], 16, [ink[0], ink[1], ink[2]], &Palette::default()); + let pixel = |x: usize, y: usize| &image.data[(y * 16 + x) * 4..][..4]; assert_eq!(pixel(8, 7), [0x20, 0x60, 0xa0, 255]); assert_eq!(pixel(8, 11), [255, 0, 0, 255]); assert_eq!(pixel(8, 2)[3], 0); @@ -560,8 +543,8 @@ mod tests { #[test] fn opacity_scales_each_paint_on_its_own() { const HALF: &str = r##""##; - let image = rasterize(&[HALF], 16.0, [0, 0], [1.0; 3], &Palette::default()); - let alpha = |x: usize, y: usize| image.data[(y * 17 + x) * 4 + 3]; + let image = rasterize(&[HALF], 16, [1.0; 3], &Palette::default()); + let alpha = |x: usize, y: usize| image.data[(y * 16 + x) * 4 + 3]; assert_eq!(alpha(8, 8), 128); // The stroke's outer half covers only what the fill leaves. assert_eq!(alpha(1, 8), 64); @@ -569,28 +552,26 @@ mod tests { #[test] fn icons_keep_transparency_and_overlays_across_scales() { - for size in [8.0, 16.0, 32.0, 57.5] { - for phase in [[0, 0], [1, 2], [3, 3]] { - let plain = rasterize(&[BOX], size, phase, [1.0; 3], &Palette::default()); - let marked = rasterize(&[BOX, MARK], size, phase, [1.0; 3], &Palette::default()); - for image in [&plain, &marked] { - assert_eq!( - image.data.len(), - (image.placement.width * image.placement.height * 4) as usize - ); - assert!(image.data.chunks_exact(4).any(|p| p[3] > 0)); - assert!(image.data.chunks_exact(4).any(|p| p[3] < 255)); - assert!( - image - .data - .chunks_exact(4) - .filter(|p| p[3] == 0) - .all(|p| p[..3] == [0; 3]) - ); - } - assert_ne!(plain.data, marked.data); - assert!(marked.data.chunks_exact(4).any(|p| p[0] > p[2] && p[3] > 0)); + for size in [8, 16, 32, 57] { + let plain = rasterize(&[BOX], size, [1.0; 3], &Palette::default()); + let marked = rasterize(&[BOX, MARK], size, [1.0; 3], &Palette::default()); + for image in [&plain, &marked] { + assert_eq!( + image.data.len(), + (image.placement.width * image.placement.height * 4) as usize + ); + assert!(image.data.chunks_exact(4).any(|p| p[3] > 0)); + assert!(image.data.chunks_exact(4).any(|p| p[3] < 255)); + assert!( + image + .data + .chunks_exact(4) + .filter(|p| p[3] == 0) + .all(|p| p[..3] == [0; 3]) + ); } + assert_ne!(plain.data, marked.data); + assert!(marked.data.chunks_exact(4).any(|p| p[0] > p[2] && p[3] > 0)); } } } diff --git a/crates/mobile/src/library.rs b/crates/mobile/src/library.rs index e38232478794070acad6ca71e0db2c6be1a8df50..c641d7c063f7e1f2e791cf4aba6e465eddbd10b4 100644 --- a/crates/mobile/src/library.rs +++ b/crates/mobile/src/library.rs @@ -33,8 +33,6 @@ const TIMEOUT: Duration = Duration::from_secs(10); const LIMIT: usize = 256 * 1024 * 1024; /// The tab colour OneNote gives a section that stores none, as the desktop shows it. const SECTION_COLOR: u32 = 0x00e4_a88a; -/// How often sections no page shows are polled, as the desktop polls them. -const BACKGROUND: Duration = Duration::from_secs(15); /// Called on a background thread when a notebook's sync status may have changed. static SYNC_WAKE: OnceLock = OnceLock::new(); @@ -239,7 +237,7 @@ fn stem(path: &str) -> String { fn tabs(folder: &Folder, tabs: &mut Vec) { for section in &folder.sections { let (name, color, readable) = match §ion.state { - SectionState::Readable { name, color } => (name.clone(), *color, true), + SectionState::Readable { name, color, .. } => (name.clone(), *color, true), SectionState::Locked | SectionState::Unreadable(_) => (None, None, false), }; tabs.push(Tab { @@ -272,8 +270,10 @@ impl Library { (None, Place::File(path.to_owned()), None) } else { let notebook = Notebook::open(path, cache)?; + // Nothing here tells a file provider's folder on this device from one elsewhere. let background = notebook.background_with( - BACKGROUND, + Background::UNWATCHED, + false, |file| Coordinated(file.to_owned()), sync_woken, )?; @@ -300,13 +300,7 @@ impl Library { pub(crate) fn server(server: Server, root: &str, cache: &Path) -> Result { let server = Arc::new(server); let connect = Arc::clone(&server); - let background = Background::smb( - root, - LIMIT, - BACKGROUND, - move || connect.connect(), - sync_woken, - )?; + let background = Background::smb(root, LIMIT, move || connect.connect(), sync_woken)?; let library = Self { notebook: Mutex::default(), place: Place::Share { diff --git a/crates/notebook/README.md b/crates/notebook/README.md index 61ba64a1660588da42bd6f3846b4771a472355b0..0b03d4ebee38b91c5fdc5230984ca04c63218230 100644 --- a/crates/notebook/README.md +++ b/crates/notebook/README.md @@ -65,14 +65,21 @@ and how many edits wait for it. `set_offline(true)` works offline as OneNote doe worker stops connecting and edits queue until `wake()` (Sync Now) or `set_offline(false)`. `session::Background` keeps the sections no session holds in sync, as OneNote 2010 keeps -every section of an open notebook: `Notebook::background(interval, notify)` for a mounted -notebook, `Background::smb(root, limit, interval, connect, notify)` on a share, then -`watch(notebook.replicas())`. Each round reads every watched file's stamp; a section whose -replica has edits waiting, or whose file moved past the replica's base, has the replica -opened for the synchronization steps that publish or rebase it and closed again. A replica -a session holds is skipped (`Error::busy`), so opening a section may wait out one step. -`status()` gives each section's `SyncStatus`, `changed()` the sections another client -changed, and `wake` and `set_offline` follow Sync Now and Work Offline. +every section of an open notebook: `Background::smb(root, limit, connect, notify)` on a +share, `Notebook::background(interval, copies, notify)` for a mounted notebook, then +`watch(notebook.replicas())`. On a share, one CHANGE_NOTIFY on the notebook's folder +(`smb::Client::watch`) reports what changed; a host watching a mounted folder passes the +changed paths to `touched`. A reported section is checked a second later, a failing one +every 31 seconds, and any other every `interval` (`Background::BACKSTOP` while a watch +reports). Newly watched sections, and every section after a reconnect, are checked one at a +time 100 ms apart. A check reads the file's stamp; a section whose replica has edits waiting, +or whose file moved past the replica's base, has the replica opened for the synchronization +steps that publish or rebase it and closed again, and a section without a replica gets one +from the file, its offline copy, on a share or with `copies`. A replica a session holds is +skipped (`Error::busy`), so opening a section may wait out one step. `status()` gives each +section's `SyncStatus`, `changed()` the sections another client changed, `wake` and +`set_offline` follow Sync Now and Work Offline, and `discard` stops the thread and deletes the +replicas holding nothing unpublished, as when the notebook closes. `Notebook::replica_path(path)` names a section's replica for either kind of notebook. `Section::resume(file, replica, notify)` starts from an owned `Replica` without @@ -373,6 +380,12 @@ are rejected with `ResourceBusy`. Notebook identities come from the files, not directory names or sizes. Missing paths, denied access and non-directory paths have distinct I/O error kinds. +`Client::watch(path, changed)` arms one CHANGE_NOTIFY with WATCH_TREE on a directory, as +OneNote 2010 watches a notebook's folder, and keeps it armed on the client's runtime without +probing the connection, so an idle watch sends nothing. `changed` hears each batch of changed +paths relative to the directory, `""` when the server lost count, and an error when the watch +ends with its connection. + `Client::read_asset(path, byte_limit)` reads an external payload under a read-only share handle that excludes writes and deletion. Empty files succeed; limits, interrupted reads and failed close never return partial bytes. This payload read diff --git a/crates/notebook/examples/launch_probe.rs b/crates/notebook/examples/launch_probe.rs new file mode 100644 index 0000000000000000000000000000000000000000..6fea1a6b45e5e4e86f3a835f7b864d53452733e7 --- /dev/null +++ b/crates/notebook/examples/launch_probe.rs @@ -0,0 +1,228 @@ +//! Temporary: measures a notebook's launch (catalog and background first pass). +//! `generate DIR N` | `local DIR CACHE` | `upload ADDRESS SHARE DIR ROOT` | +//! `smb ADDRESS SHARE ROOT CACHE [SECONDS]` + +use notebook::{ + discover::{Entry, Limits, Local, Source, discover}, + session::{Background, Notebook}, +}; +use onestore::{ + document::{Format, Layout}, + op::{Edit, Op, SectionOp}, + page::{ + Outline, Page, PageObject, PageParagraph, Paragraph, ParagraphContent, TextObject, + text::new_id, + }, +}; +use std::{ + io, + path::Path, + time::{Duration, Instant}, +}; + +type Result = std::result::Result>; + +struct Counting(S, u64, u64); + +impl Source for Counting { + fn entries(&mut self, path: &str, limit: usize) -> io::Result> { + self.2 += 1; + self.0.entries(path, limit) + } + fn read(&mut self, path: &str, limit: usize) -> io::Result> { + let bytes = self.0.read(path, limit)?; + self.1 += bytes.len() as u64; + Ok(bytes) + } +} + +const LIMITS: Limits = Limits { + entries: 100_000, + bytes_per_file: 256 * 1024 * 1024, + depth: 64, +}; + +fn page(count: usize) -> Result { + let paragraph = |i: usize| PageParagraph { + id: new_id().unwrap(), + parent: None, + level: 1, + style: None, + format: Format::default(), + content: ParagraphContent::Text(TextObject { + id: new_id().unwrap(), + date_field: None, + text: Paragraph::new( + format!("Paragraph {i} of the launch probe, with some words."), + Format::default(), + ), + tags: Vec::new(), + }), + lists: Vec::new(), + tags: Vec::new(), + media: Default::default(), + collapsed: false, + }; + Ok(Page { + title: "Probe".into(), + identity: None, + created: None, + margin_origin: [0.0; 2], + color: None, + rule_lines: None, + objects: vec![PageObject::Outline(Outline { + id: new_id()?, + title: false, + min_width: None, + layout: Layout { + x: Some(36.0), + y: Some(86.4), + ..Layout::default() + }, + indents: Vec::new(), + paragraphs: (0..count).map(paragraph).collect(), + unsupported: Vec::new(), + })], + definitions: Default::default(), + }) +} + +fn generate(root: &Path, count: usize) -> Result<()> { + let cache = tempfile::tempdir()?; + let creation = onestore::PageCreation::new(None, Some("First"), "Probe")?; + let mut notebook = Notebook::create(root, cache.path(), 0x00e4a88a, &creation)?; + notebook.create_group("", "Group A")?; + notebook.create_group("", "Group B")?; + for n in 0..count { + let folder = ["", "Group A", "Group B"][n % 3]; + let path = notebook.create_section(folder, &format!("Section {n:03}"), &creation)?; + let file = root.join(&path); + let arena = onestore::Arena::default(); + let mut section = onestore::Section::open(&arena, std::fs::read(&file)?)?; + let creation = onestore::PageCreation::new(None, Some("Probe"), "Probe")?; + section.apply( + "Probe", + &Edit { + at: 133_000_000_000_000_000, + ops: vec![Op::Section(SectionOp::Import { + creation, + page: page(1200)?, + })], + }, + )?; + let transaction = section.seal()?.ok_or("nothing sealed")?; + transaction.commit_file(&file)?; + } + Ok(()) +} + +fn size(root: &Path) -> u64 { + std::fs::read_dir(root) + .into_iter() + .flatten() + .flatten() + .map(|entry| match entry.file_type() { + Ok(kind) if kind.is_dir() => size(&entry.path()), + _ => entry.metadata().map_or(0, |metadata| metadata.len()), + }) + .sum() +} + +#[cfg(feature = "smb")] +fn client(address: &str, share: &str) -> Result { + Ok(notebook::smb::Client::connect( + address, + share, + notebook::smb::Credentials { + username: "", + password: "", + domain: "", + }, + Duration::from_secs(10), + )?) +} + +#[cfg(feature = "smb")] +fn upload(client: ¬ebook::smb::Client, local: &Path, remote: &str) -> Result<()> { + client.create_directory(remote)?; + for entry in std::fs::read_dir(local)? { + let entry = entry?; + let name = entry.file_name().into_string().map_err(|_| "name")?; + let target = format!("{remote}/{name}"); + if entry.file_type()?.is_dir() { + upload(client, &entry.path(), &target)?; + } else { + client.create(&target, &std::fs::read(entry.path())?)?; + } + } + Ok(()) +} + +fn main() -> Result<()> { + let args: Vec = std::env::args().skip(1).collect(); + match args.iter().map(String::as_str).collect::>().as_slice() { + ["generate", root, count] => { + generate(Path::new(root), count.parse()?)?; + println!("{} bytes", size(Path::new(root))); + } + ["local", root, cache] => { + let start = Instant::now(); + let notebook = Notebook::open(root, cache)?; + let replicas = notebook.replicas(); + println!( + "open: {:?}, {} sections", + start.elapsed(), + replicas.len() + ); + let mut counting = Counting(Local::open(root)?, 0, 0); + discover(&mut counting, LIMITS)?; + println!("discovery reads {} bytes, {} listings", counting.1, counting.2); + } + #[cfg(feature = "smb")] + ["upload", address, share, local, root] => { + upload(&client(address, share)?, Path::new(local), root)?; + } + #[cfg(feature = "smb")] + ["smb", address, share, root, cache, rest @ ..] => { + let seconds: u64 = rest.first().map_or(Ok(0), |s| s.parse())?; + let start = Instant::now(); + let connected = client(address, share)?; + let notebook = Notebook::open_smb(std::sync::Arc::new(connected), root, cache)?; + let replicas = notebook.replicas(); + println!( + "open: {:?}, {} sections", + start.elapsed(), + replicas.len() + ); + if seconds > 0 { + let (address, share) = (address.to_string(), share.to_string()); + let background = Background::smb( + root, + 256 * 1024 * 1024, + move || { + client(&address, &share) + .map_err(|error| io::Error::other(error.to_string())) + }, + || {}, + )?; + background.watch(replicas); + let start = Instant::now(); + loop { + let status = background.status(); + if status.iter().all(|(_, status)| status.synced.is_some()) { + println!("background first pass: {:?}", start.elapsed()); + break; + } + if start.elapsed() > Duration::from_secs(seconds) { + println!("background still going after {seconds} s"); + break; + } + std::thread::sleep(Duration::from_millis(50)); + } + std::thread::sleep(Duration::from_secs(2)); + } + } + _ => return Err("usage".into()), + } + Ok(()) +} diff --git a/crates/notebook/src/background.rs b/crates/notebook/src/background.rs index 1c327727876c1099208f13396682bca462be5fa8..db8647f83115bb7fcfdff5bd4b3908fd4c653830 100644 --- a/crates/notebook/src/background.rs +++ b/crates/notebook/src/background.rs @@ -11,19 +11,45 @@ use onestore::Stamp; use std::{ io, path::{Path, PathBuf}, - sync::{Arc, Mutex, atomic::Ordering}, + sync::{ + Arc, Mutex, Weak, + atomic::{AtomicBool, Ordering}, + }, thread, - time::Duration, + time::{Duration, Instant}, }; -/// Polls each watched section file's stamp every interval. A section whose replica has -/// edits waiting, or whose file moved past the replica's base, has its replica opened for -/// the synchronization steps that publish or rebase it, then closed again; any other costs -/// one stamp read. A replica a session holds is left to that session's worker. Dropping -/// requests cancellation without waiting for the step in flight. -pub struct Background { +/// How long a section that could not be reached waits to be tried again; OneNote 2010 +/// retries a failing section about every 31 seconds. +const RETRY: Duration = Duration::from_secs(31); +/// Between one section's first check and the next's, so opening a notebook reads its files +/// one after another rather than in a burst. +const STAGGER: Duration = Duration::from_millis(100); +/// How long a reported change settles before its section is checked, so the writes of one +/// commit cost one check. +const SETTLE: Duration = Duration::from_secs(1); + +/// Keeps each watched section in sync: a section is checked when a watch on the notebook's +/// folder reports it changed, when it could not be reached, and otherwise every interval. +/// A check costs one stamp read; a section whose replica has edits waiting, or whose file +/// moved past the replica's base, has its replica opened for the synchronization steps that +/// publish or rebase it, then closed again. A section without a replica gets one, its offline +/// copy, where the notebook keeps them. A replica a session holds is left to that session's +/// worker. Dropping requests cancellation without waiting for the step in flight. +pub struct Background(Arc); + +/// What the background thread and whatever reports changes to it share. +pub(crate) struct Shared { signal: Arc, - watched: Arc>, + watched: Mutex, + /// Asked for by `discard`, once the thread stops. + discard: AtomicBool, +} + +/// A connection's report of its notebook folder's changes, from a watch armed on connecting. +pub(crate) struct Reports { + shared: Weak, + connection: u64, } #[derive(Default)] @@ -31,6 +57,10 @@ struct Watched { sections: Vec, /// Sections whose file changed since `changed` was last asked. changed: Vec, + /// Counts connections, so a watch that ended with an earlier one is not this one's. + connection: u64, + /// The current connection's watch ended: changes since went unreported. + lost: bool, } struct Watch { @@ -39,12 +69,24 @@ struct Watch { /// The file's stamp when last reached. stamp: Option, status: SyncStatus, + /// When the section is next checked. + due: Instant, } impl Background { + /// How often each section is checked while a watch reports its folder's changes, against + /// a report that went missing; OneNote 2010, watching, checks nothing on a timer. + pub const BACKSTOP: Duration = Duration::from_secs(60 * 60); + /// How often each section is checked where nothing reports its folder's changes. + pub const UNWATCHED: Duration = Duration::from_secs(15); + + /// Starts the thread. `connect` binds each catalog path to its file, arming any watch + /// that reports the folder's changes through `Reports`, and answers how long a section + /// may go unchecked while nothing reports it; it runs again after a transport failure. + /// `copies` keeps an offline copy of every section. pub(crate) fn start( - interval: Duration, - mut connect: impl FnMut() -> io::Result + Send + 'static, + copies: bool, + mut connect: impl FnMut(Reports) -> io::Result<(B, Duration)> + Send + 'static, notify: impl Fn() + Send + 'static, ) -> Result where @@ -52,165 +94,201 @@ impl Background { B: FnMut(&str) -> R, { let (signal, receiver) = Signal::new(); - let watched = Arc::new(Mutex::new(Watched::default())); - let (shared, sections) = (Arc::clone(&signal), Arc::clone(&watched)); + let shared = Arc::new(Shared { + signal, + watched: Mutex::default(), + discard: AtomicBool::new(false), + }); + let weak = Arc::downgrade(&shared); + let owner = Arc::clone(&shared); thread::Builder::new() .name("onestore-background".into()) .spawn(move || { - let mut bound: Option = None; - while !shared.stopped.load(Ordering::Acquire) { - if shared.offline.load(Ordering::Acquire) - && !shared.requested.load(Ordering::Acquire) + let signal = &owner.signal; + let mut bound: Option<(B, Duration)> = None; + let mut news = false; + while !signal.stopped.load(Ordering::Acquire) { + if signal.offline.load(Ordering::Acquire) + && !signal.requested.load(Ordering::Acquire) { bound = None; let _ = receiver.recv(); continue; } - shared.requested.store(false, Ordering::Release); - let round: Vec<_> = match sections.lock() { - Ok(watched) => watched - .sections - .iter() - .map(|watch| { - ( - watch.path.clone(), - watch.replica.clone(), - watch.stamp.clone(), - ) - }) - .collect(), - Err(_) => return, - }; - let mut news = false; - for (path, replica, seen) in round { - if shared.stopped.load(Ordering::Acquire) { + let now = Instant::now(); + let (path, replica, seen, connection) = { + let Ok(mut watched) = owner.watched.lock() else { return; + }; + if std::mem::take(&mut watched.lost) { + bound = None; + watched.rescan(now); } - let bind = match &mut bound { - Some(bind) => bind, - None => match connect() { - Ok(bind) => bound.insert(bind), + let due = watched.next().map(|index| &watched.sections[index]); + match due.filter(|watch| watch.due <= now) { + Some(watch) => ( + watch.path.clone(), + watch.replica.clone(), + watch.stamp.clone(), + watched.connection, + ), + None => { + let wait = due.map(|watch| watch.due - now); + drop(watched); + signal.requested.store(false, Ordering::Release); + if std::mem::take(&mut news) { + notify(); + } + let _ = match wait { + Some(wait) => receiver.recv_timeout(wait).ok(), + None => receiver.recv().ok(), + }; + continue; + } + } + }; + let (bind, interval) = match &mut bound { + Some(bound) => bound, + None => { + let reports = Reports { + shared: weak.clone(), + connection: connection + 1, + }; + let connected = connect(reports); + let Ok(mut watched) = owner.watched.lock() else { + return; + }; + // Whatever ended before now ended with the last connection. + watched.connection += 1; + watched.lost = false; + match connected { + // Changes while unwatched went unreported. + Ok(connected) => { + watched.rescan(now); + bound = Some(connected); + } Err(error) => { - // Nothing on the share is reachable this round. - let Ok(mut watched) = sections.lock() else { - return; - }; + let error = Error::RemoteIo(error); for watch in &mut watched.sections { - news |= watch.fail( - &Error::RemoteIo(io::Error::new( - error.kind(), - error.to_string(), - )), - None, - ); + news |= watch.fail(&error, None); + watch.due = now + RETRY; } - break; } - }, - }; - let (queued, outcome) = - step(&mut bind(&path), replica.as_deref(), seen.as_ref()); - if matches!(outcome, Err(Error::RemoteIo(_) | Error::Remote(_))) { - bound = None; - } - let Ok(mut watched) = sections.lock() else { - return; - }; - let Watched { sections, changed } = &mut *watched; - let Some(watch) = sections.iter_mut().find(|watch| watch.path == path) - else { + } continue; - }; - news |= match outcome { - Ok((stamp, moved)) => { - watch.stamp = Some(stamp); - if moved { - changed.push(path); - } - let before = summary(&watch.status); - if let Some(queued) = queued { - watch.status = SyncStatus { - synced: Some(crate::now()), - error: None, - queued, - }; - } - moved || before != summary(&watch.status) - } - Err(error) => watch.fail(&error, queued), - }; + } + }; + let (queued, outcome) = + step(&mut bind(&path), replica.as_deref(), seen.as_ref(), copies); + let interval = *interval; + if outcome.as_ref().is_err_and(disconnected) { + bound = None; } - if news { - notify(); + let Ok(mut watched) = owner.watched.lock() else { + return; + }; + let Watched { + sections, changed, .. + } = &mut *watched; + let Some(watch) = sections.iter_mut().find(|watch| watch.path == path) else { + continue; + }; + let now = Instant::now(); + news |= match outcome { + Ok((stamp, moved)) => { + watch.stamp = Some(stamp); + watch.due = now + interval; + if moved { + changed.push(path); + } + let before = summary(&watch.status); + if let Some(queued) = queued { + watch.status = SyncStatus { + synced: Some(crate::now()), + error: None, + queued, + }; + } + moved || before != summary(&watch.status) + } + Err(error) => { + watch.due = now + RETRY; + watch.fail(&error, queued) + } + }; + } + if owner.discard.load(Ordering::Acquire) + && let Ok(watched) = owner.watched.lock() + { + for replica in watched + .sections + .iter() + .filter_map(|watch| watch.replica.as_ref()) + { + discard(replica); } - let _ = receiver.recv_timeout(interval); } })?; - Ok(Self { signal, watched }) + Ok(Self(shared)) } - /// Keeps the sections of a notebook on a share in sync while they are not open; - /// `connect` runs again after a transport failure. Paths are relative to `root`. + /// Keeps the sections of a notebook on a share in sync while they are not open, with an + /// offline copy of each, and one watch on the notebook's folder reporting what changed, + /// as OneNote 2010 watches it; `connect` runs again after a transport failure. Paths are + /// relative to `root`. #[cfg(feature = "smb")] pub fn smb( root: &str, limit: usize, - interval: Duration, mut connect: impl FnMut() -> io::Result + Send + 'static, notify: impl Fn() + Send + 'static, ) -> Result { let root = root.replace('\\', "/"); Self::start( - interval, - move || { + true, + move |reports| { let client = Arc::new(connect()?); + let interval = match client.watch(&root, move |changed| match changed { + Ok(paths) => reports.touched(&paths), + Err(_) => reports.lost(), + }) { + Ok(()) => Self::BACKSTOP, + Err(error) if error.kind() == io::ErrorKind::Unsupported => Self::UNWATCHED, + Err(error) => return Err(error), + }; let root = root.clone(); - Ok(move |path: &str| { + let bind = move |path: &str| { let file = match root.as_str() { "" => path.to_owned(), root => format!("{root}/{path}"), }; crate::SmbRemote::new(Arc::clone(&client), file, limit) - }) + }; + Ok((bind, interval)) }, notify, ) } - /// Watches these sections, by catalog path and replica (`Notebook::replicas`), from - /// the next round, which starts now. + /// Watches these sections, by catalog path and replica (`Notebook::replicas`); a section + /// not watched before is first checked soon after, the next one a little later. pub fn watch(&self, sections: Vec<(String, Option)>) { - if let Ok(mut watched) = self.watched.lock() { - let mut previous = std::mem::take(&mut watched.sections); - watched.sections = sections - .into_iter() - .map( - |(path, replica)| match previous.iter().position(|watch| watch.path == path) { - Some(index) => Watch { - replica, - ..previous.swap_remove(index) - }, - None => Watch { - path, - replica, - stamp: None, - status: SyncStatus { - synced: None, - error: None, - queued: 0, - }, - }, - }, - ) - .collect(); + if let Ok(mut watched) = self.0.watched.lock() { + watched.watch(sections, Instant::now()); } - self.signal.wake(); + self.0.signal.wake(); } - /// Each watched section's status as its last round left it, in watch order. A section a + /// Checks the sections at or below these catalog paths soon, as a watch on the + /// notebook's folder reports them changed; `""` names every section. + pub fn touched(&self, paths: &[String]) { + self.0.touched(paths); + } + + /// Each watched section's status as its last check left it, in watch order. A section a /// session holds keeps the status it had before. pub fn status(&self) -> Vec<(String, SyncStatus)> { - self.watched.lock().map_or_else( + self.0.watched.lock().map_or_else( |_| Vec::new(), |watched| { watched @@ -237,32 +315,165 @@ impl Background { /// Sections whose file changed since the last call, by catalog path. pub fn changed(&self) -> Vec { - self.watched + self.0 + .watched .lock() .map(|mut watched| std::mem::take(&mut watched.changed)) .unwrap_or_default() } - /// Runs a round now, working offline included (Sync Now). + /// Checks every section now, working offline included (Sync Now). pub fn wake(&self) { - self.signal.requested.store(true, Ordering::Release); - self.signal.wake(); + if let Ok(mut watched) = self.0.watched.lock() { + let now = Instant::now(); + for watch in &mut watched.sections { + watch.due = now; + } + } + self.0.signal.requested.store(true, Ordering::Release); + self.0.signal.wake(); } - /// Working offline, no round runs until `wake`, or until working online again. + /// Stops for good, then deletes each section's replica that holds nothing unpublished, as + /// OneNote lets go of a notebook it closes; one with edits waiting, or that a session + /// holds, stays. + pub fn discard(&self) { + self.0.discard.store(true, Ordering::Release); + self.0.signal.stopped.store(true, Ordering::Release); + self.0.signal.wake(); + } + + /// Working offline, nothing is checked until `wake`, or until working online again. pub fn set_offline(&self, offline: bool) { - self.signal.offline.store(offline, Ordering::Release); - self.signal.wake(); + self.0.signal.offline.store(offline, Ordering::Release); + self.0.signal.wake(); } } impl Drop for Background { fn drop(&mut self) { - self.signal.stopped.store(true, Ordering::Release); + self.0.signal.stopped.store(true, Ordering::Release); + self.0.signal.wake(); + } +} + +impl Shared { + fn touched(&self, paths: &[String]) { + if let Ok(mut watched) = self.watched.lock() { + watched.touched(paths, Instant::now()); + } self.signal.wake(); } } +impl Reports { + /// The sections at or below these paths changed. + pub(crate) fn touched(&self, paths: &[String]) { + if let Some(shared) = self.shared.upgrade() { + shared.touched(paths); + } + } + + /// The watch ended, and with it, as far as anyone can tell, the connection. + pub(crate) fn lost(&self) { + if let Some(shared) = self.shared.upgrade() { + if let Ok(mut watched) = shared.watched.lock() + && watched.connection == self.connection + { + watched.lost = true; + } + shared.signal.wake(); + } + } +} + +impl Watched { + fn watch(&mut self, sections: Vec<(String, Option)>, now: Instant) { + let mut previous = std::mem::take(&mut self.sections); + let mut new = 0; + self.sections = sections + .into_iter() + .map( + |(path, replica)| match previous.iter().position(|watch| watch.path == path) { + Some(index) => Watch { + replica, + ..previous.swap_remove(index) + }, + None => { + new += 1; + Watch { + path, + replica, + stamp: None, + status: SyncStatus { + synced: None, + error: None, + queued: 0, + }, + due: now + STAGGER * (new - 1), + } + } + }, + ) + .collect(); + } + + /// Checks every section from `now`, one `STAGGER` after another. + fn rescan(&mut self, now: Instant) { + for (index, watch) in (0..).zip(&mut self.sections) { + watch.due = now + STAGGER * index; + } + } + + fn touched(&mut self, paths: &[String], now: Instant) { + for watch in &mut self.sections { + if paths.iter().any(|path| within(&watch.path, path)) { + watch.due = watch.due.min(now + SETTLE); + } + } + } + + /// The section checked next. + fn next(&self) -> Option { + (0..self.sections.len()).min_by_key(|&index| self.sections[index].due) + } +} + +/// Deletes the replica at `replica` if it holds nothing unpublished and no one holds it. +fn discard(replica: &Path) { + if matches!(crate::peek(replica), Ok((_, 0))) { + for suffix in ["-wal", "-shm", ""] { + let mut file = replica.as_os_str().to_owned(); + file.push(suffix); + let _ = std::fs::remove_file(file); + } + } +} + +/// Whether the catalog path `section` is `path` or lies below it, as a share compares names. +fn within(section: &str, path: &str) -> bool { + let path = path.trim_matches('/'); + path.is_empty() + || section.len() >= path.len() + && section.is_char_boundary(path.len()) + && section[..path.len()].eq_ignore_ascii_case(path) + && matches!(section.as_bytes().get(path.len()), None | Some(b'/')) +} + +/// Whether a failed step lost the connection, rather than failing for its one file. +fn disconnected(error: &Error) -> bool { + use io::ErrorKind::*; + let kind = match error { + Error::RemoteIo(error) => error.kind(), + Error::Remote(error) => error.error.kind(), + _ => return false, + }; + matches!( + kind, + NotConnected | TimedOut | ConnectionReset | ConnectionAborted | BrokenPipe + ) +} + impl Watch { /// Records a failed step and what it found waiting, answering whether the status shown /// changes. @@ -284,20 +495,43 @@ fn summary(status: &SyncStatus) -> (bool, Option, u64) { } /// One section's step: how many of its edits wait (`None` while unknown, as while a session -/// holds its replica), then the file's stamp now and whether it changed since `seen`. +/// holds its replica), then the file's stamp now and whether it changed since `seen`. With +/// `copies`, a section without a replica gets one from the file as it is now. fn step( remote: &mut R, replica: Option<&Path>, seen: Option<&Stamp>, + copies: bool, ) -> (Option, Result<(Stamp, bool)>) { let stamp = match remote.stamp() { Ok(stamp) => stamp, Err(error) => return (None, Err(Error::RemoteIo(error))), }; let moved = seen.is_some_and(|seen| *seen != stamp); - let Some(replica) = replica.filter(|replica| replica.exists()) else { + let Some(replica) = replica else { return (Some(0), Ok((stamp, moved))); }; + if !replica.exists() { + if !copies { + return (Some(0), Ok((stamp, moved))); + } + let copied = (|| { + let image = remote.read().map_err(Error::RemoteIo)?; + if let Some(folder) = replica.parent() { + std::fs::create_dir_all(folder)?; + } + Replica::seed(replica, &image)?; + Ok(Stamp::of(&image)?) + })(); + return match copied { + Ok(stamp) => (Some(0), Ok((stamp, moved))), + // A session made it first. + Err(Error::Io(error)) if error.kind() == io::ErrorKind::AlreadyExists => { + (None, Ok((stamp, false))) + } + Err(error) => (None, Err(error)), + }; + } match crate::peek(replica) { Ok((base, 0)) if base == stamp => return (Some(0), Ok((stamp, moved))), Err(error) if error.busy() => return (None, Ok((stamp, false))), @@ -324,3 +558,176 @@ fn step( .map(|summary| summary.queued_edits); (queued, synced) } + +#[cfg(test)] +mod tests { + use super::*; + const BACKSTOP: Duration = Background::BACKSTOP; + use onestore::{CommitError, Transaction}; + use std::{ + collections::HashMap, + sync::mpsc::{self, Receiver}, + }; + + fn sections(count: usize) -> Vec<(String, Option)> { + (0..count) + .map(|n| { + let folder = if n % 2 == 0 { "" } else { "Group/" }; + (format!("{folder}Section {n:03}.one"), None) + }) + .collect() + } + + /// Runs the schedule as the thread would with every check succeeding, up to `end`, + /// answering which sections were checked when. + fn run(watched: &mut Watched, end: Instant, interval: Duration) -> Vec<(Instant, String)> { + let mut checks = Vec::new(); + while let Some(index) = watched.next() { + let watch = &mut watched.sections[index]; + if watch.due > end { + break; + } + checks.push((watch.due, watch.path.clone())); + watch.due += interval; + } + checks + } + + #[test] + fn an_idle_notebook_is_read_once_staggered_then_left_alone() { + let start = Instant::now(); + let mut watched = Watched::default(); + watched.watch(sections(200), start); + let checks = run(&mut watched, start + BACKSTOP - SETTLE, BACKSTOP); + assert_eq!(checks.len(), 200, "one first check each, then none"); + for pair in checks.windows(2) { + assert!(pair[1].0 - pair[0].0 >= STAGGER, "never a burst"); + } + // Then each section costs one backstop check an interval: over a day, 24 stamp reads + // where a 15-second poll made 5,760. + let day = run( + &mut watched, + start + Duration::from_secs(86_400) - SETTLE, + BACKSTOP, + ); + assert_eq!(day.len(), 200 * (86_400 / BACKSTOP.as_secs() as usize - 1)); + } + + #[test] + fn a_reported_change_checks_exactly_the_sections_it_names() { + let start = Instant::now(); + let mut watched = Watched::default(); + watched.watch(sections(200), start); + let now = start + Duration::from_secs(60); + run(&mut watched, now, BACKSTOP); + // A commit's several writes are reported apiece, and settle into one check. + for _ in 0..3 { + watched.touched(&["section 004.ONE".into()], now); + } + watched.touched( + &["Section 004.one".into()], + now + Duration::from_millis(300), + ); + let checks = run(&mut watched, now + Duration::from_secs(60), BACKSTOP); + assert_eq!(checks, vec![(now + SETTLE, "Section 004.one".to_owned())]); + // A group's folder names the sections in it, and "" every section. + let now = now + Duration::from_secs(60); + watched.touched(&["Group".into()], now); + let group = run(&mut watched, now + SETTLE, BACKSTOP); + assert_eq!(group.len(), 100); + assert!(group.iter().all(|(_, path)| path.starts_with("Group/"))); + watched.touched(&["Grou".into(), String::new()], now); + assert_eq!(run(&mut watched, now + SETTLE, BACKSTOP).len(), 200); + } + + /// Section files in memory, each stamp read reported as it happens. + #[derive(Clone)] + struct Files { + images: Arc>>>, + stamps: mpsc::Sender, + } + + struct File(Files, String); + + impl Remote for File { + fn read(&mut self) -> io::Result> { + let images = self.0.images.lock().unwrap(); + Ok(images[&self.1].clone()) + } + + fn stamp(&mut self) -> io::Result { + let _ = self.0.stamps.send(self.1.clone()); + Stamp::of(&self.read()?).map_err(io::Error::other) + } + + fn publish(&mut self, _: &Transaction) -> std::result::Result<(), CommitError> { + unreachable!("nothing is queued") + } + + fn confirm(&mut self, _: &Stamp) -> std::result::Result<(), CommitError> { + unreachable!("nothing is queued") + } + } + + fn quiet(stamps: &Receiver, wait: Duration) -> Vec { + let mut read = Vec::new(); + while let Ok(path) = stamps.recv_timeout(wait) { + read.push(path); + } + read + } + + #[test] + fn a_watch_wakes_the_section_it_reports_and_a_lost_one_rescans() { + let (sender, stamps) = mpsc::channel(); + let files = Files { + images: Arc::default(), + stamps: sender, + }; + let mut paths: Vec = sections(6).into_iter().map(|(path, _)| path).collect(); + paths.sort(); + for path in &paths { + let image = onestore::create_section("s.one", path, "Author").unwrap(); + files.images.lock().unwrap().insert(path.clone(), image); + } + let (armed, watches) = mpsc::channel(); + let remote = files.clone(); + let background = Background::start( + false, + move |reports| { + armed.send(reports).unwrap(); + let files = remote.clone(); + Ok(( + move |path: &str| File(files.clone(), path.to_owned()), + Duration::from_secs(3600), + )) + }, + || {}, + ) + .unwrap(); + background.watch(sections(6)); + let mut first = quiet(&stamps, STAGGER * 4); + first.sort(); + assert_eq!(first, paths, "each section's first check"); + let watch = watches.recv().unwrap(); + assert!(quiet(&stamps, SETTLE).is_empty(), "idle reads nothing"); + + let changed = &paths[3]; + files.images.lock().unwrap().insert( + changed.clone(), + onestore::create_section("s.one", "Changed", "Author").unwrap(), + ); + watch.touched(std::slice::from_ref(changed)); + assert_eq!(quiet(&stamps, SETTLE * 2), vec![changed.clone()]); + assert_eq!(background.changed(), vec![changed.clone()]); + + // A watch that ended leaves changes unreported, so everything is read again. + watch.lost(); + let rewatch = watches.recv_timeout(SETTLE).unwrap(); + assert_eq!(quiet(&stamps, STAGGER * 4).len(), 6); + // The ended watch's connection is gone; only the current one's loss counts. + watch.lost(); + assert!(quiet(&stamps, SETTLE).is_empty()); + drop(rewatch); + } +} diff --git a/crates/notebook/src/discover.rs b/crates/notebook/src/discover.rs index 60a4e136f50d9c3ad2abc4dcc3e23a5ab8449756..ea9ca8427a4f8a92ef8a1875f5ffd9f32798df7d 100644 --- a/crates/notebook/src/discover.rs +++ b/crates/notebook/src/discover.rs @@ -30,6 +30,9 @@ pub enum SectionState { name: Option, /// The tab colour as a COLORREF; OneNote assigns one when absent. color: Option, + /// The root object space's GUID, the document identity that names the section's + /// replica in a mounted notebook. + document: [u8; 16], }, Locked, Unreadable(onestore::Error), @@ -310,6 +313,7 @@ fn scan( state: SectionState::Readable { name, color: color.filter(|color| *color != 0xffff_ffff), + document: index.root.guid, }, }); } else { diff --git a/crates/notebook/src/lib.rs b/crates/notebook/src/lib.rs index 288a7e32ffcd8203755d8dff6f0bd7a669c10262..983f04abc300140f32e08b2b79eaa40686051863 100644 --- a/crates/notebook/src/lib.rs +++ b/crates/notebook/src/lib.rs @@ -105,23 +105,28 @@ impl Replica { /// Seeds a new cache from a validated section image, refusing any existing path. /// An initialization error preserves the created file for inspection. pub fn create(path: impl AsRef, source: &[u8]) -> Result { + Self::seed(path.as_ref(), source)?; + Self::start(cache_connection(path.as_ref())?) + } + + /// `create` without opening the cache it made, as for an offline copy. + pub(crate) fn seed(path: &Path, source: &[u8]) -> Result<()> { validate(source)?; - let path = path.as_ref(); - // Built under another name and linked into place whole, so that a cache that exists - // is complete: a background step or a failed creation never leaves one half made. + // Built under a name of its own and linked into place whole, so that a cache that + // exists is complete, and two threads making the same one never share a build. + static BUILDS: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); + let build = BUILDS.fetch_add(1, std::sync::atomic::Ordering::Relaxed); let mut building = path.as_os_str().to_owned(); - building.push(".creating"); + building.push(format!(".creating-{}-{build}", std::process::id())); let building = PathBuf::from(building); - for stale in ["", "-wal", "-shm"] { - let mut file = building.as_os_str().to_owned(); - file.push(stale); - let _ = std::fs::remove_file(file); - } let built = Self::build(&building, source); let linked = built.and_then(|()| Ok(std::fs::hard_link(&building, path)?)); - let _ = std::fs::remove_file(&building); - linked?; - Self::start(cache_connection(path)?) + for suffix in ["", "-wal", "-shm"] { + let mut file = building.as_os_str().to_owned(); + file.push(suffix); + let _ = std::fs::remove_file(file); + } + linked } fn build(path: &Path, source: &[u8]) -> Result<()> { diff --git a/crates/notebook/src/merge.rs b/crates/notebook/src/merge.rs index de83c4e40c61e46fec5d67449a064e81a8192d23..907fa6c694e1cd75804d27ea204ed7e8108557e2 100644 --- a/crates/notebook/src/merge.rs +++ b/crates/notebook/src/merge.rs @@ -164,6 +164,7 @@ fn named(op: &PageOp) -> Vec { PageOp::Level { paragraph, .. } | PageOp::Paragraph { paragraph, .. } | PageOp::Style { paragraph, .. } + | PageOp::Media { paragraph, .. } | PageOp::List { paragraph, .. } => vec![*paragraph], PageOp::Tags { target, .. } => vec![*target], PageOp::Add { object, .. } => vec![object.id()], @@ -736,6 +737,7 @@ impl Diff { } PageOp::Paragraph { paragraph, .. } | PageOp::Style { paragraph, .. } + | PageOp::Media { paragraph, .. } | PageOp::List { paragraph, .. } => { require(self.kept(*paragraph) && self.new_has(*paragraph))? } diff --git a/crates/notebook/src/session.rs b/crates/notebook/src/session.rs index d76676e865ef5052deed1d2e9b499d8344429edd..680c2b3c4a22c2b3d1087f305f9b60b101f9dabb 100644 --- a/crates/notebook/src/session.rs +++ b/crates/notebook/src/session.rs @@ -887,13 +887,16 @@ impl Notebook { /// `smb` on a share. It exists once the section has been opened. pub fn replica_path(&self, path: &str) -> Result { let section = self.section_path(path)?; - Ok(match &self.root { - Some(_) => { + Ok(match (&self.root, §ion.state) { + (Some(_), discover::SectionState::Readable { document, .. }) => { + replica_file(&self.cache, document) + } + (Some(_), _) => { let image = self.storage.read(§ion.path)?; let root = RevisionIndex::parse(&Store::parse(&image)?)?.root; replica_file(&self.cache, &root.guid) } - None => replica_file(&self.cache.join("smb"), §ion.file_id), + (None, _) => replica_file(&self.cache.join("smb"), §ion.file_id), }) } @@ -914,13 +917,16 @@ impl Notebook { } /// Keeps the sections of a mounted notebook in sync while they are not open - /// (`Background`), polling each file every `interval` once watched. + /// (`Background`): each file is checked every `interval`, and sooner when + /// `Background::touched` reports it changed. `copies` keeps an offline copy of every + /// section, for a folder that is not on this computer. pub fn background( &self, interval: Duration, + copies: bool, notify: impl Fn() + Send + 'static, ) -> Result { - self.background_with(interval, |file| FileRemote(file.to_owned()), notify) + self.background_with(interval, copies, |file| FileRemote(file.to_owned()), notify) } /// `background`, reaching each section file through the remote `remote` makes for it @@ -928,6 +934,7 @@ impl Notebook { pub fn background_with( &self, interval: Duration, + copies: bool, remote: impl Fn(&Path) -> R + Clone + Send + 'static, notify: impl Fn() + Send + 'static, ) -> Result { @@ -939,10 +946,10 @@ impl Notebook { .into()); }; Background::start( - interval, - move || { + copies, + move |_| { let (root, remote) = (root.clone(), remote.clone()); - Ok(move |path: &str| remote(&root.join(path))) + Ok((move |path: &str| remote(&root.join(path)), interval)) }, notify, ) diff --git a/crates/notebook/src/smb/directory.rs b/crates/notebook/src/smb/directory.rs index 1649cc9e07b18c39b8a8c2d0899c51d5e90b4a5b..6a0cc5ff34bd5d5e07c0a545c4666366dfc3fe27 100644 --- a/crates/notebook/src/smb/directory.rs +++ b/crates/notebook/src/smb/directory.rs @@ -96,6 +96,73 @@ impl Client { }) .collect()) } + + /// Watches the directory tree at `path` as OneNote 2010 watches a notebook's folder: one + /// CHANGE_NOTIFY with WATCH_TREE, kept armed on the server, so an idle watch sends + /// nothing. `changed` runs on the client's runtime with each batch of changed paths, + /// relative to `path` and `/`-separated, or `""` when the server lost count of them, then + /// `Err(_)` once the watch ends with its connection or the client. + pub fn watch( + &self, + path: &str, + changed: impl FnMut(io::Result>) + Send + 'static, + ) -> io::Result<()> { + if Handle::try_current().is_ok() { + return Err(io::ErrorKind::InvalidInput.into()); + } + let mut connection = self + .connection + .lock() + .map_err(|_| io::ErrorKind::Other)? + .clone() + .ok_or(io::ErrorKind::NotConnected)?; + // Probing a connection whose only request is the watch would end OneNote's silence. + connection.set_keepalive(None); + let runtime = self.runtime.lock().map_err(|_| io::ErrorKind::Other)?; + let runtime = runtime.as_ref().ok_or(io::ErrorKind::NotConnected)?; + let path = path.replace('\\', "/"); + let mut watcher = runtime + .block_on(async { + tokio::time::timeout(self.timeout, self.tree.watch(&mut connection, &path, true)) + .await + }) + .map_err(|_| io::Error::from(io::ErrorKind::TimedOut))? + .map_err(|error| match &error { + smb2::Error::Protocol { status, .. } if status.0 == 0xc0000034 => { + io::Error::new(io::ErrorKind::NotFound, error) + } + // STATUS_NOT_SUPPORTED, STATUS_INVALID_DEVICE_REQUEST + smb2::Error::Protocol { status, .. } + if matches!(status.0, 0xc00000bb | 0xc0000010) => + { + io::Error::new(io::ErrorKind::Unsupported, error) + } + _ => io::Error::other(error), + })?; + // However the task ends, the connection failing or the client retiring its runtime, + // `changed` hears that the watch did. + struct Ends>)>(F); + impl>)> Drop for Ends { + fn drop(&mut self) { + (self.0)(Err(io::ErrorKind::NotConnected.into())); + } + } + let mut ends = Ends(changed); + runtime.spawn(async move { + loop { + match watcher.next_events().await { + Ok(events) => { + (ends.0)(Ok(events.into_iter().map(|event| event.filename).collect())) + } + Err(smb2::Error::Protocol { status, .. }) if status.0 == 0x0000010c => { + (ends.0)(Ok(vec![String::new()])) + } + Err(_) => return, + } + } + }); + Ok(()) + } } fn decode(mut bytes: &[u8]) -> io::Result> { diff --git a/crates/notebook/src/smb/tests.rs b/crates/notebook/src/smb/tests.rs index 74340bd1f7b7fa7a5ce4b7de2c78e69834b3b12d..32fc3ec9e204498cba8f0a2b4513abbbfe17de3a 100644 --- a/crates/notebook/src/smb/tests.rs +++ b/crates/notebook/src/smb/tests.rs @@ -421,6 +421,44 @@ fn live_stamp_and_transaction() { poller.delete(&path).unwrap(); } +/// A watch names the file another client commits to in the watched folder. +#[test] +#[ignore = "requires ONESTORE_SMB_LAB pointing to disposable Samba"] +fn live_watch() { + let watcher = client(); + let writer = client(); + let folder = format!( + "watch-{}", + SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap() + .as_nanos() + ); + writer.create_directory(&folder).unwrap(); + let path = format!("{folder}/Watched.one"); + let source = onestore::create_section("Watched.one", "Watched", "Author").unwrap(); + create(&writer, &path, &source); + let (sender, changes) = std::sync::mpsc::channel(); + watcher + .watch(&folder, move |changed| { + let _ = sender.send(changed.map_err(|error| error.kind())); + }) + .unwrap(); + std::thread::sleep(Duration::from_secs(1)); + writer + .commit_transaction(&path, &replaced(&source, 0..0, "Changed ")) + .unwrap(); + let changed = changes.recv_timeout(Duration::from_secs(10)).unwrap(); + assert!(changed.unwrap().contains(&"Watched.one".to_owned())); + // An idle watch hears nothing more. + while let Ok(changed) = changes.recv_timeout(Duration::from_secs(2)) { + assert!(changed.unwrap().iter().all(|path| path == "Watched.one")); + } + drop(watcher); + writer.delete(&path).unwrap(); + writer.delete(&folder).unwrap(); +} + #[test] #[ignore = "requires ONESTORE_SMB_LAB pointing to disposable Samba"] fn live_storage_inspection() { diff --git a/crates/notebook/tests/session.rs b/crates/notebook/tests/session.rs index 0620ce0a6d3aa30a5a657104612649b32bcbc4eb..e041320398dc31de45b9ee16b9043301d8b559b0 100644 --- a/crates/notebook/tests/session.rs +++ b/crates/notebook/tests/session.rs @@ -1029,7 +1029,7 @@ fn a_closed_sections_queued_edits_publish_in_the_background_when_online() { let notebook = two_sections(directory.path()); let file = directory.path().join("Shared/Second.one"); let background = notebook - .background(Duration::from_millis(20), || {}) + .background(Duration::from_millis(20), false, || {}) .unwrap(); background.set_offline(true); background.watch(notebook.replicas()); @@ -1083,7 +1083,7 @@ fn a_remote_change_to_a_closed_section_is_noticed_and_rebases_its_replica() { let notified = Arc::new(AtomicUsize::new(0)); let counter = Arc::clone(¬ified); let background = notebook - .background(Duration::from_millis(20), move || { + .background(Duration::from_millis(20), false, move || { counter.fetch_add(1, Ordering::SeqCst); }) .unwrap(); @@ -1125,3 +1125,77 @@ fn a_remote_change_to_a_closed_section_is_noticed_and_rebases_its_replica() { }); assert_same(opened.unwrap().page(*space).unwrap(), native); } + +#[test] +fn a_reported_change_wakes_its_section_without_polling() { + let directory = tempfile::tempdir().unwrap(); + let notebook = two_sections(directory.path()); + let background = notebook + .background(Duration::from_secs(3600), false, || {}) + .unwrap(); + background.watch(notebook.replicas()); + until("both sections were reached", || { + background + .status() + .iter() + .all(|(_, status)| status.synced.is_some()) + }); + let file = directory.path().join("Shared/Second.one"); + let bytes = onestore::read_file(&file).unwrap(); + let space = notebook::session::stored_pages(&bytes).unwrap()[0].space; + let native = edited(&model_ops::page_of(&bytes, space), "Native "); + ops::save(&bytes, space, &native) + .unwrap() + .commit_file(&file) + .unwrap(); + std::thread::sleep(Duration::from_secs(2)); + assert!( + background.changed().is_empty(), + "nothing polls an idle notebook" + ); + background.touched(&["Second.one".to_owned()]); + let mut noticed = Vec::new(); + until("the reported change was noticed", || { + noticed.extend(background.changed()); + !noticed.is_empty() + }); + assert_eq!(noticed, ["Second.one"]); +} + +#[test] +fn every_section_gets_an_offline_copy_that_closing_the_notebook_discards() { + let directory = tempfile::tempdir().unwrap(); + let notebook = two_sections(directory.path()); + let replicas: Vec<_> = notebook + .replicas() + .into_iter() + .map(|(_, replica)| replica.unwrap()) + .collect(); + assert!(replicas.iter().all(|replica| !replica.exists())); + let background = notebook + .background(Duration::from_secs(3600), true, || {}) + .unwrap(); + background.watch(notebook.replicas()); + until("both sections have offline copies", || { + replicas.iter().all(|replica| replica.exists()) + && background + .status() + .iter() + .all(|(_, status)| status.synced.is_some()) + }); + // Opening a section starts from its copy, and edits made offline wait in it. + let section = notebook.section("Second.one", || {}).unwrap(); + section.set_offline(true); + let space = section.pages().unwrap()[0].0; + let before = section.page(space).unwrap(); + typed(§ion, space, &before, 0..0, "Waiting "); + section.close().unwrap(); + background.discard(); + let second = notebook.replica_path("Second.one").unwrap(); + until("the copy with nothing waiting was discarded", || { + !notebook.replica_path("First.one").unwrap().exists() + }); + assert!(second.exists(), "a copy with edits waiting stays"); + let kept = notebook::Replica::open(&second).unwrap(); + assert_eq!(kept.pending().unwrap().len(), 1); +} diff --git a/crates/onestore/examples/ink_lab.rs b/crates/onestore/examples/ink_lab.rs new file mode 100644 index 0000000000000000000000000000000000000000..7a92fcfda39d0711b662a6d02d3aeadb0f8913d9 --- /dev/null +++ b/crates/onestore/examples/ink_lab.rs @@ -0,0 +1,25 @@ +use onestore::{RevisionIndex, Store, document::Document}; +fn main() { + let path = std::env::args().nth(1).unwrap(); + let needle = std::env::args().nth(2).unwrap(); + let bytes = std::fs::read(path).unwrap(); + let store = Store::parse(&bytes).unwrap(); + let index = RevisionIndex::parse(&store).unwrap(); + let document = Document::parse(&index).unwrap(); + for (space, _) in document.pages().unwrap() { + let page = onestore::page::Page::from_space(&document, space).unwrap(); + if page.title != needle { + continue; + } + if std::env::var_os("INK_MODEL").is_some() { + for object in &page.objects { + if let onestore::page::PageObject::Ink(ink) = object { + println!("{}", serde_json::to_string(ink).unwrap()); + } + } + continue; + } + let revision = document.spaces[&space].active().unwrap(); + println!("{}", serde_json::to_string_pretty(revision).unwrap()); + } +} diff --git a/crates/onestore/examples/scratch_dump.rs b/crates/onestore/examples/scratch_dump.rs new file mode 100644 index 0000000000000000000000000000000000000000..db7d79089617abe0c8d20fe30b2c53eaaf76d8ff --- /dev/null +++ b/crates/onestore/examples/scratch_dump.rs @@ -0,0 +1,29 @@ +use onestore::{RevisionIndex, Store, document::Document}; +fn main() { + let path = std::env::args().nth(1).unwrap(); + let needle = std::env::args().nth(2).unwrap(); + let bytes = std::fs::read(path).unwrap(); + let store = Store::parse(&bytes).unwrap(); + let index = RevisionIndex::parse(&store).unwrap(); + let document = Document::parse(&index).unwrap(); + for (space, _) in document.pages().unwrap() { + let page = onestore::page::Page::from_space(&document, space).unwrap(); + if page.title != needle { continue; } + let revision = document.spaces[&space].active().unwrap(); + println!("{}", serde_json::to_string_pretty(revision).unwrap()); + if let Ok(out) = std::env::var("PAYLOAD_OUT") { + for object in &page.objects { + if let onestore::page::PageObject::Outline(o) = object { + for p in &o.paragraphs { + if let onestore::page::ParagraphContent::Attachment(a) = &p.content { + std::fs::write(format!("{out}/{}", a.filename), a.bytes.as_deref().unwrap()).unwrap(); + if let Some(i) = &a.preview { std::fs::write(format!("{out}/{}.png", a.filename), i).unwrap(); } + } + } + } + } + } + } +} +#[allow(dead_code)] +fn unused() {} diff --git a/crates/onestore/src/document.rs b/crates/onestore/src/document.rs index 4556ed6da34ba0aa5beabd308c6c5a06d34bae60..069fa5b5638cf5a02615c7833bacaa42df2e10a6 100644 --- a/crates/onestore/src/document.rs +++ b/crates/onestore/src/document.rs @@ -405,9 +405,12 @@ impl Revision<'_> { #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] /// Tag dates count seconds since 1980-01-01 UTC; status retains the ActionItemStatus bits. +/// A task tag (status bit 4) has no definition and carries its own icon and property status. pub struct Tag { pub definition: Option, pub action_type: Option, + pub shape: Option, + pub property_status: Option, pub status: u16, pub created: Option, pub completed: Option, @@ -446,6 +449,8 @@ pub enum Kind<'a> { margin_origin_x: Option, margin_origin_y: Option, rtl: Option, + /// Every audio and video recording on the page (AudioRecordingGuids). + recordings: Vec<[u8; 16]>, }, Metadata { title: Option, @@ -475,11 +480,16 @@ pub enum Kind<'a> { }, OutlineGroup, /// An ink drawing or handwriting container: strokes hang off `data`, nested containers off - /// the element's content list. Scaling multiplies stroke coordinates. + /// the element's content list. Scaling multiplies stroke coordinates. A drawn shape adds + /// its kind (11 a line, 12 a closed shape) and geometry: a line's ends, or a closed + /// shape's transform and anchors, as `page::InkShape` reads them. Ink { data: Option, scale_x: Option, scale_y: Option, + shape_kind: Option, + line: Option>, + anchors: Option>, }, InkData { strokes: Vec, @@ -555,6 +565,8 @@ pub enum Kind<'a> { source_path: Option, recording_id: Option<[u8; 16]>, recording_type: Option, + /// AudioRecordingDuration, in milliseconds. + recording_duration: Option, /// Displayed icon width and height in points. icon_width: Option, icon_height: Option, @@ -675,6 +687,16 @@ impl<'a, 'o> Fields<'a, 'o> { fn u32(&mut self, id: u32) -> Result> { Ok(self.fixed(id)?.map(u32::from_le_bytes)) } + fn guids(&mut self, id: u32) -> Result> { + let bytes = self.bytes(id)?.unwrap_or_default(); + if !bytes.len().is_multiple_of(16) { + return Err(invalid("Media identifiers have an invalid length")); + } + Ok(bytes + .chunks_exact(16) + .map(|id| id.try_into().unwrap()) + .collect()) + } fn u16(&mut self, id: u32) -> Result> { Ok(self.fixed(id)?.map(u16::from_le_bytes)) } @@ -1058,6 +1080,8 @@ impl<'a> Element<'a> { tags.push(Tag { definition: tag.one(0x20003488)?, action_type: tag.u16(0x10003463)?, + shape: tag.u16(0x10003464)?, + property_status: tag.u32(0x14003467)?, status: tag .u16(0x10003470)? .ok_or_else(|| invalid("Note tag has no status"))?, @@ -1114,6 +1138,7 @@ impl<'a> Element<'a> { margin_origin_x: f.float(0x14001d0f, 36.0)?, margin_origin_y: f.float(0x14001d10, 36.0)?, rtl: f.boolean(0x08001c92)?, + recordings: f.guids(0x1c001ca3)?, }, 0x20030 => Kind::Metadata { title: f.text(0x1c001cf3)?, @@ -1128,6 +1153,9 @@ impl<'a> Element<'a> { data: f.one(0x20003415)?, scale_x: f.float(0x14001c46, 1.0)?, scale_y: f.float(0x14001c47, 1.0)?, + shape_kind: f.bytes(0x0c001d4f)?.and_then(|b| b.first().copied()), + line: f.bytes(0x1c001dac)?.map(<[u8]>::to_vec), + anchors: f.bytes(0x1c001daa)?.map(<[u8]>::to_vec), }, 0x2003b => Kind::InkData { strokes: f.refs(0x24003416, IdStream::Objects)?, @@ -1327,6 +1355,7 @@ impl<'a> Element<'a> { source_path: f.text(0x1c001d9d)?, recording_id: f.fixed(0x1c001c97)?, recording_type: f.u32(0x14001d24)?, + recording_duration: f.u32(0x14001cfd)?, icon_width: f.float(0x140034cd, 36.0)?, icon_height: f.float(0x140034ce, 36.0)?, }, @@ -1349,14 +1378,7 @@ impl<'a> Element<'a> { }, _ => Kind::Unknown, }; - let media_ids = f.bytes(0x1c001c98)?.unwrap_or_default(); - if !media_ids.len().is_multiple_of(16) { - return Err(invalid("Media identifiers have an invalid length")); - } - let media_ids = media_ids - .chunks_exact(16) - .map(|id| id.try_into().unwrap()) - .collect(); + let media_ids = f.guids(0x1c001c98)?; Ok(Self { jcid: object.jcid, children, diff --git a/crates/onestore/src/op/apply.rs b/crates/onestore/src/op/apply.rs index 3fcab72678b98e557e839720dbe6786b0d6a4b57..e66c662e72a90eebefcda9f8203312ee67bac13a 100644 --- a/crates/onestore/src/op/apply.rs +++ b/crates/onestore/src/op/apply.rs @@ -63,9 +63,16 @@ impl<'a> Section<'a> { .collect(); for space in spaces { // A page the edit went on to delete has no cells left to check. - let Ok(page) = section.active(space) else { + if section.active(space).is_err() { continue; - }; + } + Writer { + section: &mut *section, + space, + author, + } + .playlist()?; + let page = section.active(space)?; if page.view.nodes.values().any(|node| { matches!(node.kind, Kind::Cell { .. }) && node.children.is_empty() }) { @@ -474,6 +481,41 @@ impl<'a> Writer<'_, 'a> { Ok(()) } + /// Lists the page's recordings in its AudioRecordingGuids as OneNote keeps them: in + /// the order they arrived, without those since removed, absent when there are none. + fn playlist(&mut self) -> Result<(), Failure> { + let page = self.page()?; + let [id] = page.pages[..] else { + return Ok(()); + }; + let Kind::Page { recordings, .. } = &page.view.nodes[&id].kind else { + return Ok(()); + }; + let present: Vec<[u8; 16]> = page + .view + .nodes + .values() + .filter_map(|node| match node.kind { + Kind::Attachment { recording_id, .. } => recording_id, + _ => None, + }) + .collect(); + let mut listed: Vec<[u8; 16]> = recordings + .iter() + .filter(|recording| present.contains(recording)) + .copied() + .collect(); + for recording in present { + if !listed.contains(&recording) { + listed.push(recording); + } + } + if listed == *recordings { + return Ok(()); + } + self.write(|page| content::playlist_changes(page, id, &listed)) + } + /// Requires `id` reachable on the page. fn target(&mut self, id: ExGuid) -> Result<(), Failure> { let page = self.page()?; @@ -746,6 +788,33 @@ impl<'a> Writer<'_, 'a> { let text = self.text_of(*paragraph)?; self.write(|page| properties::style_changes(page, text, *style, Some(definition))) } + PageOp::Media { paragraph, media } => { + self.target(*paragraph)?; + let page = self.page()?; + let content = match page.view.nodes[paragraph].content.as_slice() { + [content] + if matches!( + page.view.nodes[content].kind, + Kind::RichText { .. } | Kind::Attachment { .. } + ) => + { + *content + } + _ => { + return Err(OpError::Unsupported( + "Link a paragraph of text or a file to a recording", + ) + .into()); + } + }; + if media.recordings.is_empty() != media.time_ms.is_none() { + return Err(OpError::Unsupported( + "A recording link names recordings and a moment in them", + ) + .into()); + } + self.write(|page| properties::media_changes(page, content, media)) + } PageOp::List { paragraph, lists } => { self.target(*paragraph)?; let page = self.page()?; @@ -772,17 +841,16 @@ impl<'a> Writer<'_, 'a> { definitions, } => { self.target(*target)?; - let mut entries = Vec::new(); - for tag in tags { - let definition = tag - .definition - .ok_or(OpError::Unsupported("A note tag names its definition"))?; - let model = definitions - .iter() - .find(|(id, _)| *id == definition) - .map(|(_, definition)| definition); - entries.push((definition, tag, model)); - } + let entries: Vec<_> = tags + .iter() + .map(|tag| { + let model = definitions + .iter() + .find(|(id, _)| Some(*id) == tag.definition) + .map(|(_, definition)| definition); + (tag, model) + }) + .collect(); self.write(|page| properties::tag_changes(page, *target, &entries)) } PageOp::Add { object, before } => self.add(object, *before), @@ -1449,7 +1517,7 @@ fn bare(paragraph: &PageParagraph) -> Result<(), OpError> { } if paragraph.media != Default::default() { return Err(OpError::Unsupported( - "Recording annotations are made by OneNote while it records", + "Link a paragraph to a recording with its own op", )); } match ¶graph.content { @@ -1470,9 +1538,6 @@ fn bare(paragraph: &PageParagraph) -> Result<(), OpError> { } Ok(()) } - ParagraphContent::Attachment(attachment) if attachment.recording.is_some() => Err( - OpError::Unsupported("Recordings are captured by OneNote, not inserted"), - ), _ => Ok(()), } } diff --git a/crates/onestore/src/op/content.rs b/crates/onestore/src/op/content.rs index cd897b9107d76050272e315f5556969367e073ba..3f9cfb43f198f7014f7156192b69301e2c9ae66a 100644 --- a/crates/onestore/src/op/content.rs +++ b/crates/onestore/src/op/content.rs @@ -6,7 +6,7 @@ use crate::{ Error, ExGuid, active::{ActivePage, Changes}, document::{Format, Layout}, - page::{Attachment, Image, Ink, InkStroke, Paragraph}, + page::{Attachment, Image, Ink, InkStroke, Paragraph, Recording}, write::PropertyObject, }; use std::collections::BTreeMap; @@ -319,8 +319,10 @@ pub(crate) fn attachment_changes( "An attachment needs a file name without path separators", )); } - if attachment.recording.is_some() { - return Err(invalid("Recordings are captured by OneNote, not inserted")); + if Recording::kind_of(name).is_some() != attachment.recording.is_some() { + return Err(invalid( + "Audio and video files are recordings, and only they are", + )); } if attachment.bytes.is_none() { return Err(invalid("A new attachment needs its payload")); @@ -374,6 +376,19 @@ pub(crate) fn attachment_changes( if let Some(path) = &attachment.source_path { values.push((0x1c001d9d, crate::create::string(path))); } + if let Some(recording) = attachment.recording { + if recording.id == [0; 16] || !matches!(recording.kind, 1 | 2) { + return Err(invalid( + "A recording needs an identity and is audio or video", + )); + } + // AudioRecordingGuid, IRecordMedia and AudioRecordingDuration. + values.push((0x1c001c97, recording.id.to_vec())); + values.push((0x14001d24, recording.kind.to_le_bytes().to_vec())); + if let Some(duration) = recording.duration_ms { + values.push((0x14001cfd, duration.to_le_bytes().to_vec())); + } + } values.extend(layout_values(&Default::default(), &attachment.layout)?.0); let mut changed = BTreeMap::new(); let mut file = PropertyObject::file(ids.file, &payload_reference(ids.payload), extension)?; @@ -400,6 +415,21 @@ pub(crate) fn attachment_changes( Ok(changed) } +/// Page `page` listing `recordings` in its AudioRecordingGuids, or none. +pub(crate) fn playlist_changes( + active: &ActivePage<'_>, + page: ExGuid, + recordings: &[[u8; 16]], +) -> Result { + let mut node = PropertyObject::from_object(&active.live.revision.objects[&page])?; + if recordings.is_empty() { + node.remove(&[0x1c001ca3])?; + } else { + node.set(&[(0x1c001ca3, recordings.concat().as_slice())])?; + } + Ok(BTreeMap::from([(page, node)])) +} + /// A stored attachment keeps its payload and preview; its shown name, recorded source /// path and icon size change in place, as OneNote's rename does. pub(crate) fn attachment_edit_changes( @@ -415,7 +445,7 @@ pub(crate) fn attachment_edit_changes( .to_vec(), )]; if attachment.recording != stored.recording { - return Err(invalid("A recording stays the recording OneNote captured")); + return Err(invalid("A file keeps its recording")); } let name = attachment.filename.as_str(); if name.is_empty() || name.contains(['\0', '/', '\\']) { @@ -425,6 +455,11 @@ pub(crate) fn attachment_edit_changes( } let mut removed = Vec::new(); if attachment.filename != stored.filename { + if Recording::kind_of(name).is_some() != attachment.recording.is_some() { + return Err(invalid( + "Audio and video files are recordings, and only they are", + )); + } let name = crate::create::string(&attachment.filename); values.push((0x1c001c22, name.clone())); values.push((0x1c001d9c, name)); @@ -457,19 +492,31 @@ pub(crate) fn attachment_edit_changes( Ok(BTreeMap::from([(object, node)])) } -/// Creates stroke objects (numbered from `first`) and one drawing-attribute object per -/// distinct pen; returns the compact references for the data node's stroke list. +/// Creates stroke objects (numbered on from the page's last stroke) and one +/// drawing-attribute object per distinct pen; returns the compact references for the data +/// node's stroke list. A shape's strokes carry only their path and pen, as OneNote's do. fn write_strokes( + active: &ActivePage<'_>, changed: &mut Changes, data_object: &mut PropertyObject, strokes: &[(ExGuid, &InkStroke)], - first: usize, + shape: bool, filetime: u64, ) -> Result, Error> { + let first = active + .view + .nodes + .values() + .filter_map(|node| match node.kind { + crate::document::Kind::InkStroke { index, .. } => index, + _ => None, + }) + .max() + .unwrap_or(0); let mut styles: Vec<(InkPen, ExGuid)> = Vec::new(); let mut references = Vec::new(); for (offset, (stroke_id, stroke)) in strokes.iter().enumerate() { - let pen = InkPen::of(stroke); + let pen = InkPen::of(stroke, shape); let style = match styles.iter().find(|(known, _)| *known == pen) { Some((_, id)) => *id, None => { @@ -488,13 +535,14 @@ fn write_strokes( id } }; + let values = if shape { + stroke_path(stroke)? + } else { + stroke_values(stroke, first + offset as u32 + 1, filetime)? + }; let mut object = PropertyObject { jcid: 0x20047, - bytes: crate::create::properties(&stroke_values( - stroke, - (first + offset) as u32 + 1, - filetime, - )?)?, + bytes: crate::create::properties(&values)?, global_ids: std::sync::Arc::new(BTreeMap::from([(0, stroke_id.guid)])), }; object.reference(*stroke_id)?; @@ -506,7 +554,8 @@ fn write_strokes( Ok(references) } -/// The drawing attributes OneNote shares between strokes drawn with the same pen. +/// The drawing attributes OneNote shares between strokes drawn with the same pen. A pen +/// drawn with the mouse ignores pressure; a shape's pen spans every coordinate instead. #[derive(PartialEq)] struct InkPen { width: u32, @@ -514,22 +563,31 @@ struct InkPen { color: Option, transparency: Option, pen_tip: Option, + raster_operation: Option, + shape: bool, } impl InkPen { - fn of(stroke: &InkStroke) -> Self { + fn of(stroke: &InkStroke, shape: bool) -> Self { Self { width: (stroke.width * 2540.0 / 72.0).to_bits(), height: (stroke.height * 2540.0 / 72.0).to_bits(), color: stroke.color, transparency: stroke.transparency, pen_tip: stroke.pen_tip, + raster_operation: stroke.raster_operation, + shape, } } fn values(&self) -> Values { + let dimensions = if self.shape { + crate::page::ink::SHAPE_DIMENSIONS + } else { + crate::page::ink::DIMENSIONS + }; let mut values: Values = vec![ - (0x1c00340a, crate::page::ink::DIMENSIONS.to_vec()), + (0x1c00340a, dimensions.to_vec()), (0x1400340c, self.height.to_le_bytes().to_vec()), (0x1400340d, self.width.to_le_bytes().to_vec()), ]; @@ -542,11 +600,17 @@ impl InkPen { if let Some(tip) = self.pen_tip { values.push((0x0c003412, vec![tip])); } + if let Some(operation) = self.raster_operation { + values.push((0x0c003413, vec![operation])); + } + if !self.shape { + values.push((0x88003411, Vec::new())); + } values } } -fn stroke_values(stroke: &InkStroke, index: u32, filetime: u64) -> Result { +fn stroke_path(stroke: &InkStroke) -> Result { if stroke.points.is_empty() { return Err(invalid("A stroke needs at least one point")); } @@ -562,27 +626,32 @@ fn stroke_values(stroke: &InkStroke, index: u32, filetime: u64) -> Result Result { + let mut values = stroke_path(stroke)?; + let [left, top] = [0, 1].map(|axis| { + stroke + .points + .iter() + .map(|p| p[axis]) + .fold(f32::INFINITY, f32::min) + }); let mut origin = ((left - stroke.width / 2.0) / 36.0).to_le_bytes().to_vec(); origin.extend_from_slice(&((top - stroke.height / 2.0) / 36.0).to_le_bytes()); - Ok(vec![ - (0x1c00340b, stroke.packet()), + let bias = if stroke.raster_operation == Some(9) { 2 } else { 0 }; + values.extend([ (0x14003419, index.to_le_bytes().to_vec()), (0x1000341b, 0x409_u16.to_le_bytes().to_vec()), - (0x0c00341c, vec![0]), + (0x0c00341c, vec![bias]), (0x1c00341a, crate::write::fresh_guid()?.to_vec()), (0x1c00341d, filetime.to_le_bytes().to_vec()), (0x1c00345b, origin), - ]) + ]); + Ok(values) } /// New ink as OneNote 2010 stores a drawing: container `id` that the page lists as a child @@ -614,15 +683,27 @@ pub(crate) fn ink_changes( global_ids: std::sync::Arc::new(BTreeMap::from([(0, data.guid)])), }; data_object.reference(data)?; - let references = write_strokes(&mut changed, &mut data_object, strokes, 0, filetime)?; + let references = write_strokes( + active, + &mut changed, + &mut data_object, + strokes, + ink.shape.is_some(), + filetime, + )?; data_object.set(&[(0x24003416, &references)])?; changed.insert(data, data_object); + // A shape keeps its kind and geometry beside the strokes, and counts as a second kind of + // drawing. + let mut values: Values = vec![(0x14001d7a, modified.to_vec())]; + if let Some(shape) = &ink.shape { + let (kind, property, bytes) = shape.stored(); + values.extend([(0x0c001d4f, vec![kind]), (property, bytes)]); + } + values.push((0x14001d4e, (1 + u32::from(ink.shape.is_some())).to_le_bytes().to_vec())); let mut object = PropertyObject { jcid: 0x60014, - bytes: crate::create::properties(&[ - (0x14001d7a, modified.to_vec()), - (0x14001d4e, 1u32.to_le_bytes().to_vec()), - ])?, + bytes: crate::create::properties(&values)?, global_ids: std::sync::Arc::new(BTreeMap::from([(0, id.guid)])), }; object.reference(id)?; @@ -641,10 +722,12 @@ pub(crate) fn strokes_changes( kept: &[ExGuid], added: &[(ExGuid, &InkStroke)], ) -> Result { - let data = match active.view.nodes.get(&container).map(|node| &node.kind) { + let (data, shape) = match active.view.nodes.get(&container).map(|node| &node.kind) { Some(crate::document::Kind::Ink { - data: Some(data), .. - }) => *data, + data: Some(data), + shape_kind, + .. + }) => (*data, shape_kind.is_some()), _ => return Err(invalid("Stored ink has no stroke data to rewrite")), }; let (modified, filetime) = crate::create::current_timestamps()?; @@ -657,10 +740,11 @@ pub(crate) fn strokes_changes( references.extend(data_object.reference(*id)?); } references.extend(write_strokes( + active, &mut changed, &mut data_object, added, - kept.len(), + shape, filetime, )?); data_object.set(&[(0x24003416, &references)])?; diff --git a/crates/onestore/src/op/lower.rs b/crates/onestore/src/op/lower.rs index 9b8c0f043e15eb97480707642a07ef84364a8d2c..9110ef3e27dc1c72dad65128145eea70af1f6dee 100644 --- a/crates/onestore/src/op/lower.rs +++ b/crates/onestore/src/op/lower.rs @@ -225,7 +225,7 @@ pub(crate) fn collect_containers(list: &[PageParagraph], out: &mut Vec) /// What the writers keep as stored, checked before any op: titles, unsupported objects, /// definitions other than lists and tags, outline roles, paragraph styles and formats, -/// recording links, fields and content types. +/// fields and content types. pub(crate) fn validate( before: &Page, after: &Page, @@ -309,17 +309,13 @@ pub(crate) fn validate( let Some(previous) = old.paragraphs.get(id) else { continue; }; - // A join moves a text object, with its style and recording link, into an emptied - // paragraph; one new to the page starts as a copy without the link. + // A join moves a text object, with its style, into an emptied paragraph. let owner = paragraph.text().map(|text| owners.get(&text.id)); - let (style, media) = match owner { - Some(Some(owner)) => (owner.style, owner.media.clone()), - Some(None) => (paragraph.style, Default::default()), - None => (previous.style, previous.media.clone()), + let style = match owner { + Some(Some(owner)) => owner.style, + Some(None) => paragraph.style, + None => previous.style, }; - if paragraph.media != media { - return Err(invalid("Recording annotations cannot be edited")); - } if paragraph.style != style || paragraph.format != previous.format { return Err(invalid( "Paragraph styles and paragraph formatting cannot be edited", @@ -695,6 +691,8 @@ pub(crate) fn same_tags(a: &[Tag], b: &[Tag]) -> bool { ( t.definition, t.action_type, + t.shape, + t.property_status, t.status, t.created, t.completed, @@ -745,6 +743,7 @@ fn bare(paragraph: &PageParagraph) -> PageParagraph { stripped.lists.clear(); stripped.tags.clear(); stripped.collapsed = false; + stripped.media = Default::default(); match &mut stripped.content { ParagraphContent::Text(text) => text.tags.clear(), ParagraphContent::Table(table) => { @@ -820,6 +819,7 @@ impl Lowering { self.styles(after, new)?; self.lists(after, new)?; self.tags(after, new)?; + self.media(new)?; self.paragraph_formatting(new)?; self.formatting(old, new)?; self.layout(old, new)?; @@ -1775,9 +1775,9 @@ impl Lowering { } let mut definitions = Vec::new(); for tag in tags { - let definition = tag - .definition - .ok_or_else(|| invalid("A note tag names its definition"))?; + let Some(definition) = tag.definition else { + continue; + }; if let Some(model) = after.definitions.get(&definition) && !definitions.iter().any(|(id, _)| *id == definition) { @@ -1794,6 +1794,20 @@ impl Lowering { Ok(()) } + fn media(&mut self, new: &View<'_>) -> Result<(), Error> { + for (id, paragraph) in &new.paragraphs { + let current = model::paragraph(&self.current, *id) + .ok_or_else(|| invalid("A paragraph is missing after text edits"))?; + if current.media != paragraph.media { + self.emit(PageOp::Media { + paragraph: *id, + media: paragraph.media.clone(), + })?; + } + } + Ok(()) + } + fn paragraph_formatting(&mut self, new: &View<'_>) -> Result<(), Error> { for (id, paragraph) in &new.paragraphs { let Some(text) = paragraph.text() else { diff --git a/crates/onestore/src/op/mod.rs b/crates/onestore/src/op/mod.rs index 3c91882a2766cb0a9ae75ee8ffd63979af2e3ac8..4de89028000f8a4b1bc8ac8bdd4e2291aab9d105 100644 --- a/crates/onestore/src/op/mod.rs +++ b/crates/onestore/src/op/mod.rs @@ -6,8 +6,8 @@ use crate::{ ExGuid, OutlineEdit, PageCreation, PageEdit, TextAttribute, document::{Layout, Tag}, page::{ - Definition, InkStroke, Page, PageObject, PageParagraph, Paragraph, TableCell, TableColumn, - TableRow, + Definition, InkStroke, MediaIndex, Page, PageObject, PageParagraph, Paragraph, TableCell, + TableColumn, TableRow, }, }; use serde::{Deserialize, Serialize}; @@ -151,6 +151,12 @@ pub enum PageOp { style: ExGuid, definition: Definition, }, + /// Links a paragraph's text or file to a moment in recordings on the page, or with + /// empty `media` unlinks it. + Media { + paragraph: ExGuid, + media: MediaIndex, + }, /// Replaces a paragraph's list nodes; a node another paragraph owns is copied. List { paragraph: ExGuid, diff --git a/crates/onestore/src/op/model.rs b/crates/onestore/src/op/model.rs index c03231f41104c61b5297c58b2de9dfdcf00cfab1..e7b320617de5fa7793cc25a84698601400166849 100644 --- a/crates/onestore/src/op/model.rs +++ b/crates/onestore/src/op/model.rs @@ -468,6 +468,7 @@ fn stroked(stroke: &crate::page::InkStroke) -> crate::page::InkStroke { fn inked(ink: &crate::page::Ink) -> crate::page::Ink { crate::page::Ink { strokes: ink.strokes.iter().map(stroked).collect(), + shape: ink.shape.as_ref().map(crate::page::ink::InkShape::snapped), ..ink.clone() } } @@ -1042,6 +1043,21 @@ fn interpret(page: &mut Page, op: &PageOp) -> Result<(), Error> { text.text = Paragraph::from_runs(runs); } } + PageOp::Media { paragraph, media } => { + let paragraph = paragraph_mut(page, *paragraph)?; + if !matches!( + paragraph.content, + ParagraphContent::Text(_) | ParagraphContent::Attachment(_) + ) { + return Err(invalid("Link a paragraph of text or a file to a recording")); + } + if media.recordings.is_empty() != media.time_ms.is_none() { + return Err(invalid( + "A recording link names recordings and a moment in them", + )); + } + paragraph.media = media.clone(); + } PageOp::List { paragraph, lists } => { paragraph_mut(page, *paragraph)?.lists = lists.iter().map(|(id, _)| *id).collect(); for (id, definition) in lists { diff --git a/crates/onestore/src/op/properties.rs b/crates/onestore/src/op/properties.rs index fd0e7c8f8c71714f3f8f3119643d428c382c5171..4d7aadd7f94dc035e14cc157128f5ee21ab2e1df 100644 --- a/crates/onestore/src/op/properties.rs +++ b/crates/onestore/src/op/properties.rs @@ -5,7 +5,7 @@ use crate::{ Error, ExGuid, PropertySets, Value, active::{ActivePage, Changes}, document::{Kind, Tag}, - page::Definition, + page::{Definition, MediaIndex}, write::PropertyObject, }; use std::collections::{BTreeMap, BTreeSet}; @@ -165,54 +165,78 @@ fn tag_definition_values(definition: &Definition) -> Result { Ok(values) } -/// Replaces the note tags of `object`. Each entry names the stored identity of its tag's -/// definition and, where the tag does not carry its action type or the page lacks the -/// definition, the definition itself. +/// Replaces the note tags of `object`. A normal tag comes with its definition where the page +/// lacks it; a task tag has none and stores its own icon, in the order OneNote writes it. pub(crate) fn tag_changes( active: &ActivePage<'_>, object: ExGuid, - tags: &[(ExGuid, &Tag, Option<&Definition>)], + tags: &[(&Tag, Option<&Definition>)], ) -> Result { let raw = &active.live.revision; let mut definitions: Vec<(ExGuid, Option)> = Vec::new(); - let mut sets: Vec<(usize, Values)> = Vec::new(); + let mut sets: Vec<(Option, Values)> = Vec::new(); let mut action_types = BTreeSet::new(); - for (written, tag, definition) in tags { - let action_type = if tag.status & 4 != 0 { - tag.action_type - } else { - match definition.map(|d| &d.kind) { - Some(Kind::TagDefinition { action_type, .. }) => *action_type, - _ => return Err(invalid("A note tag must reference a tag definition")), - } - }; - if !action_types.insert(action_type.unwrap_or(0)) { - return Err(invalid("An element holds one note tag per action type")); - } - let index = match definitions.iter().position(|(id, _)| id == written) { - Some(index) => index, - None => { - let values = if raw.objects.contains_key(written) { - None - } else { - let definition = definition - .ok_or_else(|| invalid("A note tag references a missing tag definition"))?; - Some(tag_definition_values(definition)?) - }; - definitions.push((*written, values)); - definitions.len() - 1 - } - }; - let mut fields: Values = Vec::new(); - if let Some(action_type) = tag.action_type { - fields.push((0x10003463, action_type.to_le_bytes().to_vec())); - } - for (id, value) in [ + for (tag, definition) in tags { + let dates = [ (0x1400346e, tag.created), (0x1400346f, tag.completed), (0x1400346a, tag.start), (0x1400346b, tag.due), - ] { + ]; + let mut fields: Values = Vec::new(); + if tag.status & 4 != 0 { + let (Some(action_type), Some(shape), Some(property_status)) = + (tag.action_type, tag.shape, tag.property_status) + else { + return Err(invalid( + "A task tag stores its action type, icon and property status", + )); + }; + if !action_types.insert(action_type) { + return Err(invalid("An element holds one note tag per action type")); + } + fields.extend([ + (0x0c003473, vec![0]), + (0x10003463, action_type.to_le_bytes().to_vec()), + (0x10003464, shape.to_le_bytes().to_vec()), + (0x14003467, property_status.to_le_bytes().to_vec()), + ]); + if let Some(task) = tag.task_id { + fields.push((0x1c003469, task.to_vec())); + } + for (id, value) in [dates[2], dates[3], dates[0], dates[1]] { + if let Some(value) = value { + fields.push((id, value.to_le_bytes().to_vec())); + } + } + fields.push((0x10003470, tag.status.to_le_bytes().to_vec())); + sets.push((None, fields)); + continue; + } + let written = tag + .definition + .ok_or_else(|| invalid("A note tag must reference a tag definition"))?; + let action_type = match definition.map(|d| &d.kind) { + Some(Kind::TagDefinition { action_type, .. }) => *action_type, + _ => return Err(invalid("A note tag must reference a tag definition")), + }; + if !action_types.insert(action_type.unwrap_or(0)) { + return Err(invalid("An element holds one note tag per action type")); + } + if !definitions.iter().any(|(id, _)| *id == written) { + let values = if raw.objects.contains_key(&written) { + None + } else { + let definition = definition + .ok_or_else(|| invalid("A note tag references a missing tag definition"))?; + Some(tag_definition_values(definition)?) + }; + definitions.push((written, values)); + } + if let Some(action_type) = tag.action_type { + fields.push((0x10003463, action_type.to_le_bytes().to_vec())); + } + for (id, value) in dates { if let Some(value) = value { fields.push((id, value.to_le_bytes().to_vec())); } @@ -221,7 +245,7 @@ pub(crate) fn tag_changes( if let Some(task) = tag.task_id { fields.push((0x1c003469, task.to_vec())); } - sets.push((index, fields)); + sets.push((Some(written), fields)); } let parents = active.editable_parents(object)?; let modified = crate::create::current_timestamps()?.0.to_le_bytes(); @@ -240,9 +264,12 @@ pub(crate) fn tag_changes( } let mut target = PropertyObject::from_object(&raw.objects[&object])?; let mut encoded = Vec::new(); - for (index, fields) in &sets { - let reference = target.reference(definitions[*index].0)?; - let mut set = vec![(0x20003488, reference.to_vec())]; + for (definition, fields) in &sets { + let mut set = Vec::new(); + if let Some(definition) = definition { + let reference = target.reference(*definition)?; + set.push((0x20003488, reference.to_vec())); + } set.extend(fields.iter().cloned()); encoded.push(set); } @@ -380,3 +407,29 @@ pub(crate) fn paragraph_format_changes( crate::formatting::touch_ancestors(raw, parents, object, &modified, &mut changed)?; Ok(changed) } + +/// Links content `object` to a moment in recordings (MediaIndex), or unlinks it. +pub(crate) fn media_changes( + active: &ActivePage<'_>, + object: ExGuid, + media: &MediaIndex, +) -> Result { + let parents = active.editable_parents(object)?; + let modified = crate::create::current_timestamps()?.0.to_le_bytes(); + let raw = &active.live.revision; + let mut target = PropertyObject::from_object(&raw.objects[&object])?; + match media.time_ms { + Some(time) => { + let recordings = media.recordings.concat(); + target.set(&[ + (0x1c001c98, recordings.as_slice()), + (0x14001c99, &time.to_le_bytes()), + ])?; + } + None => target.remove(&[0x1c001c98, 0x14001c99])?, + } + target.set(&[(0x14001d7a, &modified)])?; + let mut changed = BTreeMap::from([(object, target)]); + crate::formatting::touch_ancestors(raw, parents, object, &modified, &mut changed)?; + Ok(changed) +} diff --git a/crates/onestore/src/op/tests.rs b/crates/onestore/src/op/tests.rs index d88b3eb776b3e5cf1616609583b735b9293bfe98..0fda9207f9db8c27833a6beb61ae7ec6b7994d5a 100644 --- a/crates/onestore/src/op/tests.rs +++ b/crates/onestore/src/op/tests.rs @@ -834,6 +834,8 @@ pub(super) fn mutate(page: &mut Page, rng: &mut Rng, family: Family) -> bool { text.tags.push(crate::document::Tag { definition: Some(definition), action_type: None, + shape: None, + property_status: None, status, created: Some(1_262_401_445), completed: (status == 1).then_some(1_262_402_000), @@ -1159,6 +1161,7 @@ pub(super) fn mutate(page: &mut Page, rng: &mut Rng, family: Family) -> bool { layout: Layout::default(), strokes: vec![stroke(rng)], groups: Vec::new(), + shape: None, }) }; page.objects.insert(at, object); @@ -1251,6 +1254,8 @@ fn stroke(rng: &mut Rng) -> InkStroke { color: Some(0x0000ff), transparency: None, pen_tip: None, + // Every other stroke a highlighter's. + raster_operation: (x as u32).is_multiple_of(2).then_some(9), } } @@ -1595,6 +1600,8 @@ fn random_op(page: &Page, rng: &mut Rng) -> Option { tags: vec![crate::document::Tag { definition: Some(definition), action_type: None, + shape: None, + property_status: None, status: 0, created: Some(1_262_401_445), completed: None, @@ -1639,15 +1646,20 @@ fn random_op(page: &Page, rng: &mut Rng) -> Option { }), before: None, }, - _ => PageOp::Add { - object: PageObject::Ink(Ink { - id: id(), - layout: Layout::default(), - strokes: vec![stroke(rng)], - groups: Vec::new(), - }), - before: None, - }, + _ => { + let stroke = stroke(rng); + let ends = [stroke.points[0], stroke.points[stroke.points.len() - 1]]; + PageOp::Add { + object: PageObject::Ink(Ink { + id: id(), + layout: Layout::default(), + strokes: vec![stroke], + groups: Vec::new(), + shape: Some(crate::page::ink::InkShape::Line(ends)), + }), + before: None, + } + } }) } diff --git a/crates/onestore/src/page/ink.rs b/crates/onestore/src/page/ink.rs index 881a2222ddac97a8fdccae1d17a4a85596bfc483..2e3da7394a66d91130d5b4e14500a09dd529c4df 100644 --- a/crates/onestore/src/page/ink.rs +++ b/crates/onestore/src/page/ink.rs @@ -32,6 +32,119 @@ pub struct Ink { pub strokes: Vec, /// Nested ink containers, as newer OneNote versions group handwriting. pub groups: Vec, + /// The shape drawn, for a drawing made from Draw's Insert Shapes. + #[serde(default)] + pub shape: Option, +} + +/// The shapes of Draw's Insert Shapes that Snowbound draws. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum ShapeKind { + Line, + Arrow, + Rectangle, + Ellipse, +} + +impl Ink { + /// A shape dragged from `from` to `to` in page points, drawn with `pen`'s width and + /// colour as OneNote 2010 draws it (`corpus/ink-tools`): a rectangle clockwise from its + /// top left corner, an ellipse in 50 steps clockwise from its right end, and an arrow's + /// head as a second stroke of two barbs at atan(1/2) to the line, 8 points plus twice + /// the pen's width long. The result reads back unchanged once stored. + pub fn drawn( + kind: ShapeKind, + from: [f32; 2], + to: [f32; 2], + pen: &InkStroke, + ) -> Result { + let [x0, y0] = [from[0].min(to[0]), from[1].min(to[1])]; + let [x1, y1] = [from[0].max(to[0]), from[1].max(to[1])]; + let (shape, paths) = match kind { + ShapeKind::Line | ShapeKind::Arrow => { + let mut paths = vec![vec![from, to]]; + let [dx, dy] = [to[0] - from[0], to[1] - from[1]]; + if kind == ShapeKind::Arrow && dx.hypot(dy) > 0.0 { + let back = (-dy).atan2(-dx); + let length = 8.0 + 2.0 * pen.width; + let barb = |turn: f32| { + let angle = back + turn; + [to[0] + length * angle.cos(), to[1] + length * angle.sin()] + }; + let spread = 0.5f32.atan(); + paths.push(vec![barb(spread), to, barb(-spread)]); + } + (InkShape::Line([from, to]), paths) + } + ShapeKind::Rectangle | ShapeKind::Ellipse => { + let [w, h] = [x1 - x0, y1 - y0]; + let transform = [w, 0.0, 0.0, h, x0, y0]; + let (anchors, path): (&[[f32; 2]], Vec<[f32; 2]>) = if kind == ShapeKind::Rectangle + { + ( + &[ + [0.0, 0.0], + [0.5, 0.0], + [1.0, 0.0], + [1.0, 0.5], + [1.0, 1.0], + [0.5, 1.0], + [0.0, 1.0], + [0.0, 0.5], + ], + vec![[x0, y0], [x1, y0], [x1, y1], [x0, y1], [x0, y0]], + ) + } else { + let [cx, cy, rx, ry] = [x0 + w / 2.0, y0 + h / 2.0, w / 2.0, h / 2.0]; + ( + &[[0.5, 0.0], [1.0, 0.5], [0.5, 1.0], [0.0, 0.5]], + (0..=50) + .map(|step| { + let angle = step as f32 * std::f32::consts::TAU / 50.0; + [cx + rx * angle.cos(), cy + ry * angle.sin()] + }) + .collect(), + ) + }; + ( + InkShape::Closed { + transform, + anchors: anchors.to_vec(), + }, + vec![path], + ) + } + }; + let mut strokes = Vec::new(); + for path in paths { + strokes.push(InkStroke { + id: super::text::new_id()?, + points: path.into_iter().map(|p| p.map(snap)).collect(), + ..pen.clone() + }); + } + Ok(Self { + id: super::text::new_id()?, + layout: Layout::default(), + strokes, + groups: Vec::new(), + shape: Some(shape.snapped()), + }) + } +} + +/// What OneNote keeps beside a drawn shape's strokes to edit it by (`corpus/ink-tools`), in +/// page points before the drawing's offset. +#[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)] +pub enum InkShape { + /// A line or arrow from one end to the other. + Line([[f32; 2]; 2]), + /// A closed shape: `transform` (`[a, b, c, d, x, y]`) takes its anchors, in the unit + /// square, onto the page. + Closed { + transform: [f32; 6], + anchors: Vec<[f32; 2]>, + }, } #[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)] @@ -48,6 +161,10 @@ pub struct InkStroke { pub transparency: Option, /// 0 is a round (ball) tip, 1 a rectangle. pub pen_tip: Option, + /// The ISF raster operation: 9 (MaskPen) for a highlighter, whose colour multiplies + /// what lies beneath. + #[serde(default)] + pub raster_operation: Option, } impl Ink { @@ -61,6 +178,9 @@ impl Ink { data, scale_x, scale_y, + shape_kind, + line, + anchors, } = &node.kind else { unreachable!() @@ -98,6 +218,7 @@ impl Ink { layout: node.layout.clone(), strokes, groups, + shape: InkShape::read(*shape_kind, line.as_deref(), anchors.as_deref()), }) } @@ -151,6 +272,7 @@ impl InkStroke { color, transparency, pen_tip, + raster_operation, .. } = &style.kind else { @@ -190,10 +312,78 @@ impl InkStroke { color: *color, transparency: *transparency, pen_tip: *pen_tip, + raster_operation: *raster_operation, }) } } +impl InkShape { + /// Any geometry this does not recognise reads as no shape: the strokes still draw it. + fn read(kind: Option, line: Option<&[u8]>, anchors: Option<&[u8]>) -> Option { + let floats = |bytes: &[u8]| -> Vec { + bytes + .chunks_exact(4) + .map(|b| f32::from_le_bytes(b.try_into().unwrap())) + .collect() + }; + match (kind?, line, anchors) { + (11, Some(line), _) if line.len() == 16 => { + let v = floats(line); + Some(Self::Line([[v[0], v[1]], [v[2], v[3]]].map(|end| end.map(|v| v * 36.0)))) + .filter(Self::finite) + } + (12, _, Some(bytes)) if bytes.len() >= 24 && (bytes.len() - 24) % 8 == 0 => { + let v = floats(bytes); + Some(Self::Closed { + transform: std::array::from_fn(|i| v[i] * 36.0), + anchors: v[6..].chunks_exact(2).map(|p| [p[0], p[1]]).collect(), + }) + .filter(Self::finite) + } + _ => None, + } + } + + fn finite(&self) -> bool { + match self { + Self::Line(ends) => ends.iter().flatten().all(|v| v.is_finite()), + Self::Closed { transform, anchors } => transform + .iter() + .chain(anchors.iter().flatten()) + .all(|v| v.is_finite()), + } + } + + /// The kind, property and bytes OneNote stores the shape as. + pub(crate) fn stored(&self) -> (u8, u32, Vec) { + let half = |v: &f32| (v / 36.0).to_le_bytes(); + match self { + Self::Line(ends) => (11, 0x1c001dac, ends.iter().flatten().flat_map(half).collect()), + Self::Closed { transform, anchors } => ( + 12, + 0x1c001daa, + transform + .iter() + .flat_map(half) + .chain(anchors.iter().flatten().flat_map(|v| v.to_le_bytes())) + .collect(), + ), + } + } + + /// This shape as it reads back once stored. + pub fn snapped(&self) -> Self { + let half = |v: f32| (v / 36.0) * 36.0; + match self { + Self::Line(ends) => Self::Line(ends.map(|end| end.map(half))), + Self::Closed { transform, anchors } => Self::Closed { + transform: transform.map(half), + anchors: anchors.clone(), + }, + } + } +} + /// ISF multi-byte decoding: a count, then that many 7-bit little-endian varints whose low bit /// is the sign. fn multi_byte(bytes: &[u8]) -> Option> { @@ -268,6 +458,15 @@ pub(crate) const DIMENSIONS: [u8; 64] = [ 0x00, 0x00, 0x00, 0x00, 0x38, 0x04, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x8b, 0xc5, 0xe2, 0x41, ]; +/// The dimension table OneNote 2010 writes for a drawn shape's pen: X and Y in HIMETRIC over +/// the whole 32-bit range, at a resolution of 1000. +pub(crate) const SHAPE_DIMENSIONS: [u8; 64] = [ + 0x8f, 0x6a, 0x8a, 0x59, 0xc0, 0x52, 0xa0, 0x4b, 0x93, 0xaf, 0xaf, 0x35, 0x74, 0x11, 0xa5, 0x61, + 0x00, 0x00, 0x00, 0x80, 0xff, 0xff, 0xff, 0x7f, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x7a, 0x44, + 0x75, 0x9f, 0x3f, 0xb5, 0xe0, 0x04, 0x98, 0x44, 0xa7, 0xee, 0xc3, 0x0d, 0xbb, 0x5a, 0x90, 0x11, + 0x00, 0x00, 0x00, 0x80, 0xff, 0xff, 0xff, 0x7f, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x7a, 0x44, +]; + #[cfg(test)] mod tests { use super::*; @@ -282,6 +481,7 @@ mod tests { color: None, transparency: None, pen_tip: None, + raster_operation: None, }; let values = multi_byte(&stroke.packet()).unwrap(); assert_eq!(values.len(), 6); diff --git a/crates/onestore/src/page/mod.rs b/crates/onestore/src/page/mod.rs index 54e6a9a648bd982a20eca80f23a3a5f643714b08..0dabb489978a4275ed06e2119b81d61524adfc95 100644 --- a/crates/onestore/src/page/mod.rs +++ b/crates/onestore/src/page/mod.rs @@ -282,21 +282,34 @@ pub struct Attachment { pub bytes: Option>, #[serde(with = "payload")] pub preview: Option>, - /// Set when the file is an audio or video recording OneNote captured; annotations on - /// the page refer to it by identity (`PageParagraph::media`). + /// Set when the file is audio or video OneNote plays; notes on the page refer to it by + /// identity (`PageParagraph::media`). pub recording: Option, } -/// A recording's identity and OneNote's type code for it. +/// A recording's identity, whether it is audio (1) or video (2), and its length. #[derive(Clone, Copy, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)] pub struct Recording { pub id: [u8; 16], pub kind: u32, + pub duration_ms: Option, +} + +impl Recording { + /// The kind of recording MS-ONE requires a file named `filename` to be, if any: audio + /// for .wma, .mp3 and .wav, video for .wmv, .avi and .mpg. + pub fn kind_of(filename: &str) -> Option { + let extension = filename.rsplit_once('.')?.1.to_ascii_lowercase(); + match extension.as_str() { + "wma" | "mp3" | "wav" => Some(1), + "wmv" | "avi" | "mpg" => Some(2), + _ => None, + } + } } /// A paragraph's link to a moment in recordings on the page: OneNote plays from -/// `time_ms` when the paragraph is chosen. Read from native pages and preserved through -/// edits; OneNote alone creates them while recording. +/// `time_ms` when the paragraph is chosen, and links what is written while it records. #[derive(Clone, Debug, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)] pub struct MediaIndex { pub recordings: Vec<[u8; 16]>, @@ -328,6 +341,7 @@ impl Attachment { source_path, recording_id, recording_type, + recording_duration, icon_width, icon_height, } = &node.kind @@ -357,7 +371,10 @@ impl Attachment { preview: payload(preview)?, recording: recording_id.map(|id| Recording { id, - kind: recording_type.unwrap_or(0), + kind: recording_type + .or_else(|| Recording::kind_of(filename.as_deref().unwrap_or_default())) + .unwrap_or(1), + duration_ms: *recording_duration, }), }) } @@ -1369,6 +1386,7 @@ mod tests { margin_origin_x: Some(36.0), margin_origin_y: Some(12.0), rtl: None, + recordings: Vec::new(), }); page.children.push(id(2)); let mut outline = element(Kind::Outline { @@ -1621,6 +1639,8 @@ mod tests { source.nodes.get_mut(&id(4)).unwrap().tags.push(Tag { definition: Some(id(7)), action_type: None, + shape: None, + property_status: None, status: 0, created: Some(123), completed: None, diff --git a/crates/onestore/tests/page_ink.rs b/crates/onestore/tests/page_ink.rs index 329e50b6662427da2472c06b69b0668b86b71b28..3d8faa34d709aad5d8c40148b6c9d5b48a9faedc 100644 --- a/crates/onestore/tests/page_ink.rs +++ b/crates/onestore/tests/page_ink.rs @@ -88,6 +88,7 @@ fn stroke(points: &[[f32; 2]], color: Option) -> InkStroke { color, transparency: None, pen_tip: None, + raster_operation: None, } } @@ -109,6 +110,7 @@ fn drawing() -> Ink { stroke(&[[300.0, 120.0], [360.0, 180.0]], Some(0x0000ff)), ], groups: Vec::new(), + shape: None, } } @@ -232,3 +234,372 @@ fn stored_strokes_keep_their_paths() { } assert!(ops::saved(NATIVE, space, &moved).is_err()); } + +/// OneNote 2010's Draw tab used with the mouse (`corpus/ink-tools/native-ui`): every favourite +/// pen and highlighter, shapes, and ink erased, moved and deleted. +const TOOLS: &[u8] = include_bytes!("../../../corpus/ink-tools/native-ui/notebook/ink.one"); + +fn titled(bytes: &[u8], title: &str) -> (ExGuid, Page) { + let store = Store::parse(bytes).unwrap(); + let index = RevisionIndex::parse(&store).unwrap(); + let document = Document::parse(&index).unwrap(); + document + .pages() + .unwrap() + .into_iter() + .map(|(space, _)| (space, Page::from_space(&document, space).unwrap())) + .find(|(_, page)| page.title == title) + .unwrap() +} + +fn himetric(value: f32) -> f32 { + value * 72.0 / 2540.0 +} + +use onestore::page::ink::{InkShape, ShapeKind}; + +#[test] +fn native_pens_highlighters_and_shapes_read_as_drawn() { + let (_, gallery) = titled(TOOLS, "Gallery"); + let pens: Vec<_> = drawings(&gallery) + .iter() + .map(|ink| { + let [stroke] = ink.strokes.as_slice() else { + panic!("OneNote keeps each stroke a drawing of its own") + }; + ( + (stroke.width / himetric(1.0)).round(), + (stroke.height / himetric(1.0)).round(), + stroke.color, + stroke.raster_operation, + ) + }) + .collect(); + let (red, blue, green, grey) = (0x241ced, 0xbb6531, 0x367d17, 0x808080); + let pen = |width, color| (width, width, color, None); + let marker = |color| (70.0, 400.0, Some(color), Some(9)); + assert_eq!( + pens, + [ + pen(35.0, None), + pen(35.0, Some(red)), + pen(35.0, Some(blue)), + pen(35.0, Some(green)), + pen(35.0, Some(grey)), + marker(0x00ffff), + marker(0xffff00), + pen(50.0, None), + pen(50.0, Some(red)), + pen(50.0, Some(blue)), + pen(50.0, Some(green)), + pen(50.0, Some(grey)), + marker(0x00ff00), + marker(0xff00ff), + ] + ); + let highlighter = &drawings(&gallery)[5].strokes[0]; + assert_eq!( + (highlighter.transparency, highlighter.pen_tip), + (Some(127), Some(1)) + ); + + let (_, pens) = titled(TOOLS, "Pens"); + let shapes: Vec<&Ink> = drawings(&pens) + .into_iter() + .filter(|ink| ink.shape.is_some()) + .collect(); + let [rectangle, line, arrow, ellipse] = shapes.as_slice() else { + panic!("{}", shapes.len()) + }; + let pen = InkStroke { + points: Vec::new(), + ..rectangle.strokes[0].clone() + }; + assert!((pen.width - himetric(50.0)).abs() < 1e-5); + let close_points = |a: &[[f32; 2]], b: &[[f32; 2]]| { + a.len() == b.len() + && a.iter() + .zip(b) + .all(|(a, b)| (a[0] - b[0]).abs() < 0.05 && (a[1] - b[1]).abs() < 0.05) + }; + for (native, kind, from, to) in [ + (rectangle, ShapeKind::Rectangle, [252.0, 86.4], [324.0, 122.4]), + (line, ShapeKind::Line, [342.0, 86.4], [414.0, 122.4]), + (arrow, ShapeKind::Arrow, [252.0, 158.4], [324.0, 194.4]), + (ellipse, ShapeKind::Ellipse, [342.0, 158.4], [414.0, 194.4]), + ] { + let drawn = Ink::drawn(kind, from, to, &pen).unwrap(); + assert_eq!(drawn.strokes.len(), native.strokes.len(), "{kind:?}"); + for (drawn, native) in drawn.strokes.iter().zip(&native.strokes) { + assert!( + close_points(&drawn.points, &native.points), + "{kind:?}: {:?} against {:?}", + drawn.points, + native.points + ); + } + match (drawn.shape.unwrap(), native.shape.clone().unwrap()) { + (InkShape::Line(a), InkShape::Line(b)) => assert!(close_points(&a, &b)), + ( + InkShape::Closed { transform, anchors }, + InkShape::Closed { + transform: native_transform, + anchors: native_anchors, + }, + ) => { + assert_eq!(anchors, native_anchors); + assert!( + transform + .iter() + .zip(native_transform) + .all(|(a, b)| (a - b).abs() < 0.05) + ); + } + shapes => panic!("{shapes:?}"), + } + } + + // Erased and deleted drawings leave the page; a moved one keeps its strokes and takes + // an offset. + let (_, edits) = titled(TOOLS, "Edits"); + let moved: Vec<_> = drawings(&edits) + .into_iter() + .filter(|ink| ink.shape.is_none()) + .map(|ink| (ink.layout.x, ink.layout.y)) + .collect(); + assert_eq!(moved.len(), 2); + assert!(moved.iter().any(|&(x, y)| x == Some(45.0) + && y.is_some_and(|y| (y - 15.0).abs() < 1e-4))); +} + +fn tool_page(source: &[u8]) -> (ExGuid, Page, Vec) { + let (space, page) = first_page(source); + let accent = InkStroke { + id: new_id().unwrap(), + points: Vec::new(), + width: snap(himetric(35.0)), + height: snap(himetric(35.0)), + color: Some(0x7a9a1f), + transparency: None, + pen_tip: None, + raster_operation: None, + }; + let marker = InkStroke { + width: himetric(70.0), + height: himetric(400.0), + color: Some(0x00ffff), + transparency: Some(127), + pen_tip: Some(1), + raster_operation: Some(9), + ..accent.clone() + }; + let shape_pen = InkStroke { + width: himetric(50.0), + height: himetric(50.0), + ..accent.clone() + }; + let free = |pen: &InkStroke, points: Vec<[f32; 2]>| Ink { + id: new_id().unwrap(), + layout: Default::default(), + strokes: vec![InkStroke { + id: new_id().unwrap(), + points: points.into_iter().map(|p| p.map(snap)).collect(), + ..pen.clone() + }], + groups: Vec::new(), + shape: None, + }; + let wave: Vec<[f32; 2]> = (0..=40) + .map(|step| { + let x = 60.0 + step as f32 * 4.0; + [x, 150.0 + 12.0 * (x / 18.0).sin()] + }) + .collect(); + let inks = vec![ + free(&marker, vec![[60.0, 190.0], [220.0, 190.0]]), + free(&accent, wave), + Ink::drawn(ShapeKind::Rectangle, [252.0, 140.4], [324.0, 194.4], &shape_pen).unwrap(), + Ink::drawn(ShapeKind::Ellipse, [342.0, 140.4], [432.0, 194.4], &shape_pen).unwrap(), + Ink::drawn(ShapeKind::Arrow, [252.0, 230.4], [360.0, 266.4], &shape_pen).unwrap(), + Ink::drawn(ShapeKind::Line, [378.0, 230.4], [450.0, 230.4], &shape_pen).unwrap(), + ]; + (space, page, inks) +} + +fn added(source: &[u8], space: ExGuid, inks: &[Ink]) -> Vec { + let mut bytes = source.to_vec(); + for ink in inks { + bytes = ops::page_edited( + &bytes, + space, + vec![onestore::op::PageOp::Add { + object: PageObject::Ink(ink.clone()), + before: None, + }], + ) + .unwrap(); + } + bytes +} + +/// `ONESTORE_INK_TOOLS_EXPORT` names a new directory receiving the candidate for a cold +/// reopen. +#[test] +fn pens_highlighters_and_shapes_are_written_as_onenote_draws_them() { + use onestore::document::{FieldValue, Kind}; + let source = onestore::create_section("ink.one", "Drawn in Snowbound", "Author").unwrap(); + let (space, before, inks) = tool_page(&source); + // Each stroke is one edit, as the canvas stores it. + let written = added(&source, space, &inks); + let stored = page_in(&written, space); + let mut expected = before.clone(); + expected + .objects + .extend(inks.iter().cloned().map(PageObject::Ink)); + expected.title = stored.title.clone(); + assert_eq!(stored, expected); + export("ONESTORE_INK_TOOLS_EXPORT", "ink.one", &written); + + let store = Store::parse(&written).unwrap(); + let index = RevisionIndex::parse(&store).unwrap(); + let document = Document::parse(&index).unwrap(); + let revision = document.active(space).unwrap(); + let node = |id: ExGuid| &revision.nodes[&id]; + let extra = |id: ExGuid, property: u32| { + node(id) + .extra + .iter() + .flatten() + .find(|field| field.id == property) + .map(|field| match field.value { + FieldValue::Bytes(bytes) => bytes.to_vec(), + _ => Vec::new(), + }) + }; + let style = |stroke: ExGuid| match node(stroke).kind { + Kind::InkStroke { + style: Some(style), + bias, + index, + .. + } => (node(style).kind.clone(), bias, index), + _ => panic!(), + }; + let [marker, wave, rectangle, _, arrow, _] = inks.as_slice() else { + unreachable!() + }; + match style(marker.strokes[0].id) { + ( + Kind::InkStyle { + raster_operation: Some(9), + pen_tip: Some(1), + transparency: Some(127), + ignore_pressure: Some(true), + .. + }, + Some(2), + Some(1), + ) => {} + other => panic!("{other:?}"), + } + match style(wave.strokes[0].id) { + ( + Kind::InkStyle { + raster_operation: None, + ignore_pressure: Some(true), + color: Some(0x7a9a1f), + .. + }, + Some(0), + Some(2), + ) => {} + other => panic!("{other:?}"), + } + for stroke in &arrow.strokes { + match style(stroke.id) { + ( + Kind::InkStyle { + ignore_pressure: None, + .. + }, + None, + None, + ) => {} + other => panic!("{other:?}"), + } + } + assert_eq!(extra(wave.id, 0x14001d4e), Some(1u32.to_le_bytes().to_vec())); + assert_eq!(extra(rectangle.id, 0x14001d4e), Some(2u32.to_le_bytes().to_vec())); + match &node(rectangle.id).kind { + Kind::Ink { + shape_kind: Some(12), + anchors: Some(anchors), + .. + } => assert_eq!(anchors.len(), 24 + 8 * 8), + other => panic!("{other:?}"), + } +} + +/// `ONESTORE_INK_TOOLS_EDIT_EXPORT` names a new directory receiving the candidate for a cold +/// reopen: OneNote's own drawings after Snowbound erases one, moves another and draws more. +#[test] +fn onenote_drawings_take_erasing_moving_and_more_strokes() { + use onestore::{OutlineEdit, op::PageOp}; + let (space, page) = titled(TOOLS, "Pens"); + let native = drawings(&page); + let zigzag = native + .iter() + .find(|ink| ink.strokes[0].color == Some(0x241ced)) + .unwrap() + .id; + let line = native[1].id; + let edited = ops::page_edited( + TOOLS, + space, + vec![ + PageOp::Delete { object: zigzag }, + PageOp::Outline { + object: line, + edit: OutlineEdit::Position { x: 36.0, y: 18.0 }, + }, + ], + ) + .unwrap(); + let (_, _, inks) = tool_page(&onestore::create_section("x.one", "x", "x").unwrap()); + let moved: Vec = inks + .into_iter() + .map(|mut ink| { + for stroke in &mut ink.strokes { + for point in &mut stroke.points { + point[1] = snap(point[1] + 200.0); + } + } + if let Some(shape) = &mut ink.shape { + *shape = match shape { + InkShape::Line([a, b]) => InkShape::Line([[a[0], a[1] + 200.0], [b[0], b[1] + 200.0]]), + InkShape::Closed { transform, anchors } => InkShape::Closed { + transform: { + let mut t = *transform; + t[5] += 200.0; + t + }, + anchors: anchors.clone(), + }, + } + .snapped(); + } + ink + }) + .collect(); + let written = added(&edited, space, &moved); + let stored = page_in(&written, space); + let found = drawings(&stored); + assert!(!found.iter().any(|ink| ink.id == zigzag)); + let line = found.iter().find(|ink| ink.id == line).unwrap(); + assert_eq!((line.layout.x, line.layout.y), (Some(36.0), Some(18.0))); + assert_eq!(line.strokes, native[1].strokes); + for ink in &moved { + assert_eq!(found.iter().find(|found| found.id == ink.id), Some(&ink)); + } + export("ONESTORE_INK_TOOLS_EDIT_EXPORT", "ink.one", &written); +} diff --git a/crates/onestore/tests/page_media.rs b/crates/onestore/tests/page_media.rs index c18be4ca6372b03407a962d7dce9f053837ab5dc..e465e939296c5691b9a8b9f96e20112bf0b0f93a 100644 --- a/crates/onestore/tests/page_media.rs +++ b/crates/onestore/tests/page_media.rs @@ -103,18 +103,29 @@ fn editing_a_recorded_page_keeps_the_recording_and_its_annotation() { assert_eq!(recording(&stored), recording(&before)); assert_eq!(stored.objects.len(), before.objects.len()); - let mut annotated = after.clone(); - let outline = annotated - .objects - .iter_mut() - .find_map(|object| match object { - PageObject::Outline(outline) if !outline.title => Some(outline), - _ => None, - }) - .unwrap(); - let last = outline.paragraphs.last_mut().unwrap(); - last.media = recording(&before).1; - assert!(ops::saved(written.as_slice(), space, &annotated).is_err()); + // A note linked to the recording, then unlinked, as Snowbound links notes taken while + // it records. + let annotate = |page: &Page, media: MediaIndex| { + let mut page = page.clone(); + for object in &mut page.objects { + if let PageObject::Outline(outline) = object + && !outline.title + { + outline.paragraphs.last_mut().unwrap().media = media.clone(); + } + } + page + }; + let linked = ops::saved( + written.as_slice(), + space, + &annotate(&stored, recording(&before).1), + ) + .unwrap(); + let (_, relinked) = page(linked.as_slice()); + assert_eq!(relinked, annotate(&stored, recording(&before).1)); + let unlinked = ops::saved(linked.as_slice(), space, &stored).unwrap(); + assert_eq!(page(unlinked.as_slice()).1, stored); let mut recaptured = after.clone(); for object in &mut recaptured.objects { if let PageObject::Outline(outline) = object { diff --git a/crates/onestore/tests/page_tags.rs b/crates/onestore/tests/page_tags.rs index eec322c51ca55e55bd0f5e1c8199fb83bda499e6..1f2c2d8d638bdbcec471a1308aae9f1c410778b8 100644 --- a/crates/onestore/tests/page_tags.rs +++ b/crates/onestore/tests/page_tags.rs @@ -113,6 +113,8 @@ fn tag(definition: ExGuid, completed: bool) -> Tag { Tag { definition: Some(definition), action_type: None, + shape: None, + property_status: None, status: u16::from(completed), created: Some(1_262_401_445), completed: completed.then_some(1_262_405_106), @@ -305,3 +307,49 @@ fn new_definitions_and_tags_publish_on_a_fresh_page() { .unwrap(); } } + +/// An Outlook task tag names no definition: rewritten beside a normal tag it keeps its icon, +/// property status and dates, and one missing its icon is refused. +#[test] +fn a_task_tag_rewrites_without_a_definition() { + let features = include_bytes!("../../../corpus/m6/native-features-01/notebook/Features.one"); + let (space, before) = page_by_title(features, "Task controls"); + let at = tagged(&before); + let task = body_paragraphs(&mut before.clone())[at] + .text() + .unwrap() + .tags[0] + .clone(); + assert_eq!( + ( + task.definition, + task.shape, + task.property_status, + task.status + ), + (None, Some(89), Some(0), 6) + ); + let mut after = before.clone(); + let id = new_id().unwrap(); + after.definitions.insert(id, definition("To Do", 3)); + body_paragraphs(&mut after)[at] + .text_mut() + .unwrap() + .tags + .insert(0, tag(id, false)); + let written = ops::saved(features, space, &after).unwrap(); + let stored = assert_same(written.as_slice(), space, &after); + let mut untagged = stored.clone(); + body_paragraphs(&mut untagged)[at] + .text_mut() + .unwrap() + .tags + .remove(0); + untagged.definitions.remove(&id); + let again = ops::saved(&written, space, &untagged).unwrap(); + assert_same(again.as_slice(), space, &before); + let mut shapeless = before.clone(); + body_paragraphs(&mut shapeless)[at].text_mut().unwrap().tags[0].shape = None; + body_paragraphs(&mut shapeless)[at].text_mut().unwrap().tags[0].created = Some(1); + assert!(ops::saved(features, space, &shapeless).is_err()); +} diff --git a/crates/onestore/tests/support/page_edits.rs b/crates/onestore/tests/support/page_edits.rs index 53718a2e02fe072b70ffe31f93b7dcffe64c86f6..dd9aa7507cfa66de67cc9725f840754b739265cc 100644 --- a/crates/onestore/tests/support/page_edits.rs +++ b/crates/onestore/tests/support/page_edits.rs @@ -256,6 +256,8 @@ fn mutate(page: &mut Page, bytes: &mut Bytes<'_>) { text.tags.push(onestore::document::Tag { definition: Some(definition), action_type: None, + shape: None, + property_status: None, status: u16::from(bytes.next().unwrap_or(0) % 2), created: Some(1_262_401_445), completed: None, diff --git a/crates/snowbound/src/library.rs b/crates/snowbound/src/library.rs index d5ffe0f22f05b120971698491cc4ef16139f6f1d..15dc47f8d7bfddc3548dce0c46fd3754cbeb08ef 100644 --- a/crates/snowbound/src/library.rs +++ b/crates/snowbound/src/library.rs @@ -33,10 +33,6 @@ pub fn on_background(notify: impl Fn() + Send + Sync + 'static) { let _ = NOTIFY.set(Box::new(notify)); } -/// How often the sections no tab shows are polled; OneNote 2010 read a closed section -/// fifteen seconds after another client changed it. -const BACKGROUND: Duration = Duration::from_secs(15); - fn notify_background() { if let Some(notify) = NOTIFY.get() { notify(); @@ -171,6 +167,8 @@ pub struct Library { /// Syncs the notebook's sections while no tab shows them, as OneNote syncs every /// section of an open notebook. pub background: Option>, + /// Reports the changes to a folder on this computer to `background`. + watch: Option>, } impl Library { @@ -192,10 +190,15 @@ impl Library { } } let notebook = Notebook::open(location, cache); + let (background, watch) = match ¬ebook { + Ok(notebook) => local_background(notebook, location), + Err(_) => (None, None), + }; Self { location: location.to_owned(), name: file_name(Path::new(location)), - background: notebook.as_ref().ok().and_then(local_background), + background, + watch, notebook: notebook.map(Some).map_err(|error| error.to_string()), server: None, notice, @@ -219,7 +222,6 @@ impl Library { let background = Background::smb( &server.mount.root, LIMIT, - BACKGROUND, move || connect.connect(), notify_background, ) @@ -234,6 +236,7 @@ impl Library { notice: None, cache: cache.to_owned(), background: Some(Arc::new(background)), + watch: None, }) } @@ -260,15 +263,18 @@ impl Library { notice: self.notice.clone(), cache: self.cache.clone(), background: self.background.clone(), + watch: self.watch.clone(), } } /// A notebook `Notebook::create` just made in the folder at `location`. pub fn created(location: &str, notebook: Notebook, cache: &Path) -> Self { + let (background, watch) = local_background(¬ebook, location); Self { location: location.to_owned(), name: file_name(Path::new(location)), - background: local_background(¬ebook), + background, + watch, notebook: Ok(Some(notebook)), server: None, notice: None, @@ -286,6 +292,7 @@ impl Library { notice: None, cache: cache.to_owned(), background: None, + watch: None, } } @@ -296,7 +303,8 @@ impl Library { path: &str, notify: impl Fn() + Send + 'static, ) -> Result> { - // The background may hold the replica for a step, which takes a network round trip. + // The background may hold the replica for a step, which takes a network round trip, + // or be making it as an offline copy. let deadline = Instant::now() + TIMEOUT * 3; let notify = Arc::new(Mutex::new(notify)); let notifier = || { @@ -335,7 +343,12 @@ impl Library { (Err(error), _) => return Err(error.clone().into()), }; match opened { - Err(error) if error.busy() && Instant::now() < deadline => { + Err(error) + if (error.busy() + || matches!(&error, notebook::Error::Io(error) + if error.kind() == io::ErrorKind::AlreadyExists)) + && Instant::now() < deadline => + { std::thread::sleep(Duration::from_millis(50)); } opened => break opened?, @@ -446,15 +459,37 @@ pub fn recycle_bin(path: &str) -> bool { path.rsplit('/').next() == Some("OneNote_RecycleBin") } -/// A mounted notebook's background sync, following Work Offline. -fn local_background(notebook: &Notebook) -> Option> { - let background = notebook - .background(BACKGROUND, notify_background) +/// A mounted notebook's background sync, following Work Offline, and the watch that reports +/// the folder's changes to it. Only a folder on this computer reports every change; one on a +/// network volume is checked more often and kept in offline copies, as on a share. +fn local_background( + notebook: &Notebook, + location: &str, +) -> (Option>, Option>) { + let reports = Arc::new(OnceLock::>::new()); + let report = Arc::clone(&reports); + let watch = crate::watch::watch(Path::new(location), move |paths| { + if let Some(background) = report.get().and_then(std::sync::Weak::upgrade) { + background.touched(&paths); + } + }); + let remote = !crate::watch::on_this_computer(Path::new(location)); + let interval = match watch { + Some(_) if !remote => Background::BACKSTOP, + _ => Background::UNWATCHED, + }; + let Some(background) = notebook + .background(interval, remote, notify_background) .inspect_err(|error| eprintln!("Background sync did not start: {error}")) - .ok()?; + .ok() + else { + return (None, None); + }; background.set_offline(offline()); background.watch(notebook.replicas()); - Some(Arc::new(background)) + let background = Arc::new(background); + let _ = reports.set(Arc::downgrade(&background)); + (Some(background), watch.map(Arc::new)) } pub fn file_name(path: &Path) -> String { @@ -483,7 +518,7 @@ fn tabs(catalog: &Folder) -> Vec { .sections .iter() .filter_map(|section| match §ion.state { - SectionState::Readable { name, color } => Some(Tab { + SectionState::Readable { name, color, .. } => Some(Tab { name: section_name(§ion.path, name), path: section.path.clone(), color: *color, @@ -668,6 +703,7 @@ mod tests { notice: None, cache: PathBuf::new(), background: None, + watch: None, }; let shown = |root, file| library(root).local(Path::new(file)); let under = Some(PathBuf::from("/Volumes/agent/lab/Group/New Section 1.one")); diff --git a/crates/snowbound/src/linux.rs b/crates/snowbound/src/linux.rs index a700797a0f7b2cb98b94398d14cac4f70b633c47..396c5b0e28a9b8d4bd37ef31947e5af0998a8484 100644 --- a/crates/snowbound/src/linux.rs +++ b/crates/snowbound/src/linux.rs @@ -25,6 +25,8 @@ use winit::{ const APP_ID: &str = "snowbound"; /// The title bar's leading margin; the window controls sit at its trailing end. pub const LEADING: f32 = 8.0; +/// The margin past the window controls. +pub const TRAILING: f32 = 8.0; /// How far inside the window's edges a press resizes it, and how far along them a corner /// reaches, in logical pixels. const EDGE: f32 = 5.0; diff --git a/crates/snowbound/src/macos.rs b/crates/snowbound/src/macos.rs index d4fccd8adaf5efa67099198b7b6e2675c6756482..29464bf36949dba8d816b49dc439045a3f3a7171 100644 --- a/crates/snowbound/src/macos.rs +++ b/crates/snowbound/src/macos.rs @@ -86,6 +86,8 @@ fn ns_window(window: &Window) -> Retained { /// Room the traffic lights take at the title bar's leading edge. pub const LEADING: f32 = 78.0; +/// The title bar's trailing margin, the gap the traffic lights leave before the toolbar. +pub const TRAILING: f32 = 12.0; /// How far AppKit rounds a window's corners. pub const CORNER_RADIUS: f32 = 10.0; diff --git a/crates/snowbound/src/main.rs b/crates/snowbound/src/main.rs index 21289acf0965534b1da6def0f7372e5f022a7b95..bea24a9a40cf6a7735f348210cf34fa8ebfae38c 100644 --- a/crates/snowbound/src/main.rs +++ b/crates/snowbound/src/main.rs @@ -31,6 +31,7 @@ mod surface; mod sync; mod tags; mod templates; +mod watch; use canvas::gpu::page::PageScene; use canvas::interaction::{Cursor, PageView, Place, Request, Response, TextColors, accessibility}; @@ -1200,6 +1201,13 @@ impl State { if self.session.is_some() { self.page_tools(&theme); } + self.ui.leaf( + "trail", + Spec { + size: [px(self.trailing() - FRAME), px(1.0)], + ..Spec::default() + }, + ); self.ui.close(); self.ui.open_as( @@ -1268,6 +1276,23 @@ impl State { } } self.ui.close(); + if let Some(task) = self.view.task_under_pointer() { + let [left, top, ..] = self.ui.rect(page()).unwrap_or_default(); + let scale = self.ui.scale(); + let [x0, y0, x1, y1] = task; + let part = [ + left + x0 / scale, + top + y0 / scale, + left + x1 / scale, + top + y1 / scale, + ]; + ui::popup::tooltip_over( + &mut self.ui, + part, + "Outlook task", + Some("Edit it in OneNote with Outlook"), + ); + } let signal = self.ui.signal(page()); let focused = self.ui.window_focused && self.ui.focused() == Some(page()); if focused != self.page_focused { @@ -1411,6 +1436,16 @@ impl State { ); } + /// The toolbar and tab row's margin at the window's trailing side, which matches the + /// leading gap where the row is the title bar. + fn trailing(&self) -> f32 { + if platform::system_titlebar(&self.window) { + FRAME + } else { + platform::TRAILING + } + } + /// Whether the chrome's empty space drags the window: where it is the title bar, or /// over the title's gradient. fn chrome_drags(&self) -> bool { @@ -1468,7 +1503,7 @@ impl State { self.ui.leaf( "trail", Spec { - size: [px(FRAME - GAP), px(1.0)], + size: [px(self.trailing() - GAP), px(1.0)], ..Spec::default() }, ); diff --git a/crates/snowbound/src/manage.rs b/crates/snowbound/src/manage.rs index 48fcc4edfc83221b32b995d19c257721598b5dc7..6df5eb3400a46d1b973e624a7aa84217d9388847 100644 --- a/crates/snowbound/src/manage.rs +++ b/crates/snowbound/src/manage.rs @@ -189,9 +189,13 @@ impl State { Ok(()) } - /// Closes `library`: its files stay, and the sidebar and the next launch leave it out. + /// Closes `library`: its files stay, its offline copies go unless edits wait in them, + /// and the sidebar and the next launch leave it out. pub(crate) fn close_notebook(&mut self, library: &Arc) { self.notebooks.retain(|open| !Arc::ptr_eq(open, library)); + if let Some(background) = &library.background { + background.discard(); + } let shown = |shown: &Arc| shown.location == library.location; if self.sectionless.as_ref().is_some_and(shown) || self diff --git a/crates/snowbound/src/meeting.rs b/crates/snowbound/src/meeting.rs index f289ef2cb784afc795a94e92106c6cb3593db422..e2c3eb2d816872860f97450788a2e95cf38d02b2 100644 --- a/crates/snowbound/src/meeting.rs +++ b/crates/snowbound/src/meeting.rs @@ -164,6 +164,8 @@ pub fn content(page: &Page) -> Result> { vec![Tag { definition: Some(to_do), action_type: None, + shape: None, + property_status: None, status: 0, created: Some(created), completed: Some(0), diff --git a/crates/snowbound/src/sidebar.rs b/crates/snowbound/src/sidebar.rs index 939124e4bf13141c52146d4c54ade3c7f4da925f..424fba2f1cd2c9a6875fe69b4d07b844363631ce 100644 --- a/crates/snowbound/src/sidebar.rs +++ b/crates/snowbound/src/sidebar.rs @@ -260,7 +260,7 @@ fn folder( }; for section in &folder.sections { let (name, color, readable) = match §ion.state { - SectionState::Readable { name, color } => ( + SectionState::Readable { name, color, .. } => ( crate::library::section_name(§ion.path, name), *color, true, diff --git a/crates/snowbound/src/tags.rs b/crates/snowbound/src/tags.rs index 48caad352c92fb43c36cb95572541b638a35d51a..b4a5113514e2788b29e377d2738795216d2e9c85 100644 --- a/crates/snowbound/src/tags.rs +++ b/crates/snowbound/src/tags.rs @@ -103,13 +103,13 @@ fn label(ui: &mut Ui, part: &str, text: &str, tag: &NoteTag) { ); } -fn icon(ui: &mut Ui, part: &str, art: Option<&'static [&'static str]>) { +fn icon(ui: &mut Ui, part: &str, art: Option<&'static [&'static str]>, tint: [f32; 4]) { ui.leaf( part, Spec { size: [px(SIDE + 4.0), px(ROW)], icon: art, - color: Some([1.0; 4]), + color: Some(tint), center: true, ..Spec::default() }, @@ -324,7 +324,11 @@ impl State { ..Spec::default() }, ); - icon(ui, "icon", artwork(tag)); + // OneNote's list marks a tag without a symbol with its "None" symbol art. + match artwork(tag) { + Some(art) => icon(ui, "icon", Some(art), [1.0; 4]), + None => icon(ui, "icon", Some(crate::art::FONT_COLOR), theme.text), + } let text = match commands::tag_chord(place) { Some(chord) => format!("{} ({})", tag.label, chord.label(Platform::CURRENT)), None => tag.label.clone(), @@ -585,7 +589,7 @@ fn tag_editor( ..Spec::default() }, ); - icon(ui, "icon", artwork(tag)); + icon(ui, "icon", artwork(tag), [1.0; 4]); label(ui, "label", &tag.label, tag); ui.leaf( "end", diff --git a/crates/snowbound/src/watch.rs b/crates/snowbound/src/watch.rs new file mode 100644 index 0000000000000000000000000000000000000000..d6d95ad4c2e099a3a16d61a22f2a003dd66814fe --- /dev/null +++ b/crates/snowbound/src/watch.rs @@ -0,0 +1,441 @@ +//! Changes below a notebook folder on this computer, as the system reports them: FSEvents on +//! macOS, inotify on Linux. A notebook on a share learns of its changes from the server +//! instead (`notebook::session::Background::smb`). + +use std::path::Path; + +/// Reports until dropped. +pub struct Watch { + _stream: platform::Stream, +} + +/// Watches the folder at `root`: `changed` hears the paths that changed below it, relative to +/// it and `/`-separated, or `""` when the system can only say something did. `None` where the +/// system cannot watch it. +pub fn watch(root: &Path, changed: impl Fn(Vec) + Send + Sync + 'static) -> Option { + let root = root.canonicalize().ok()?; + let folder = root.clone(); + let relative = move |paths: Vec| { + changed( + paths + .iter() + .map(|path| match path.strip_prefix(&folder) { + Ok(path) => path.to_string_lossy().replace('\\', "/"), + Err(_) => String::new(), + }) + .collect(), + ) + }; + platform::Stream::start(&root, Box::new(relative)) + .inspect_err(|error| eprintln!("{}: changes go unwatched: {error}", root.display())) + .ok() + .map(|stream| Watch { _stream: stream }) +} + +/// Whether the folder at `path` is on this computer's own disks, where every change to it is +/// one the system reports, rather than on a network volume. +pub fn on_this_computer(path: &Path) -> bool { + use std::os::unix::ffi::OsStrExt; + let Ok(name) = std::ffi::CString::new(path.as_os_str().as_bytes()) else { + return true; + }; + let mut volume: libc::statfs = unsafe { std::mem::zeroed() }; + if unsafe { libc::statfs(name.as_ptr(), &mut volume) } != 0 { + return true; + } + #[cfg(target_os = "macos")] + return volume.f_flags & libc::MNT_LOCAL as u32 != 0; + // NFS, SMB, CIFS, SMB2, FUSE (sshfs and the like), 9P, Ceph, AFS. + #[cfg(target_os = "linux")] + return ![ + 0x6969, + 0x517b, + 0xff53_4d42, + 0xfe53_4d42, + 0x6573_5546, + 0x0102_1997, + 0x00c3_6400, + 0x5346_414f, + ] + .contains(&(volume.f_type as u64)); +} + +type Changed = Box) + Send + Sync>; + +#[cfg(target_os = "macos")] +mod platform { + use super::Changed; + use std::{ + ffi::{CStr, c_char, c_void}, + io, + os::unix::ffi::OsStrExt, + path::{Path, PathBuf}, + }; + + type Stream_ = *mut c_void; + + #[repr(C)] + struct Context { + version: isize, + info: *mut c_void, + retain: *const c_void, + release: Option, + description: *const c_void, + } + + type Callback = + unsafe extern "C" fn(Stream_, *mut c_void, usize, *mut c_void, *const u32, *const u64); + + #[link(name = "CoreServices", kind = "framework")] + unsafe extern "C" { + fn FSEventStreamCreate( + allocator: *const c_void, + callback: Callback, + context: *const Context, + paths: *const c_void, + since: u64, + latency: f64, + flags: u32, + ) -> Stream_; + fn FSEventStreamSetDispatchQueue(stream: Stream_, queue: *mut c_void); + fn FSEventStreamStart(stream: Stream_) -> u8; + fn FSEventStreamStop(stream: Stream_); + fn FSEventStreamInvalidate(stream: Stream_); + fn FSEventStreamRelease(stream: Stream_); + } + + #[link(name = "CoreFoundation", kind = "framework")] + unsafe extern "C" { + static kCFTypeArrayCallBacks: c_void; + fn CFStringCreateWithBytes( + allocator: *const c_void, + bytes: *const u8, + length: isize, + encoding: u32, + external: u8, + ) -> *const c_void; + fn CFArrayCreate( + allocator: *const c_void, + values: *const *const c_void, + count: isize, + callbacks: *const c_void, + ) -> *const c_void; + fn CFRelease(object: *const c_void); + } + + unsafe extern "C" { + fn dispatch_queue_create(label: *const c_char, attributes: *const c_void) -> *mut c_void; + fn dispatch_sync_f( + queue: *mut c_void, + context: *mut c_void, + work: extern "C" fn(*mut c_void), + ); + fn dispatch_release(object: *mut c_void); + } + + const SINCE_NOW: u64 = u64::MAX; + const NO_DEFER: u32 = 0x2; + const WATCH_ROOT: u32 = 0x4; + /// Mac OS X 10.7 and later; 10.6 reports the folders that changed. + const FILE_EVENTS: u32 = 0x10; + const UTF8: u32 = 0x0800_0100; + + pub struct Stream { + stream: Stream_, + queue: *mut c_void, + } + + // The stream and queue are only started and torn down, each call safe from any thread. + unsafe impl Send for Stream {} + unsafe impl Sync for Stream {} + + unsafe extern "C" fn events( + _: Stream_, + info: *mut c_void, + count: usize, + paths: *mut c_void, + _: *const u32, + _: *const u64, + ) { + let changed = unsafe { &*(info as *const Changed) }; + let paths = unsafe { std::slice::from_raw_parts(paths as *const *const c_char, count) }; + changed( + paths + .iter() + .map(|&path| { + let path = unsafe { CStr::from_ptr(path) }; + PathBuf::from(std::ffi::OsStr::from_bytes(path.to_bytes())) + }) + .collect(), + ); + } + + unsafe extern "C" fn release(info: *const c_void) { + drop(unsafe { Box::from_raw(info as *mut Changed) }); + } + + extern "C" fn drained(_: *mut c_void) {} + + /// Whether this is Mac OS X 10.7 or later (Darwin 11). + fn file_events() -> bool { + let mut name: libc::utsname = unsafe { std::mem::zeroed() }; + (unsafe { libc::uname(&mut name) }) == 0 + && unsafe { CStr::from_ptr(name.release.as_ptr()) } + .to_str() + .ok() + .and_then(|release| release.split('.').next()?.parse::().ok()) + .is_some_and(|major| major >= 11) + } + + impl Stream { + pub fn start(root: &Path, changed: Changed) -> io::Result { + let root = root.as_os_str().as_bytes(); + let flags = NO_DEFER | WATCH_ROOT | if file_events() { FILE_EVENTS } else { 0 }; + unsafe { + let path = CFStringCreateWithBytes( + std::ptr::null(), + root.as_ptr(), + root.len() as isize, + UTF8, + 0, + ); + let paths = CFArrayCreate(std::ptr::null(), &path, 1, &kCFTypeArrayCallBacks); + CFRelease(path); + let context = Context { + version: 0, + info: Box::into_raw(Box::new(changed)).cast(), + retain: std::ptr::null(), + release: Some(release), + description: std::ptr::null(), + }; + let stream = FSEventStreamCreate( + std::ptr::null(), + events, + &context, + paths, + SINCE_NOW, + 0.3, + flags, + ); + CFRelease(paths); + if stream.is_null() { + release(context.info); + return Err(io::Error::other("FSEvents refused the folder")); + } + let queue = dispatch_queue_create(c"snowbound.watch".as_ptr(), std::ptr::null()); + FSEventStreamSetDispatchQueue(stream, queue); + let stream = Self { stream, queue }; + if FSEventStreamStart(stream.stream) == 0 { + return Err(io::Error::other("FSEvents did not start")); + } + Ok(stream) + } + } + } + + impl Drop for Stream { + fn drop(&mut self) { + unsafe { + FSEventStreamStop(self.stream); + FSEventStreamInvalidate(self.stream); + // A callback already on the queue finishes before the closure goes. + dispatch_sync_f(self.queue, std::ptr::null_mut(), drained); + FSEventStreamRelease(self.stream); + dispatch_release(self.queue); + } + } + } +} + +#[cfg(target_os = "linux")] +mod platform { + use super::Changed; + use std::{ + collections::HashMap, + ffi::CString, + io, + os::unix::ffi::OsStrExt, + path::{Path, PathBuf}, + thread::JoinHandle, + }; + + const MASK: u32 = libc::IN_MODIFY + | libc::IN_CLOSE_WRITE + | libc::IN_ATTRIB + | libc::IN_CREATE + | libc::IN_DELETE + | libc::IN_MOVED_FROM + | libc::IN_MOVED_TO + | libc::IN_DELETE_SELF + | libc::IN_MOVE_SELF; + + /// inotify watches one folder at a time, so every folder below the root has its own. + pub struct Stream { + inotify: i32, + stop: i32, + thread: Option>, + } + + fn check(result: i32) -> io::Result { + if result < 0 { + Err(io::Error::last_os_error()) + } else { + Ok(result) + } + } + + fn add(inotify: i32, folder: &Path, folders: &mut HashMap) { + let Ok(name) = CString::new(folder.as_os_str().as_bytes()) else { + return; + }; + if let Ok(watch) = check(unsafe { libc::inotify_add_watch(inotify, name.as_ptr(), MASK) }) { + folders.insert(watch, folder.to_owned()); + } + for entry in std::fs::read_dir(folder).into_iter().flatten().flatten() { + if entry.file_type().is_ok_and(|kind| kind.is_dir()) { + add(inotify, &entry.path(), folders); + } + } + } + + impl Stream { + pub fn start(root: &Path, changed: Changed) -> io::Result { + let inotify = check(unsafe { libc::inotify_init1(libc::IN_CLOEXEC) })?; + let stop = match check(unsafe { libc::eventfd(0, libc::EFD_CLOEXEC) }) { + Ok(stop) => stop, + Err(error) => { + unsafe { libc::close(inotify) }; + return Err(error); + } + }; + let mut folders = HashMap::new(); + add(inotify, root, &mut folders); + let root = root.to_owned(); + let thread = std::thread::Builder::new() + .name("snowbound-watch".into()) + .spawn(move || { + let mut buffer = vec![0u8; 64 * 1024]; + loop { + let mut polled = [ + libc::pollfd { + fd: inotify, + events: libc::POLLIN, + revents: 0, + }, + libc::pollfd { + fd: stop, + events: libc::POLLIN, + revents: 0, + }, + ]; + if unsafe { libc::poll(polled.as_mut_ptr(), 2, -1) } < 0 { + if io::Error::last_os_error().kind() == io::ErrorKind::Interrupted { + continue; + } + return; + } + if polled[1].revents != 0 { + return; + } + let read = unsafe { + libc::read(inotify, buffer.as_mut_ptr().cast(), buffer.len()) + }; + let Ok(read) = usize::try_from(read) else { + return; + }; + let mut paths = Vec::new(); + let mut at = 0; + while at + std::mem::size_of::() <= read { + let event: libc::inotify_event = + unsafe { std::ptr::read_unaligned(buffer[at..].as_ptr().cast()) }; + let start = at + std::mem::size_of::(); + let end = (start + event.len as usize).min(read); + at = end; + if event.mask & libc::IN_Q_OVERFLOW != 0 { + paths.push(root.clone()); + continue; + } + let Some(folder) = folders.get(&event.wd).cloned() else { + continue; + }; + let name = &buffer[start..end]; + let name = + &name[..name.iter().position(|&b| b == 0).unwrap_or(name.len())]; + let path = if name.is_empty() { + folder + } else { + folder.join(std::ffi::OsStr::from_bytes(name)) + }; + if event.mask & libc::IN_ISDIR != 0 + && event.mask & (libc::IN_CREATE | libc::IN_MOVED_TO) != 0 + { + add(inotify, &path, &mut folders); + } + if event.mask & libc::IN_IGNORED != 0 { + folders.remove(&event.wd); + } + paths.push(path); + } + if !paths.is_empty() { + changed(paths); + } + } + })?; + Ok(Self { + inotify, + stop, + thread: Some(thread), + }) + } + } + + impl Drop for Stream { + fn drop(&mut self) { + unsafe { libc::write(self.stop, (&1u64 as *const u64).cast(), 8) }; + if let Some(thread) = self.thread.take() { + let _ = thread.join(); + } + unsafe { + libc::close(self.inotify); + libc::close(self.stop); + } + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::{sync::mpsc, time::Duration}; + + #[test] + fn a_folder_on_this_computer_reports_the_file_that_changed() { + let folder = std::env::temp_dir().join(format!("snowbound-watch-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&folder); + std::fs::create_dir_all(folder.join("Group")).unwrap(); + std::fs::write(folder.join("Group/Section.one"), b"before").unwrap(); + assert!(on_this_computer(&folder)); + let (sender, changes) = mpsc::channel(); + let sender = std::sync::Mutex::new(sender); + let watching = watch(&folder, move |paths| { + let _ = sender.lock().unwrap().send(paths); + }) + .unwrap(); + // FSEvents starts reporting a moment after the stream starts. + std::thread::sleep(Duration::from_millis(500)); + std::fs::write(folder.join("Group/Section.one"), b"after").unwrap(); + let mut reported = Vec::new(); + while let Ok(paths) = changes.recv_timeout(Duration::from_secs(5)) { + reported.extend(paths); + if reported.iter().any(|path| path == "Group/Section.one") { + break; + } + } + drop(watching); + std::fs::remove_dir_all(&folder).unwrap(); + assert!( + reported.iter().any(|path| path == "Group/Section.one"), + "{reported:?}" + ); + } +} diff --git a/crates/ui/src/popup.rs b/crates/ui/src/popup.rs index 10c173f9c128fedcae9fd22afc2cea9bd8cbbfad..577dd9450f4a640d90450c2b9b1c28d2bb8f6553 100644 --- a/crates/ui/src/popup.rs +++ b/crates/ui/src/popup.rs @@ -138,6 +138,22 @@ pub fn menu( /// delay, or at once while another has just shown. A press or the wheel hides it until the /// pointer leaves. pub fn tooltip(ui: &mut Ui, title: &str, keys: &str, description: Option<&str>) { + tooltip_below(ui, None, title, keys, description); +} + +/// A `tooltip` below `part`, a window rectangle within the box built last, as a custom +/// box names what it draws. +pub fn tooltip_over(ui: &mut Ui, part: [f32; 4], title: &str, description: Option<&str>) { + tooltip_below(ui, Some(part), title, "", description); +} + +fn tooltip_below( + ui: &mut Ui, + part: Option<[f32; 4]>, + title: &str, + keys: &str, + description: Option<&str>, +) { let Some(&index) = ui.nodes[*ui.stack.last().unwrap()].children.last() else { return; }; @@ -183,7 +199,7 @@ pub fn tooltip(ui: &mut Ui, title: &str, keys: &str, description: Option<&str>) } else { format!("{title} ({keys})") }; - let [left, top, right, bottom] = ui.rect(id).unwrap_or_default(); + let [left, top, right, bottom] = part.or(ui.rect(id)).unwrap_or_default(); let theme = &ui.theme; let spec = Spec { axis: Axis::Y, diff --git a/tools/test_task_tags.py b/tools/test_task_tags.py new file mode 100644 index 0000000000000000000000000000000000000000..5fc8cc24949ac95e4f4f4f5cbfd786532cf23a2b --- /dev/null +++ b/tools/test_task_tags.py @@ -0,0 +1,52 @@ +from pathlib import Path +import re +import runpy +import shutil +from tempfile import TemporaryDirectory +import unittest +import xml.etree.ElementTree as ET + +from native_xml import ns + +ROOT = Path(__file__).resolve().parent.parent +FIXTURE = ROOT / 'corpus/task-tags' +compare = runpy.run_path(str(ROOT / 'tools/verify-document.py'))['compare'] + + +def tasks(read): + """Each page's paragraphs by title: their text, task identities and normal tag count.""" + result = {} + for path in sorted(read.glob('page-*.xml')): + page = ET.parse(path).getroot() + title = page.find('one:Title/one:OE/one:T', ns) + if title is None: + continue + result[title.text] = [(re.sub(r'<[^>]*>', '', oe.find('one:T', ns).text or ''), + [(t.get('guidTask'), t.get('disabled'), t.get('startDate'), t.get('dueDate')) + for t in oe.findall('one:OutlookTask', ns)], + len(oe.findall('one:Tag', ns))) + for oe in page.find('one:Outline', ns).iter('{%s}OE' % ns['one'])] + return result + + +class TaskTagsTest(unittest.TestCase): + def test_removed_and_restored_task_tags_reopen_natively(self): + with TemporaryDirectory() as temporary: + read = Path(temporary) / 'read' + shutil.copytree(FIXTURE / 'cold/read', read) + compare(FIXTURE / 'candidate', read) + native = tasks(FIXTURE / 'native/read') + cold = tasks(FIXTURE / 'cold/read') + self.assertEqual(cold['Delete and undo'], native['Delete and undo']) + removed = native['Remove a task'] + self.assertEqual([len(t) for _, t, _ in removed], [1, 1, 1, 0]) + self.assertEqual(cold['Remove a task'], [ + (removed[0][0], [], 0), + removed[1], + (removed[2][0], removed[2][1], 0), + removed[3], + ]) + + +if __name__ == '__main__': + unittest.main()