authorgravatar for git@paperclover.netclover caruso <git@paperclover.net> 2026-09-12 17:52:17-07:00
committergravatar for git@paperclover.netclover caruso <git@paperclover.net> 2026-09-25 20:26:20-07:00
logdb49720208e16a932d4c13ebe1172ab872638ac7
treedb0d588f0b60bfdbc452233f7aca5c66ea8ab638
parent237d4668ad21d8c6275da60e59bc4acb066fb853
signature Signed by SSH key SHA256:52mNGHRsVFBDED9IAX5pe+LRWUefqTbxEReunq21QvU

feat: parse equations into a tree that renders the MathML OneNote exports

An equation is stored as linear text where U+FDD0 opens an inline object, U+FDEE separates its arguments and U+FDEF closes it, with the object's kind (the text object model's OBJECTTYPE: 31 superscript, 16 fraction, …), argument count and building symbols on the text-run data of the opening character. The reader now carries that data on each run's format as Format::math_object, and page::Math parses a paragraph into a tree of identifiers, numbers, operators and objects and renders it as MathML in the form OneNote's own export uses (mml: prefix, one mi per letter, math-italic Latin letters plain, other alphabets as stored, single-node arguments bare and multi-node ones in mrow). corpus/m6/native-math-01 is the oracle: its three equation-editor expressions render byte for byte to the MathML the COM read exported (superscripts, a fraction and inline Greek). Subscripts, radicals, limits and brackets follow the OBJECTTYPE meaning without a native check yet. Assisted-by: claude-fable-5.1

5 files changed, 346 insertions(+), 4 deletions(-)

crates/onestore/README.md+3-3
...@@ -43,7 +43,7 @@ harness also accepts `--client-profile release`....@@ -43,7 +43,7 @@ harness also accepts `--client-profile release`.
43| `Object::file_reference`, `Store::file_data` | Identify internal/external payloads and read internal payload bytes |43| `Object::file_reference`, `Store::file_data` | Identify internal/external payloads and read internal payload bytes |
44| `document::Document`, `Revision::text_runs` | Interpret document objects and inherited text formatting while retaining unknown properties and revision identities |44| `document::Document`, `Revision::text_runs` | Interpret document objects and inherited text formatting while retaining unknown properties and revision identities |
45| `Document::active`, `Document::pages_in`, `Revision::parents`, `RevisionIndex::active` | Resolve the active revision, the pages of a space and parent links without repeating the lookups |45| `Document::active`, `Document::pages_in`, `Revision::parents`, `RevisionIndex::active` | Resolve the active revision, the pages of a space and parent links without repeating the lookups |
46| `page::Page`, `page::Paragraph`, `page::Ink` | Build an editable page model (title, outlines, paragraphs with coalesced text spans, tables, images, attachments, ink drawings and handwriting decoded to stroke polylines in page points) with stored identities; content outside the model is retained as `Unsupported` |46| `page::Page`, `page::Paragraph`, `page::Ink`, `page::Math` | Build an editable page model (title, outlines, paragraphs with coalesced text spans, tables, images, attachments, ink drawings and handwriting decoded to stroke polylines in page points) with stored identities; equations parse from their linear text and run data into a tree that renders the MathML OneNote exports; content outside the model is retained as `Unsupported` |
47| `protected::UnlockedSection` (optional feature) | Own decoded buffers for explicit known-password inspection; clear those buffers on drop; derived document strings/exports remain caller-owned |47| `protected::UnlockedSection` (optional feature) | Own decoded buffers for explicit known-password inspection; clear those buffers on drop; derived document strings/exports remain caller-owned |
48| `create_section` | Create one page containing one plain-text paragraph and an author, including Unicode |48| `create_section` | Create one page containing one plain-text paragraph and an author, including Unicode |
49| `PageCreation`, `PreparedEdit::create_page` | Add an empty top-level page and its section entry atomically, retaining page identities across retries |49| `PageCreation`, `PreparedEdit::create_page` | Add an empty top-level page and its section entry atomically, retaining page identities across retries |
...@@ -174,8 +174,8 @@ cargo run --example random_edit -- /tmp/one-demo/synthetic.one /tmp/edited.one 4...@@ -174,8 +174,8 @@ cargo run --example random_edit -- /tmp/one-demo/synthetic.one /tmp/edited.one 4
174The report contains readable pages, document JSON, assets, source identities and174The report contains readable pages, document JSON, assets, source identities and
175coordinates. It preserves paragraph nesting, lists, tables, links and tags.175coordinates. It preserves paragraph nesting, lists, tables, links and tags.
176Historical contexts, recycle-bin pages and default templates are represented176Historical contexts, recycle-bin pages and default templates are represented
177separately. Native ink is decoded to strokes; structured equations retain their177separately. Native ink is decoded to strokes and equations to MathML; both
178source data and appear explicitly as opaque content. The report is a reading view; its native178retain their source data. The report is a reading view; its native
179PDF references supply the original canvas layout.179PDF references supply the original canvas layout.
180180
181Read and validate a snapshot before interpreting its graph:181Read and validate a snapshot before interpreting its graph:
crates/onestore/src/document.rs+48-1
...@@ -125,6 +125,18 @@ format_fields! {...@@ -125,6 +125,18 @@ format_fields! {
125 space_after: f32,125 space_after: f32,
126 line_spacing: f32,126 line_spacing: f32,
127 list_spacing: f32,127 list_spacing: f32,
128 math_object: MathObject,
129}
130
131/// The inline math object a run belongs to, from its text-run data: `kind` follows the
132/// `OBJECTTYPE` values of the Windows text object model (31 superscript, 16 fraction, …),
133/// `arguments` counts the object's arguments on its opening run, and `symbols` are the
134/// characters the object builds with (a bracket pair, an n-ary operator, `^` or `/`).
135#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
136pub struct MathObject {
137 pub kind: u32,
138 pub arguments: Option<u32>,
139 pub symbols: Vec<char>,
128}140}
129141
130#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]142#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
...@@ -194,12 +206,46 @@ impl Revision<'_> {...@@ -194,12 +206,46 @@ impl Revision<'_> {
194 return Err(invalid("Text-run boundary splits a surrogate pair"));206 return Err(invalid("Text-run boundary splits a surrogate pair"));
195 }207 }
196 let fragment = &text[start..byte];208 let fragment = &text[start..byte];
197 let format = run209 let mut format = run
198 .format210 .format
199 .map(style)211 .map(style)
200 .transpose()?212 .transpose()?
201 .unwrap_or(&default)213 .unwrap_or(&default)
202 .inherit(&base);214 .inherit(&base);
215 if let Some(set) = run.extra_set.and_then(|index| node.extra.get(index)) {
216 let scalar = |id: u32| {
217 set.iter().find_map(|field| match field.value {
218 FieldValue::Bytes(bytes) if field.id == id => Some(bytes),
219 _ => None,
220 })
221 };
222 if let Some(kind) = scalar(0x1400344f) {
223 format.math_object = Some(MathObject {
224 kind: u32::from_le_bytes(
225 kind.try_into()
226 .map_err(|_| invalid("Math object type has an invalid length"))?,
227 ),
228 arguments: scalar(0x14003450)
229 .map(|b| {
230 b.try_into().map(u32::from_le_bytes).map_err(|_| {
231 invalid("Math argument count has an invalid length")
232 })
233 })
234 .transpose()?,
235 symbols: [0x10003453, 0x10003454, 0x10003455]
236 .into_iter()
237 .filter_map(scalar)
238 .map(|b| {
239 b.try_into()
240 .ok()
241 .map(u16::from_le_bytes)
242 .and_then(|unit| char::from_u32(u32::from(unit)))
243 .ok_or_else(|| invalid("Math symbol is not a character"))
244 })
245 .collect::<Result<_>>()?,
246 });
247 }
248 }
203 let mut link = None;249 let mut link = None;
204 if format.hyperlink != Some(true) {250 if format.hyperlink != Some(true) {
205 target = None;251 target = None;
...@@ -943,6 +989,7 @@ impl<'a> Element<'a> {...@@ -943,6 +989,7 @@ impl<'a> Element<'a> {
943 space_after: f.float(0x1400342f, 36.0)?,989 space_after: f.float(0x1400342f, 36.0)?,
944 line_spacing: f.float(0x14003430, 36.0)?,990 line_spacing: f.float(0x14003430, 36.0)?,
945 list_spacing: f.float(0x14001ccb, 36.0)?,991 list_spacing: f.float(0x14001ccb, 36.0)?,
992 math_object: None,
946 };993 };
947 let mut tags = Vec::new();994 let mut tags = Vec::new();
948 if let Some(property) = f.take(0x40003489) {995 if let Some(property) = f.take(0x40003489) {
crates/onestore/src/page/math.rs created+217
...@@ -0,0 +1,217 @@
1//! Equations as OneNote stores them: a linear text where U+FDD0 opens an inline object,
2//! U+FDEE separates its arguments and U+FDEF closes it, with the object's kind on the run
3//! data of the opening character. `Math::parse` builds the tree and `mathml` renders it the
4//! way OneNote's own export does for the kinds a native fixture has verified (superscript
5//! and fraction); other kinds follow the text object model's meaning without a native check.
6
7use super::text::Paragraph;
8use crate::document::MathObject;
9
10#[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)]
11pub enum Math {
12 Identifier(char),
13 Number(String),
14 Operator(char),
15 Object {
16 kind: u32,
17 symbols: Vec<char>,
18 arguments: Vec<Vec<Math>>,
19 },
20}
21
22const OBJECT_START: char = '\u{fdd0}';
23const ARGUMENT_SEPARATOR: char = '\u{fdee}';
24const OBJECT_END: char = '\u{fdef}';
25
26impl Math {
27 /// Whether a paragraph's text is stored as an equation.
28 pub fn is_equation(paragraph: &Paragraph) -> bool {
29 paragraph.text().contains(OBJECT_START)
30 || paragraph
31 .spans()
32 .iter()
33 .any(|span| span.format.math == Some(true))
34 }
35
36 /// The equation a paragraph's text holds, as a sequence of nodes.
37 pub fn parse(paragraph: &Paragraph) -> Result<Vec<Math>, crate::Error> {
38 let invalid = |message| crate::Error { offset: 0, message };
39 let object_at = |offset: usize| -> Option<&MathObject> {
40 paragraph
41 .format_at(u32::try_from(offset).ok()?)
42 .ok()
43 .and_then(|f| f.math_object.as_ref())
44 };
45 struct Frame {
46 kind: u32,
47 symbols: Vec<char>,
48 arguments: Vec<Vec<Math>>,
49 }
50 let mut frames: Vec<Frame> = Vec::new();
51 let mut sequences: Vec<Vec<Math>> = vec![Vec::new()];
52 let mut number = String::new();
53 let mut offset = 0;
54 for character in paragraph.text().chars() {
55 let here = offset;
56 offset += character.len_utf16();
57 if character.is_ascii_digit() {
58 number.push(character);
59 continue;
60 }
61 if !number.is_empty() {
62 let number = std::mem::take(&mut number);
63 sequences.last_mut().unwrap().push(Math::Number(number));
64 }
65 match character {
66 OBJECT_START => {
67 // The span lookup treats a boundary offset as the span ending there, so the
68 // opening character's own span is the one containing the offset after it.
69 let object = object_at(here + 1)
70 .ok_or_else(|| invalid("An equation object has no run data"))?;
71 frames.push(Frame {
72 kind: object.kind,
73 symbols: object.symbols.clone(),
74 arguments: Vec::new(),
75 });
76 sequences.push(Vec::new());
77 }
78 ARGUMENT_SEPARATOR => {
79 let argument = sequences.pop().unwrap();
80 frames
81 .last_mut()
82 .ok_or_else(|| invalid("An equation argument separator has no object"))?
83 .arguments
84 .push(argument);
85 sequences.push(Vec::new());
86 }
87 OBJECT_END => {
88 let argument = sequences.pop().unwrap();
89 let mut frame = frames
90 .pop()
91 .ok_or_else(|| invalid("An equation object ends before it starts"))?;
92 frame.arguments.push(argument);
93 sequences.last_mut().unwrap().push(Math::Object {
94 kind: frame.kind,
95 symbols: frame.symbols,
96 arguments: frame.arguments,
97 });
98 }
99 c if c.is_whitespace() => {}
100 c if c.is_alphabetic() => sequences
101 .last_mut()
102 .unwrap()
103 .push(Math::Identifier(plain(c))),
104 c => sequences.last_mut().unwrap().push(Math::Operator(c)),
105 }
106 }
107 if !number.is_empty() {
108 sequences.last_mut().unwrap().push(Math::Number(number));
109 }
110 if !frames.is_empty() {
111 return Err(invalid("An equation object never ends"));
112 }
113 Ok(sequences.pop().unwrap())
114 }
115
116 /// MathML for a node sequence, without the `math` wrapper, in OneNote's export form
117 /// (`mml:` prefixed elements, one `mi` per letter).
118 pub fn mathml(nodes: &[Math]) -> String {
119 let mut out = String::new();
120 for node in nodes {
121 node.write(&mut out);
122 }
123 out
124 }
125
126 fn write(&self, out: &mut String) {
127 match self {
128 Math::Identifier(c) => tag(out, "mi", &c.to_string()),
129 Math::Number(n) => tag(out, "mn", n),
130 Math::Operator(c) => tag(out, "mo", &c.to_string()),
131 Math::Object {
132 kind,
133 symbols,
134 arguments,
135 } => {
136 let argument = |out: &mut String, index: usize| match arguments
137 .get(index)
138 .map(Vec::as_slice)
139 {
140 Some([single]) => single.write(out),
141 Some(many) => {
142 out.push_str("<mml:mrow>");
143 for node in many {
144 node.write(out);
145 }
146 out.push_str("</mml:mrow>");
147 }
148 None => out.push_str("<mml:mrow/>"),
149 };
150 let wrapped = |out: &mut String, element: &str, order: &[usize]| {
151 out.push_str(&format!("<mml:{element}>"));
152 for index in order {
153 argument(out, *index);
154 }
155 out.push_str(&format!("</mml:{element}>"));
156 };
157 match (kind, arguments.len()) {
158 (31, 2) => wrapped(out, "msup", &[0, 1]),
159 (29, 2) => wrapped(out, "msub", &[0, 1]),
160 (30, 3) => wrapped(out, "msubsup", &[0, 1, 2]),
161 (16, 2) | (26, 2) => wrapped(out, "mfrac", &[0, 1]),
162 (25, 2) if !arguments[0].is_empty() => wrapped(out, "mroot", &[1, 0]),
163 (25, _) => wrapped(out, "msqrt", &[arguments.len() - 1]),
164 (19, 2) => wrapped(out, "munder", &[0, 1]),
165 (33, 2) => wrapped(out, "mover", &[0, 1]),
166 (13, _) => {
167 out.push_str("<mml:mrow>");
168 if let Some(open) = symbols.first() {
169 tag(out, "mo", &open.to_string());
170 }
171 for index in 0..arguments.len() {
172 argument(out, index);
173 }
174 if let Some(close) = symbols.get(1) {
175 tag(out, "mo", &close.to_string());
176 }
177 out.push_str("</mml:mrow>");
178 }
179 _ => {
180 out.push_str("<mml:mrow>");
181 for symbol in symbols {
182 tag(out, "mo", &symbol.to_string());
183 }
184 for index in 0..arguments.len() {
185 argument(out, index);
186 }
187 out.push_str("</mml:mrow>");
188 }
189 }
190 }
191 }
192 }
193}
194
195fn tag(out: &mut String, element: &str, content: &str) {
196 out.push_str(&format!("<mml:{element}>"));
197 for c in content.chars() {
198 match c {
199 '<' => out.push_str("&lt;"),
200 '&' => out.push_str("&amp;"),
201 '>' => out.push_str("&gt;"),
202 c => out.push(c),
203 }
204 }
205 out.push_str(&format!("</mml:{element}>"));
206}
207
208/// OneNote exports mathematical italic Latin letters as their plain letters and leaves every
209/// other alphabet (Greek, double-struck, …) as stored.
210fn plain(c: char) -> char {
211 match u32::from(c) {
212 code @ 0x1d434..=0x1d44d => char::from_u32(code - 0x1d434 + u32::from('A')).unwrap(),
213 code @ 0x1d44e..=0x1d467 => char::from_u32(code - 0x1d44e + u32::from('a')).unwrap(),
214 0x210e => 'h',
215 _ => c,
216 }
217}
crates/onestore/src/page/mod.rs+2
...@@ -12,9 +12,11 @@ use std::{...@@ -12,9 +12,11 @@ use std::{
1212
13pub mod ink;13pub mod ink;
14pub mod link;14pub mod link;
15pub mod math;
15pub mod text;16pub mod text;
1617
17pub use ink::{Ink, InkStroke};18pub use ink::{Ink, InkStroke};
19pub use math::Math;
18pub(crate) mod write;20pub(crate) mod write;
19pub use text::Paragraph;21pub use text::Paragraph;
2022
crates/onestore/tests/page_math.rs created+76
...@@ -0,0 +1,76 @@
1use onestore::{
2 RevisionIndex, Store,
3 document::Document,
4 page::{Math, Page, PageObject},
5};
6
7/// Three equations authored through OneNote 2010's equation editor, whose COM read exports
8/// MathML (`corpus/m6/native-math-01`).
9const NATIVE: &[u8] = include_bytes!("../../../corpus/m6/native-math-01/notebook/synthetic.one");
10const NATIVE_READ: &str = include_str!("../../../corpus/m6/native-math-01/read/page-000.xml");
11
12fn native_mathml() -> Vec<String> {
13 NATIVE_READ
14 .split("<mml:math")
15 .skip(1)
16 .map(|rest| {
17 let body = &rest[rest.find('>').unwrap() + 1..rest.find("</mml:math>").unwrap()];
18 let mut decoded = String::new();
19 let mut remainder = body;
20 while let Some(start) = remainder.find("&#") {
21 decoded.push_str(&remainder[..start]);
22 let end = start + remainder[start..].find(';').unwrap();
23 let code: u32 = remainder[start + 2..end].parse().unwrap();
24 decoded.push(char::from_u32(code).unwrap());
25 remainder = &remainder[end + 1..];
26 }
27 decoded.push_str(remainder);
28 decoded
29 })
30 .collect()
31}
32
33#[test]
34fn native_equations_render_to_the_mathml_onenote_exports() {
35 let store = Store::parse(NATIVE).unwrap();
36 let index = RevisionIndex::parse(&store).unwrap();
37 let document = Document::parse(&index).unwrap();
38 let page = document
39 .pages()
40 .unwrap()
41 .into_iter()
42 .map(|(space, _)| Page::from_space(&document, space).unwrap())
43 .find(|page| page.title == "Equation controls")
44 .unwrap();
45 let mut rendered = Vec::new();
46 for object in &page.objects {
47 let PageObject::Outline(outline) = object else {
48 continue;
49 };
50 for paragraph in &outline.paragraphs {
51 let Some(text) = paragraph.text() else {
52 continue;
53 };
54 if !Math::is_equation(&text.text) {
55 continue;
56 }
57 // Inline math follows ordinary text in the same paragraph; only the math runs
58 // form the equation.
59 let mut at = 0;
60 let math_runs: Vec<(String, onestore::document::Format)> = text
61 .text
62 .spans()
63 .iter()
64 .map(|span| {
65 let piece = text.text.text()[at..span.end].to_owned();
66 at = span.end;
67 (piece, span.format.clone())
68 })
69 .filter(|(_, format)| format.math == Some(true))
70 .collect();
71 let equation = onestore::page::Paragraph::from_runs(math_runs);
72 rendered.push(Math::mathml(&Math::parse(&equation).unwrap()));
73 }
74 }
75 assert_eq!(rendered, native_mathml());
76}