| author | |
| committer | |
| log | d2b738a4ff1796471a1f7581f203e652392fc309 |
| tree | 4f49e7e6a4a645a74c64b567e4cf129848d60768 |
| parent | 132fa1379f8ba53b504a5bdda15a9cc58d5f44cd |
| signature | Signed by SSH key SHA256:52mNGHRsVFBDED9IAX5pe+LRWUefqTbxEReunq21QvU |
After a failed COMMIT, a cold reopen decides whether the commit landed. When
that reopen fails for now (too many open files, a busy cache, memory
pressure), the cache stays closed instead of failing the section: the edits
wait in memory, the reopen is retried once a second however busy the section
is, and the commit is decided once it succeeds, so its edits land exactly
once.
The closed cache stays owned. The warm connection, never used again, keeps
SQLite's lock until the replica holds <cache>-lock and lists the cache file
for the process in its stead, and every cache open fails busy while either is
held; the check runs once the opener holds SQLite's lock, so no second owner
can publish and acknowledge the undecided commit and have it queued again. A
replica dropped meanwhile keeps retrying and keeps the cache owned.
An edit a caller blocks on is refused at once, as under #136. The reread that
refusal needs waits for the cache: page reads answer from the section as it
was, later edits wait to apply to the reread section, and once the cache
reopens only the closed commit is decided before the edits apply again, as
#136 does, so an edit on a refused page is refused rather than queued. The
session reports the section's pages as changed after such a reread.
Meanwhile reads of the cache answer WouldBlock and the sync worker carries on;
sync_status reports the edits queued and waiting and a failure instead of
erring, and the desktop counts unsaved changes through it. Only a cache file
that is gone, or no longer a database, is lost: every waiting edit is refused
with the error and the section stays closed. A cold open that comes back
read-only is retried.
- Edits made while Snowbound cannot reopen its cache after a failed save now keep waiting and save once it can.
Fixes #148
Assisted-by: claude-opus-5.59 files changed, 1301 insertions(+), 334 deletions(-)
crates/notebook/README.md+9-3| ... | ... | @@ -53,11 +53,17 @@ connection holds exclusive ownership between transactions, and a second open fai |
| 53 | 53 | busy. No network wait occurs in a local edit. An edit a session submitted that the cache |
| 54 | 54 | cannot write for now (a full disk, an I/O error) stays applied in memory and is written |
| 55 | 55 | again, in order, each second; meanwhile `written` returns `WouldBlock`. A replica dropped |
| 56 | meanwhile keeps its cache open and its edits retrying until they are written or the cache is | |
| 56 | meanwhile keeps its cache owned and its edits retrying until they are written or the cache is | |
| 57 | 57 | deleted, and only then releases it; `notebook::unwritten()` counts the edits waiting in the |
| 58 | 58 | process, for a host to ask before it quits. After a commit fails, a fresh connection |
| 59 | decides whether it landed. An edit the cache could never write is refused alone, and the | |
| 60 | section thread rereads the queue. | |
| 59 | decides whether it landed. Until the cache reopens (too many open files, a busy cache), the | |
| 60 | edits wait the same way, reads of the cache answer `WouldBlock`, and the cache stays owned: | |
| 61 | the replica locks `<cache>-lock` and lists the cache for the process, and every open fails | |
| 62 | busy while it does. A cache gone or no longer a database refuses every edit waiting, as an | |
| 63 | error, and stays closed. An edit a caller blocks on, or one the cache could never write, is | |
| 64 | refused at once, and the section thread rereads the queue; while the cache is closed, that | |
| 65 | reread waits for it, page reads answer from the section as it was, and later edits wait to | |
| 66 | apply to the reread section; the session then reports the section's pages as `Changed`. | |
| 61 | 67 | |
| 62 | 68 | ## Sessions |
| 63 | 69 |
crates/notebook/src/lib.rs+129-6| ... | ... | @@ -281,8 +281,8 @@ impl Replica { |
| 281 | 281 | }) |
| 282 | 282 | } |
| 283 | 283 | |
| 284 | fn lock(&self) -> Result<MutexGuard<'_, Connection>> { | |
| 285 | lock(&self.section.connection) | |
| 284 | fn lock(&self) -> Result<Open<'_>> { | |
| 285 | lock(&self.section.cache) | |
| 286 | 286 | } |
| 287 | 287 | |
| 288 | 288 | /// Hands `request` to the section thread. |
| ... | ... | @@ -494,10 +494,41 @@ impl Drop for Replica { |
| 494 | 494 | } |
| 495 | 495 | } |
| 496 | 496 | |
| 497 | fn lock(connection: &Mutex<Connection>) -> Result<MutexGuard<'_, Connection>> { | |
| 498 | connection | |
| 497 | /// The open cache's connection, for as long as its lock is held. | |
| 498 | struct Open<'a>(MutexGuard<'a, working::Cache>); | |
| 499 | ||
| 500 | impl std::ops::Deref for Open<'_> { | |
| 501 | type Target = Connection; | |
| 502 | ||
| 503 | fn deref(&self) -> &Connection { | |
| 504 | match &*self.0 { | |
| 505 | working::Cache::Open(connection) => connection, | |
| 506 | _ => unreachable!("Only an open cache is lent"), | |
| 507 | } | |
| 508 | } | |
| 509 | } | |
| 510 | ||
| 511 | impl std::ops::DerefMut for Open<'_> { | |
| 512 | fn deref_mut(&mut self) -> &mut Connection { | |
| 513 | match &mut *self.0 { | |
| 514 | working::Cache::Open(connection) => connection, | |
| 515 | _ => unreachable!("Only an open cache is lent"), | |
| 516 | } | |
| 517 | } | |
| 518 | } | |
| 519 | ||
| 520 | /// The cache's connection; `WouldBlock` while a failed commit keeps the cache closed. | |
| 521 | fn lock(cache: &Mutex<working::Cache>) -> Result<Open<'_>> { | |
| 522 | let cache = cache | |
| 499 | 523 | .lock() |
| 500 | .map_err(|_| io::Error::other("Cache owner panicked").into()) | |
| 524 | .map_err(|_| io::Error::other("Cache owner panicked"))?; | |
| 525 | match &*cache { | |
| 526 | working::Cache::Open(_) => Ok(Open(cache)), | |
| 527 | working::Cache::Closed { .. } => { | |
| 528 | Err(io::Error::new(io::ErrorKind::WouldBlock, "The cache reopens once it can").into()) | |
| 529 | } | |
| 530 | working::Cache::Lost(error) => Err(io::Error::other(error.clone()).into()), | |
| 531 | } | |
| 501 | 532 | } |
| 502 | 533 | |
| 503 | 534 | /// Tells the worker a burst of local edits is durable. |
| ... | ... | @@ -558,9 +589,101 @@ fn signed(value: u64) -> Result<i64> { |
| 558 | 589 | Ok(i64::try_from(value).map_err(|_| io::Error::from(io::ErrorKind::InvalidInput))?) |
| 559 | 590 | } |
| 560 | 591 | |
| 592 | /// `connect`, busy while anyone owns the cache: SQLite's lock, or the `Owned` a section holds | |
| 593 | /// while a failed commit keeps the cache closed, checked once SQLite's is held, as that | |
| 594 | /// section takes its own before letting SQLite's go. | |
| 595 | fn cache_connection(path: &Path) -> Result<Connection> { | |
| 596 | let connection = connect(path)?; | |
| 597 | connection.query_row("SELECT count(*) FROM sqlite_schema", [], |_| Ok(()))?; | |
| 598 | let owned = identity(path) | |
| 599 | .is_ok_and(|identity| OWNED.lock().is_ok_and(|owned| owned.contains(&identity))) | |
| 600 | || std::fs::File::open(owner_lock(path)).is_ok_and(|lock| { | |
| 601 | matches!( | |
| 602 | lock.try_lock_shared(), | |
| 603 | Err(std::fs::TryLockError::WouldBlock) | |
| 604 | ) | |
| 605 | }); | |
| 606 | if owned { | |
| 607 | return Err(rusqlite::Error::SqliteFailure( | |
| 608 | rusqlite::ffi::Error::new(rusqlite::ffi::SQLITE_BUSY), | |
| 609 | Some("A section holds the cache while it reopens".into()), | |
| 610 | ) | |
| 611 | .into()); | |
| 612 | } | |
| 613 | Ok(connection) | |
| 614 | } | |
| 615 | ||
| 616 | /// The file beside a cache whose lock keeps it owned while a failed commit keeps it closed. | |
| 617 | fn owner_lock(cache: &Path) -> PathBuf { | |
| 618 | let mut lock = cache.as_os_str().to_owned(); | |
| 619 | lock.push("-lock"); | |
| 620 | lock.into() | |
| 621 | } | |
| 622 | ||
| 623 | /// Caches an `Owned` holds in this process, where a lock on a network filesystem (`flock` over | |
| 624 | /// NFS on Linux) need not exclude. | |
| 625 | static OWNED: Mutex<std::collections::BTreeSet<Identity>> = | |
| 626 | Mutex::new(std::collections::BTreeSet::new()); | |
| 627 | ||
| 628 | /// A cache file however its path is spelled: its device and inode, or elsewhere its | |
| 629 | /// canonical path. | |
| 630 | #[cfg(unix)] | |
| 631 | type Identity = (u64, u64); | |
| 632 | #[cfg(not(unix))] | |
| 633 | type Identity = PathBuf; | |
| 634 | ||
| 635 | fn identity(cache: &Path) -> io::Result<Identity> { | |
| 636 | #[cfg(unix)] | |
| 637 | { | |
| 638 | use std::os::unix::fs::MetadataExt; | |
| 639 | let metadata = std::fs::metadata(cache)?; | |
| 640 | Ok((metadata.dev(), metadata.ino())) | |
| 641 | } | |
| 642 | #[cfg(not(unix))] | |
| 643 | std::fs::canonicalize(cache) | |
| 644 | } | |
| 645 | ||
| 646 | /// Ownership of a closed cache, which every open honours until it drops: `<cache>-lock` | |
| 647 | /// locked, and the cache listed in `OWNED`. | |
| 648 | struct Owned { | |
| 649 | cache: PathBuf, | |
| 650 | identity: Identity, | |
| 651 | _lock: std::fs::File, | |
| 652 | } | |
| 653 | ||
| 654 | impl Owned { | |
| 655 | fn take(cache: &Path) -> Result<Self> { | |
| 656 | let lock = std::fs::File::options() | |
| 657 | .create(true) | |
| 658 | .truncate(false) | |
| 659 | .write(true) | |
| 660 | .open(owner_lock(cache))?; | |
| 661 | lock.try_lock().map_err(io::Error::from)?; | |
| 662 | let identity = identity(cache)?; | |
| 663 | #[cfg_attr(unix, allow(clippy::clone_on_copy))] | |
| 664 | (OWNED.lock()) | |
| 665 | .map_err(|_| io::Error::other("Cache owners panicked"))? | |
| 666 | .insert(identity.clone()); | |
| 667 | Ok(Self { | |
| 668 | cache: cache.to_owned(), | |
| 669 | identity, | |
| 670 | _lock: lock, | |
| 671 | }) | |
| 672 | } | |
| 673 | } | |
| 674 | ||
| 675 | impl Drop for Owned { | |
| 676 | fn drop(&mut self) { | |
| 677 | if let Ok(mut owned) = OWNED.lock() { | |
| 678 | owned.remove(&self.identity); | |
| 679 | } | |
| 680 | let _ = std::fs::remove_file(owner_lock(&self.cache)); | |
| 681 | } | |
| 682 | } | |
| 683 | ||
| 561 | 684 | /// Opens a cache without writing to it: exclusive locking, a full sync of every commit |
| 562 | 685 | /// (`F_FULLFSYNC` on macOS, the WAL's syncs included) and foreign keys, each queried back. |
| 563 | fn cache_connection(path: &Path) -> Result<Connection> { | |
| 686 | fn connect(path: &Path) -> Result<Connection> { | |
| 564 | 687 | #[cfg(target_arch = "wasm32")] |
| 565 | 688 | fs::install_sqlite(); |
| 566 | 689 | let connection = Connection::open_with_flags(path, OpenFlags::SQLITE_OPEN_READ_WRITE)?; |
crates/notebook/src/location.rs+16-6| ... | ... | @@ -191,13 +191,23 @@ fn adopt(from: &Path, to: &Path) -> io::Result<()> { |
| 191 | 191 | Ok(()) |
| 192 | 192 | } |
| 193 | 193 | |
| 194 | /// Removes a replica folder that holds no replica. | |
| 194 | /// Removes a replica folder that holds no replica, with the owner lock a crash may have left | |
| 195 | /// beside a cache moved away since (`crate::Owned`). | |
| 195 | 196 | fn tidy(folder: &Path) { |
| 196 | let empty = fs::read_dir(folder).is_ok_and(|mut entries| { | |
| 197 | entries.all(|entry| entry.is_ok_and(|entry| entry.file_name() == LOCATION)) | |
| 198 | }); | |
| 199 | if empty { | |
| 200 | let _ = fs::remove_file(folder.join(LOCATION)); | |
| 197 | let Ok(entries) = fs::read_dir(folder).and_then(|entries| { | |
| 198 | entries | |
| 199 | .map(|entry| Ok(entry?.file_name())) | |
| 200 | .collect::<io::Result<Vec<_>>>() | |
| 201 | }) else { | |
| 202 | return; | |
| 203 | }; | |
| 204 | let left = |name: &std::ffi::OsString| { | |
| 205 | name == LOCATION || name.to_string_lossy().ends_with(".sqlite-lock") | |
| 206 | }; | |
| 207 | if entries.iter().all(left) { | |
| 208 | for name in entries { | |
| 209 | let _ = fs::remove_file(folder.join(name)); | |
| 210 | } | |
| 201 | 211 | let _ = fs::remove_dir(folder); |
| 202 | 212 | } |
| 203 | 213 | } |
crates/notebook/src/session.rs+32-11| ... | ... | @@ -2186,7 +2186,8 @@ pub struct SyncStatus { |
| 2186 | 2186 | pub synced: Option<u64>, |
| 2187 | 2187 | /// Why the last step failed, until one reaches the section file again. |
| 2188 | 2188 | pub error: Option<io::Error>, |
| 2189 | /// Edits the section file does not hold yet, uncertain attempts included. | |
| 2189 | /// Edits the section file does not hold yet, uncertain attempts and edits waiting for the | |
| 2190 | /// cache included; while a failed commit keeps the cache closed, those queued before it. | |
| 2190 | 2191 | pub queued: u64, |
| 2191 | 2192 | } |
| 2192 | 2193 | |
| ... | ... | @@ -2237,11 +2238,23 @@ pub(crate) fn sync_status( |
| 2237 | 2238 | synced: impl FnOnce() -> Option<u64>, |
| 2238 | 2239 | observed: &Mutex<Observed>, |
| 2239 | 2240 | ) -> Result<SyncStatus> { |
| 2240 | let queued: i64 = crate::lock(&replica.section.connection)?.query_row( | |
| 2241 | "SELECT count(*) FROM edits", | |
| 2242 | [], | |
| 2243 | |row| row.get(0), | |
| 2244 | )?; | |
| 2241 | let waiting = replica | |
| 2242 | .section | |
| 2243 | .waiting | |
| 2244 | .load(std::sync::atomic::Ordering::Relaxed) as u64; | |
| 2245 | let (queued, failed) = match &*(replica.section.cache.lock()) | |
| 2246 | .map_err(|_| io::Error::other("Cache owner panicked"))? | |
| 2247 | { | |
| 2248 | crate::working::Cache::Open(connection) => { | |
| 2249 | let queued: i64 = | |
| 2250 | connection.query_row("SELECT count(*) FROM edits", [], |row| row.get(0))?; | |
| 2251 | (crate::unsigned(queued)?, None) | |
| 2252 | } | |
| 2253 | crate::working::Cache::Closed { queued, .. } => { | |
| 2254 | (*queued, Some("The cache reopens once it can".to_owned())) | |
| 2255 | } | |
| 2256 | crate::working::Cache::Lost(error) => (0, Some(error.clone())), | |
| 2257 | }; | |
| 2245 | 2258 | let observed = observed |
| 2246 | 2259 | .lock() |
| 2247 | 2260 | .map_err(|_| io::Error::other("Synchronization observer panicked"))?; |
| ... | ... | @@ -2249,11 +2262,11 @@ pub(crate) fn sync_status( |
| 2249 | 2262 | // Read after `observed`: the worker stores the time before it clears the error, so a |
| 2250 | 2263 | // cleared error never shows beside the time of the step before it. |
| 2251 | 2264 | synced: synced(), |
| 2252 | error: observed | |
| 2253 | .as_ref() | |
| 2254 | .and_then(Option::as_ref) | |
| 2255 | .map(|error| io::Error::new(error.kind(), error.to_string())), | |
| 2256 | queued: crate::unsigned(queued)?, | |
| 2265 | error: failed.map(io::Error::other).or_else(|| { | |
| 2266 | (observed.as_ref().and_then(Option::as_ref)) | |
| 2267 | .map(|error| io::Error::new(error.kind(), error.to_string())) | |
| 2268 | }), | |
| 2269 | queued: queued + waiting, | |
| 2257 | 2270 | }) |
| 2258 | 2271 | } |
| 2259 | 2272 | |
| ... | ... | @@ -2475,6 +2488,14 @@ impl Section { |
| 2475 | 2488 | } |
| 2476 | 2489 | }); |
| 2477 | 2490 | let observed: Arc<Mutex<Observed>> = Arc::new(Mutex::new(None)); |
| 2491 | if let Ok(mut reread) = replica.section.reread.lock() { | |
| 2492 | let (sender, notify) = (sender.clone(), Arc::clone(&notify)); | |
| 2493 | *reread = Some(Box::new(move |spaces| { | |
| 2494 | if sender.send(Event::Changed(spaces)).is_ok() { | |
| 2495 | notify(); | |
| 2496 | } | |
| 2497 | })); | |
| 2498 | } | |
| 2478 | 2499 | let mut section = Self { |
| 2479 | 2500 | file, |
| 2480 | 2501 | replica, |
crates/notebook/src/session/tests.rs+21| ... | ... | @@ -854,3 +854,24 @@ fn edits_the_cache_refuses_wait_in_order_and_land_once() { |
| 854 | 854 | assert_eq!(waited[0].page.color, Some(0x0065_4321)); |
| 855 | 855 | assert!(Replica::open(&cache).unwrap().pending().unwrap().is_empty()); |
| 856 | 856 | } |
| 857 | ||
| 858 | #[test] | |
| 859 | fn a_cache_owned_while_closed_opens_for_nobody_else() { | |
| 860 | let temporary = tempfile::tempdir().unwrap(); | |
| 861 | let cache = temporary.path().join("owned.sqlite"); | |
| 862 | let source = onestore::create_section("Owned.one", "Original", "Fixture").unwrap(); | |
| 863 | drop(Replica::create(&cache, &source).unwrap()); | |
| 864 | let named = PathBuf::from(crate::connect(&cache).unwrap().path().unwrap()); | |
| 865 | let owned = crate::Owned::take(&named).unwrap(); | |
| 866 | assert!(Replica::open(&cache).is_err_and(|error| error.busy())); | |
| 867 | assert!(crate::closed(&cache).is_err_and(|error| error.busy())); | |
| 868 | drop(owned); | |
| 869 | assert!(!crate::owner_lock(&named).exists()); | |
| 870 | drop(Replica::open(&cache).unwrap()); | |
| 871 | // Another process holds only the lock beside the cache. | |
| 872 | let lock = std::fs::File::create(crate::owner_lock(&named)).unwrap(); | |
| 873 | lock.try_lock().unwrap(); | |
| 874 | assert!(Replica::open(&cache).is_err_and(|error| error.busy())); | |
| 875 | drop(lock); | |
| 876 | drop(Replica::open(&cache).unwrap()); | |
| 877 | } |
crates/notebook/src/working.rs+467-234| ... | ... | @@ -15,7 +15,7 @@ use rusqlite::{Connection, OptionalExtension, TransactionBehavior, params}; |
| 15 | 15 | use std::{ |
| 16 | 16 | collections::{BTreeMap, BTreeSet, VecDeque}, |
| 17 | 17 | io, |
| 18 | path::PathBuf, | |
| 18 | path::{Path, PathBuf}, | |
| 19 | 19 | sync::{ |
| 20 | 20 | Arc, Mutex, Weak, |
| 21 | 21 | atomic::{AtomicUsize, Ordering}, |
| ... | ... | @@ -25,6 +25,7 @@ use std::{ |
| 25 | 25 | |
| 26 | 26 | pub(crate) type Reply<T> = Box<dyn FnOnce(Result<T>) + Send>; |
| 27 | 27 | pub(crate) type Notice = Box<dyn Fn(&str) + Send>; |
| 28 | type Changed = Box<dyn Fn(Vec<ExGuid>) + Send>; | |
| 28 | 29 | |
| 29 | 30 | pub(crate) enum Request { |
| 30 | 31 | Apply(Accepted), |
| ... | ... | @@ -87,6 +88,34 @@ pub(crate) struct Sealed { |
| 87 | 88 | pub transaction: Option<Transaction>, |
| 88 | 89 | } |
| 89 | 90 | |
| 91 | /// The cache's connection, or what a failed commit leaves: the cache closed until it reopens | |
| 92 | /// and reads cold whether the commit whose last edit is `last` landed, `ids` naming its edits | |
| 93 | /// still waiting and `queued` the edits queued before it; or lost where it never will reopen. | |
| 94 | pub(crate) enum Cache { | |
| 95 | Open(Connection), | |
| 96 | Closed { | |
| 97 | last: u64, | |
| 98 | ids: Vec<u64>, | |
| 99 | queued: u64, | |
| 100 | owner: Owner, | |
| 101 | }, | |
| 102 | Lost(String), | |
| 103 | } | |
| 104 | ||
| 105 | /// What keeps everyone else out of a closed cache: the warm connection, unused, until | |
| 106 | /// `crate::Owned` can be taken in its stead. | |
| 107 | pub(crate) enum Owner { | |
| 108 | Warm(#[allow(dead_code)] Connection), | |
| 109 | Owned(#[allow(dead_code)] crate::Owned), | |
| 110 | } | |
| 111 | ||
| 112 | /// Whether a failed commit keeps the cache closed. | |
| 113 | fn closed(cache: &Mutex<Cache>) -> bool { | |
| 114 | cache | |
| 115 | .lock() | |
| 116 | .is_ok_and(|cache| matches!(*cache, Cache::Closed { .. })) | |
| 117 | } | |
| 118 | ||
| 90 | 119 | impl Request { |
| 91 | 120 | fn fail(self, kind: io::ErrorKind, message: &str) { |
| 92 | 121 | let error = || io::Error::new(kind, message.to_owned()).into(); |
| ... | ... | @@ -112,7 +141,10 @@ impl Request { |
| 112 | 141 | /// keeps answering page reads from the section as it was; the new thread then takes the |
| 113 | 142 | /// requests over, so reads never wait for a rebuild. |
| 114 | 143 | pub(crate) struct Thread { |
| 115 | pub(crate) connection: Mutex<Connection>, | |
| 144 | pub(crate) cache: Mutex<Cache>, | |
| 145 | /// Where the cache reopens after a failed commit; none in the browser, where reopening | |
| 146 | /// through its file system is unverified. | |
| 147 | path: Option<PathBuf>, | |
| 116 | 148 | /// The sync worker, which a durable burst of edits wakes. |
| 117 | 149 | pub(crate) worker: Mutex<Weak<Signal>>, |
| 118 | 150 | /// Taken when the replica drops, which ends every section thread. |
| ... | ... | @@ -126,7 +158,9 @@ pub(crate) struct Thread { |
| 126 | 158 | /// A password-protected section's key, under which its queue is sealed too. |
| 127 | 159 | pub(crate) key: Option<Key>, |
| 128 | 160 | /// The edits the cache last refused for now, as `crate::unwritten` counts them. |
| 129 | waiting: AtomicUsize, | |
| 161 | pub(crate) waiting: AtomicUsize, | |
| 162 | /// Told the pages a reread after a refusal may change, which their views reload. | |
| 163 | pub(crate) reread: Mutex<Option<Changed>>, | |
| 130 | 164 | /// Last, so the cache closes before background synchronization takes it over. |
| 131 | 165 | pub(crate) released: crate::Released, |
| 132 | 166 | } |
| ... | ... | @@ -309,18 +343,20 @@ mod serving { |
| 309 | 343 | } |
| 310 | 344 | } |
| 311 | 345 | |
| 312 | /// The next request; a second passes without one while `retry`. | |
| 346 | /// How often the cache is tried again while edits wait for it or it is closed. | |
| 347 | const RETRY: std::time::Duration = std::time::Duration::from_secs(1); | |
| 348 | ||
| 349 | /// The next request, or a timeout once `retry` passes without one. | |
| 313 | 350 | #[cfg(not(target_arch = "wasm32"))] |
| 314 | 351 | async fn receive( |
| 315 | 352 | requests: &mpsc::Receiver<Request>, |
| 316 | retry: bool, | |
| 353 | retry: Option<std::time::Duration>, | |
| 317 | 354 | ) -> std::result::Result<Request, mpsc::RecvTimeoutError> { |
| 318 | if retry { | |
| 319 | requests.recv_timeout(std::time::Duration::from_secs(1)) | |
| 320 | } else { | |
| 321 | requests | |
| 355 | match retry { | |
| 356 | Some(wait) => requests.recv_timeout(wait), | |
| 357 | None => requests | |
| 322 | 358 | .recv() |
| 323 | .map_err(|_| mpsc::RecvTimeoutError::Disconnected) | |
| 359 | .map_err(|_| mpsc::RecvTimeoutError::Disconnected), | |
| 324 | 360 | } |
| 325 | 361 | } |
| 326 | 362 | |
| ... | ... | @@ -329,7 +365,7 @@ async fn receive( |
| 329 | 365 | #[cfg(target_arch = "wasm32")] |
| 330 | 366 | async fn receive( |
| 331 | 367 | requests: &mpsc::Receiver<Request>, |
| 332 | _retry: bool, | |
| 368 | _retry: Option<std::time::Duration>, | |
| 333 | 369 | ) -> std::result::Result<Request, mpsc::RecvTimeoutError> { |
| 334 | 370 | std::future::poll_fn(|_| match requests.try_recv() { |
| 335 | 371 | Ok(request) => std::task::Poll::Ready(Ok(request)), |
| ... | ... | @@ -349,7 +385,10 @@ pub(crate) fn spawn(connection: Connection, key: Option<Key>) -> Result<(Arc<Thr |
| 349 | 385 | #[cfg(target_arch = "wasm32")] |
| 350 | 386 | let (durable_wake, woken) = crate::task::channel(); |
| 351 | 387 | let shared = Arc::new(Thread { |
| 352 | connection: Mutex::new(connection), | |
| 388 | path: (connection.path()) | |
| 389 | .filter(|_| cfg!(not(target_arch = "wasm32"))) | |
| 390 | .map(PathBuf::from), | |
| 391 | cache: Mutex::new(Cache::Open(connection)), | |
| 353 | 392 | worker: Mutex::new(Weak::new()), |
| 354 | 393 | sender: Mutex::new(Some(sender)), |
| 355 | 394 | #[cfg(target_arch = "wasm32")] |
| ... | ... | @@ -360,6 +399,7 @@ pub(crate) fn spawn(connection: Connection, key: Option<Key>) -> Result<(Arc<Thr |
| 360 | 399 | threads: Mutex::new(Vec::new()), |
| 361 | 400 | key, |
| 362 | 401 | waiting: AtomicUsize::new(0), |
| 402 | reread: Mutex::new(None), | |
| 363 | 403 | released: crate::Released::default(), |
| 364 | 404 | }); |
| 365 | 405 | #[cfg(target_arch = "wasm32")] |
| ... | ... | @@ -393,7 +433,7 @@ async fn run( |
| 393 | 433 | ) { |
| 394 | 434 | loop { |
| 395 | 435 | let arena = Arena::default(); |
| 396 | let working = match Working::open(&arena, &shared.connection, shared.key.as_ref()) { | |
| 436 | let working = match Working::open(&arena, &shared.cache, shared.key.as_ref()) { | |
| 397 | 437 | Ok(working) => working, |
| 398 | 438 | Err(error) => { |
| 399 | 439 | if let Some(ready) = ready.take() { |
| ... | ... | @@ -451,19 +491,17 @@ enum Job { |
| 451 | 491 | fn build(shared: Arc<Thread>, job: Job, signal: mpsc::Sender<Request>) { |
| 452 | 492 | let answer: Box<dyn FnOnce() + Send> = match job { |
| 453 | 493 | Job::Reopen { reply } => Box::new(move || reply(Ok(()))), |
| 454 | Job::Rebase { image, reply } => { | |
| 455 | match rebase(&shared.connection, image, shared.key.as_ref()) { | |
| 456 | Ok(changed) => Box::new(move || reply(Ok(changed))), | |
| 457 | Err(error) => { | |
| 458 | reply(Err(error)); | |
| 459 | let _ = signal.send(Request::Resume); | |
| 460 | return; | |
| 461 | } | |
| 494 | Job::Rebase { image, reply } => match rebase(&shared.cache, image, shared.key.as_ref()) { | |
| 495 | Ok(changed) => Box::new(move || reply(Ok(changed))), | |
| 496 | Err(error) => { | |
| 497 | reply(Err(error)); | |
| 498 | let _ = signal.send(Request::Resume); | |
| 499 | return; | |
| 462 | 500 | } |
| 463 | } | |
| 501 | }, | |
| 464 | 502 | }; |
| 465 | 503 | let arena = Arena::default(); |
| 466 | let opened = Working::open(&arena, &shared.connection, shared.key.as_ref()); | |
| 504 | let opened = Working::open(&arena, &shared.cache, shared.key.as_ref()); | |
| 467 | 505 | let (to, from) = mpsc::sync_channel(1); |
| 468 | 506 | if signal.send(Request::Handover(to)).is_err() { |
| 469 | 507 | return; |
| ... | ... | @@ -492,40 +530,65 @@ fn build(shared: Arc<Thread>, job: Job, signal: mpsc::Sender<Request>) { |
| 492 | 530 | fn inline(shared: &Thread, job: Job) -> bool { |
| 493 | 531 | match job { |
| 494 | 532 | Job::Reopen { reply } => reply(Ok(())), |
| 495 | Job::Rebase { image, reply } => { | |
| 496 | match rebase(&shared.connection, image, shared.key.as_ref()) { | |
| 497 | Ok(changed) => reply(Ok(changed)), | |
| 498 | Err(error) => { | |
| 499 | reply(Err(error)); | |
| 500 | return false; | |
| 501 | } | |
| 533 | Job::Rebase { image, reply } => match rebase(&shared.cache, image, shared.key.as_ref()) { | |
| 534 | Ok(changed) => reply(Ok(changed)), | |
| 535 | Err(error) => { | |
| 536 | reply(Err(error)); | |
| 537 | return false; | |
| 502 | 538 | } |
| 503 | } | |
| 539 | }, | |
| 504 | 540 | } |
| 505 | 541 | true |
| 506 | 542 | } |
| 507 | 543 | |
| 508 | /// Writes again each second the edits a dropped replica left waiting, until none do or their | |
| 509 | /// cache is gone, keeping the cache open and `Released` from firing meanwhile. | |
| 544 | /// Writes again each second the edits a dropped replica left waiting, until none do and the | |
| 545 | /// cache is open, or the cache is gone; the cache stays owned and `Released` unfired meanwhile. | |
| 546 | /// With `reread`, the closed commit is settled first and the edits applied again to the queue | |
| 547 | /// as it then stands, refusing those that no longer apply, as the section thread would have. | |
| 510 | 548 | #[cfg(not(target_arch = "wasm32"))] |
| 511 | fn orphan(shared: &Arc<Thread>, unwritten: Vec<Accepted>) { | |
| 549 | fn orphan(shared: &Arc<Thread>, mut open: Option<i64>, mut unwritten: Vec<Accepted>, reread: bool) { | |
| 512 | 550 | let shared = Arc::clone(shared); |
| 513 | let path = lock(&shared.connection) | |
| 514 | .ok() | |
| 515 | .and_then(|connection| connection.path().map(PathBuf::from)); | |
| 516 | 551 | let spawned = std::thread::Builder::new() |
| 517 | 552 | .name("onestore-unwritten".into()) |
| 518 | 553 | .spawn(move || { |
| 519 | let arena = Arena::default(); | |
| 520 | let Ok(mut working) = Working::open(&arena, &shared.connection, shared.key.as_ref()) | |
| 521 | else { | |
| 522 | return; | |
| 523 | }; | |
| 524 | working.unwritten = unwritten; | |
| 525 | 554 | // A cache deleted, as with its notebook, takes no edits. |
| 526 | while !working.unwritten.is_empty() && path.as_ref().is_some_and(|path| path.exists()) { | |
| 527 | std::thread::sleep(std::time::Duration::from_secs(1)); | |
| 528 | let _ = working.flush(&shared); | |
| 555 | let kept = || shared.path.as_ref().is_some_and(|path| path.exists()); | |
| 556 | let second = || std::thread::sleep(std::time::Duration::from_secs(1)); | |
| 557 | while closed(&shared.cache) && kept() { | |
| 558 | second(); | |
| 559 | let _ = write(&shared, &mut open, &mut unwritten, !reread); | |
| 560 | } | |
| 561 | // An edit that applies part way is refused, and the others apply again to a fresh | |
| 562 | // reading of the queue. | |
| 563 | if reread { | |
| 564 | 'reread: loop { | |
| 565 | let arena = Arena::default(); | |
| 566 | let mut working = | |
| 567 | match Working::open(&arena, &shared.cache, shared.key.as_ref()) { | |
| 568 | Ok(working) => working, | |
| 569 | Err(error) => { | |
| 570 | for edit in unwritten.drain(..) { | |
| 571 | (edit.reply)(Err(io::Error::other(error.to_string()).into())); | |
| 572 | } | |
| 573 | break; | |
| 574 | } | |
| 575 | }; | |
| 576 | let mut edits = std::mem::take(&mut unwritten).into_iter(); | |
| 577 | while let Some(edit) = edits.next() { | |
| 578 | if working.accept(edit) { | |
| 579 | unwritten = std::mem::take(&mut working.unwritten); | |
| 580 | unwritten.extend(edits); | |
| 581 | continue 'reread; | |
| 582 | } | |
| 583 | } | |
| 584 | open = working.open; | |
| 585 | unwritten = std::mem::take(&mut working.unwritten); | |
| 586 | break; | |
| 587 | } | |
| 588 | } | |
| 589 | while !unwritten.is_empty() && kept() { | |
| 590 | second(); | |
| 591 | let _ = write(&shared, &mut open, &mut unwritten, true); | |
| 529 | 592 | } |
| 530 | 593 | }); |
| 531 | 594 | if let Err(error) = spawned { |
| ... | ... | @@ -556,7 +619,7 @@ struct Working<'a> { |
| 556 | 619 | } |
| 557 | 620 | |
| 558 | 621 | impl<'a> Working<'a> { |
| 559 | fn open(arena: &'a Arena, connection: &Mutex<Connection>, key: Option<&Key>) -> Result<Self> { | |
| 622 | fn open(arena: &'a Arena, connection: &Mutex<Cache>, key: Option<&Key>) -> Result<Self> { | |
| 560 | 623 | let (section, open, touched) = replay(arena, &*lock(connection)?, key)?; |
| 561 | 624 | Ok(Self { |
| 562 | 625 | section, |
| ... | ... | @@ -572,23 +635,53 @@ impl<'a> Working<'a> { |
| 572 | 635 | requests: &mpsc::Receiver<Request>, |
| 573 | 636 | backlog: &mut VecDeque<Request>, |
| 574 | 637 | ) -> Next { |
| 575 | let connection = &shared.connection; | |
| 638 | let connection = &shared.cache; | |
| 576 | 639 | // While another thread rebuilds the section, reads answer from this one and every |
| 577 | 640 | // other request waits for the rebuilt section. |
| 578 | 641 | let mut held: Option<VecDeque<Request>> = None; |
| 642 | // While a reread waits for the closed cache, reads answer from this section and edits | |
| 643 | // wait for the reread, as one this section refused may underlie them. | |
| 644 | let mut stale: Option<VecDeque<Request>> = None; | |
| 645 | let mut attempted = web_time::Instant::now(); | |
| 579 | 646 | loop { |
| 580 | 647 | let mut burst = std::mem::take(backlog); |
| 581 | // Only a burst that took an edit, or a retry's tick, writes at its end: reads never. | |
| 648 | // Only a burst that took an edit, or a retry falling due, writes at its end: reads | |
| 649 | // never. | |
| 582 | 650 | let mut write = false; |
| 651 | let retry = !self.unwritten.is_empty() || closed(connection); | |
| 583 | 652 | if burst.is_empty() { |
| 584 | match receive(requests, !self.unwritten.is_empty()).await { | |
| 653 | let wait = retry.then(|| RETRY.saturating_sub(attempted.elapsed())); | |
| 654 | match receive(requests, wait).await { | |
| 585 | 655 | Ok(request) => burst.push_back(request), |
| 586 | Err(mpsc::RecvTimeoutError::Timeout) => write = true, | |
| 656 | Err(mpsc::RecvTimeoutError::Timeout) => {} | |
| 587 | 657 | Err(mpsc::RecvTimeoutError::Disconnected) => { |
| 588 | let _ = self.flush(shared); | |
| 658 | let stale_section = stale.is_some(); | |
| 659 | // A refusal now leaves the section holding an edit the queue lacks too. | |
| 660 | #[cfg_attr(target_arch = "wasm32", allow(unused_variables))] | |
| 661 | let reread = crate::working::write( | |
| 662 | shared, | |
| 663 | &mut self.open, | |
| 664 | &mut self.unwritten, | |
| 665 | !stale_section, | |
| 666 | ) | |
| 667 | .is_err() | |
| 668 | || stale_section; | |
| 669 | for request in stale.into_iter().flatten() { | |
| 670 | match request { | |
| 671 | Request::Apply(accepted) => self.unwritten.push(accepted), | |
| 672 | request => { | |
| 673 | request.fail(io::ErrorKind::Other, "The section thread stopped") | |
| 674 | } | |
| 675 | } | |
| 676 | } | |
| 589 | 677 | #[cfg(not(target_arch = "wasm32"))] |
| 590 | if !self.unwritten.is_empty() { | |
| 591 | orphan(shared, std::mem::take(&mut self.unwritten)); | |
| 678 | if !self.unwritten.is_empty() || closed(connection) { | |
| 679 | orphan( | |
| 680 | shared, | |
| 681 | self.open, | |
| 682 | std::mem::take(&mut self.unwritten), | |
| 683 | reread, | |
| 684 | ); | |
| 592 | 685 | } |
| 593 | 686 | return Next::Stop; |
| 594 | 687 | } |
| ... | ... | @@ -599,24 +692,11 @@ impl<'a> Working<'a> { |
| 599 | 692 | let mut later = VecDeque::new(); |
| 600 | 693 | let mut reopen = false; |
| 601 | 694 | while !reopen && let Some(request) = burst.pop_front().or_else(|| later.pop_front()) { |
| 695 | let Some(request) = self.answer(request) else { | |
| 696 | continue; | |
| 697 | }; | |
| 602 | 698 | if let Some(waiting) = &mut held { |
| 603 | 699 | match request { |
| 604 | Request::Page { space, reply } => { | |
| 605 | reply(self.section.page(space).map_err(Into::into)) | |
| 606 | } | |
| 607 | Request::Pages { reply } => reply(self.section.pages().map_err(Into::into)), | |
| 608 | Request::Conflicts { reply } => { | |
| 609 | reply(self.section.conflicts().map_err(Into::into)) | |
| 610 | } | |
| 611 | Request::Versions { reply } => { | |
| 612 | reply(self.section.versions().map_err(Into::into)) | |
| 613 | } | |
| 614 | Request::Version { | |
| 615 | space, | |
| 616 | version, | |
| 617 | reply, | |
| 618 | } => reply(self.section.version(space, version).map_err(Into::into)), | |
| 619 | Request::Moving { space, reply } => reply(self.moving(space)), | |
| 620 | 700 | Request::Handover(to) => { |
| 621 | 701 | let mut waiting = held.take().unwrap_or_default(); |
| 622 | 702 | waiting.extend(burst.drain(..).chain(later.drain(..))); |
| ... | ... | @@ -630,6 +710,25 @@ impl<'a> Working<'a> { |
| 630 | 710 | } |
| 631 | 711 | continue; |
| 632 | 712 | } |
| 713 | if let Some(waiting) = &mut stale { | |
| 714 | match request { | |
| 715 | Request::Apply( | |
| 716 | accepted @ Accepted { | |
| 717 | waiting: Some(_), .. | |
| 718 | }, | |
| 719 | ) => { | |
| 720 | if let Some(notice) = &accepted.waiting { | |
| 721 | notice("The cache reopens once it can"); | |
| 722 | } | |
| 723 | waiting.push_back(Request::Apply(accepted)); | |
| 724 | } | |
| 725 | request @ Request::Reopen { .. } => waiting.push_back(request), | |
| 726 | request => { | |
| 727 | request.fail(io::ErrorKind::WouldBlock, "The cache reopens once it can") | |
| 728 | } | |
| 729 | } | |
| 730 | continue; | |
| 731 | } | |
| 633 | 732 | if matches!(request, Request::Flush { .. } | Request::Seal { .. }) |
| 634 | 733 | && !burst.is_empty() |
| 635 | 734 | { |
| ... | ... | @@ -641,7 +740,7 @@ impl<'a> Working<'a> { |
| 641 | 740 | Request::Flush { .. } | Request::Seal { .. } | Request::Rebase { .. } |
| 642 | 741 | ) { |
| 643 | 742 | let failed = self.flush(shared).is_err(); |
| 644 | if failed || !self.unwritten.is_empty() { | |
| 743 | if failed || !self.unwritten.is_empty() || closed(connection) { | |
| 645 | 744 | // The worker retries later; the edits wait here meanwhile. |
| 646 | 745 | request.fail(io::ErrorKind::WouldBlock, "Edits wait to be written"); |
| 647 | 746 | reopen = failed; |
| ... | ... | @@ -650,36 +749,17 @@ impl<'a> Working<'a> { |
| 650 | 749 | } |
| 651 | 750 | reopen = match request { |
| 652 | 751 | Request::Apply(accepted) => { |
| 653 | write = true; | |
| 654 | self.accept(accepted) | |
| 655 | } | |
| 656 | Request::Page { space, reply } => { | |
| 657 | reply(self.section.page(space).map_err(Into::into)); | |
| 658 | false | |
| 659 | } | |
| 660 | Request::Pages { reply } => { | |
| 661 | reply(self.section.pages().map_err(Into::into)); | |
| 662 | false | |
| 663 | } | |
| 664 | Request::Conflicts { reply } => { | |
| 665 | reply(self.section.conflicts().map_err(Into::into)); | |
| 666 | false | |
| 667 | } | |
| 668 | Request::Versions { reply } => { | |
| 669 | reply(self.section.versions().map_err(Into::into)); | |
| 670 | false | |
| 671 | } | |
| 672 | Request::Version { | |
| 673 | space, | |
| 674 | version, | |
| 675 | reply, | |
| 676 | } => { | |
| 677 | reply(self.section.version(space, version).map_err(Into::into)); | |
| 678 | false | |
| 679 | } | |
| 680 | Request::Moving { space, reply } => { | |
| 681 | reply(self.moving(space)); | |
| 682 | false | |
| 752 | // While the cache is closed, an edit nobody waits for is refused before | |
| 753 | // it applies, instead of showing until a reread can drop it. | |
| 754 | if accepted.waiting.is_none() | |
| 755 | && let Err(error) = lock(connection).map(drop) | |
| 756 | { | |
| 757 | (accepted.reply)(Err(error)); | |
| 758 | false | |
| 759 | } else { | |
| 760 | write = true; | |
| 761 | self.accept(accepted) | |
| 762 | } | |
| 683 | 763 | } |
| 684 | 764 | Request::Flush { reply, durable } => { |
| 685 | 765 | #[cfg(target_arch = "wasm32")] |
| ... | ... | @@ -715,6 +795,12 @@ impl<'a> Working<'a> { |
| 715 | 795 | } |
| 716 | 796 | Request::Reopen { reply } => { |
| 717 | 797 | let _ = self.flush(shared); |
| 798 | if closed(connection) { | |
| 799 | stale | |
| 800 | .get_or_insert_default() | |
| 801 | .push_back(Request::Reopen { reply }); | |
| 802 | continue; | |
| 803 | } | |
| 718 | 804 | #[cfg(target_arch = "wasm32")] |
| 719 | 805 | let reopen = inline(shared, Job::Reopen { reply }); |
| 720 | 806 | #[cfg(not(target_arch = "wasm32"))] |
| ... | ... | @@ -726,20 +812,86 @@ impl<'a> Working<'a> { |
| 726 | 812 | }; |
| 727 | 813 | reopen |
| 728 | 814 | } |
| 729 | Request::Handover(_) | Request::Resume => false, | |
| 815 | Request::Page { .. } | |
| 816 | | Request::Pages { .. } | |
| 817 | | Request::Conflicts { .. } | |
| 818 | | Request::Versions { .. } | |
| 819 | | Request::Version { .. } | |
| 820 | | Request::Moving { .. } | |
| 821 | | Request::Handover(_) | |
| 822 | | Request::Resume => false, | |
| 823 | }; | |
| 824 | } | |
| 825 | // Retries fall due once a second however busy the section is; a closed cache is | |
| 826 | // tried then alone, not with each edit. | |
| 827 | if retry { | |
| 828 | let due = attempted.elapsed() >= RETRY; | |
| 829 | write = if closed(connection) { | |
| 830 | due | |
| 831 | } else { | |
| 832 | write || due | |
| 730 | 833 | }; |
| 731 | 834 | } |
| 732 | if reopen || write && self.flush(shared).is_err() { | |
| 733 | backlog.extend( | |
| 734 | self.unwritten | |
| 735 | .drain(..) | |
| 736 | .map(Request::Apply) | |
| 737 | .chain(burst.drain(..)) | |
| 738 | .chain(later.drain(..)), | |
| 739 | ); | |
| 740 | return Next::Reopen; | |
| 835 | if write { | |
| 836 | attempted = web_time::Instant::now(); | |
| 837 | } | |
| 838 | if let Some(waiting) = &stale { | |
| 839 | // Only whether the closed commit landed is settled before the reread. | |
| 840 | if write { | |
| 841 | let _ = | |
| 842 | crate::working::write(shared, &mut self.open, &mut self.unwritten, false); | |
| 843 | } | |
| 844 | if closed(connection) { | |
| 845 | let held = waiting | |
| 846 | .iter() | |
| 847 | .filter(|request| matches!(request, Request::Apply(_))) | |
| 848 | .count(); | |
| 849 | shared.wait(self.unwritten.len() + held); | |
| 850 | continue; | |
| 851 | } | |
| 852 | } else if reopen || write && self.flush(shared).is_err() { | |
| 853 | if closed(connection) { | |
| 854 | stale = Some(VecDeque::new()); | |
| 855 | backlog.extend(burst.drain(..).chain(later.drain(..))); | |
| 856 | continue; | |
| 857 | } | |
| 858 | } else { | |
| 859 | continue; | |
| 860 | } | |
| 861 | backlog.extend( | |
| 862 | self.unwritten | |
| 863 | .drain(..) | |
| 864 | .map(Request::Apply) | |
| 865 | .chain(stale.into_iter().flatten()) | |
| 866 | .chain(burst.drain(..)) | |
| 867 | .chain(later.drain(..)), | |
| 868 | ); | |
| 869 | // What the host asks next answers from the reread section. | |
| 870 | if let (Ok(pages), Ok(reread)) = (self.section.pages(), shared.reread.lock()) | |
| 871 | && let Some(reread) = reread.as_ref() | |
| 872 | { | |
| 873 | reread(pages.into_iter().map(|(space, ..)| space).collect()); | |
| 741 | 874 | } |
| 875 | return Next::Reopen; | |
| 876 | } | |
| 877 | } | |
| 878 | ||
| 879 | /// Answers `request` where it reads the section; any other comes back. | |
| 880 | fn answer(&mut self, request: Request) -> Option<Request> { | |
| 881 | match request { | |
| 882 | Request::Page { space, reply } => reply(self.section.page(space).map_err(Into::into)), | |
| 883 | Request::Pages { reply } => reply(self.section.pages().map_err(Into::into)), | |
| 884 | Request::Conflicts { reply } => reply(self.section.conflicts().map_err(Into::into)), | |
| 885 | Request::Versions { reply } => reply(self.section.versions().map_err(Into::into)), | |
| 886 | Request::Version { | |
| 887 | space, | |
| 888 | version, | |
| 889 | reply, | |
| 890 | } => reply(self.section.version(space, version).map_err(Into::into)), | |
| 891 | Request::Moving { space, reply } => reply(self.moving(space)), | |
| 892 | request => return Some(request), | |
| 742 | 893 | } |
| 894 | None | |
| 743 | 895 | } |
| 744 | 896 | |
| 745 | 897 | /// Starts a thread that runs `job` and rereads the section; false when the replica is |
| ... | ... | @@ -807,109 +959,12 @@ impl<'a> Working<'a> { |
| 807 | 959 | } |
| 808 | 960 | } |
| 809 | 961 | |
| 810 | /// Writes the unwritten edits in one transaction, then answers their senders. A write | |
| 811 | /// that may succeed later keeps the edits with a notice, telling the first; one that never | |
| 812 | /// will refuses the edit it failed on. An error means the section holds an edit the queue | |
| 813 | /// never will, or the cache closed, and must be reread. | |
| 814 | 962 | fn flush(&mut self, shared: &Arc<Thread>) -> Result<()> { |
| 815 | if self.unwritten.is_empty() { | |
| 816 | return Ok(()); | |
| 817 | } | |
| 818 | let mut connection = lock(&shared.connection)?; | |
| 819 | let mut ids = Vec::new(); | |
| 820 | let committed = connection | |
| 821 | .transaction_with_behavior(TransactionBehavior::Immediate) | |
| 822 | .map_err(crate::Error::from) | |
| 823 | .and_then(|transaction| { | |
| 824 | let batch = match self.open { | |
| 825 | Some(batch) => batch, | |
| 826 | None => { | |
| 827 | transaction.execute("INSERT INTO batches DEFAULT VALUES", [])?; | |
| 828 | transaction.last_insert_rowid() | |
| 829 | } | |
| 830 | }; | |
| 831 | for edit in &self.unwritten { | |
| 832 | ids.push(queue::insert( | |
| 833 | &transaction, | |
| 834 | shared.key.as_ref(), | |
| 835 | None, | |
| 836 | batch, | |
| 837 | &edit.author, | |
| 838 | &edit.edit, | |
| 839 | )?); | |
| 840 | } | |
| 841 | Ok((batch, transaction.commit().err())) | |
| 842 | }); | |
| 843 | // The error, and the edit it refuses when it would fail again. | |
| 844 | let written = match committed { | |
| 845 | Ok((batch, None)) => Ok(batch), | |
| 846 | Ok((batch, Some(error))) => { | |
| 847 | if landed(&mut connection, ids.last().copied().unwrap_or_default())? { | |
| 848 | Ok(batch) | |
| 849 | } else { | |
| 850 | Err((error.into(), None)) | |
| 851 | } | |
| 852 | } | |
| 853 | Err(error) => { | |
| 854 | let refused = (!transient(&error)).then_some(ids.len()); | |
| 855 | Err((error, refused)) | |
| 856 | } | |
| 857 | }; | |
| 858 | drop(connection); | |
| 859 | let batch = match written { | |
| 860 | Ok(batch) => batch, | |
| 861 | Err((error, refused)) => { | |
| 862 | let message = error.to_string(); | |
| 863 | let before = self.unwritten.len(); | |
| 864 | let mut at = 0; | |
| 865 | for edit in self.unwritten.extract_if(.., |edit| { | |
| 866 | at += 1; | |
| 867 | edit.waiting.is_none() || refused == Some(at - 1) | |
| 868 | }) { | |
| 869 | (edit.reply)(Err(io::Error::other(message.clone()).into())); | |
| 870 | } | |
| 871 | shared.wait(self.unwritten.len()); | |
| 872 | if let Some(notice) = self | |
| 873 | .unwritten | |
| 874 | .first() | |
| 875 | .and_then(|edit| edit.waiting.as_ref()) | |
| 876 | { | |
| 877 | notice(&message); | |
| 878 | } | |
| 879 | return if self.unwritten.len() == before { | |
| 880 | Ok(()) | |
| 881 | } else { | |
| 882 | Err(error) | |
| 883 | }; | |
| 884 | } | |
| 885 | }; | |
| 886 | self.open = Some(batch); | |
| 887 | #[cfg(target_arch = "wasm32")] | |
| 888 | shared.receipt( | |
| 889 | self.unwritten | |
| 890 | .drain(..) | |
| 891 | .zip(ids) | |
| 892 | .map(|(edit, id)| { | |
| 893 | let reply: Reply<()> = | |
| 894 | Box::new(move |result| (edit.reply)(result.map(|()| id))); | |
| 895 | reply | |
| 896 | }) | |
| 897 | .collect(), | |
| 898 | true, | |
| 899 | ); | |
| 900 | #[cfg(not(target_arch = "wasm32"))] | |
| 901 | { | |
| 902 | for (edit, id) in self.unwritten.drain(..).zip(ids) { | |
| 903 | (edit.reply)(Ok(id)); | |
| 904 | } | |
| 905 | crate::edited(&shared.worker); | |
| 906 | } | |
| 907 | shared.wait(0); | |
| 908 | Ok(()) | |
| 963 | write(shared, &mut self.open, &mut self.unwritten, true) | |
| 909 | 964 | } |
| 910 | 965 | |
| 911 | 966 | /// Seals the open batch unless a sealed batch still waits for publication. |
| 912 | fn seal(&mut self, connection: &Mutex<Connection>) -> Result<Option<Sealed>> { | |
| 967 | fn seal(&mut self, connection: &Mutex<Cache>) -> Result<Option<Sealed>> { | |
| 913 | 968 | let Some(batch) = self.open else { |
| 914 | 969 | return Ok(None); |
| 915 | 970 | }; |
| ... | ... | @@ -954,7 +1009,7 @@ impl<'a> Working<'a> { |
| 954 | 1009 | /// version the remote could not take gains a conflict page holding it, queued as a new edit. |
| 955 | 1010 | /// Returns the pages the remote changed. |
| 956 | 1011 | fn rebase( |
| 957 | connection: &Mutex<Connection>, | |
| 1012 | connection: &Mutex<Cache>, | |
| 958 | 1013 | image: Option<Vec<u8>>, |
| 959 | 1014 | key: Option<&Key>, |
| 960 | 1015 | ) -> Result<Vec<ExGuid>> { |
| ... | ... | @@ -1186,28 +1241,206 @@ fn transient(error: &crate::Error) -> bool { |
| 1186 | 1241 | if matches!(error.code, SystemIoFailure | DiskFull | DatabaseBusy | DatabaseLocked | CannotOpen)) |
| 1187 | 1242 | } |
| 1188 | 1243 | |
| 1189 | /// Whether the commit whose last edit is `last` reached the cache after it failed, as a cold | |
| 1190 | /// open reads it, a commit landing whole: the warm connection may not know of frames its WAL | |
| 1191 | /// holds, and must never write again. When the cache cannot be reopened, an inert connection | |
| 1192 | /// stays, failing every statement. | |
| 1193 | fn landed(connection: &mut Connection, last: u64) -> Result<bool> { | |
| 1194 | let path = connection.path().map(PathBuf::from); | |
| 1195 | let inert = Connection::open_in_memory()?; | |
| 1196 | inert.execute_batch("PRAGMA query_only=1")?; | |
| 1197 | std::mem::replace(connection, inert) | |
| 1198 | .close() | |
| 1199 | .map_err(|(_, error)| error)?; | |
| 1244 | /// Writes `unwritten` in one transaction after the open batch `open`, then answers their | |
| 1245 | /// senders. A write that may succeed later keeps the edits with a notice; one that never will | |
| 1246 | /// refuses the edit it failed on, and an edit nobody waits for is refused at once. A failed | |
| 1247 | /// commit leaves the cache closed, still owned, until it reopens to decide whether the commit | |
| 1248 | /// landed; a lost cache refuses every edit. An error means the section holds an edit the | |
| 1249 | /// queue never will, or the cache is lost, and must be reread. | |
| 1250 | fn write( | |
| 1251 | shared: &Arc<Thread>, | |
| 1252 | open: &mut Option<i64>, | |
| 1253 | unwritten: &mut Vec<Accepted>, | |
| 1254 | commit: bool, | |
| 1255 | ) -> Result<()> { | |
| 1256 | let mut cache = (shared.cache.lock()).map_err(|_| io::Error::other("Cache owner panicked"))?; | |
| 1257 | if unwritten.is_empty() && !matches!(*cache, Cache::Closed { .. }) { | |
| 1258 | return Ok(()); | |
| 1259 | } | |
| 1260 | let path = shared.path.as_deref(); | |
| 1261 | // The ids the cache holds for the first of the edits. | |
| 1262 | let mut ids = Vec::new(); | |
| 1263 | // The error, and the edit it refuses when it would fail again. | |
| 1264 | let failed = (|| { | |
| 1265 | ids = reopen(&mut cache, path, open).map_err(|error| (error, None))?; | |
| 1266 | if !commit || ids.len() == unwritten.len() { | |
| 1267 | return Ok(()); | |
| 1268 | } | |
| 1269 | let Cache::Open(connection) = &mut *cache else { | |
| 1270 | unreachable!("A cache that does not reopen fails"); | |
| 1271 | }; | |
| 1272 | let mut inserted = Vec::new(); | |
| 1273 | let mut queued = 0; | |
| 1274 | let committed = connection | |
| 1275 | .transaction_with_behavior(TransactionBehavior::Immediate) | |
| 1276 | .map_err(crate::Error::from) | |
| 1277 | .and_then(|transaction| { | |
| 1278 | queued = transaction | |
| 1279 | .query_row("SELECT count(*) FROM edits", [], |row| row.get::<_, i64>(0))?; | |
| 1280 | let batch = match *open { | |
| 1281 | Some(batch) => batch, | |
| 1282 | None => { | |
| 1283 | transaction.execute("INSERT INTO batches DEFAULT VALUES", [])?; | |
| 1284 | transaction.last_insert_rowid() | |
| 1285 | } | |
| 1286 | }; | |
| 1287 | for edit in &unwritten[ids.len()..] { | |
| 1288 | inserted.push(queue::insert( | |
| 1289 | &transaction, | |
| 1290 | shared.key.as_ref(), | |
| 1291 | None, | |
| 1292 | batch, | |
| 1293 | &edit.author, | |
| 1294 | &edit.edit, | |
| 1295 | )?); | |
| 1296 | } | |
| 1297 | Ok((batch, transaction.commit().err())) | |
| 1298 | }); | |
| 1299 | match committed { | |
| 1300 | Ok((batch, None)) => *open = Some(batch), | |
| 1301 | Ok((_, Some(error))) => { | |
| 1302 | // The warm connection may not know of frames its WAL holds, and must never | |
| 1303 | // write again; it keeps the cache owned until the owner lock can. | |
| 1304 | if let Cache::Open(warm) = | |
| 1305 | std::mem::replace(&mut *cache, Cache::Lost(String::new())) | |
| 1306 | { | |
| 1307 | *cache = Cache::Closed { | |
| 1308 | last: inserted[inserted.len() - 1], | |
| 1309 | ids: inserted, | |
| 1310 | queued: queued.unsigned_abs(), | |
| 1311 | owner: Owner::Warm(warm), | |
| 1312 | }; | |
| 1313 | } | |
| 1314 | inserted = reopen(&mut cache, path, open).map_err(|error| (error, None))?; | |
| 1315 | if inserted.is_empty() { | |
| 1316 | return Err((error.into(), None)); | |
| 1317 | } | |
| 1318 | } | |
| 1319 | Err(error) => { | |
| 1320 | let refused = (!transient(&error)).then_some(inserted.len()); | |
| 1321 | return Err((error, refused)); | |
| 1322 | } | |
| 1323 | } | |
| 1324 | ids.extend(inserted); | |
| 1325 | Ok(()) | |
| 1326 | })(); | |
| 1327 | let written: Vec<_> = unwritten.drain(..ids.len()).zip(ids).collect(); | |
| 1328 | let refused: Vec<_> = match &failed { | |
| 1329 | Ok(()) => Vec::new(), | |
| 1330 | Err((_, failing)) => { | |
| 1331 | let lost = matches!(*cache, Cache::Lost(_)); | |
| 1332 | let (mut at, mut kept) = (0, 0); | |
| 1333 | (unwritten.extract_if(.., |edit| { | |
| 1334 | at += 1; | |
| 1335 | let refuse = lost || edit.waiting.is_none() || *failing == Some(at - 1); | |
| 1336 | if !refuse { | |
| 1337 | kept += 1; | |
| 1338 | } else if let Cache::Closed { ids, .. } = &mut *cache | |
| 1339 | && kept < ids.len() | |
| 1340 | { | |
| 1341 | // Whether its commit landed, it is answered now; a reread shows which. | |
| 1342 | ids.remove(kept); | |
| 1343 | } | |
| 1344 | refuse | |
| 1345 | })) | |
| 1346 | .collect() | |
| 1347 | } | |
| 1348 | }; | |
| 1349 | drop(cache); | |
| 1350 | // Counted before anyone hears, as a host hearing may ask. | |
| 1351 | shared.wait(unwritten.len()); | |
| 1352 | if !written.is_empty() { | |
| 1353 | #[cfg(target_arch = "wasm32")] | |
| 1354 | shared.receipt( | |
| 1355 | written | |
| 1356 | .into_iter() | |
| 1357 | .map(|(edit, id)| { | |
| 1358 | let reply: Reply<()> = | |
| 1359 | Box::new(move |result| (edit.reply)(result.map(|()| id))); | |
| 1360 | reply | |
| 1361 | }) | |
| 1362 | .collect(), | |
| 1363 | true, | |
| 1364 | ); | |
| 1365 | #[cfg(not(target_arch = "wasm32"))] | |
| 1366 | { | |
| 1367 | for (edit, id) in written { | |
| 1368 | (edit.reply)(Ok(id)); | |
| 1369 | } | |
| 1370 | crate::edited(&shared.worker); | |
| 1371 | } | |
| 1372 | } | |
| 1373 | let Err((error, _)) = failed else { | |
| 1374 | return Ok(()); | |
| 1375 | }; | |
| 1376 | let message = error.to_string(); | |
| 1377 | let kept = refused.is_empty(); | |
| 1378 | for edit in refused { | |
| 1379 | (edit.reply)(Err(io::Error::other(message.clone()).into())); | |
| 1380 | } | |
| 1381 | if let Some(notice) = unwritten.iter().find_map(|edit| edit.waiting.as_ref()) { | |
| 1382 | notice(&message); | |
| 1383 | } | |
| 1384 | if kept { Ok(()) } else { Err(error) } | |
| 1385 | } | |
| 1386 | ||
| 1387 | /// Reopens the cache a failed commit closed and reads cold whether the commit landed, as one | |
| 1388 | /// lands whole: the ids of its edits still waiting when it did, its batch becoming `open`; | |
| 1389 | /// none where it did not or the cache was open. The closed cache stays owned meanwhile, and | |
| 1390 | /// is lost once the file is gone or no longer a database. | |
| 1391 | fn reopen(cache: &mut Cache, path: Option<&Path>, open: &mut Option<i64>) -> Result<Vec<u64>> { | |
| 1392 | let (last, ids, owner) = match cache { | |
| 1393 | Cache::Open(_) => return Ok(Vec::new()), | |
| 1394 | Cache::Closed { | |
| 1395 | last, ids, owner, .. | |
| 1396 | } => (*last, ids, owner), | |
| 1397 | Cache::Lost(error) => return Err(io::Error::other(error.clone()).into()), | |
| 1398 | }; | |
| 1200 | 1399 | // Reopening through the browser's file system is unverified: it stays closed. |
| 1201 | let path = path | |
| 1202 | .filter(|_| cfg!(not(target_arch = "wasm32"))) | |
| 1203 | .ok_or_else(|| io::Error::other("The cache could not confirm a write"))?; | |
| 1204 | let cold = crate::cache_connection(&path)?; | |
| 1205 | crate::write_ahead(&cold)?; | |
| 1206 | let found: bool = cold.query_row( | |
| 1207 | "SELECT EXISTS(SELECT 1 FROM edits WHERE id=?1)", | |
| 1208 | [crate::signed(last)?], | |
| 1209 | |row| row.get(0), | |
| 1210 | )?; | |
| 1211 | *connection = cold; | |
| 1212 | Ok(found) | |
| 1400 | let Some(path) = path else { | |
| 1401 | let error = "The cache could not confirm a write"; | |
| 1402 | *cache = Cache::Lost(error.to_owned()); | |
| 1403 | return Err(io::Error::other(error).into()); | |
| 1404 | }; | |
| 1405 | let reopened = (|| -> Result<(Connection, Option<i64>)> { | |
| 1406 | if let Owner::Warm(_) = owner { | |
| 1407 | *owner = Owner::Owned(crate::Owned::take(path)?); | |
| 1408 | } | |
| 1409 | let cold = crate::connect(path)?; | |
| 1410 | crate::write_ahead(&cold)?; | |
| 1411 | if cold.is_readonly(rusqlite::MAIN_DB)? { | |
| 1412 | return Err(io::Error::from(io::ErrorKind::ReadOnlyFilesystem).into()); | |
| 1413 | } | |
| 1414 | let batch = cold | |
| 1415 | .query_row( | |
| 1416 | "SELECT batch FROM edits WHERE id=?1", | |
| 1417 | [crate::signed(last)?], | |
| 1418 | |row| row.get(0), | |
| 1419 | ) | |
| 1420 | .optional()?; | |
| 1421 | Ok((cold, batch)) | |
| 1422 | })(); | |
| 1423 | match reopened { | |
| 1424 | Ok((cold, batch)) => { | |
| 1425 | let landed = match batch { | |
| 1426 | Some(batch) => { | |
| 1427 | *open = Some(batch); | |
| 1428 | std::mem::take(ids) | |
| 1429 | } | |
| 1430 | None => Vec::new(), | |
| 1431 | }; | |
| 1432 | *cache = Cache::Open(cold); | |
| 1433 | Ok(landed) | |
| 1434 | } | |
| 1435 | Err(error) => { | |
| 1436 | use rusqlite::ErrorCode::*; | |
| 1437 | let lost = !path.exists() | |
| 1438 | || matches!(&error, crate::Error::Database(rusqlite::Error::SqliteFailure(error, _)) | |
| 1439 | if matches!(error.code, NotADatabase | DatabaseCorrupt)); | |
| 1440 | if lost { | |
| 1441 | *cache = Cache::Lost(error.to_string()); | |
| 1442 | } | |
| 1443 | Err(error) | |
| 1444 | } | |
| 1445 | } | |
| 1213 | 1446 | } |
crates/notebook/tests/commit_fault.rs+546-55| ... | ... | @@ -1,20 +1,47 @@ |
| 1 | //! Commits whose WAL write or sync fails, through a shim over the platform's VFS. Its own | |
| 2 | //! binary, as the shim becomes the process's default VFS. | |
| 1 | //! Commits whose WAL write or sync fails, and caches that will not reopen after one, through a | |
| 2 | //! shim over the platform's VFS. Its own binary, as the shim becomes the process's default VFS. | |
| 3 | 3 | #![cfg(unix)] |
| 4 | 4 | |
| 5 | 5 | use notebook::{ |
| 6 | Replica, | |
| 7 | session::{Event, Section}, | |
| 6 | EditStatus, Error, Replica, | |
| 7 | session::{Event, Section, SyncState}, | |
| 8 | }; | |
| 9 | use onestore::{ | |
| 10 | PageCreation, | |
| 11 | op::{Edit, Op, PageOp, SectionOp}, | |
| 12 | }; | |
| 13 | use std::{ | |
| 14 | io, | |
| 15 | path::{Path, PathBuf}, | |
| 16 | sync::{MutexGuard, PoisonError, atomic::Ordering, mpsc}, | |
| 17 | time::{Duration, Instant}, | |
| 8 | 18 | }; |
| 9 | use onestore::op::{Edit, Op, PageOp}; | |
| 10 | use std::{path::Path, sync::atomic::Ordering, sync::mpsc, time::Duration}; | |
| 11 | 19 | |
| 12 | 20 | #[path = "support/vfs.rs"] |
| 13 | 21 | mod vfs; |
| 14 | 22 | use vfs::{ |
| 15 | ARMED, KEEP_WAL, OPEN, SYNC_AFTER, SYNC_BEFORE, WAL_WRITES_AFTER_FAULT, WRITE_COMMIT, install, | |
| 23 | ARMED, EXCLUSIVE, FAILING_OPENS, KEEP_WAL, SYNC_AFTER, SYNC_BEFORE, WAL_WRITES_AFTER_FAULT, | |
| 24 | WRITE_COMMIT, install, | |
| 16 | 25 | }; |
| 17 | 26 | |
| 27 | /// Holds the shim for one test, disarmed of what a failed test left. | |
| 28 | fn exclusive() -> MutexGuard<'static, ()> { | |
| 29 | install(); | |
| 30 | let exclusive = EXCLUSIVE.lock().unwrap_or_else(PoisonError::into_inner); | |
| 31 | ARMED.store(0, Ordering::SeqCst); | |
| 32 | FAILING_OPENS.store(0, Ordering::SeqCst); | |
| 33 | exclusive | |
| 34 | } | |
| 35 | ||
| 36 | /// A section file in `root` and its cache, which the returned path names. | |
| 37 | fn seed(root: &Path) -> PathBuf { | |
| 38 | let source = onestore::create_section("Fault.one", "Original", "Fixture").unwrap(); | |
| 39 | std::fs::write(root.join("Fault.one"), &source).unwrap(); | |
| 40 | let cache = root.join("replica.sqlite"); | |
| 41 | drop(Replica::create(&cache, &source).unwrap()); | |
| 42 | cache | |
| 43 | } | |
| 44 | ||
| 18 | 45 | fn colour(space: onestore::ExGuid, colour: u32) -> Edit { |
| 19 | 46 | Edit { |
| 20 | 47 | at: 133_000_000_000_000_000 + u64::from(colour), |
| ... | ... | @@ -25,17 +52,49 @@ fn colour(space: onestore::ExGuid, colour: u32) -> Edit { |
| 25 | 52 | } |
| 26 | 53 | } |
| 27 | 54 | |
| 55 | /// The queued edits a cold open of `cache` reads, with the colour each gives the page. | |
| 56 | fn colours(cache: &Path) -> Vec<(u64, u32)> { | |
| 57 | Replica::open(cache) | |
| 58 | .unwrap() | |
| 59 | .pending() | |
| 60 | .unwrap() | |
| 61 | .iter() | |
| 62 | .map(|edit| match &edit.edit.ops[..] { | |
| 63 | [ | |
| 64 | Op::Page { | |
| 65 | op: PageOp::Color(Some(colour)), | |
| 66 | .. | |
| 67 | }, | |
| 68 | ] => (edit.id, *colour), | |
| 69 | ops => panic!("{ops:?}"), | |
| 70 | }) | |
| 71 | .collect() | |
| 72 | } | |
| 73 | ||
| 74 | fn until(section: &Section, seen: impl Fn(&Event) -> bool) { | |
| 75 | let deadline = Instant::now() + Duration::from_secs(10); | |
| 76 | while !section.events().iter().any(&seen) { | |
| 77 | assert!(Instant::now() < deadline); | |
| 78 | std::thread::sleep(Duration::from_millis(10)); | |
| 79 | } | |
| 80 | } | |
| 81 | ||
| 82 | fn would_block<T>(result: Result<T, Error>) -> bool { | |
| 83 | matches!(result, Err(Error::Io(error)) if error.kind() == io::ErrorKind::WouldBlock) | |
| 84 | } | |
| 85 | ||
| 28 | 86 | /// What the section answered for the edit committed under `fault` and the one after it, |
| 29 | 87 | /// whether a failed write was reported, and the queue as the section saw it. |
| 30 | 88 | struct Outcome { |
| 31 | 89 | replies: [Result<u64, String>; 2], |
| 32 | 90 | failed: bool, |
| 33 | pending: Option<Vec<u64>>, | |
| 91 | pending: Vec<u64>, | |
| 34 | 92 | } |
| 35 | 93 | |
| 36 | 94 | /// Seeds the WAL with one edit, so the armed commit's sync is the only one it makes, then |
| 37 | /// commits under `fault` and once more after it; with `crash`, leaves the cache open. | |
| 38 | fn scenario(root: &Path, fault: u8, crash: bool) -> Outcome { | |
| 95 | /// commits under `fault`, the cache failing to reopen `opens` times, and once more after it; | |
| 96 | /// with `crash`, leaves the cache open. | |
| 97 | fn scenario(root: &Path, fault: u8, opens: usize, crash: bool) -> Outcome { | |
| 39 | 98 | install(); |
| 40 | 99 | let replica = Replica::open(root.join("replica.sqlite")).unwrap(); |
| 41 | 100 | let section = Section::resume(root.join("Fault.one"), replica, true, || {}).unwrap(); |
| ... | ... | @@ -54,9 +113,15 @@ fn scenario(root: &Path, fault: u8, crash: bool) -> Outcome { |
| 54 | 113 | }; |
| 55 | 114 | apply(1).unwrap(); |
| 56 | 115 | WAL_WRITES_AFTER_FAULT.store(0, Ordering::SeqCst); |
| 116 | FAILING_OPENS.store(opens, Ordering::SeqCst); | |
| 57 | 117 | ARMED.store(fault, Ordering::SeqCst); |
| 58 | 118 | let first = apply(2); |
| 59 | 119 | assert_eq!(ARMED.load(Ordering::SeqCst), 0, "fault {fault} never fired"); |
| 120 | assert_eq!( | |
| 121 | FAILING_OPENS.load(Ordering::SeqCst), | |
| 122 | 0, | |
| 123 | "the cache never reopened" | |
| 124 | ); | |
| 60 | 125 | assert_eq!( |
| 61 | 126 | WAL_WRITES_AFTER_FAULT.load(Ordering::SeqCst), |
| 62 | 127 | 0, |
| ... | ... | @@ -69,8 +134,10 @@ fn scenario(root: &Path, fault: u8, crash: bool) -> Outcome { |
| 69 | 134 | .any(|event| matches!(event, Event::Failed(_))); |
| 70 | 135 | let pending = section |
| 71 | 136 | .pending() |
| 72 | .ok() | |
| 73 | .map(|pending| pending.iter().map(|edit| edit.id).collect()); | |
| 137 | .unwrap() | |
| 138 | .iter() | |
| 139 | .map(|edit| edit.id) | |
| 140 | .collect(); | |
| 74 | 141 | if crash { |
| 75 | 142 | std::mem::forget(section); |
| 76 | 143 | } else { |
| ... | ... | @@ -85,14 +152,15 @@ fn scenario(root: &Path, fault: u8, crash: bool) -> Outcome { |
| 85 | 152 | |
| 86 | 153 | #[test] |
| 87 | 154 | fn commit_fault_child() { |
| 88 | let (Some(root), Some(fault)) = ( | |
| 155 | let (Some(root), Some(fault), Some(opens)) = ( | |
| 89 | 156 | std::env::var_os("COMMIT_FAULT_CHILD"), |
| 90 | 157 | std::env::var("COMMIT_FAULT").ok(), |
| 158 | std::env::var("COMMIT_FAULT_OPENS").ok(), | |
| 91 | 159 | ) else { |
| 92 | 160 | return; |
| 93 | 161 | }; |
| 94 | 162 | let root = Path::new(&root); |
| 95 | let outcome = scenario(root, fault.parse().unwrap(), true); | |
| 163 | let outcome = scenario(root, fault.parse().unwrap(), opens.parse().unwrap(), true); | |
| 96 | 164 | let ids = |result: &Result<u64, String>| result.as_ref().map_or(0, |id| *id); |
| 97 | 165 | std::fs::write( |
| 98 | 166 | root.join("outcome"), |
| ... | ... | @@ -110,68 +178,491 @@ fn commit_fault_child() { |
| 110 | 178 | |
| 111 | 179 | #[test] |
| 112 | 180 | fn a_failed_commit_is_decided_by_a_cold_read_and_never_replayed_twice() { |
| 113 | install(); | |
| 114 | // (fault, whether the edit reached the cache, whether the cache stays closed) | |
| 115 | for (fault, landed, closed) in [ | |
| 116 | (SYNC_BEFORE, false, false), | |
| 117 | (SYNC_AFTER, false, false), | |
| 118 | (WRITE_COMMIT, false, false), | |
| 119 | (SYNC_AFTER | KEEP_WAL, true, false), | |
| 120 | (SYNC_BEFORE | OPEN, false, true), | |
| 181 | let _exclusive = exclusive(); | |
| 182 | // (fault, whether the edit reached the cache, how many reopens fail) | |
| 183 | for (fault, landed, opens) in [ | |
| 184 | (SYNC_BEFORE, false, 0), | |
| 185 | (SYNC_AFTER, false, 0), | |
| 186 | (WRITE_COMMIT, false, 0), | |
| 187 | (SYNC_AFTER | KEEP_WAL, true, 0), | |
| 188 | (SYNC_BEFORE, false, 2), | |
| 189 | (SYNC_AFTER | KEEP_WAL, true, 2), | |
| 121 | 190 | ] { |
| 122 | 191 | for crash in [false, true] { |
| 123 | 192 | let directory = tempfile::tempdir().unwrap(); |
| 124 | 193 | let root = directory.path(); |
| 125 | let source = onestore::create_section("Fault.one", "Original", "Fixture").unwrap(); | |
| 126 | std::fs::write(root.join("Fault.one"), &source).unwrap(); | |
| 127 | drop(Replica::create(root.join("replica.sqlite"), &source).unwrap()); | |
| 194 | let cache = seed(root); | |
| 128 | 195 | let (first, second, failed) = if crash { |
| 129 | 196 | let output = std::process::Command::new(std::env::current_exe().unwrap()) |
| 130 | 197 | .args(["--exact", "commit_fault_child", "--nocapture"]) |
| 131 | 198 | .env("COMMIT_FAULT_CHILD", root) |
| 132 | 199 | .env("COMMIT_FAULT", fault.to_string()) |
| 200 | .env("COMMIT_FAULT_OPENS", opens.to_string()) | |
| 133 | 201 | .output() |
| 134 | 202 | .unwrap(); |
| 135 | 203 | assert!(output.status.success(), "{output:?}"); |
| 136 | 204 | let outcome = std::fs::read_to_string(root.join("outcome")).unwrap(); |
| 137 | 205 | let mut fields = outcome.split(' '); |
| 138 | let mut id = || Some(fields.next()?.parse::<u64>().unwrap()).filter(|id| *id > 0); | |
| 206 | let mut id = || fields.next().unwrap().parse::<u64>().unwrap(); | |
| 139 | 207 | let (first, second) = (id(), id()); |
| 140 | 208 | (first, second, fields.next() == Some("true")) |
| 141 | 209 | } else { |
| 142 | let outcome = scenario(root, fault, false); | |
| 143 | let [first, second] = outcome.replies; | |
| 144 | if !closed { | |
| 145 | let ids = [first.clone().unwrap(), second.clone().unwrap()]; | |
| 146 | assert_eq!(&outcome.pending.unwrap()[1..], ids); | |
| 147 | } else { | |
| 148 | assert!(outcome.pending.is_none()); | |
| 149 | } | |
| 150 | (first.ok(), second.ok(), outcome.failed) | |
| 210 | let outcome = scenario(root, fault, opens, false); | |
| 211 | let ids = outcome.replies.map(Result::unwrap); | |
| 212 | assert_eq!(outcome.pending[1..], ids); | |
| 213 | (ids[0], ids[1], outcome.failed) | |
| 151 | 214 | }; |
| 152 | assert_eq!(failed, !landed, "fault {fault}"); | |
| 153 | let cold = Replica::open(root.join("replica.sqlite")).unwrap(); | |
| 154 | let colours: Vec<_> = cold | |
| 155 | .pending() | |
| 215 | // An edit waiting for the cache to reopen is reported, as one that did not land. | |
| 216 | assert_eq!(failed, !landed || opens > 0, "fault {fault}, {opens} opens"); | |
| 217 | assert_eq!( | |
| 218 | colours(&cache)[1..], | |
| 219 | [(first, 2), (second, 3)], | |
| 220 | "fault {fault}, {opens} opens" | |
| 221 | ); | |
| 222 | } | |
| 223 | } | |
| 224 | } | |
| 225 | ||
| 226 | #[test] | |
| 227 | fn edits_wait_while_the_cache_cannot_reopen_and_land_once() { | |
| 228 | let _exclusive = exclusive(); | |
| 229 | for close in [false, true] { | |
| 230 | let directory = tempfile::tempdir().unwrap(); | |
| 231 | let root = directory.path(); | |
| 232 | let cache = seed(root); | |
| 233 | let section = Section::resume( | |
| 234 | root.join("Fault.one"), | |
| 235 | Replica::open(&cache).unwrap(), | |
| 236 | true, | |
| 237 | || {}, | |
| 238 | ) | |
| 239 | .unwrap(); | |
| 240 | let space = section.pages().unwrap()[0].0; | |
| 241 | let (replied, replies) = mpsc::channel(); | |
| 242 | let apply = |value| { | |
| 243 | let replied = replied.clone(); | |
| 244 | section | |
| 245 | .apply_with("Fixture", colour(space, value), move |result| { | |
| 246 | replied | |
| 247 | .send(result.map_err(|error| error.to_string())) | |
| 248 | .unwrap(); | |
| 249 | }) | |
| 250 | .unwrap(); | |
| 251 | }; | |
| 252 | let reply = || { | |
| 253 | replies | |
| 254 | .recv_timeout(Duration::from_secs(10)) | |
| 156 | 255 | .unwrap() |
| 157 | .iter() | |
| 158 | .map(|edit| match &edit.edit.ops[..] { | |
| 159 | [ | |
| 160 | Op::Page { | |
| 161 | op: PageOp::Color(Some(colour)), | |
| 162 | .. | |
| 163 | }, | |
| 164 | ] => (edit.id, *colour), | |
| 165 | ops => panic!("{ops:?}"), | |
| 256 | .unwrap() | |
| 257 | }; | |
| 258 | apply(1); | |
| 259 | let seeded = reply(); | |
| 260 | FAILING_OPENS.store(1000, Ordering::SeqCst); | |
| 261 | ARMED.store(SYNC_BEFORE, Ordering::SeqCst); | |
| 262 | apply(2); | |
| 263 | until(&section, |event| matches!(event, Event::Failed(_))); | |
| 264 | // The cache is tried again once a second. | |
| 265 | let left = FAILING_OPENS.load(Ordering::SeqCst); | |
| 266 | std::thread::sleep(Duration::from_secs(3)); | |
| 267 | let tries = left - FAILING_OPENS.load(Ordering::SeqCst); | |
| 268 | // Never faster than once a second; as slow as the machine is busy. | |
| 269 | assert!((1..=4).contains(&tries), "{tries} tries in 3 s"); | |
| 270 | apply(3); | |
| 271 | // A caller blocking on its edit learns at once, and the waiting edits stay. | |
| 272 | assert!(would_block( | |
| 273 | section.replica().apply("Fixture", colour(space, 4)) | |
| 274 | )); | |
| 275 | // A sync step meets the closed cache and the worker carries on. | |
| 276 | section.wake(); | |
| 277 | until( | |
| 278 | &section, | |
| 279 | |event| matches!(event, Event::Failed(error) if error == "The cache reopens once it can"), | |
| 280 | ); | |
| 281 | // Each second the cache is tried again, and the edits still wait. | |
| 282 | until(&section, |event| matches!(event, Event::Failed(_))); | |
| 283 | assert!(replies.try_recv().is_err()); | |
| 284 | assert_eq!(section.page(space).unwrap().color, Some(3)); | |
| 285 | assert!(would_block(section.written())); | |
| 286 | assert!(would_block(section.pending())); | |
| 287 | assert_eq!(notebook::unwritten(), 2); | |
| 288 | let status = section.sync_status().unwrap(); | |
| 289 | // The seed, queued before the commit that failed, and the two waiting. | |
| 290 | assert_eq!((status.state(), status.queued), (SyncState::Failed, 3)); | |
| 291 | // Nobody else takes the cache meanwhile, nor once the section closes. | |
| 292 | assert!(Replica::open(&cache).is_err_and(|error| error.busy())); | |
| 293 | let open = if close { | |
| 294 | section.close().unwrap(); | |
| 295 | assert!(Replica::open(&cache).is_err_and(|error| error.busy())); | |
| 296 | None | |
| 297 | } else { | |
| 298 | Some(section) | |
| 299 | }; | |
| 300 | FAILING_OPENS.store(0, Ordering::SeqCst); | |
| 301 | let ids = [reply(), reply()]; | |
| 302 | if let Some(section) = open { | |
| 303 | section.wake(); | |
| 304 | until(&section, |event| { | |
| 305 | matches!(event, Event::Attempt { id, status: EditStatus::Published { .. } } | |
| 306 | if *id == ids[1]) | |
| 307 | }); | |
| 308 | section.close().unwrap(); | |
| 309 | let arena = onestore::Arena::default(); | |
| 310 | let stored = | |
| 311 | onestore::Section::open(&arena, std::fs::read(root.join("Fault.one")).unwrap()) | |
| 312 | .unwrap(); | |
| 313 | assert_eq!(stored.page(space).unwrap().color, Some(3)); | |
| 314 | assert!(colours(&cache).is_empty()); | |
| 315 | } else { | |
| 316 | // Its cache is released once the edits are written. | |
| 317 | let deadline = Instant::now() + Duration::from_secs(10); | |
| 318 | while Replica::open(&cache).is_err() { | |
| 319 | assert!(Instant::now() < deadline); | |
| 320 | std::thread::sleep(Duration::from_millis(10)); | |
| 321 | } | |
| 322 | assert_eq!(colours(&cache), [(seeded, 1), (ids[0], 2), (ids[1], 3)]); | |
| 323 | } | |
| 324 | assert_eq!(notebook::unwritten(), 0); | |
| 325 | } | |
| 326 | } | |
| 327 | ||
| 328 | #[test] | |
| 329 | fn a_lost_cache_refuses_the_edits_that_wait() { | |
| 330 | let _exclusive = exclusive(); | |
| 331 | for corrupt in [false, true] { | |
| 332 | let directory = tempfile::tempdir().unwrap(); | |
| 333 | let root = directory.path(); | |
| 334 | let cache = seed(root); | |
| 335 | let section = Section::resume( | |
| 336 | root.join("Fault.one"), | |
| 337 | Replica::open(&cache).unwrap(), | |
| 338 | true, | |
| 339 | || {}, | |
| 340 | ) | |
| 341 | .unwrap(); | |
| 342 | let space = section.pages().unwrap()[0].0; | |
| 343 | let (replied, replies) = mpsc::channel(); | |
| 344 | let apply = |value| { | |
| 345 | let replied = replied.clone(); | |
| 346 | section | |
| 347 | .apply_with("Fixture", colour(space, value), move |result| { | |
| 348 | replied | |
| 349 | .send(result.map_err(|error| error.to_string())) | |
| 350 | .unwrap(); | |
| 166 | 351 | }) |
| 167 | .collect(); | |
| 168 | if closed { | |
| 169 | assert_eq!((first, second), (None, None)); | |
| 170 | assert_eq!(colours.len(), 1, "fault {fault}"); | |
| 171 | } else { | |
| 172 | let (first, second) = (first.unwrap(), second.unwrap()); | |
| 173 | assert_eq!(&colours[1..], [(first, 2), (second, 3)], "fault {fault}"); | |
| 352 | .unwrap(); | |
| 353 | }; | |
| 354 | let reply = || replies.recv_timeout(Duration::from_secs(10)).unwrap(); | |
| 355 | apply(1); | |
| 356 | reply().unwrap(); | |
| 357 | FAILING_OPENS.store(usize::MAX, Ordering::SeqCst); | |
| 358 | ARMED.store(SYNC_BEFORE, Ordering::SeqCst); | |
| 359 | apply(2); | |
| 360 | until(&section, |event| matches!(event, Event::Failed(_))); | |
| 361 | for suffix in ["-wal", "-shm"] { | |
| 362 | let mut file = cache.clone().into_os_string(); | |
| 363 | file.push(suffix); | |
| 364 | let _ = std::fs::remove_file(file); | |
| 365 | } | |
| 366 | if corrupt { | |
| 367 | std::fs::write(&cache, [0xab; 4096]).unwrap(); | |
| 368 | } else { | |
| 369 | std::fs::remove_file(&cache).unwrap(); | |
| 370 | } | |
| 371 | FAILING_OPENS.store(0, Ordering::SeqCst); | |
| 372 | assert!(reply().is_err()); | |
| 373 | assert_eq!(notebook::unwritten(), 0); | |
| 374 | let pending = section.pending(); | |
| 375 | assert!(pending.is_err() && !would_block(pending)); | |
| 376 | apply(3); | |
| 377 | assert!(reply().is_err()); | |
| 378 | let status = section.sync_status().unwrap(); | |
| 379 | assert_eq!((status.state(), status.queued), (SyncState::Failed, 0)); | |
| 380 | // The sync worker meets the lost cache and stops. | |
| 381 | std::thread::sleep(Duration::from_millis(100)); | |
| 382 | section.events(); | |
| 383 | section.wake(); | |
| 384 | until(&section, |event| matches!(event, Event::Failed(_))); | |
| 385 | assert!(section.close().is_err()); | |
| 386 | } | |
| 387 | } | |
| 388 | ||
| 389 | #[test] | |
| 390 | fn a_closed_cache_stays_owned_and_its_landed_commit_counts_once() { | |
| 391 | let _exclusive = exclusive(); | |
| 392 | for close in [false, true] { | |
| 393 | let directory = tempfile::tempdir().unwrap(); | |
| 394 | let root = directory.path(); | |
| 395 | let cache = seed(root); | |
| 396 | let section = Section::resume( | |
| 397 | root.join("Fault.one"), | |
| 398 | Replica::open(&cache).unwrap(), | |
| 399 | true, | |
| 400 | || {}, | |
| 401 | ) | |
| 402 | .unwrap(); | |
| 403 | let space = section.pages().unwrap()[0].0; | |
| 404 | let (replied, replies) = mpsc::channel::<Result<u64, String>>(); | |
| 405 | let reply = || { | |
| 406 | replies | |
| 407 | .recv_timeout(Duration::from_secs(10)) | |
| 408 | .unwrap() | |
| 409 | .unwrap() | |
| 410 | }; | |
| 411 | let apply = |value| { | |
| 412 | let replied = replied.clone(); | |
| 413 | section | |
| 414 | .apply_with("Fixture", colour(space, value), move |result| { | |
| 415 | replied | |
| 416 | .send(result.map_err(|error| error.to_string())) | |
| 417 | .unwrap(); | |
| 418 | }) | |
| 419 | .unwrap(); | |
| 420 | }; | |
| 421 | apply(1); | |
| 422 | let seeded = reply(); | |
| 423 | FAILING_OPENS.store(usize::MAX, Ordering::SeqCst); | |
| 424 | ARMED.store(SYNC_AFTER | KEEP_WAL, Ordering::SeqCst); | |
| 425 | // A caller blocking on an edit whose commit fails hears at once, though it landed. | |
| 426 | let asked = Instant::now(); | |
| 427 | assert!( | |
| 428 | section | |
| 429 | .replica() | |
| 430 | .apply("Fixture", colour(space, 2)) | |
| 431 | .is_err() | |
| 432 | ); | |
| 433 | assert!(asked.elapsed() < Duration::from_secs(5)); | |
| 434 | // Waits after the commit the closed cache holds, in the same batch. | |
| 435 | apply(3); | |
| 436 | until(&section, |event| matches!(event, Event::Failed(_))); | |
| 437 | // Nobody else takes the cache to publish and acknowledge that commit meanwhile. | |
| 438 | assert!(Replica::open(&cache).is_err_and(|error| error.busy())); | |
| 439 | let open = if close { | |
| 440 | section.close().unwrap(); | |
| 441 | assert!(Replica::open(&cache).is_err_and(|error| error.busy())); | |
| 442 | None | |
| 443 | } else { | |
| 444 | Some(section) | |
| 445 | }; | |
| 446 | FAILING_OPENS.store(0, Ordering::SeqCst); | |
| 447 | let third = reply(); | |
| 448 | if let Some(section) = open { | |
| 449 | // Reread once the cache reopens, the section is the queue. | |
| 450 | let deadline = Instant::now() + Duration::from_secs(10); | |
| 451 | while !section.pending().is_ok_and(|pending| pending.len() == 3) { | |
| 452 | assert!(Instant::now() < deadline); | |
| 453 | std::thread::sleep(Duration::from_millis(10)); | |
| 454 | } | |
| 455 | assert_eq!(section.page(space).unwrap().color, Some(3)); | |
| 456 | section.close().unwrap(); | |
| 457 | } else { | |
| 458 | let deadline = Instant::now() + Duration::from_secs(10); | |
| 459 | while Replica::open(&cache).is_err() { | |
| 460 | assert!(Instant::now() < deadline); | |
| 461 | std::thread::sleep(Duration::from_millis(10)); | |
| 174 | 462 | } |
| 175 | 463 | } |
| 464 | let stored = colours(&cache); | |
| 465 | assert_eq!(stored.len(), 3, "{stored:?}"); | |
| 466 | assert_eq!( | |
| 467 | (stored[0], stored[1].1, stored[2]), | |
| 468 | ((seeded, 1), 2, (third, 3)) | |
| 469 | ); | |
| 176 | 470 | } |
| 177 | 471 | } |
| 472 | ||
| 473 | #[test] | |
| 474 | fn closed_crash_child() { | |
| 475 | let (Some(root), Some(fault)) = ( | |
| 476 | std::env::var_os("CLOSED_CRASH_CHILD"), | |
| 477 | std::env::var("COMMIT_FAULT").ok(), | |
| 478 | ) else { | |
| 479 | return; | |
| 480 | }; | |
| 481 | install(); | |
| 482 | let root = Path::new(&root); | |
| 483 | let replica = Replica::open(root.join("replica.sqlite")).unwrap(); | |
| 484 | let section = Section::resume(root.join("Fault.one"), replica, true, || {}).unwrap(); | |
| 485 | let space = section.pages().unwrap()[0].0; | |
| 486 | let (replied, replies) = mpsc::channel(); | |
| 487 | let replied = move |result: Result<u64, Error>| replied.send(result.is_ok()).unwrap(); | |
| 488 | let apply = |value, replied: Box<dyn FnOnce(Result<u64, Error>) + Send>| { | |
| 489 | section | |
| 490 | .apply_with("Fixture", colour(space, value), replied) | |
| 491 | .unwrap(); | |
| 492 | }; | |
| 493 | apply(1, Box::new(replied)); | |
| 494 | assert!(replies.recv_timeout(Duration::from_secs(10)).unwrap()); | |
| 495 | FAILING_OPENS.store(usize::MAX, Ordering::SeqCst); | |
| 496 | ARMED.store(fault.parse().unwrap(), Ordering::SeqCst); | |
| 497 | apply(2, Box::new(drop)); | |
| 498 | until(&section, |event| matches!(event, Event::Failed(_))); | |
| 499 | apply(3, Box::new(drop)); | |
| 500 | std::thread::sleep(Duration::from_millis(100)); | |
| 501 | // Ends as a crash would, the cache still closed. | |
| 502 | std::process::exit(0); | |
| 503 | } | |
| 504 | ||
| 505 | #[test] | |
| 506 | fn a_crash_while_the_cache_is_closed_leaves_what_landed() { | |
| 507 | for (fault, landed) in [(SYNC_BEFORE, false), (SYNC_AFTER | KEEP_WAL, true)] { | |
| 508 | let directory = tempfile::tempdir().unwrap(); | |
| 509 | let root = directory.path(); | |
| 510 | let cache = seed(root); | |
| 511 | let output = std::process::Command::new(std::env::current_exe().unwrap()) | |
| 512 | .args(["--exact", "closed_crash_child", "--nocapture"]) | |
| 513 | .env("CLOSED_CRASH_CHILD", root) | |
| 514 | .env("COMMIT_FAULT", fault.to_string()) | |
| 515 | .output() | |
| 516 | .unwrap(); | |
| 517 | assert!(output.status.success(), "{output:?}"); | |
| 518 | let colours: Vec<_> = colours(&cache) | |
| 519 | .into_iter() | |
| 520 | .map(|(_, colour)| colour) | |
| 521 | .collect(); | |
| 522 | assert_eq!( | |
| 523 | colours, | |
| 524 | if landed { vec![1, 2] } else { vec![1] }, | |
| 525 | "fault {fault}" | |
| 526 | ); | |
| 527 | } | |
| 528 | } | |
| 529 | ||
| 530 | /// A section, offline, over a seeded cache in `root`, with its first page's space. | |
| 531 | fn opened(root: &Path) -> (PathBuf, Section, onestore::ExGuid) { | |
| 532 | let cache = seed(root); | |
| 533 | let section = Section::resume( | |
| 534 | root.join("Fault.one"), | |
| 535 | Replica::open(&cache).unwrap(), | |
| 536 | true, | |
| 537 | || {}, | |
| 538 | ) | |
| 539 | .unwrap(); | |
| 540 | let space = section.pages().unwrap()[0].0; | |
| 541 | (cache, section, space) | |
| 542 | } | |
| 543 | ||
| 544 | #[test] | |
| 545 | fn edits_wait_for_the_reread_a_refused_edit_needs_while_the_cache_is_closed() { | |
| 546 | let _exclusive = exclusive(); | |
| 547 | for close in [false, true] { | |
| 548 | let directory = tempfile::tempdir().unwrap(); | |
| 549 | let root = directory.path(); | |
| 550 | let (cache, section, space) = opened(root); | |
| 551 | let (replied, replies) = mpsc::channel::<(u32, Result<u64, String>)>(); | |
| 552 | let apply = |space, value| { | |
| 553 | let replied = replied.clone(); | |
| 554 | section | |
| 555 | .apply_with("Fixture", colour(space, value), move |result| { | |
| 556 | let result = result.map_err(|error| error.to_string()); | |
| 557 | replied.send((value, result)).unwrap(); | |
| 558 | }) | |
| 559 | .unwrap(); | |
| 560 | }; | |
| 561 | let reply = || replies.recv_timeout(Duration::from_secs(10)).unwrap(); | |
| 562 | apply(space, 1); | |
| 563 | let seeded = reply().1.unwrap(); | |
| 564 | FAILING_OPENS.store(usize::MAX, Ordering::SeqCst); | |
| 565 | ARMED.store(SYNC_BEFORE, Ordering::SeqCst); | |
| 566 | // A page a caller blocks on is refused, its commit never landing. | |
| 567 | let creation = PageCreation::new(None, Some("Refused"), "Fixture").unwrap(); | |
| 568 | let made = creation.space(); | |
| 569 | let create = Edit { | |
| 570 | at: 133_000_000_000_000_100, | |
| 571 | ops: vec![Op::Section(SectionOp::Create(creation))], | |
| 572 | }; | |
| 573 | assert!(section.replica().apply("Fixture", create).is_err()); | |
| 574 | // Until the cache reopens to reread the section, edits wait to apply, one on the | |
| 575 | // refused page among them. | |
| 576 | apply(made, 2); | |
| 577 | apply(space, 3); | |
| 578 | std::thread::sleep(Duration::from_millis(1500)); | |
| 579 | assert!(replies.try_recv().is_err()); | |
| 580 | assert_eq!(section.page(space).unwrap().color, Some(1)); | |
| 581 | assert!(would_block(section.written())); | |
| 582 | assert_eq!(notebook::unwritten(), 2); | |
| 583 | let open = if close { | |
| 584 | section.close().unwrap(); | |
| 585 | None | |
| 586 | } else { | |
| 587 | Some(section) | |
| 588 | }; | |
| 589 | FAILING_OPENS.store(0, Ordering::SeqCst); | |
| 590 | // A host reading the section steadily, as one typing does, keeps no retry back. | |
| 591 | let deadline = Instant::now() + Duration::from_secs(10); | |
| 592 | let mut answered = Vec::new(); | |
| 593 | while answered.len() < 2 { | |
| 594 | assert!(Instant::now() < deadline); | |
| 595 | if let Some(section) = &open { | |
| 596 | let _ = section.page(space); | |
| 597 | } | |
| 598 | answered.extend(replies.recv_timeout(Duration::from_millis(300))); | |
| 599 | } | |
| 600 | let mut answered: [_; 2] = answered.try_into().unwrap(); | |
| 601 | answered.sort_by_key(|(value, _)| *value); | |
| 602 | let [(_, dependent), (_, kept)] = answered; | |
| 603 | assert!(dependent.is_err(), "{dependent:?}"); | |
| 604 | let kept = kept.unwrap(); | |
| 605 | if let Some(section) = open { | |
| 606 | // The reread tells the host to reload what it shows. | |
| 607 | until( | |
| 608 | &section, | |
| 609 | |event| matches!(event, Event::Changed(spaces) if spaces.contains(&made)), | |
| 610 | ); | |
| 611 | assert!( | |
| 612 | section | |
| 613 | .pages() | |
| 614 | .unwrap() | |
| 615 | .iter() | |
| 616 | .all(|(listed, ..)| *listed != made) | |
| 617 | ); | |
| 618 | assert_eq!(section.page(space).unwrap().color, Some(3)); | |
| 619 | section.close().unwrap(); | |
| 620 | } | |
| 621 | let deadline = Instant::now() + Duration::from_secs(10); | |
| 622 | while Replica::open(&cache).is_err_and(|error| error.busy()) { | |
| 623 | assert!(Instant::now() < deadline); | |
| 624 | std::thread::sleep(Duration::from_millis(10)); | |
| 625 | } | |
| 626 | assert_eq!(colours(&cache), [(seeded, 1), (kept, 3)]); | |
| 627 | } | |
| 628 | } | |
| 629 | ||
| 630 | #[test] | |
| 631 | fn a_cache_whose_owner_lock_cannot_be_taken_stays_held_by_its_warm_connection() { | |
| 632 | let _exclusive = exclusive(); | |
| 633 | let directory = tempfile::tempdir().unwrap(); | |
| 634 | let root = directory.path(); | |
| 635 | let (cache, section, space) = opened(root); | |
| 636 | let (replied, replies) = mpsc::channel(); | |
| 637 | let apply = |value| { | |
| 638 | let replied = replied.clone(); | |
| 639 | section | |
| 640 | .apply_with("Fixture", colour(space, value), move |result| { | |
| 641 | replied | |
| 642 | .send(result.map_err(|error| error.to_string())) | |
| 643 | .unwrap(); | |
| 644 | }) | |
| 645 | .unwrap(); | |
| 646 | }; | |
| 647 | let reply = || { | |
| 648 | replies | |
| 649 | .recv_timeout(Duration::from_secs(10)) | |
| 650 | .unwrap() | |
| 651 | .unwrap() | |
| 652 | }; | |
| 653 | apply(1); | |
| 654 | let seeded = reply(); | |
| 655 | let mut lock = cache.clone().into_os_string(); | |
| 656 | lock.push("-lock"); | |
| 657 | std::fs::create_dir(&lock).unwrap(); | |
| 658 | ARMED.store(SYNC_BEFORE, Ordering::SeqCst); | |
| 659 | apply(2); | |
| 660 | until(&section, |event| matches!(event, Event::Failed(_))); | |
| 661 | std::thread::sleep(Duration::from_millis(1500)); | |
| 662 | assert!(replies.try_recv().is_err()); | |
| 663 | assert!(Replica::open(&cache).is_err_and(|error| error.busy())); | |
| 664 | std::fs::remove_dir(&lock).unwrap(); | |
| 665 | let second = reply(); | |
| 666 | section.close().unwrap(); | |
| 667 | assert_eq!(colours(&cache), [(seeded, 1), (second, 2)]); | |
| 668 | } |
crates/notebook/tests/support/vfs.rs+24-7| ... | ... | @@ -1,6 +1,6 @@ |
| 1 | 1 | //! A VFS over the platform's, made the process's default by `install`, that fails what a |
| 2 | //! test arms: once, the WAL's sync or commit frame or the next database open; or every WAL | |
| 3 | //! write of the database `FAILING` names, for as long as it names it. | |
| 2 | //! test arms: once, the WAL's sync or commit frame; the replica's next `FAILING_OPENS` opens; | |
| 3 | //! or every WAL write of the database `FAILING` names, for as long as it names it. | |
| 4 | 4 | #![allow(dead_code)] |
| 5 | 5 | |
| 6 | 6 | use rusqlite::ffi; |
| ... | ... | @@ -19,8 +19,6 @@ pub const SYNC_BEFORE: u8 = 1; |
| 19 | 19 | pub const SYNC_AFTER: u8 = 2; |
| 20 | 20 | /// The commit frame's header is never written. |
| 21 | 21 | pub const WRITE_COMMIT: u8 = 4; |
| 22 | /// The next open of a database fails, as the cold reopen after a failed commit. | |
| 23 | pub const OPEN: u8 = 8; | |
| 24 | 22 | /// After a fault, the next sync fails, as the checkpoint closing the warm connection makes, |
| 25 | 23 | /// which keeps the WAL. |
| 26 | 24 | pub const KEEP_WAL: u8 = 16; |
| ... | ... | @@ -29,12 +27,15 @@ pub static ARMED: AtomicU8 = AtomicU8::new(0); |
| 29 | 27 | /// Between a fault and the next database open, as the replica reopens the cache cold. |
| 30 | 28 | static FIRED: AtomicBool = AtomicBool::new(false); |
| 31 | 29 | pub static WAL_WRITES_AFTER_FAULT: AtomicUsize = AtomicUsize::new(0); |
| 30 | /// How many database opens on a replica's own threads still fail, as the cold reopen after a | |
| 31 | /// failed commit can, of the database `FAILING` names where it names one; a test's opens never do. | |
| 32 | pub static FAILING_OPENS: AtomicUsize = AtomicUsize::new(0); | |
| 32 | 33 | static REAL: OnceLock<usize> = OnceLock::new(); |
| 33 | 34 | /// The platform's WAL file methods, the shim's copy of them, and the copy failing every write. |
| 34 | 35 | static WAL: OnceLock<(usize, usize, usize)> = OnceLock::new(); |
| 35 | 36 | /// While set, every write to this database's WAL fails, for WALs opened while it was set. |
| 36 | 37 | pub static FAILING: Mutex<Option<PathBuf>> = Mutex::new(None); |
| 37 | /// Held by a test for as long as it sets `FAILING`, which one test sets at a time. | |
| 38 | /// Held by a test for as long as it arms a fault, which one test does at a time. | |
| 38 | 39 | pub static EXCLUSIVE: Mutex<()> = Mutex::new(()); |
| 39 | 40 | |
| 40 | 41 | fn fire(fault: u8) -> bool { |
| ... | ... | @@ -58,8 +59,24 @@ unsafe extern "C" fn open( |
| 58 | 59 | ) -> c_int { |
| 59 | 60 | if flags & ffi::SQLITE_OPEN_MAIN_DB != 0 { |
| 60 | 61 | FIRED.store(false, Ordering::SeqCst); |
| 61 | if fire(OPEN) { | |
| 62 | FIRED.store(false, Ordering::SeqCst); | |
| 62 | let named = unsafe { CStr::from_ptr(name) } | |
| 63 | .to_string_lossy() | |
| 64 | .into_owned(); | |
| 65 | let chosen = FAILING | |
| 66 | .lock() | |
| 67 | .unwrap() | |
| 68 | .as_ref() | |
| 69 | .is_none_or(|database| named.starts_with(&*database.to_string_lossy())); | |
| 70 | if chosen | |
| 71 | && std::thread::current() | |
| 72 | .name() | |
| 73 | .is_some_and(|name| name.starts_with("onestore-")) | |
| 74 | && FAILING_OPENS | |
| 75 | .try_update(Ordering::SeqCst, Ordering::SeqCst, |opens| { | |
| 76 | opens.checked_sub(1) | |
| 77 | }) | |
| 78 | .is_ok() | |
| 79 | { | |
| 63 | 80 | unsafe { (*file).pMethods = std::ptr::null() }; |
| 64 | 81 | return ffi::SQLITE_CANTOPEN; |
| 65 | 82 | } |
crates/snowbound/src/manage.rs+57-12| ... | ... | @@ -436,12 +436,7 @@ impl State { |
| 436 | 436 | state.vanishing.remove(&location); |
| 437 | 437 | let waiting = (state.session.as_ref()) |
| 438 | 438 | .filter(|session| session.library.location == location) |
| 439 | .is_some_and(|session| { | |
| 440 | session | |
| 441 | .section | |
| 442 | .pending() | |
| 443 | .is_ok_and(|pending| !pending.is_empty()) | |
| 444 | }); | |
| 439 | .is_some_and(|session| unsaved(&session.section) > 0); | |
| 445 | 440 | let listed = (state.notebooks.iter()).find(|open| open.location == location); |
| 446 | 441 | if let Some(library) = listed.cloned().filter(|_| gone && !waiting) { |
| 447 | 442 | state.close_notebook(&library); |
| ... | ... | @@ -583,10 +578,8 @@ impl State { |
| 583 | 578 | .filter(|library| library.in_icloud()) |
| 584 | 579 | .cloned() |
| 585 | 580 | .collect(); |
| 586 | let waiting = (self.sessions()).filter(|session| { | |
| 587 | session.library.in_icloud() | |
| 588 | && (session.section.pending()).is_ok_and(|pending| !pending.is_empty()) | |
| 589 | }); | |
| 581 | let waiting = (self.sessions()) | |
| 582 | .filter(|session| session.library.in_icloud() && unsaved(&session.section) > 0); | |
| 590 | 583 | for session in waiting { |
| 591 | 584 | let Some(folder) = platform::settings_dir().map(|folder| folder.join("Recovery")) |
| 592 | 585 | else { |
| ... | ... | @@ -653,8 +646,7 @@ impl State { |
| 653 | 646 | self.persist().unwrap_or_else(|error| eprintln!("{error}")); |
| 654 | 647 | let open = (self.session.as_ref()) |
| 655 | 648 | .filter(|session| Arc::ptr_eq(&session.library, &library)) |
| 656 | .and_then(|session| session.section.pending().ok()) | |
| 657 | .map_or(0, |pending| pending.len() as u64); | |
| 649 | .map_or(0, |session| unsaved(&session.section)); | |
| 658 | 650 | let waiting = (library.background.iter()) |
| 659 | 651 | .flat_map(|background| background.status()) |
| 660 | 652 | .map(|(_, sync)| sync.queued) |
| ... | ... | @@ -1211,6 +1203,12 @@ fn art_ops(page: &Page, art: Vec<Image>) -> Vec<PageOp> { |
| 1211 | 1203 | ops |
| 1212 | 1204 | } |
| 1213 | 1205 | |
| 1206 | /// The changes `section` holds that its file does not yet, as many as can be told, and at least | |
| 1207 | /// one where none can. | |
| 1208 | fn unsaved(section: &notebook::session::Section) -> u64 { | |
| 1209 | section.sync_status().map_or(1, |sync| sync.queued) | |
| 1210 | } | |
| 1211 | ||
| 1214 | 1212 | #[cfg(test)] |
| 1215 | 1213 | mod tests { |
| 1216 | 1214 | use super::*; |
| ... | ... | @@ -1225,6 +1223,53 @@ mod tests { |
| 1225 | 1223 | creation.dated(&date, &time).unwrap() |
| 1226 | 1224 | } |
| 1227 | 1225 | |
| 1226 | #[test] | |
| 1227 | fn unsaved_counts_what_a_closed_cache_holds() { | |
| 1228 | use std::sync::atomic::Ordering; | |
| 1229 | crate::vfs::install(); | |
| 1230 | let _exclusive = | |
| 1231 | (crate::vfs::EXCLUSIVE.lock()).unwrap_or_else(std::sync::PoisonError::into_inner); | |
| 1232 | let temporary = | |
| 1233 | std::env::temp_dir().join(format!("snowbound-unsaved-{}", std::process::id())); | |
| 1234 | let _ = notebook::fs::remove_dir_all(&temporary); | |
| 1235 | notebook::fs::create_dir_all(&temporary).unwrap(); | |
| 1236 | let file = temporary.join("Unsaved.one"); | |
| 1237 | let cache = temporary.join("unsaved.sqlite"); | |
| 1238 | let source = onestore::create_section("Unsaved.one", "Original", AUTHOR).unwrap(); | |
| 1239 | notebook::fs::write(&file, &source).unwrap(); | |
| 1240 | let open = || { | |
| 1241 | let replica = notebook::Replica::open(&cache).unwrap(); | |
| 1242 | notebook::session::Section::resume(&file, replica, true, || {}).unwrap() | |
| 1243 | }; | |
| 1244 | drop(notebook::Replica::create(&cache, &source).unwrap()); | |
| 1245 | let section = open(); | |
| 1246 | let space = section.pages().unwrap()[0].0; | |
| 1247 | let colour = |colour| Edit { | |
| 1248 | at: crate::filetime(), | |
| 1249 | ops: vec![Op::Page { | |
| 1250 | space, | |
| 1251 | op: PageOp::Color(Some(colour)), | |
| 1252 | }], | |
| 1253 | }; | |
| 1254 | section.replica().apply(AUTHOR, colour(1)).unwrap(); | |
| 1255 | section.close().unwrap(); | |
| 1256 | // The cache's next commit fails, and it cannot reopen; the edit is refused. | |
| 1257 | *crate::vfs::FAILING.lock().unwrap() = Some(cache.canonicalize().unwrap()); | |
| 1258 | crate::vfs::FAILING_OPENS.store(usize::MAX, Ordering::SeqCst); | |
| 1259 | let section = open(); | |
| 1260 | assert_eq!(unsaved(&section), 1); | |
| 1261 | assert!(section.replica().apply(AUTHOR, colour(2)).is_err()); | |
| 1262 | assert!(section.pending().is_err()); | |
| 1263 | assert_eq!(unsaved(&section), 1); | |
| 1264 | *crate::vfs::FAILING.lock().unwrap() = None; | |
| 1265 | crate::vfs::FAILING_OPENS.store(0, Ordering::SeqCst); | |
| 1266 | section.close().unwrap(); | |
| 1267 | while notebook::Replica::open(&cache).is_err() { | |
| 1268 | std::thread::sleep(std::time::Duration::from_millis(10)); | |
| 1269 | } | |
| 1270 | let _ = notebook::fs::remove_dir_all(&temporary); | |
| 1271 | } | |
| 1272 | ||
| 1228 | 1273 | #[test] |
| 1229 | 1274 | fn repeated_section_renames_follow_identity_and_keep_the_latest_label() { |
| 1230 | 1275 | let temporary = |