| 1 | # SSD read cache |
| 2 | |
| 3 | Guest disks remain on the HDD pool. The optional L2ARC holds disposable copies |
| 4 | of recently used blocks on the OS SSD. Cache exhaustion recycles those copies; |
| 5 | guest capacity continues to come from HDD storage. Cache reads that fail are |
| 6 | retried against the HDD pool. Writes still require HDD I/O. |
| 7 | |
| 8 | The host configuration is [zfs-read-cache.nix](../nixos/zfs-read-cache.nix). |
| 9 | It owns the capacity, minimum free space at allocation, and cache-file path; the pool name |
| 10 | comes from the existing host configuration. The service runs after pool import |
| 11 | without making guest startup depend on it. Stopping the service leaves the |
| 12 | cache attached, so normal service restarts preserve its contents. |
| 13 | |
| 14 | The helper preallocates the entire file before attaching it with `zpool add |
| 15 | ... cache ...`. It never adds normal storage, a special vdev, or a log device. |
| 16 | The file and directory are restricted to root. Insufficient free space causes |
| 17 | setup to fail before allocation; other host software can still consume the |
| 18 | remaining free space afterward. An existing attached cache must have the |
| 19 | configured size; an unknown file at the destination is preserved and rejected. |
| 20 | |
| 21 | ## Inspection and recovery |
| 22 | |
| 23 | Use `systemctl status snowglobe-zfs-read-cache.service`, its journal, and |
| 24 | `zpool status -P` to distinguish service setup failure from pool health. A |
| 25 | missing cache can appear as unavailable while the pool remains online. |
| 26 | |
| 27 | If allocation was interrupted, restarting the service clears its abandoned |
| 28 | temporary allocation. If interruption occurred after publishing the file but |
| 29 | before attaching it, setup preserves that unattached file. Verify that the |
| 30 | exact file is absent from every pool's device list before removing it and |
| 31 | retrying the service. |
| 32 | |
| 33 | To replace or resize a cache, stop its setup service and confirm the exact |
| 34 | device appears under **cache** in the pool status. Remove only that cache |
| 35 | device from the pool before deleting its file. Update the capacity declaration, |
| 36 | deploy the reviewed configuration, and start the service to allocate a fresh |
| 37 | cache. Guest disks stay in place and the cache warms again. Do not truncate, |
| 38 | punch holes in, or replace an attached file. |
| 39 | |
| 40 | Changes to the production pool follow the repository's production deployment |
| 41 | workflow. A staged dashboard cannot exercise this host-level configuration. |
| 42 | |
| 43 | ## Verification |
| 44 | |
| 45 | [The isolated NixOS test](../nixos/tests/zfs-read-cache.nix) exercises allocation, |
| 46 | idempotent startup, warmed read hits, bounded cache recycling, import with and |
| 47 | without the cache, writable storage after cache loss, safe replacement, and |
| 48 | rejection of unknown files, symlinks, and insufficient free space. Its small |
| 49 | ARC limit belongs only to the disposable test VM. |
| 50 | |
| 51 | The 2026-10-20 test passed in 64 seconds, recording 97,453 cache hits and |
| 52 | 1,234,213,376 cache-write bytes through a fixed 512 MiB cache. Data checksums |
| 53 | matched after import with the cache removed, and writes still succeeded. |
| 54 | |
| 55 | On 2026-10-20, a separate disposable pool on Snow Globe accepted a fully |
| 56 | preallocated 150 GiB cache file on the existing ext4 SSD. Repeated setup kept |
| 57 | the same file. The fixture and allocation were removed afterward; the |
| 58 | production pool was unchanged. The SSD had approximately 200 GiB available |
| 59 | before allocation. Its largest consumers were Nix store contents (93 GiB), |
| 60 | container layers (72 GiB), and VM experiments/templates (about 49 GiB). |
| 61 | |
| 62 | The Windows provisioning samples from that day included Windows Update |
| 63 | servicing. They do not establish a cache-related startup improvement. L2ARC |
| 64 | benefit requires cache hits beyond those already served by RAM; a virtualized |
| 65 | correctness test is not a physical SSD/HDD performance benchmark. |
| 66 | |
| 67 | See [OpenZFS caching documentation](https://openzfs.github.io/openzfs-docs/Basic%20Concepts/Pool%20Structure/Caching.html). |
| 68 | |
| 69 | ## Production rollout |
| 70 | |
| 71 | Release `40a5a3b0e724162a` activated the cache on 2026-10-20 from commit |
| 72 | `d7a27fd3`, after backup `20261020T195536Z-88c9a1`. The cache was online at its |
| 73 | configured capacity, and restarting its setup service preserved its device |
| 74 | GUID and contents. The pool stayed healthy with no cache I/O or checksum errors. |
| 75 | Approximately 50 GiB remained free on the OS SSD. |
| 76 | |
| 77 | All six previously running guests retained their VM IDs. A new disposable |
| 78 | guest placed its disk under `/srv/vm`, displayed its firmware shell through |
| 79 | the production browser console, accepted a keyboard command, and shut down |
| 80 | normally. Its definition and disk were removed afterward. The release's |
| 81 | service health checks passed. Cache warming was observed; these checks do not |
| 82 | establish a Windows startup speedup. |
| 83 | |
| 84 | Subsequent template preparation and fresh-clone timings are recorded in |
| 85 | [Windows presets](windows-presets.md). Those observations do not isolate the |
| 86 | read cache's contribution. |