| 1 | # Snow Globe hardware and recovery |
| 2 | |
| 3 | ## Physical host |
| 4 | |
| 5 | Snow Globe is the existing Ryzen 9 5950X computer, with 128 GB RAM, an RTX 3090, |
| 6 | a 500 GB WD Blue SN5000 NVMe OS disk, and four 8 TB WD80EFPX disks in the |
| 7 | `globe` RAIDZ1 pool. The onboard RTL8168 Ethernet connection is 1 Gb/s. |
| 8 | [The host configuration](../nixos/snowglobe.nix), |
| 9 | [network configuration](../nixos/snowglobe-network.nix), and |
| 10 | [filesystem configuration](../nixos/hardware-configuration.nix) own installed |
| 11 | settings. Use the release deployment workflow for host changes. |
| 12 | |
| 13 | Read-only inspection on 2026-10-16 confirmed the physical machine at |
| 14 | `root@10.0.0.1:22`, a healthy `globe` pool with zero known data errors, and |
| 15 | mounted production, staging, Clover, and Media datasets. The host configuration |
| 16 | names it `snowglobe`. Renaming the computer preserves the existing hardware, |
| 17 | pool, and application state. |
| 18 | |
| 19 | ## Storage boundaries |
| 20 | |
| 21 | | Dataset | Mount | Role | |
| 22 | | --- | --- | --- | |
| 23 | | `globe` | `/srv` | Pool root; unencrypted | |
| 24 | | `globe/clover` | `/srv/clover` | Personal files; independent encryption root | |
| 25 | | `globe/clover/Media` | `/srv/clover/Media` | Media; independent encryption root | |
| 26 | | `globe/prod` | `/srv/prod` | Encrypted managed service data | |
| 27 | | `globe/staging` | `/srv/staging` | Encrypted preview data | |
| 28 | | `globe/logs` | `/srv/logs` | Independent encrypted dataset | |
| 29 | |
| 30 | Old datasets are not active deployment sources. Preserve private migration |
| 31 | evidence when removing obsolete storage. |
| 32 | |
| 33 | Clover and Media are case-insensitive POSIX-ACL filesystems. Their current |
| 34 | properties are `acltype=posix`, `aclmode=discard`, `aclinherit=discard`. |
| 35 | Production and staging also use POSIX ACLs. Preserve allocated UIDs and the |
| 36 | shared group 3000. Test permission changes on a clone; do not recursively |
| 37 | change personal ownership to repair one service. |
| 38 | |
| 39 | The [site mapping](../config/site.pkl) owns media folder names. Saved application |
| 40 | paths are preserved by the service bind mounts: qBittorrent sees Seedbox at |
| 41 | both `/data/media/Seedbox` and `/data/media/torrent`; Jellyfin keeps its |
| 42 | `/media/jellyfin/...` paths. Preserve those mappings and YouTube archive files |
| 43 | alongside their videos when reorganizing media. |
| 44 | |
| 45 | Samba serves these datasets directly using POSIX permissions. Its declaration |
| 46 | preserves shared editing with `0660` files and `2770` directories and uses |
| 47 | kernel oplocks for service-write visibility. Run |
| 48 | `swift tools/samba-macos-test.swift <mounted-share>` against disposable files |
| 49 | after changing permissions or SMB configuration. The probe exercises Cocoa |
| 50 | safe saves, packages, metadata/resource forks, and cleanup. |
| 51 | |
| 52 | ## Unlock and recovery |
| 53 | |
| 54 | The host configuration lists five encrypted roots and their systemd credential |
| 55 | names. TPM-encrypted blobs live in `/etc/credstore.encrypted`; the unlock unit |
| 56 | runs after pool import and before mounting. Plaintext keys must stay outside |
| 57 | the Nix store and tracked configuration. Nomad refuses to start without the |
| 58 | actual pool, Clover, Media, production, and staging mounts. |
| 59 | |
| 60 | Credentials are bound to the machine's TPM and PCR 7. Clearing/replacing the |
| 61 | TPM or changing firmware can require manual unlock and resealing. Secure Boot |
| 62 | is disabled, so automatic TPM unlock does not establish a trusted boot chain. |
| 63 | Keep independent recovery keys when replacing hardware. |
| 64 | |
| 65 | The private off-server recovery archive on Clover's Mac includes the additional |
| 66 | production/staging keys in `postgres-backups/new-os-dataset-keys.json` and the |
| 67 | verified pre-install PostgreSQL logical/physical exports in |
| 68 | `postgres-backups/20261004T230507Z-e75419.tar`. Original dataset keys |
| 69 | and the former OS configuration are in the same private recovery collection. |
| 70 | These paths document locations only; they do not establish current backup |
| 71 | completeness or authorize opening private contents. |
| 72 | |
| 73 | For manual unlock on a confirmed recovery host, recover the required key into |
| 74 | a root-only file in `/run`, then run these commands one at a time with the |
| 75 | verified dataset substituted: |
| 76 | |
| 77 | ```sh |
| 78 | zfs load-key -L file:///run/<private-key-file> <dataset> |
| 79 | rm /run/<private-key-file> |
| 80 | zfs mount <dataset> |
| 81 | findmnt --mountpoint <expected-mountpoint> |
| 82 | ``` |
| 83 | |
| 84 | Loading a key does not mount the filesystem. Mount parents before children; |
| 85 | never overlay a nonempty mountpoint to bypass a mount failure. |
| 86 | |
| 87 | ## Control state and backups |
| 88 | |
| 89 | The NVMe contains both Nomad state and `/var/lib/snowglobe`: service identities, |
| 90 | ACL token files, dashboard accounts, release history, and backup manifests and |
| 91 | database dumps. Read-only `findmnt` inspection on 2026-10-16 confirmed both |
| 92 | control-state locations remain on the ext4 OS filesystem, outside the ZFS |
| 93 | service datasets. Replacing the OS disk without recovering this state loses |
| 94 | more than service files. A Nomad snapshot contains jobs, ACLs, and secret |
| 95 | variables; it does not contain the separate dashboard account database or UID |
| 96 | registry. See [data restore](data-restore.md) for service recovery boundaries. |
| 97 | |
| 98 | No automatic ZFS snapshot or release-backup retention schedule is configured. |
| 99 | Release backups do not substitute for independent off-server recovery copies. |
| 100 | Before destructive storage or OS work, verify key recovery and application/ |
| 101 | database restores from at least two restricted copies stored separately off |
| 102 | this computer. Preserve the existing encrypted roots, pool host ID, numeric |
| 103 | owners, SSH trust, and retained recovery sources through a rebuild. |
| 104 | |
| 105 | ## Namespace migration, 2026-10-21 |
| 106 | |
| 107 | Production's kernel hostname, static hostname, and Nomad node name are |
| 108 | `snowglobe`. Clover manages the Tailscale machine name separately. |
| 109 | |
| 110 | The final telemetry cutover validated 6,949,996 custom metric samples across |
| 111 | 458 series and all 308,283 stored spans, including 4,953,978 renamed samples |
| 112 | and 3,861 renamed spans. Metrics retained every series and timestamp within |
| 113 | the store's numeric precision; two existing conflicting timestamps retained |
| 114 | one of their original values. Traces retained their timestamps, stream labels, |
| 115 | and every field, verified with a normalized all-field digest. Other metrics |
| 116 | and logs remained intact. The original metrics and traces directories are |
| 117 | preserved in `/srv/prod/victoria/archives/namespace-20261021`. Root-only |
| 118 | migration reports and the executed script live under |
| 119 | `/var/lib/snowglobe/migrations/namespace-final-20261021`. |
| 120 | |
| 121 | The 265 pre-rename releases are preserved under |
| 122 | `/opt/snowglobe/archives/releases`; they are archives, excluded from executable |
| 123 | rollback history. `/opt/snowglobe/archives/namespace-20261021` preserves the |
| 124 | complete prior deployment history, retired manager compatibility files and |
| 125 | ACL policy definitions, and the former `/etc/nixos` source tree. The installed |
| 126 | configuration now follows `/opt/snowglobe/current`. Old path aliases, stale |
| 127 | runtime sockets, the unused preview host unit, and unreferenced old ACL |
| 128 | policies have been removed. Numeric identities and active service data paths |
| 129 | were preserved. |