1# Snow Globe hardware and recovery
2
3## Physical host
4
5Snow Globe is the existing Ryzen 9 5950X computer, with 128 GB RAM, an RTX 3090,
6a 500 GB WD Blue SN5000 NVMe OS disk, and four 8 TB WD80EFPX disks in the
7`globe` RAIDZ1 pool. The onboard RTL8168 Ethernet connection is 1 Gb/s.
8[The host configuration](../nixos/snowglobe.nix),
9[network configuration](../nixos/snowglobe-network.nix), and
10[filesystem configuration](../nixos/hardware-configuration.nix) own installed
11settings. Use the release deployment workflow for host changes.
12
13Read-only inspection on 2026-10-16 confirmed the physical machine at
14`root@10.0.0.1:22`, a healthy `globe` pool with zero known data errors, and
15mounted production, staging, Clover, and Media datasets. The host configuration
16names it `snowglobe`. Renaming the computer preserves the existing hardware,
17pool, and application state.
18
19## Storage boundaries
20
21| Dataset | Mount | Role |
22| --- | --- | --- |
23| `globe` | `/srv` | Pool root; unencrypted |
24| `globe/clover` | `/srv/clover` | Personal files; independent encryption root |
25| `globe/clover/Media` | `/srv/clover/Media` | Media; independent encryption root |
26| `globe/prod` | `/srv/prod` | Encrypted managed service data |
27| `globe/staging` | `/srv/staging` | Encrypted preview data |
28| `globe/logs` | `/srv/logs` | Independent encrypted dataset |
29
30Old datasets are not active deployment sources. Preserve private migration
31evidence when removing obsolete storage.
32
33Clover and Media are case-insensitive POSIX-ACL filesystems. Their current
34properties are `acltype=posix`, `aclmode=discard`, `aclinherit=discard`.
35Production and staging also use POSIX ACLs. Preserve allocated UIDs and the
36shared group 3000. Test permission changes on a clone; do not recursively
37change personal ownership to repair one service.
38
39The [site mapping](../config/site.pkl) owns media folder names. Saved application
40paths are preserved by the service bind mounts: qBittorrent sees Seedbox at
41both `/data/media/Seedbox` and `/data/media/torrent`; Jellyfin keeps its
42`/media/jellyfin/...` paths. Preserve those mappings and YouTube archive files
43alongside their videos when reorganizing media.
44
45Samba serves these datasets directly using POSIX permissions. Its declaration
46preserves shared editing with `0660` files and `2770` directories and uses
47kernel oplocks for service-write visibility. Run
48`swift tools/samba-macos-test.swift <mounted-share>` against disposable files
49after changing permissions or SMB configuration. The probe exercises Cocoa
50safe saves, packages, metadata/resource forks, and cleanup.
51
52## Unlock and recovery
53
54The host configuration lists five encrypted roots and their systemd credential
55names. TPM-encrypted blobs live in `/etc/credstore.encrypted`; the unlock unit
56runs after pool import and before mounting. Plaintext keys must stay outside
57the Nix store and tracked configuration. Nomad refuses to start without the
58actual pool, Clover, Media, production, and staging mounts.
59
60Credentials are bound to the machine's TPM and PCR 7. Clearing/replacing the
61TPM or changing firmware can require manual unlock and resealing. Secure Boot
62is disabled, so automatic TPM unlock does not establish a trusted boot chain.
63Keep independent recovery keys when replacing hardware.
64
65The private off-server recovery archive on Clover's Mac includes the additional
66production/staging keys in `postgres-backups/new-os-dataset-keys.json` and the
67verified pre-install PostgreSQL logical/physical exports in
68`postgres-backups/20261004T230507Z-e75419.tar`. Original dataset keys
69and the former OS configuration are in the same private recovery collection.
70These paths document locations only; they do not establish current backup
71completeness or authorize opening private contents.
72
73For manual unlock on a confirmed recovery host, recover the required key into
74a root-only file in `/run`, then run these commands one at a time with the
75verified dataset substituted:
76
77```sh
78zfs load-key -L file:///run/<private-key-file> <dataset>
79rm /run/<private-key-file>
80zfs mount <dataset>
81findmnt --mountpoint <expected-mountpoint>
82```
83
84Loading a key does not mount the filesystem. Mount parents before children;
85never overlay a nonempty mountpoint to bypass a mount failure.
86
87## Control state and backups
88
89The NVMe contains both Nomad state and `/var/lib/snowglobe`: service identities,
90ACL token files, dashboard accounts, release history, and backup manifests and
91database dumps. Read-only `findmnt` inspection on 2026-10-16 confirmed both
92control-state locations remain on the ext4 OS filesystem, outside the ZFS
93service datasets. Replacing the OS disk without recovering this state loses
94more than service files. A Nomad snapshot contains jobs, ACLs, and secret
95variables; it does not contain the separate dashboard account database or UID
96registry. See [data restore](data-restore.md) for service recovery boundaries.
97
98No automatic ZFS snapshot or release-backup retention schedule is configured.
99Release backups do not substitute for independent off-server recovery copies.
100Before destructive storage or OS work, verify key recovery and application/
101database restores from at least two restricted copies stored separately off
102this computer. Preserve the existing encrypted roots, pool host ID, numeric
103owners, SSH trust, and retained recovery sources through a rebuild.
104
105## Namespace migration, 2026-10-21
106
107Production's kernel hostname, static hostname, and Nomad node name are
108`snowglobe`. Clover manages the Tailscale machine name separately.
109
110The final telemetry cutover validated 6,949,996 custom metric samples across
111458 series and all 308,283 stored spans, including 4,953,978 renamed samples
112and 3,861 renamed spans. Metrics retained every series and timestamp within
113the store's numeric precision; two existing conflicting timestamps retained
114one of their original values. Traces retained their timestamps, stream labels,
115and every field, verified with a normalized all-field digest. Other metrics
116and logs remained intact. The original metrics and traces directories are
117preserved in `/srv/prod/victoria/archives/namespace-20261021`. Root-only
118migration reports and the executed script live under
119`/var/lib/snowglobe/migrations/namespace-final-20261021`.
120
121The 265 pre-rename releases are preserved under
122`/opt/snowglobe/archives/releases`; they are archives, excluded from executable
123rollback history. `/opt/snowglobe/archives/namespace-20261021` preserves the
124complete prior deployment history, retired manager compatibility files and
125ACL policy definitions, and the former `/etc/nixos` source tree. The installed
126configuration now follows `/opt/snowglobe/current`. Old path aliases, stale
127runtime sockets, the unused preview host unit, and unreferenced old ACL
128policies have been removed. Numeric identities and active service data paths
129were preserved.