1# Zenith source host inventory
2
3NixOS was installed onto the NVMe on October 4, 2026 and passed two boots over SSH at the existing addresses. The final boot reported `running` with zero failed units: the data pool imported healthy, all six encrypted roots unlocked and mounted automatically through the TPM credentials, and Nomad, Caddy, SSH, the host service, and NVIDIA persistence started successfully. Original dataset GUIDs and all PostgreSQL handoff file hashes matched their pre-install records. The RTX 3090 uses driver 595.71.05; `hardware.graphics.enable` supplies the driver library link needed by its persistence daemon. The historical TrueNAS inventory below describes the earlier layout.
4
5Read-only inspection on 2026-09-27 found a BIOS-booted Ryzen 9 5950X host with a Realtek RTL8111/8168/8411 NIC (`r8169`), RTX 3090, one 500 GB WD Blue SN5000 NVMe boot disk, and four 8 TB WD80EFPX disks. The NVMe is `/dev/disk/by-id/nvme-WD_Blue_SN5000_500GB_24261Z806200`; its GPT has a 1 MB BIOS boot partition, a 512 MB EFI partition, and a TrueNAS `boot-pool` partition with the running root at `boot-pool/ROOT/25.04.2.4`. The four other disks form the healthy `storage1` RAIDZ1 pool. The VM's UEFI boot configuration does not describe this host.
6
7`enp4s0` has static `10.0.0.1/24` and `192.168.0.1/24` addresses, default gateway `10.0.0.2`, and resolvers `1.1.1.1` and `1.0.0.1`. The NixOS target retains these; DHCP would not preserve the NAS address.
8
9`storage1` is unencrypted and mounted at `/mnt/storage1`. Its `apps`, `clover`, and `media` children are separate AES-256-GCM encryption roots. `apps` uses POSIX ACLs; `clover` and `media` use NFSv4 ACLs. Their mountpoints are `/mnt/storage1/apps`, `/mnt/storage1/clover`, and `/mnt/storage1/media`. The pool reported about 16.1 TB free. The [ACL cutover](storage-acl-cutover.md) and [Media cutover](media-cutover.md) describe the data-side migration; no disk or pool changes were made during this inventory.
10
11On 2026-09-27, Zenith ran OpenZFS 2.3.0 and reported `storage1` healthy. The pinned NixOS VM ran OpenZFS 2.4.4; its `zpool upgrade -v` listed every feature currently enabled or active on `storage1`, including encryption and block cloning. This checks feature support, not an actual import of the four-disk pool.
12
13TrueNAS's previous schedule snapshotted Clover hourly for one week, daily for four weeks, and monthly for two years; apps hourly for one week and daily for one month. Media had no scheduled snapshot task. NixOS does not yet replace this retention policy. The pinned NixOS `services.zfs.autoSnapshot` module has global retention counts, so it cannot express these different dataset schedules. Its `services.sanoid` module supports retention per dataset; the owner is choosing the replacement policy before it is enabled.
14
15The October 4 installer boots in UEFI mode on the same hardware. The `zenith` target now uses systemd-boot, host ID `4fa19ccb`, the renamed `globe` pool, and `paperclover.net`. The owner completed the dataset renames in [media-cutover.md](media-cutover.md); retained apps stay at `/mnt/storage1/apps`. Encrypted `globe/prod` and `globe/staging` roots were created with POSIX ACLs for the new service volumes. Nomad's startup guard requires the real pool, Clover, Media, production, and staging mounts; deployment separately checks their ACL types.
16
17The October 4 personal ACL cutover retained `globe/clover@personal-acl-20261004` and `globe/clover/Media@personal-acl-20261004`. Snapshot clones first passed read/write tests for UIDs 3000, 3106, 3114, and 3116 with group 3000. Both original datasets now use `acltype=posix`, `aclmode=discard`, and `aclinherit=discard`; retaining the legacy `aclmode=restricted` caused `EPERM` during rehearsal. One metadata pass processed 1,448,236 Clover entries and 31,187 Media entries without changing owner UIDs: group 3000 receives `g+rwX`, directories have setgid and inheritable group ACLs, and other access is `---`. Traversal stayed on each filesystem and did not follow symlinks. `/var/lib/studio/personal-acl-verified.json` records the completed original passes. The test clones were destroyed after verification; the original snapshots remain.
18
19The ASUS firmware TPM2 successfully sealed and decrypted every ZFS encryption-root key. [zenith.nix](../nixos/zenith.nix) loads encrypted systemd credentials from `/etc/credstore.encrypted` and runs `studio-zfs-unlock` after pool import and before `zfs-mount`. Plaintext keys remain outside the Nix store and NVMe; the running service's credentials exist in RAM. The credentials are bound to this TPM and PCR 7, so firmware changes affecting Secure Boot or clearing/replacing the TPM can require manual recovery. Secure Boot is currently disabled. The original dataset keys and TrueNAS configuration remain in the private Mac recovery archive; new production/staging keys are backed up separately at `/Users/clo/Library/Application Support/Zenith Recovery/postgres-backups/new-os-dataset-keys.json`.
20
21The NVMe installation replaces the old TrueNAS boot pool with a 1 GiB FAT32 EFI partition and an ext4 root partition occupying the remaining space. The only erase target is `/dev/disk/by-id/nvme-WD_Blue_SN5000_500GB_24261Z806200`; no HDD partition is formatted. [legacy-handoff.md](legacy-handoff.md) records the verified exports and off-server backups made before replacing the OS. The generated [hardware configuration](../nixos/hardware-configuration.nix) identifies the new root and EFI filesystems. The installed source lives at `/etc/nixos`; subsequent OS changes use `nixos-rebuild switch --flake /etc/nixos#zenith`.
22
23The target authorizes this Mac's existing ED25519 key for `clo` and root SSH. Its fingerprint `SHA256:52mNGHRsVFBDED9IAX5pe+LRWUefqTbxEReunq21QvU` matches the key Zenith currently accepts from this Mac. The other three keys in Zenith's `clo` authorized-keys file are not copied into the new root account. A synthetic NixOS evaluation confirmed both accounts receive exactly this key.
24
25The original TrueNAS ED25519, ECDSA, and RSA SSH host keys were recovered from the configuration backup, checked against their public keys, and restored into `/etc/ssh` on NixOS. ED25519 fingerprint `SHA256:P8pRWNrL6tceq9KN41Hdb0v0kmIboMf/zFBgOtHJfm0` matches this Mac's existing `git.paperclover.net` trust entry; a fresh connection with strict host checking passed after reloading SSH. The installer keys used during installation are retained privately in `/root/infra2-installer-host-keys`. [Shale migration](shale-migration.md) records the intended Git service scope, transport check, and replacement of the legacy SSH router.
26
27[Nomad stores its server state under `data_dir`](https://developer.hashicorp.com/nomad/docs/configuration), including [variables and their encrypted secret values](https://developer.hashicorp.com/nomad/docs/concepts/variables). Snow Globe stores generated service UIDs, deployment history, backup manifests/dumps, and dashboard state under `/var/lib/studio`. Both paths are on the VM's OS disk, outside its ZFS service datasets. Nomad used `66 MB` at inspection; Snow Globe's `8.7 GB` includes an `8.5 GB` VM-only `zpool.img`, while its release backups used `275 MB` across 25 runs. Reusing or replacing Zenith's NVMe would lose the control records unless they are migrated to encrypted ZFS or backed up separately. The pinned NixOS Nomad module accepts `services.nomad.settings.data_dir = "/srv/prod/nomad"` when `dropPrivileges = false`; a synthetic target evaluation passed. The corresponding Snow Globe state mount and dataset boundary await the storage policy decision before the physical cutover.
28
29A `nomad operator snapshot save` on the VM produced a private, compressed 143,608-byte snapshot in `/run`. Inspection reported 46 variables, 31 jobs, and 25 ACL policies. The snapshot restored into a fresh server-only Nomad agent in an isolated network namespace: its API listed all 31 jobs and 46 variables, and a hash comparison of Shale's secret variable items matched the live server without printing the values. The temporary agent, data, and snapshot were removed; the live leader and jobs stayed healthy. This proves [Nomad's server-state snapshot and restore](https://developer.hashicorp.com/nomad/commands/operator/snapshot/restore) on the pinned version, but Snow Globe's separate UID registry and backup files still need durable storage.
30
31The production target selects NixOS's stable NVIDIA driver for the RTX 3090 and its headless persistence daemon; the synthetic install evaluated with driver 595.71.05. Jellyfin has no GPU device allocation yet, so hardware transcoding still needs a physical-host test.
32
33With root SSH access and Nomad running, `STUDIO_DEPLOY_HOST=root@10.0.0.1 STUDIO_DEPLOY_PORT=22 python3 tools/deploy.py bootstrap` creates the Nomad ACL tokens and service datasets before checking external secrets. The first run reports all missing secret names; load those with `tools/import-legacy-secrets.sh` or `deploy.py secrets`, then rerun bootstrap to launch the jobs. Generated service secrets are created automatically. No Snow Globe job starts before the external-secret check passes. A bootstrap interrupted after switching the release link can be retried until deployment history records success.
34
35The ACL bootstrap ran twice against a disposable Nomad dev agent on the VM, created management, router, and dashboard token files, and left 14 policies. The dev agent and temporary state were removed; the VM's production Nomad agent was untouched.