1# Media dataset cutover
2
3Before cutover, TrueNAS mounted `storage1/media` at `/mnt/storage1/media`. It is one 3.97 TiB ZFS filesystem containing `jellyfin`, `music`, `music_intake`, and `torrent`; `jellyfin` and `torrent` report the same filesystem device. The dataset had one snapshot with no clones at inspection. The VM reads it through a read-only mount at `/srv/clover/Media` without copying its contents.
4
5The Mac mounted the VM's `media` SMB share over Tailscale, listed the source folders, and received a write error for a test file. The `clover` share is writable in the VM; both shares use the same `clo` account.
6
7Zenith's NFSv4 ACLs need a separate [permission cutover](storage-acl-cutover.md) before this dataset is served by NixOS.
8
9The new layout needs that same dataset at `/srv/clover/Media`. A disposable VM test moved a ZFS filesystem beneath a different parent with `zfs rename`, changed its mountpoint, and confirmed that its snapshot, inode, and hardlinks survived. A second test renamed an independent encrypted root beneath another encrypted root; it remained its own encryption root. These test the ZFS operations, not the live Zenith cutover.
10
11Run the layout commands as root in the live installer, one step at a time. Stop on any error. The old writers and share services must be stopped; booting the installer leaves the old TrueNAS services stopped.
12
13### Reopen the inspected pool for writes
14
15The recovery inspection imported `storage1` read-only with an alternate root under `/run/infra2-inspection`. Export releases that import; it does not delete the pool. Reimport the verified data pool by its numeric ID, without an alternate root. `-N` leaves every dataset unmounted, `cachefile=none` avoids recording this temporary import, and `-f` permits the installer to take over the pool last used by TrueNAS. This ID identifies the four-disk data pool, not the NVMe's `boot-pool`.
16
17```sh
18zpool export storage1
19zpool import -N -f -d /dev/disk/by-id -o cachefile=none 8944904502819631621
20zpool get readonly,altroot storage1
21```
22
23The last command must show `readonly=off` and `altroot=-` before continuing. Leave `boot-pool` alone.
24
25### Record identity and take a snapshot
26
27Keep this shell open: `media_guid` records the dataset's identity for the check after renaming. The recursive snapshot captures all existing datasets' file state before cutover. It does not undo dataset names or mountpoint properties.
28
29```sh
30media_guid=$(zfs get -H -o value guid storage1/media)
31zfs snapshot -r storage1@before-infra2-layout
32```
33
34### Rename the pool to globe
35
36The top-level dataset has the pool's name. Rename it by exporting and importing the pool with a new name, not with `zfs rename`. These commands also work midway through the mountpoint changes below, provided the datasets remain unmounted.
37
38```sh
39zpool export storage1
40zpool import -N -d /dev/disk/by-id -o cachefile=none 8944904502819631621 globe
41zpool get guid,readonly,altroot globe
42zfs list -r -o name,mountpoint,mounted globe
43```
44
45Expect the same pool GUID `8944904502819631621`, `readonly=off`, and `altroot=-`. Every dataset and snapshot prefix becomes `globe/`; their GUIDs, encryption keys, and snapshots are retained. The explicit `/srv` mountpoint and pinned `/mnt/storage1/apps` path remain unchanged. The production configuration uses `globe` for imports and service startup checks. See [OpenZFS pool import](https://openzfs.github.io/openzfs-docs/man/master/8/zpool-import.8.html).
46
47From here onward, use `globe` in ZFS arguments. Keep `/mnt/storage1/apps` as a filesystem path: the legacy importers still read it there. Existing recovery exports label their keys with the original `storage1/...` names; those same keys now unlock the corresponding `globe/...` datasets.
48
49### Set the layout
50
51`mountpoint` controls where a dataset appears in the filesystem. `rename` changes its name and parent in the ZFS hierarchy. The `-u` flags update metadata without mounting datasets. Encrypted datasets can be renamed while locked; these commands do not change their keys or copy their file contents. See [OpenZFS rename](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-rename.8.html) and [mountpoint properties](https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html).
52
53| Command | Effect |
54| --- | --- |
55| `zfs set -u mountpoint=/mnt/storage1/apps globe/apps` | Pin the retained app tree at the path used by the legacy importers, before moving the pool root. |
56| `zfs set -u mountpoint=/srv globe` | Move the pool's root mountpoint; default-mountpoint children such as backups, logs, and mirrors follow it. |
57| `zfs set -u mountpoint=/srv/clover globe/clover` | Set Clover's mount location. Its dataset name stays `globe/clover`. |
58| `zfs rename -u globe/media globe/clover/Media` | Move the existing Media dataset beneath Clover and capitalize its name. Its snapshots stay attached to the same dataset. |
59| `zfs set -u mountpoint=/srv/clover/Media globe/clover/Media` | Set Media's mount location. It remains an independent encryption root with its original key. |
60
61The destination dataset must not already exist, and the `Media` directory inside Clover must be absent or empty before mounting there. The live inspection checked both on 2026-10-04: the dataset and directory were absent.
62
63### Verify before mounting
64
65```sh
66test "$(zfs get -H -o value guid globe/clover/Media)" = "$media_guid"
67zfs list -o name,mountpoint,mounted globe globe/apps globe/clover globe/clover/Media
68zfs get encryptionroot globe/apps globe/clover globe/clover/Media
69zfs list -t snapshot -r -o name globe/clover/Media
70```
71
72The identity check must succeed. Expect these mountpoint properties, with `mounted=no` throughout:
73
74```text
75globe /srv
76globe/apps /mnt/storage1/apps
77globe/clover /srv/clover
78globe/clover/Media /srv/clover/Media
79```
80
81Media's `encryptionroot` must be `globe/clover/Media`, not `globe/clover`; apps and Clover must remain their own encryption roots. Media's original snapshots should now use its new dataset prefix, alongside `@before-infra2-layout`. Loading keys and mounting the filesystems follow this metadata check; permission conversion is covered in [storage-acl-cutover.md](storage-acl-cutover.md). Preserve the old NVMe until the PostgreSQL handoff in [legacy-handoff.md](legacy-handoff.md) is verified.
82
83A disposable pool with an altroot reproduced the default child's move when the parent mountpoint changed, then kept the old child path after an explicit mountpoint was set. After loading keys and mounting, Clover and Media must resolve to distinct ZFS filesystems before starting Snow Globe; `tools/studio.py pool` enforces this. Check a known hardlinked download/library pair by device and inode, then verify Jellyfin and Navidrome library paths inside their containers.
84
85The root change also relocates other default-mountpoint children: `backup` (including the 2.07 TB `backup/sandwich`), `mirrors` (266 GB), `agent`, `homes`, `logs`, and `.ix-virt`. They remain datasets under `/srv` until deliberately renamed; the move does not copy their data. `ix-apps` has a local `/mnt/.ix-apps` mountpoint and `.system` uses `legacy`, so neither follows the root. Preserve these datasets during cutover.
86
87### Remove retired app clones and the rehearsal store
88
89The five `apps-hourly-…-clone` entries are cloned filesystems, distinct from the original `globe/apps@hourly-…` snapshots. The rehearsal store is `globe/apps/snowglobe-rehearsal-20261002` (59.8 GiB at inspection). The live installer inspection on 2026-10-04 confirmed that all six are unmounted, and `zfs destroy -nvr` previews succeeded: each would delete only the named filesystem and its own `@before-infra2-layout` snapshot.
90
91These commands permanently remove those six retired filesystems and their snapshots. They preserve `globe/apps`, its original hourly snapshots, and the other datasets' cutover snapshots. Run one command at a time and stop on an error. To preview a command again, replace `-vr` with `-nvr`; lowercase `-r` includes the target's children and snapshots. Do not use uppercase `-R`, which also destroys dependent clones outside the target hierarchy. See [OpenZFS destroy](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-destroy.8.html).
92
93```sh
94zfs destroy -vr globe/apps-hourly-2026-09-06_21-00-clone
95zfs destroy -vr globe/apps-hourly-2026-09-08_23-00-clone
96zfs destroy -vr globe/apps-hourly-2026-09-11_07-00-clone
97zfs destroy -vr globe/apps-hourly-2026-09-11_12-00-clone
98zfs destroy -vr globe/apps-hourly-2026-09-12_14-00-clone
99zfs destroy -vr globe/apps/snowglobe-rehearsal-20261002
100```
101
102Check the remaining app datasets and source snapshots afterward:
103
104```sh
105zfs list -r -o name,used,mountpoint globe/apps
106zfs list -t snapshot -d 1 -o name globe/apps
107```
108
109### Unlock and mount Clover and Media
110
111Clover and Media are independent encryption roots, so each needs its original key. The recovery export labels these `storage1/clover` and `storage1/media`, respectively. For the installer commands below, the recovered keys must be available as `/run/infra2-keys/clover.hex` and `/run/infra2-keys/media.hex`, with directory mode `0700` and file mode `0600`, owned by root. These temporary files disappear at reboot; recover them from the private archive again if needed. Keep keys out of this repository and terminal output.
112
113Run one command at a time and stop on errors. Mount the pool root if needed, then verify that `/srv` is its actual mountpoint:
114
115```sh
116if [ "$(zfs get -H -o value mounted globe)" = no ]; then zfs mount globe; fi
117test "$(findmnt -n -o SOURCE --mountpoint /srv)" = globe
118```
119
120Load the two keys, mount Clover before its Media child, then remove the temporary key files after both mounts succeed. `-L` overrides the key source for this command without changing the datasets' persistent `keylocation=prompt`. Loading keys alone does not mount them; see [OpenZFS load-key](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-load-key.8.html). Use ordinary [ZFS mounts](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-mount.8.html), without overlay options; stop if the Media mountpoint is nonempty.
121
122```sh
123zfs load-key -L file:///run/infra2-keys/clover.hex globe/clover
124zfs load-key -L file:///run/infra2-keys/media.hex globe/clover/Media
125zfs mount globe/clover
126zfs mount globe/clover/Media
127rm /run/infra2-keys/clover.hex /run/infra2-keys/media.hex
128test "$(findmnt -n -o SOURCE --mountpoint /srv/clover)" = globe/clover
129test "$(findmnt -n -o SOURCE --mountpoint /srv/clover/Media)" = globe/clover/Media
130findmnt -rn -o SOURCE,TARGET,FSTYPE --mountpoint /srv/clover/Media
131```
132
133The last command must show `globe/clover/Media /srv/clover/Media zfs`. This mounts only Clover and Media beneath the pool root. Continue to the folder renames below after these checks succeed. Root can perform these renames; service access still requires the separate ACL cutover.
134
135### Unlock retained apps for the legacy handoff
136
137The original apps key is labeled `storage1/apps` in the recovery export. Prepare it as `/run/infra2-keys/apps.hex` with the same root-only permissions as the Clover and Media keys. Unlock and mount the retained dataset at its pinned legacy path:
138
139```sh
140zfs load-key -L file:///run/infra2-keys/apps.hex globe/apps
141zfs mount globe/apps
142rm /run/infra2-keys/apps.hex
143test "$(findmnt -n -o SOURCE --mountpoint /mnt/storage1/apps)" = globe/apps
144findmnt -rn -o SOURCE,TARGET,FSTYPE --mountpoint /mnt/storage1/apps
145```
146
147Expect `globe/apps /mnt/storage1/apps zfs`. Keep this path intact for the [legacy handoff](legacy-handoff.md); mounting the dataset does not start the old apps.
148
149### Keys after installation
150
151The recovery keys remain in the private recovery archive on the Mac. The `/run/infra2-keys` files above are temporary transfer copies; after `load-key`, ZFS holds the loaded key material in memory, so deleting those files does not relock mounted datasets. Reboot loses the loaded keys. The datasets retain `keylocation=prompt` for manual recovery; the installed target loads TPM-encrypted credentials before mounting them. Nomad's mount checks prevent it from starting against locked data.
152
153Zenith's enabled TPM 2.0 (`/dev/tpm0` and `/dev/tpmrm0`) sealed and decrypted all six encryption-root keys during installation preparation on 2026-10-04. A systemd service test successfully loaded the encrypted credentials and unlocked the previously locked logs dataset. [zenith-hardware.md](zenith-hardware.md) records their recovery locations and the installed boot service; the root names and credential paths have one home in [zenith.nix](../nixos/zenith.nix).
154
155[TPM-encrypted systemd credentials](https://github.com/systemd/systemd/blob/main/man/systemd-creds.xml) keep encrypted key blobs on the NVMe and supply decrypted credentials in RAM. These credentials bind to the original TPM and PCR 7. Secure Boot is disabled, so this supplies convenient automatic unlock and protection for detached drives, without establishing a trusted boot chain. Firmware changes affecting PCR 7 or a TPM reset can require resealing the credentials from the recovery keys.
156
157Manual unlock over root SSH remains the recovery path: recover the relevant key to a private `/run` file, use `zfs load-key -L file:///run/<key-file> <dataset>`, remove that file, then mount the dataset. Keep the off-server keys when replacing hardware; encrypted blobs from the old TPM cannot unlock on a replacement machine.
158
159### Rename the media folders
160
161Clover and Media both report `casesensitivity=insensitive` on the live pool. Names differing only by capitalization resolve to the same entry. For a case-only folder rename, use an unused temporary name; ordinary `mv music Music` interprets the existing destination directory as a request to move `music` inside itself. From the Media directory:
162
163```sh
164test ! -e .music-case-rename && test ! -L .music-case-rename &&
165mv -T -- music .music-case-rename &&
166mv -T -- .music-case-rename Music
167```
168
169Lowercase configured paths still resolve after a case-only rename; do not create a separate `music -> Music` or `seedbox -> Seedbox` alias, because those names collide on this filesystem. The distinct `torrent` alias below preserves the old host path. qBittorrent also receives a container mount at `/data/media/torrent` pointing to `Seedbox`, so its saved torrent paths remain valid.
170
171Zenith's qBittorrent `BT_backup` is 21 MB. A checksum-verified copy contained 544 complete bencoded resume files. Their `save_path` values include 498 at `/data/media/torrent`, three in its subfolders, and 43 in Jellyfin folders; 543 also set `qBt-downloadPath` to `/data/media/torrent`. Keep these saved paths working by renaming the real media folder and leaving a relative compatibility symlink, after the media dataset is mounted at its new path and the old writers are stopped:
172
173```sh
174test -d /srv/clover/Media/torrent
175test ! -e /srv/clover/Media/Seedbox
176mv /srv/clover/Media/torrent /srv/clover/Media/Seedbox
177ln -s Seedbox /srv/clover/Media/torrent
178test "$(stat -c %d:%i /srv/clover/Media/Seedbox)" = "$(stat -Lc %d:%i /srv/clover/Media/torrent)"
179```
180
181The same-dataset rename preserves existing hardlinks. New downloads use `/data/media/Seedbox`; saved torrents continue to resolve through `/data/media/torrent`. If `Seedbox` and the `torrent` alias already exist, skip the rename and verify their device/inode identity. Rename the existing music intake folder after stopping YouTube Triage, then keep its current container path working through a relative symlink:
182
183```sh
184test -d /srv/clover/Media/music_intake
185test ! -e '/srv/clover/Media/Intake - Music'
186mv /srv/clover/Media/music_intake '/srv/clover/Media/Intake - Music'
187ln -s 'Intake - Music' /srv/clover/Media/music_intake
188```
189
190### Service paths for the reorganized tree
191
192The service definitions read the current capitalized host folders. Jellyfin and qBittorrent retain their saved paths inside the containers through individual bind mounts; the host needs no `jellyfin` compatibility directory. The shared folder mapping is defined once in [config/site.pkl](../config/site.pkl), under `jellyfinFolders`. Jellyfin mounts its entries beneath `/media/jellyfin`; qBittorrent mounts them beneath `/data/media/jellyfin` and mounts `Seedbox` at both `/data/media/Seedbox` and `/data/media/torrent`.
193
194Navidrome mounts `Music` at `/music`. Jellyfin and Navidrome mounts remain read-only; qBittorrent follows `site.mediaReadOnly`, so read-only rehearsals stay read-only. Keeping the recorded Jellyfin paths follows its [migration guidance](https://jellyfin.org/docs/general/administration/migrate/) and avoids rewriting its database paths or library settings.
195
196Before reorganization, YouTube Archiver kept its download archive JSON beside the videos in `Media/jellyfin/Independent`, not in its `/config` cache ([upstream archive behavior](https://github.com/jmbannon/ytdl-sub/wiki/5.-Optimizing-Your-First-Config)). The source inspection found 12 archive files there (52,345 bytes), and all 12 were visible through the VM's read-only media mount. That folder now lives at `Media/Videos/Independent`; preserve its archive files with the videos. The old app directory contained only a lock, cache, and empty work directory.
197
198The complete 33 MB qBittorrent config was also copied into a disposable ZFS clone in the VM. The pinned container started with `--network none` and no media mount; its API loaded all 544 torrents. All reported `missingFiles`, as expected without `/data/media`. One resume file without `qBt-savePath` adopted the new default `/data/media/seedbox`; the other 543 retained their saved paths. The container, clone, and copied config were removed after the test.
199
200A second disposable restore copied Zenith's qBittorrent profile into the VM and mounted the real read-only media through a temporary `/data/media/seedbox` alias, with `torrent -> seedbox` alongside it. The container ran as UID 3000 with no network and a read-only root filesystem. Its API loaded all 544 torrents; after resume checks, all 544 reported 100% progress, with 539 `queuedUP`, three `stalledUP`, two `forcedUP`, and none `missingFiles`. The container resolved the legacy symlink, and its temporary profile was removed. This tests the saved paths and source files through the VM mount; the actual Zenith folder has not yet been renamed.
201
202The old resume files are owned by UID 3000, while Snow Globe assigns qBittorrent UID 3114. [import-qbittorrent.sh](import-qbittorrent.sh) copies the stopped old profile into the stopped Snow Globe service's dataset, snapshots the destination, checks the copy, changes ownership, applies the new save path, and starts the service. It refuses to run while Zenith's qBittorrent is active. In a disposable VM restore, UID 3114 loaded all 544 imported torrents without a media mount; all correctly reported `missingFiles`. With the same read-only media alias and legacy symlink as above, all 544 reached 100% progress and none remained `missingFiles`.
203
204The production importer requires the real Media ZFS mount and a `torrent` symlink resolving to the same directory as `Seedbox` before it starts qBittorrent. It compares device/inode identity, so `torrent -> Seedbox/` is accepted. For a same-machine OS replacement, set `STUDIO_LEGACY_HANDOFF` to the [offline handoff](legacy-handoff.md) directory; the profile copy then reads the retained old apps dataset without old Docker.
205
206The PIA credentials are the two lines in `~/pia.txt` on the Mac (username, then password). With `STUDIO_DEPLOY_HOST` and `STUDIO_DEPLOY_PORT` set for production, import them through `python3 tools/deploy.py secrets qbittorrent --file ~/pia.txt --key vpn_user --key vpn_pass` before deploying qBittorrent. The CLI requires a private file, sends its contents to the target over SSH, and does not print the values.
207
208At the 2026-09-26 check, Zenith's Sonarr and Radarr databases each had an enabled qBittorrent client and enabled RSS indexers; the VM's production and preview copies had both disabled. The preview setup disables every indexer's RSS and automatic search, including Sonarr's built-in EZTV indexer, while leaving the source untouched. A production [ARR import](import-arr.sh) now requires the corresponding Zenith container to be stopped before copying its database, and checks it remains stopped when the copy finishes. Reenable downloads only after the media dataset is writable at the final mountpoint and qBittorrent's imported profile is serving the same paths.
209
210After a same-machine OS replacement, `STUDIO_LEGACY_HANDOFF` makes the Sonarr and Radarr importer read their stopped databases and app files from the mounted old apps dataset on the new host. A read-only SQLite backup and integrity check passed for each old database.
211
212The VM runs Sonarr `4.0.20.3014` and Radarr `6.4.4.10685`. [Sonarr v4's External mode](https://wiki.servarr.com/sonarr/settings) delegates authentication to a reverse proxy, and [Radarr's current FAQ](https://wiki.servarr.com/radarr/faq) says native OIDC is unsupported. Sonarr's [v5 development settings](https://github.com/Sonarr/Sonarr/blob/v5-develop/src/Sonarr.Api.V5/Settings/GeneralSettingsResource.cs) contain OIDC fields; the latest stable [Sonarr release](https://github.com/Sonarr/Sonarr/releases) remains v4 at this check. Both pinned services use Caddy Forward Auth with External mode. Unauthenticated UI and API requests redirected to sign-in, and forged identity headers sent from the Mac still received HTTP 302 over trusted TLS. Nomad advertised their application ports only on `127.0.0.1`.
213
214[import-jackett.sh](import-jackett.sh) makes the Jackett dependency repeatable: previews copy the latest read-only Zenith app snapshot; production import requires both old and new Jackett stopped. The restored preview kept the same API key as Zenith and the VM's production Jackett, preserved all three indexer definitions, and served `t=caps` for Nyaa and Pirate Bay with HTTP 200. The imported Sonarr and Radarr Jackett indexers use that same key and point to the current Nomad Jackett endpoint.
215
216With `STUDIO_LEGACY_HANDOFF` set, production Jackett import reads its retained config from the new host's mounted old apps dataset after reboot; it no longer needs the old Docker host.
217
218Sonarr and Radarr persist their Jackett and qBittorrent endpoints in their databases, so those two internal ports are static (`30017` and `30038`) across allocation changes. On the VM, both apps' saved Jackett URLs matched the current reserved port. Their public HTTPS routes still use separate Caddy endpoints and forward auth.
219
220[import-navidrome.sh](import-navidrome.sh) restores the old `navidrone` app directory into Navidrome's managed dataset. The preview imported Zenith's `storage1/apps@hourly-2026-09-26_02-00` snapshot, migrated the database under the pinned image, and passed SQLite integrity and HTTP health checks. Its three users, 10,945 media files, 1,046 albums, and 2,655 artists matched the source snapshot; `snow` remained an administrator. A `Remote-User: snow` request to the production Navidrome API returned the existing `snow` user, proving that its external-auth setting accepts the proxy's header. The preview's database is retained, but its job is stopped while the Jackett preview uses the VM's limited memory. Production import requires both Navidrome jobs stopped and the real Media dataset mounted as ZFS.
221
222For a same-machine OS replacement, set `STUDIO_LEGACY_HANDOFF` to the [offline handoff](legacy-handoff.md) directory when importing production Jellyfin or Navidrome. Both importers read their retained app data directly from the mounted old apps dataset on the new host; the Media library stays on its ZFS dataset.