| 1 | # Shale migration |
| 2 | |
| 3 | Shale is the sole Git server on Zenith. Personal Forgejo is retired. Retained Forgejo repositories and database exports remain migration sources; copying the existing Shale directory does not import repositories that only existed in Forgejo. The legacy multi-server SSH router is not part of the target setup. |
| 4 | |
| 5 | The 2026-10-04 personal inventory found 12 Shale owned repositories, no Shale mirrors, and 19 retained Forgejo bare repositories: 18 under `clo` and one under `nix`. Forgejo's retained LFS directory contains no files. Complete both imports before Shale's first production activation. |
| 6 | |
| 7 | The production data migration on 2026-10-04 completed before activating either production Shale or Postgres. The retained Shale copy passed checksum and SQLite checks, then all 19 Forgejo histories imported with verified object-file hashes and ref mappings. The result contains 21 repository records. All 11 private Forgejo sources retain private access; existing Shale identities, tokens, and sessions survive. The original session secret is preserved in the Nomad variable and its private recovery copy. Production import evidence is `/var/lib/studio/shale-forgejo-import-ldfl9zuy/repositories.json`; the earlier service dataset state is `globe/prod/shale@before-shale-import-1791160403-42496`. |
| 8 | |
| 9 | After copying the retained Shale state into `/srv/prod/shale`, [import-forgejo-to-shale.py](import-forgejo-to-shale.py) imports the additional histories while Shale stays stopped. Its metadata JSON comes only from the separately restored **personal** Forgejo PostgreSQL database: |
| 10 | |
| 11 | ```sql |
| 12 | SELECT coalesce(json_agg(x), '[]') FROM ( |
| 13 | SELECT r.id, r.owner_id, u.lower_name AS owner_name, |
| 14 | r.lower_name, r.name, r.is_private, r.default_branch, |
| 15 | r.is_empty, r.is_mirror |
| 16 | FROM repository r JOIN "user" u ON u.id = r.owner_id |
| 17 | ORDER BY u.lower_name, r.lower_name |
| 18 | ) x; |
| 19 | ``` |
| 20 | |
| 21 | ```sh |
| 22 | python3 tools/import-forgejo-to-shale.py \ |
| 23 | --target /srv/prod/shale \ |
| 24 | --metadata /run/personal-forgejo-repositories.json |
| 25 | ``` |
| 26 | |
| 27 | The importer preserves existing Shale identities and repository records. Clover's repositories retain their names; the other namespace becomes `nix/config`. It copies and hashes Git objects without copying Forgejo hooks or configuration. Missing branches, tags, pull refs, and notes retain their names. A divergent branch or tag is retained under `refs/heads/forgejo/<owner>/...` or `refs/tags/forgejo/<owner>/...`; other conflicting refs use `refs/forgejo/<owner>/...`. Existing Shale refs retain their values, including newer histories already migrated from old mirrors. Former Forgejo mirrors become owned histories, with their original mirror status recorded in the private evidence. That directory contains every original ref, its destination, all copied object hashes, and the previous SQLite database. |
| 28 | |
| 29 | New repository access follows verified Forgejo visibility, with pushes restricted to the owner and issue submission disabled. When private Forgejo history joins an existing repository, public or unlisted permissions are tightened to private **before objects are copied**; existing `off` permissions remain off. An import failure must leave Shale stopped. Restoring only the previous database can re-expose imported private objects through its old public permissions, so recovery must preserve the tightened access or restore the complete service dataset. |
| 30 | |
| 31 | Existing Shale account IDs and OIDC subjects must survive the cutover. A deliberate issuer move uses `service/shale/prepare.py` to rebind the provider before startup; duplicate bindings stop the migration. Shale caches repository metadata and access at startup, so finish SQLite changes before starting the application. An isolated pinned-image test proved direct registration of a new repository: private anonymous web access returned 404 and Git discovery returned 401; after public permissions and a restart, its page and Git advertisement returned 200 with the original branch object ID. The test used copied data, `--network none`, and no published ports. |
| 32 | |
| 33 | A full-data test imported all 19 retained Forgejo histories into a disposable copy of the existing Shale state, yielding 21 repository records. Its conservative fixture metadata marked every incoming repository private. Every copied object file and every source ref passed verification. The pinned application then denied anonymous access to a new private repository and a formerly public collision, while preserving an unaffected public repository. On that isolated copy, public test permissions proved HTTP pages and Git advertisements for `bgds`, `home-infra`, and `nix/config`; `home-infra` advertised the original Forgejo `main` and `vllm`. An actual smart HTTP fetch returned a 53-object pack with a verified pack checksum. Production visibility must come from the restored Forgejo metadata, not this test fixture. |
| 34 | |
| 35 | After migration and authenticated browser checks, push the new infra `main` to `https://shale.paperclover.net/home-infra.git` with a Shale personal access token. The existing `home-infra` record and UUID are retained; the imported Forgejo branches remain available alongside the tested final `main`. |
| 36 | |
| 37 | The October 5 issue migration uses `tools/import-forgejo-issues.py` and the verified personal PostgreSQL export at `/var/lib/studio/forgejo-issue-migration/source.json`. Its sibling `source-proof.json` records the source checksum. The export contains 452 issues, 1,429 comments and events, 60 labels, 24 attachments, and 312 history revisions. Pull request discussions become issues with their original branch and merge metadata. Historical authors retain their names through non-login identities; `paperclover` maps to the existing Clover account. Original creation, update, and comment timestamps are preserved. Every source row is retained in the private `studio_forgejo_records` ledger, including revision history that Shale cannot present as editable comments. |
| 38 | |
| 39 | Run this while the target job is stopped and its allocations have exited: |
| 40 | |
| 41 | ```sh |
| 42 | python3 tools/import-forgejo-issues.py \ |
| 43 | --target /srv/prod/shale \ |
| 44 | --source /var/lib/studio/forgejo-issue-migration/source.json |
| 45 | ``` |
| 46 | |
| 47 | For an isolated stage, add `--rehearsal` and use `/srv/staging/shale-preview-<id>`. The importer verifies the export checksum, checks that the target job is stopped before writing and immediately before commit, backs up SQLite into a private evidence directory, verifies native issue records remain unchanged, and supports an idempotent rerun of the same export. Take a full service ZFS snapshot before the production import as well. Do not replace production with the stage clone: import again into the stopped production dataset so newer native issues survive. |
| 48 | |
| 49 | Six occupied numbers are remapped: `chat` #1→#6 and #2→#7; `home-infra` #1→#38, #2→#39, and #3→#40; `react-mutation` #1→#19. All other original numbers remain. Explicit Forgejo issue links are rewritten and remapped bodies identify their original number. Bare `#references` and code text remain untouched. Attachments are copied with verified SHA-256 hashes and served through an authorization check against their associated Shale issue. Their local files are readable only by root and Caddy; private issue attachments remain private. |
| 50 | |
| 51 | The writable ZFS rehearsal `shale-preview-0242cee6` starts from all 104 existing native issues and imports to 556 total. It preserves the original Clover OIDC identity. September's `r1616-ga87d2f5.zig.0.16.0` avoids the `r1758` anonymous deleted-comment crash, but its Markdown scanner uses a shared capture buffer and crashes under concurrent fenced-code rendering. The documented `NPROC=1` worker setting avoids that race. With this setting, 904 authenticated/anonymous imported-issue requests passed with eight clients, all 24 attachment hashes and access checks passed, and browser closing of a disposable copy of `chat` #1 succeeded. Production `chat` #1 remains untouched. |
| 52 | |
| 53 | Production issue import completed on October 5 under release `dda941e618934ad9`, from committed main `51be72a9`. Recovery snapshot: `globe/prod/shale@before-forgejo-issues-20261005T080143Z-14c71a`. The private import report and SQLite backup live at `/var/lib/studio/forgejo-issue-migration/issue-import-c726euv2`. All 104 native issues survived alongside the 452 imported issues. After the route reload settled, 904 concurrent HTTPS issue reads passed (644 successful reads, 260 expected private-page denials), all 24 attachment hashes and access checks passed, SQLite integrity passed, and a forged-Origin request returned 403. Browser navigation confirmed historical timestamps, comments, labels, and status on a live imported issue. Native `chat` #1 remains Todo. The disposable Keycloak rehearsal was destroyed after verification; the production recovery snapshot and private evidence remain. |
| 54 | |
| 55 | This older build predates hidden form CSRF tokens. The router requires the exact HTTPS Origin for every Shale request using the `SessionID` cookie and a mutating method. The guard precedes all Shale handlers inside an explicit Caddy `route`; otherwise default directive ordering can bypass it. Missing, wrong, and suffix-forged origins returned 403 without a database change on the clone. The valid site origin allowed a status change, while Basic-auth Git requests still reached Shale. The MCP adapter permits tokenless issue forms only with the exact verified `r1616` structural footer and no CSRF-token input anywhere on the page. Newer or mixed-token markup remains strict. |
| 56 | |
| 57 | The October 5 `r1763-g9f5b1c7.zig.0.16.0` upgrade rehearsal passed anonymous reads of the formerly crashing issues, Unicode issue creation and comments, issue closing with a comment Delete form present, access denial, restart persistence, token revocation, and logout. The Rust adapter parsed its actual owner issue markup and accepted its status/comment CSRF fields. All migrated SQLite rows remained identical. Eight-worker concurrent Markdown rendering still crashed; keep `NPROC=1`, which passed 280 concurrent fenced-code requests. The injected issue-form script is no longer needed. Private evidence lives at `/var/lib/studio/shale-upgrade-0680d28c`; its disposable containers were removed after testing. |
| 58 | |
| 59 | Shale allocates an issue number from the issue with the highest SQLite row ID. The importer now inserts by destination number, including remapped collisions. The already-imported database needs `--repair-issue-order` once while its job is stopped: the guarded repair saves a SQLite backup, swaps six imported surrogate IDs, rewrites foreign keys and ledger references, and verifies every table's contents using issue UUIDs. Native IDs, public numbers, timestamps, comments, labels, attachments and history remain intact. It refuses duplicate numbers or changes to native issue IDs. On the repaired native-auth clone, the actual Rust Backend created `home-infra` #41, `react-mutation` #20 and `chat` #8, then commented on and closed the disposable `home-infra` issue. Fresh import, repeated import, repeated repair, and refusal checks passed. |
| 60 | |
| 61 | Production upgraded to `r1763` on October 5 in release `9ef0e1fd9c7a72bc`, from main `38ee0683`. The pre-cutover Shale recovery snapshot is `globe/prod/shale@native-auth-20261005T091312Z`. Fresh normal and guest sign-in callbacks returned 200; the existing Clover session and identity remained intact. All 452 imported issue routes passed concurrent anonymous checks (192 successful reads and 260 expected private-page denials), all 24 attachment hash/access checks passed, and all 556 issues remained unique with valid foreign keys. Native `chat` #1 remains Todo. The live proof is recorded in `/var/lib/studio/shale-upgrade-0680d28c/upgrade-validation.json`. |
| 62 | |
| 63 | The October 4 transport check inspected the then-pinned image in disposable containers without mounting real app data. Its embedded Git endpoint and account settings use HTTP and personal access tokens; no SSH listener, authorized-key interface, or forced-command handler was found. The [official installation](https://astheno.software/shale/installation/) and [configuration reference](https://astheno.software/shale/reference/environment/) also expose HTTP serving and OAuth login without SSH configuration. A `git` account must either use a Shale-aware SSH bridge or await native SSH support. Direct filesystem Git commands would bypass Shale's authorization. |
| 64 | |
| 65 | Zenith's Shale app directory contains a small SQLite database and 419 MB of owned repositories. `bash tools/import-shale.sh shale-preview-4eea0e3b` copied `data`, `repositories_owned`, and `repositories_mirrors` opaquely from the read-only `storage1/apps@hourly-2026-09-26_05-00` snapshot. It verified checksums and SQLite integrity, then restarted the preview. Both sides had 11 top-level owned repository directories; the preview had one healthy Nomad allocation and returned HTTPS 200. Repository contents were not inspected. |
| 66 | |
| 67 | For the production copy, stop Zenith's Shale container and the Snow Globe Shale job, set `STUDIO_DEPLOY_HOST` and `STUDIO_DEPLOY_PORT` for the new host, then run `bash tools/import-shale.sh shale`. The importer checks both jobs remain stopped, snapshots the destination dataset, verifies all three copied directories, and leaves Snow Globe stopped. Start the new job after the copy, check the SQLite state and a known login through the new Keycloak client, then switch the public route. The destination snapshot printed by the importer remains available for recovery. |
| 68 | |
| 69 | After a same-machine OS replacement, set `STUDIO_LEGACY_HANDOFF` to the [offline handoff](legacy-handoff.md) directory as well. The importer then reads the retained Shale directory from the new host's mounted old apps dataset, with no old Docker dependency. |
| 70 | |
| 71 | ## Imported repository HTTP pushes |
| 72 | |
| 73 | The Forgejo importer originally omitted `http.receivepack=true`, which Shale sets when it creates an owned repository. Shale checks its push ACL before invoking `git-http-backend`, but does not set `REMOTE_USER`; Git's default therefore rejected authorized pushes to imported repositories. The importer now writes the setting, and Shale's prepare step repairs missing settings without replacing an explicit disable. Production `config` remains owner-only for pushes. |
| 74 | |
| 75 | Shale also forwards CGI `Status` as an ordinary header on HTTP 200. The Git-specific Caddy handler translates Git's error statuses into HTTP statuses and removes the CGI header. A disposable production database/repository copy verified anonymous and invalid credentials return 401, authenticated non-owners return 403, owner discovery advertises `main=37665e69e1aa954f0dec06b1cafa1612f44dc907`, and an actual `jj git push --bookmark main` advances the clone to `b8e734ce6aabd94a7e1aacfc989467662ca27dbe`. Disabled receive-pack returns actual HTTP 403; invalid method/content type return 400/415 without a `Status` header. No production token was added. |