| 1 | # Shared project commands |
| 2 | |
| 3 | `tools/project-tools/` owns the VM's `jj`, `pnpm`, and `project` wrappers. |
| 4 | The config repo's `users/clover/agents/vm` entries link here; installed wrappers |
| 5 | use `~/.local/share/project-tools/current`. Keep native executables adjacent to |
| 6 | the public commands as `jj-real` and `pnpm-real`. |
| 7 | |
| 8 | ## Defaults |
| 9 | |
| 10 | - `project read rg ARGS...` searches on Linux beside ZFS and streams normal |
| 11 | ripgrep output and exit status. `project read rg --files` lists files. |
| 12 | `.gitignore` works in Jujutsu checkouts without a visible `.git` directory; |
| 13 | `--require-git` restores ripgrep's usual Git-only policy. Search preprocessors |
| 14 | are disabled. `project read sha256sum FILE...` hashes named shared files |
| 15 | without transferring their contents over NFS. Run these from the project |
| 16 | share; read requests have read-only mounts, no network, and no repository lock. |
| 17 | - Noninteractive `jj` operations in `/Volumes/Code/project` run on Snow Globe. |
| 18 | This includes workspace creation, snapshots, rebases, restores, and reads. |
| 19 | The guest's resolved configuration, identity, ignores, and paths travel with |
| 20 | the request. Both sides must run exactly the same Jujutsu build. |
| 21 | - Editors, external diff tools, signing, network Git commands, and local-disk |
| 22 | repositories retain native execution. Explicit file paths for a shared |
| 23 | repository cannot escape the project share, including through symlinks. |
| 24 | - `pnpm install`, `i`, `add`, `update`, and `remove` resolve, fetch, and import |
| 25 | packages on Linux, including changed manifests and lockfiles. The runtime must |
| 26 | match the project's exact pin: 10.32.1, 11.10.0, or 12.14.0. Directory selection, |
| 27 | workspace filters, frozen/offline installs, and lockfile-only updates retain |
| 28 | their normal meanings. `--ignore-scripts` skips the Mac hook phase. |
| 29 | - Activation links Vite+'s installed managed pnpm launchers to this same wrapper, |
| 30 | preserving each original as the adjacent `pnpm-real`. This covers the |
| 31 | project-local JavaScript `vp` launcher, which otherwise bypasses pnpm on PATH. |
| 32 | Vite+ keeps package-manager detection and argument handling; its notifier |
| 33 | setting does not prevent offloading. Explicit `VP_PACKAGE_MANAGER` overrides |
| 34 | retain native execution. Reactivate after Vite+ downloads another pnpm version. |
| 35 | - `pnpm run`, `exec`, and script shortcuts execute natively. Their implicit |
| 36 | dependency installs use Linux too; dependency verification opt-outs and custom |
| 37 | verification policies are preserved. Inherited npm client metadata and Node |
| 38 | heap limits retain offloading; `NODE_ENV=production` reaches the pinned pnpm |
| 39 | unchanged on both hosts. Unsupported options, runtime versions, other custom |
| 40 | install environment settings, and unreviewed pnpmfile hooks retain native |
| 41 | execution. Environment fallbacks name the variables without exposing values. |
| 42 | A failed remote operation is reported once. |
| 43 | - `project worktree PROJECT NAME` uses those same `jj` and `pnpm` commands. |
| 44 | `--no-install` skips dependencies. The checkout path is printed only after |
| 45 | installation and Mac hooks succeed, including with `--warm-from`. |
| 46 | New workspaces wait for NFS visibility. |
| 47 | |
| 48 | Linux installs with lifecycle scripts disabled, selects Darwin |
| 49 | ARM64 packages, validates the guest's Node version, and requires ZFS reflinks. |
| 50 | Knife's reviewed `.pnpmfile.cjs` runs in the Linux sandbox: its peer-range change |
| 51 | and Windows-only architecture selection behave identically on Linux and Darwin. |
| 52 | The backend checks its exact SHA-256 before offloading. Changed hooks, other |
| 53 | pnpmfiles, and config dependencies retain native installation. |
| 54 | Pending dependency and workspace hooks finish natively on the Mac. pnpm 10/11 |
| 55 | use `pnpm --recursive --include-workspace-root rebuild --pending`. pnpm 12 uses |
| 56 | a frozen install with the original filters and dependency groups, preserving |
| 57 | prepared patches. Its repeat-install shortcut and build-cache reads/writes are |
| 58 | disabled for this phase: restoring cached generated files over NFS can take |
| 59 | longer than running their hooks. Ordinary native installs retain their cache policy. |
| 60 | The wrapper announces the Mac handoff. |
| 61 | It also announces storage-host installation and package-store lock contention. |
| 62 | The two phases share policy and registry metadata in the project's `.pnpm-cache` |
| 63 | directory, outside its workspaces. This avoids repeating network verification on |
| 64 | the Mac. The handoff waits until NFS exposes the server's new modules manifest. |
| 65 | Repository configuration is not temporarily rewritten; manifests and |
| 66 | lockfiles change only as requested by the dependency command. |
| 67 | |
| 68 | The root-owned `profiles.json` binds manifests, lockfiles, workspace settings, |
| 69 | npmrc, pnpmfiles, patches, and local tarballs. Matching alphaXiv pnpm 10.32.1 and |
| 70 | Knife pnpm 11.10.0 profiles on Node 26 can additionally restore reviewed Darwin |
| 71 | side effects on Linux after SHA-512 verification using `project-pnpm/`'s patches. |
| 72 | A profile miss still installs on Linux; it leaves native builds pending for the |
| 73 | Mac instead of replaying an older build. Validation does not snapshot the working |
| 74 | copy. Ordinary Linux side-effect cache reuse is disabled. |
| 75 | |
| 76 | ## Locking and storage |
| 77 | |
| 78 | The Mac wrapper uses `flock`; Linux uses a POSIX record lock on the same |
| 79 | `.jj/repo/command.lock` inode. macOS NFS translates `flock` into that server lock. |
| 80 | The executing Jujutsu process inherits the descriptor. The forwarding guest |
| 81 | never holds a second lock. Ctrl-C cancels a waiter; never unlink the lock inode. |
| 82 | A command lock does not reserve revisions across a sequence of commands. |
| 83 | |
| 84 | The Code export is one ZFS dataset. `project worktree --warm-from DONOR` uses |
| 85 | per-file FICLONE, preserving independent inodes and source timestamps, excluding |
| 86 | root repository metadata and Mac sidecars. It restores the requested revision |
| 87 | and retains ignored build outputs. The guest holds the donor's Cargo locks; |
| 88 | editors and other writers must also be idle. No byte-copy fallback is allowed. |
| 89 | |
| 90 | The dedicated SSH key forces the unprivileged uid-3000 Bubblewrap launcher. |
| 91 | Jujutsu requests have no network. Dependency requests get network access with |
| 92 | DNS and the host CA bundle; lifecycle scripts remain disabled on Linux. Neither path |
| 93 | exposes host credentials, forwarding, or snapshot privileges. |
| 94 | The sandbox binds projects and package stores at their guest paths as well as |
| 95 | host paths. Identical store paths allow repeated installs without repaving |
| 96 | `node_modules`. Artifacts remain under Code. pnpm 11 uses a guest-local SQLite |
| 97 | index under `~/Library/Caches/project/PROJECT/v11`; pnpm 12 uses the shared |
| 98 | `/Volumes/Code/cache/pnpm12` file index. Both phases use workspace-local virtual |
| 99 | stores so platform-specific build outputs belong to that workspace. |
| 100 | |
| 101 | ## Updating the existing deployment |
| 102 | |
| 103 | Upload `project-store-install.py`, `project-store.py`, `project-store-test.py`, |
| 104 | `project-tools/`, and `project-pnpm/` together to a private staging directory on |
| 105 | Snow Globe. Run the installer there as root: |
| 106 | |
| 107 | ```sh |
| 108 | python3 project-store-install.py stage RELEASE |
| 109 | ``` |
| 110 | |
| 111 | It verifies pinned upstream downloads, applies the cache patches, preserves the |
| 112 | reviewed profiles, and builds a versioned runtime under |
| 113 | `/opt/studio-project-store/releases/`. Existing `project-store-*` Nix GC roots |
| 114 | pin Python, Node, Bubblewrap, and coreutils. `current` is unchanged by staging. |
| 115 | Test the staged `ssh-command` as `clo` with disposable projects and a temporary |
| 116 | restricted key. Do not run Jujutsu as root against shared repositories. |
| 117 | |
| 118 | Upload `project-tools/` to a guest staging directory and run its installer as |
| 119 | `admin`. After the staged host and guest tests pass: |
| 120 | |
| 121 | ```sh |
| 122 | # Snow Globe |
| 123 | python3 project-store-install.py activate RELEASE |
| 124 | # VM, using tools/project-tools/install.py |
| 125 | python3 install.py stage RELEASE |
| 126 | python3 install.py activate RELEASE |
| 127 | ``` |
| 128 | |
| 129 | Guest activation saves the prior tools under `project-tools/previous`, preserves |
| 130 | the native executables, and changes symlinks atomically. Subsequent updates switch |
| 131 | the complete wrapper directory through `current`. Keep the previous host release |
| 132 | and guest tools for rollback. Roll back both `current` links; do not remove |
| 133 | workspaces, locks, stores, or caches. |
| 134 | |
| 135 | ## Verification |
| 136 | |
| 137 | ```sh |
| 138 | python3 tools/project-store-test.py |
| 139 | python3 tools/project-tools/test.py |
| 140 | python3 tools/project-tools/jj-test.py |
| 141 | ``` |
| 142 | |
| 143 | Run `project-tools/platform-test.py` inside the development VM against the |
| 144 | normal endpoint. Its disposable project checks Darwin-only optional dependencies |
| 145 | immediately after the Linux import, before Mac build scripts can repair them. |
| 146 | `project-tools/finish-test.py` checks native hooks, denied scripts, patch inode |
| 147 | preservation, repeated installs, filtered dependency groups, and nested mutations. |
| 148 | |
| 149 | The backend tests create disposable repositories. `PROJECT_STORE_REAL_REFLINK=1` |
| 150 | with `TMPDIR` on disposable Linux/ZFS storage exercises real clone independence, |
| 151 | timestamps, donor edits, path rejection, and failure cleanup. The config repo's |
| 152 | `vm/test.py` covers overlay preparation, worktrees, and shared memory links. |
| 153 | `project-pnpm/restore-test.cjs` checks cache integrity and path guards on Linux/ZFS. |
| 154 | |
| 155 | On 2026-10-22, the endpoint created an 801-file workspace through the VM in 2.9 |
| 156 | seconds and rebased a two-commit fixture in 1.6 seconds. Cross-client lock |
| 157 | contention and cancellation passed. Disposable pnpm 10, 11, and 12 projects |
| 158 | installed changed dependencies on Linux and ran Darwin dependency and workspace |
| 159 | hooks on the Mac. Frozen-lockfile failures, lockfile-only updates, filtered |
| 160 | add/update/remove commands, ignored scripts, and native esbuild execution passed. |
| 161 | A pnpm 12 filtered update took 12.4 seconds, including 1.4 seconds on Linux; |
| 162 | a subsequent script ran in 1.2 seconds without reinstalling. Nested pnpm scripts |
| 163 | and dependency-verification opt-outs passed. An existing global virtual store |
| 164 | migrated successfully. After activation, ordinary commands against the real |
| 165 | shared store passed: a fresh install took 70.5 seconds, a filtered add took 32.9 |
| 166 | seconds, and a script took 2.5 seconds. Native rebuilding left all 1,048 sampled |
| 167 | lodash files' inodes and mtimes unchanged. These are fixture measurements with other agents |
| 168 | active, not latency bounds or full-project build measurements. |
| 169 | |
| 170 | Vite+ 1.3.0 was also verified through the normal endpoint with pnpm 12.14.0 |
| 171 | and Knife's reviewed `.pnpmfile.cjs`: package imports ran on Linux/ZFS, while |
| 172 | the workspace postinstall and native esbuild ran on Darwin ARM64. Frozen/offline |
| 173 | installs, add/remove, lockfile-only, ignored scripts, script execution, and frozen |
| 174 | lockfile failure propagation passed. Editing the hook selected Mac installation. |
| 175 | The host and VM tools were activated as `20261022-vp-preview-2`. |
| 176 | |
| 177 | pnpm 12 needs `supportedArchitectures: [darwin-arm64]` for host-side package |
| 178 | selection; the older `config.os`/`config.cpu` overrides do not select its optional |
| 179 | dependencies. This is passed as a command setting without editing the workspace. |
| 180 | The platform regression fails against the old endpoint and passes with the fix. |
| 181 | The full Knife workspace imported 75 missing Darwin packages, and its Vite+, |
| 182 | Vite, Vitest, Rolldown, and linter bindings loaded successfully before rebuilding. |
| 183 | With Mac build-cache writes disabled, a normal frozen install in `knife-merge` |
| 184 | completed successfully in 278 seconds, including the mobile dependency's native |
| 185 | asset downloads and the root prepare hook. The host and guest were activated as |
| 186 | `20261023-vp-platform-3`. Linux side-effect cache reads and writes are explicitly |
| 187 | disabled with pnpm 12's supported boolean settings. |
| 188 | After installation, `pnpm exec vp --version`, the TypeScript compiler, and the |
| 189 | Vitest CLI started successfully. The node-pty addon loaded, and sharp encoded a |
| 190 | small PNG using its Darwin native module. |
| 191 | |
| 192 | A follow-up trace in `knife-disk` found the Mac phase copying patched packages |
| 193 | and restoring cached mobile assets over NFS. The frozen-install completion |
| 194 | described above disables those cache transfers. Five disposable VM cases cover |
| 195 | native and workspace hooks, denied scripts, patch preservation, filtered |
| 196 | production/development installs, nested adds, repeated installs, and inherited |
| 197 | agent environments. A deployed |
| 198 | frozen install in `knife-disk` completed in 43.5 seconds with its native assets |
| 199 | already present. Vite+, TypeScript, and the node-pty native binding loaded. |
| 200 | The real project-local `vp i --frozen-lockfile` then completed in 28.8 seconds |
| 201 | through Vite+'s managed pnpm launcher and the shared wrapper. Exporting |
| 202 | `VP_PNPM_MANAGER=no` alone did not route that JavaScript launcher through PATH; |
| 203 | the obsolete VM environment setting was removed. |
| 204 | |
| 205 | The host and guest were activated as `20261023-vp-env-4`. Staged pnpm 10, 11, |
| 206 | and 12 installs with `NODE_ENV=production` preserved their native dependency |
| 207 | selection; production mode is forwarded as environment, without inventing a |
| 208 | `--prod` flag. The VM fixture also checked that Mac hooks receive that value |
| 209 | while npm client metadata and Node memory limits remain compatible with offloading. |
| 210 | A real `vp i --frozen-lockfile` with those inherited metadata and memory settings |
| 211 | completed in 23.2 seconds on `knife-disk`, including Mac hooks with native assets |
| 212 | already present. Vite+'s tools, TypeScript, and the node-pty Darwin binding loaded. |
| 213 | |
| 214 | Release `20261023-tools-6` adds the read-only storage tools and waits for NFS |
| 215 | visibility before opening warm workspaces. A VM request listed 2,001 fixture |
| 216 | files in 1.93 seconds and hashed a 330 MiB fixture in 0.95 seconds. Staged |
| 217 | checks verified read-only mounts, network isolation, ignore rules, normal |
| 218 | search exit statuses, hash equality, and preprocessor rejection. A real warm |
| 219 | checkout installed a changed dependency and its Darwin hook in 7.81 seconds; |
| 220 | the donor and an explicit `--no-install` copy retained the original dependency. |