| 1 | # cross-version-test |
| 2 | |
| 3 | Deploys a tiny app — a Cloudflare **state store** + one **Worker** — across a |
| 4 | range of alchemy versions, **upgrading the same app in place**, to catch |
| 5 | regressions in the version-to-version upgrade path. |
| 6 | |
| 7 | `run.ts` runs two tests **back to back** (sequentially — they share one |
| 8 | account-wide state store and one fixed worker name, so they can never overlap): |
| 9 | |
| 10 | 1. **`upgrade` — sequential 1-by-1 upgrade.** Deploy the oldest version, then |
| 11 | upgrade the same app through every version in order |
| 12 | (v4 → v5 → v6 → v7 → current), walking the state store up one version at a |
| 13 | time. |
| 14 | 2. **`jump` — direct-to-latest.** For each older example, deploy it and then |
| 15 | upgrade the same app **directly to latest** (the current branch), skipping |
| 16 | the intermediate versions — a big-jump upgrade (e.g. state store v4 → v7 in |
| 17 | one step). |
| 18 | |
| 19 | Both assert, after every deploy, that the live worker serves the `marker` baked |
| 20 | into the version just deployed — proving the running code was actually replaced |
| 21 | in place. Each also exercises the **state-store upgrade** path: alchemy's |
| 22 | Cloudflare state store is an account-wide, version-stamped singleton that |
| 23 | `alchemy provider cloudflare bootstrap` migrates between versions. |
| 24 | |
| 25 | **Fresh state store per scenario.** Each scenario ("unit" — the whole `upgrade` |
| 26 | chain is one unit; each `jump` is its own unit) owns a brand-new state store: |
| 27 | the runner tears the store (worker + secrets) down before and after every unit. |
| 28 | So the `upgrade` chain runs on its own store, that store is destroyed, then each |
| 29 | `jump` deploys a fresh store. Pass `--reuse-store` to skip the teardowns and |
| 30 | share one store across units (faster, less isolated). |
| 31 | |
| 32 | ## Layout |
| 33 | |
| 34 | ``` |
| 35 | cross-version-test/ |
| 36 | run.ts # orchestrator — runs the stages in order |
| 37 | test/ |
| 38 | 01-beta.39/ # alchemy@2.0.0-beta.39 (last npm release with state store v4) |
| 39 | 02-beta.44/ # alchemy@2.0.0-beta.44 (last npm release with state store v5) |
| 40 | 03-beta.45/ # alchemy@2.0.0-beta.45 (last npm release with state store v6) |
| 41 | 04-beta.59/ # alchemy@2.0.0-beta.59 (latest v2 on npm, state store v7) |
| 42 | 05-current/ # current branch (workspace source, state store v7) |
| 43 | ``` |
| 44 | |
| 45 | Each stage folder is a standalone alchemy app: `alchemy.run.ts` (identical |
| 46 | across stages) + `src/worker.ts` (differs only by a `marker` and, for older |
| 47 | versions, `main: import.meta.filename` vs HEAD's `import.meta.url`) + |
| 48 | `test/integ.test.ts` (an example-style integration test). |
| 49 | |
| 50 | - **npm stages** (`01`, `02`) pin an exact `alchemy` version **and a matching |
| 51 | `effect` version** in `package.json`, and get their own isolated |
| 52 | `node_modules` via `bun install`. The `effect` pin matters: alchemy floats on |
| 53 | the `effect` 4.0 beta line, so an old alchemy paired with a too-new `effect` |
| 54 | crashes (e.g. beta.45 breaks on `effect` ≥ beta.84 via a `SchemaAST` change). |
| 55 | Each stage therefore pins the `effect` version contemporaneous with its |
| 56 | alchemy — its peer-dependency floor (beta.39 & beta.44 → `effect@4.0.0-beta.66`; |
| 57 | beta.45 → `beta.74`; beta.59 → `beta.84`). |
| 58 | - **workspace stage** (`05-current`) has no dependencies — it resolves |
| 59 | `alchemy`/`effect` from the monorepo (repo-root `node_modules`) so it runs the |
| 60 | **current branch source**. |
| 61 | |
| 62 | All stages share one identity — `Stack("CrossVersionApp")`, a fixed worker |
| 63 | `name`, the same account — which is what makes each deploy an in-place upgrade |
| 64 | rather than a new app. (TEST 2 uses the stage `<stage>-jump` to keep its state |
| 65 | separate from TEST 1's.) |
| 66 | |
| 67 | ## Running |
| 68 | |
| 69 | ```sh |
| 70 | # from the repo root — runs BOTH tests back to back |
| 71 | bun run test/cross-version-test/run.ts --profile <cloudflare-profile> |
| 72 | # or |
| 73 | ALCHEMY_PROFILE=<cloudflare-profile> bun run test/cross-version-test/run.ts |
| 74 | # run just one test |
| 75 | bun run test/cross-version-test/run.ts --profile <p> --test upgrade |
| 76 | bun run test/cross-version-test/run.ts --profile <p> --test jump |
| 77 | ``` |
| 78 | |
| 79 | ### Flags |
| 80 | |
| 81 | | Flag | Default | Meaning | |
| 82 | | ---------------- | ------------------ | ------------------------------------------------------------- | |
| 83 | | `--profile <p>` | `$ALCHEMY_PROFILE` | Cloudflare auth profile (`~/.alchemy/profiles.json`). Required. | |
| 84 | | `--test <names>` | `upgrade,jump` | Comma-separated tests to run: `upgrade`, `jump`. | |
| 85 | | `--stage <s>` | `xver` | Base alchemy stage name (TEST 2 uses `<stage>-jump`). | |
| 86 | | `--only <dirs>` | all | Comma-separated stage dirs (e.g. `01-beta.39,02-beta.44`). | |
| 87 | | `--no-install` | off | Skip `bun install` in npm stages (reuse existing installs). | |
| 88 | | `--keep` | off | Leave the final app + state store deployed (skip the last unit's teardown). | |
| 89 | | `--reuse-store` | off | Reuse one state store across units (faster, less isolated) instead of a fresh one per unit. | |
| 90 | |
| 91 | ## Per-version integration tests |
| 92 | |
| 93 | Each stage folder also has a standalone `test/integ.test.ts` (mirrors |
| 94 | `examples/*/test/integ.test.ts`): it deploys the stack in `beforeAll`, makes an |
| 95 | HTTP request to the worker and asserts the live `marker`, and destroys in |
| 96 | `afterAll`. Run one from its folder: |
| 97 | |
| 98 | ```sh |
| 99 | cd test/cross-version-test/test/05-current |
| 100 | ALCHEMY_PROFILE=<profile> bun test # bun test buffers — no live output until it finishes |
| 101 | ``` |
| 102 | |
| 103 | These run on a dedicated stage (`integ`) so they don't touch the orchestrator's |
| 104 | `xver` state. They deploy through the **account-wide state store**, so it must |
| 105 | already be at the version matching that folder's alchemy (**v4** beta.39, **v5** |
| 106 | beta.44, **v6** beta.45, **v7** beta.59/current). Either run `run.ts` (which |
| 107 | bootstraps per stage), or bootstrap first: |
| 108 | |
| 109 | ```sh |
| 110 | cd test/cross-version-test/test/01-beta.39 |
| 111 | bun run alc -- cloudflare bootstrap --profile <profile> # brings the store to v4 |
| 112 | bun test |
| 113 | ``` |
| 114 | |
| 115 | ## Tearing down the state store |
| 116 | |
| 117 | `run.ts` tears the state store down automatically between units (unless |
| 118 | `--reuse-store`), and the last unit's store is removed at the end too (unless |
| 119 | `--keep`). So a normal run leaves the account clean. |
| 120 | |
| 121 | To remove the state store manually — the `alchemy-state-store` worker, the |
| 122 | bearer-token + encryption-key secrets, and the now-empty Secrets Store (e.g. |
| 123 | after a `--keep` run or an aborted run): |
| 124 | |
| 125 | ```sh |
| 126 | cd test/cross-version-test/test/05-current # uses the current-branch CLI |
| 127 | bun run alc -- provider cloudflare teardown --profile <profile> |
| 128 | ``` |
| 129 | |
| 130 | `provider cloudflare teardown` is the inverse of |
| 131 | `provider cloudflare bootstrap`. It's idempotent and only removes resources |
| 132 | Alchemy created. It leaves a Secrets Store in place if it still contains |
| 133 | foreign secrets. |
| 134 | |
| 135 | ## ⚠️ Use a dedicated Cloudflare account |
| 136 | |
| 137 | The Cloudflare state store is **account-wide** and shared by every stack on the |
| 138 | account/profile. Stage `01` bootstraps state store **v4**, which **downgrades** |
| 139 | the store if the account already had a higher version — that can break other stacks using |
| 140 | the same account. **Point this at a dedicated/throwaway Cloudflare account.** |
| 141 | |
| 142 | The runner tears the state store down between units and at the end, so a normal |
| 143 | run leaves the account clean. |
| 144 | |
| 145 | ## Known-bad upgrade paths |
| 146 | |
| 147 | Two upgrade paths are **known broken** and are therefore **not tested** |
| 148 | (commented out of `EDGES` in `run.ts`). Both are deterministic and reproduced |
| 149 | across runs: |
| 150 | |
| 151 | | Path | Status | Why | |
| 152 | | --- | --- | --- | |
| 153 | | v4 → v5 | ✅ works | | |
| 154 | | **v5 → v6** | ❌ **known bad** | **v6 (beta.45) can't read pre-v6 state** — `500 GET DecodeError` | |
| 155 | | v6 → v7 | ✅ works | | |
| 156 | | v5/v6/v7 → worktree | ✅ works | | |
| 157 | | **v4 → worktree** | ❌ **known bad** | **a v4 store can't be upgraded to current** — `415` on write | |
| 158 | |
| 159 | Details: |
| 160 | |
| 161 | - **`v5 → v6` (beta.44 → beta.45):** after the store is upgraded to v6, reading |
| 162 | a record written under ≤v5 fails with `500 GET … DecodeError`. beta.45's |
| 163 | legacy-record read (createdAt/updatedAt reshape, PR #427) is broken; **fixed |
| 164 | in v7** — `v5 → worktree` (skipping beta.45) works. Don't step pre-v6 state |
| 165 | through beta.45. |
| 166 | - **`v4 → worktree` (beta.39 → current):** the current state-store client can't |
| 167 | write to a v4-format store — the upgrade fails with `415 Unsupported Media |
| 168 | Type` on `PUT …/StateStoreEncryptionKey`. v4 (beta.37–39) predates the RPC |
| 169 | state-store rewrite at v5, so its store HTTP API is wire-incompatible with |
| 170 | current. Step a v4 store up to ≥v5 first (`v5/v6/v7 → worktree` all work). |
| 171 | |
| 172 | Everything else passes. Transient `404/500` while bootstrapping a freshly |
| 173 | (re)deployed state-store worker is expected (the worker isn't serving yet when |
| 174 | bootstrap hoists into it) and is absorbed by the per-step retry. |
| 175 | |
| 176 | ## Adding versions / scenarios |
| 177 | |
| 178 | Add a new `test/NN-<label>/` folder (copy an existing one), pin its `alchemy` |
| 179 | version in `package.json`, set the `marker` + `main` form appropriately in |
| 180 | `src/worker.ts`, then add an entry to the `STAGES` array in `run.ts`. Keep |
| 181 | `Stack("CrossVersionApp")` and the worker `name` identical so it upgrades the |
| 182 | same app. |
| 183 | |
| 184 | ### Version → state-store-version reference |
| 185 | |
| 186 | | alchemy | state store version | pinned `effect` | notes | |
| 187 | | ---------------- | ------------------- | --------------- | ---------------------------------- | |
| 188 | | beta.29 … 30 | 1 | — | version-gating introduced (#156) | |
| 189 | | beta.31 … 32 | 2 | — | | |
| 190 | | beta.33 … 36 | 3 | — | | |
| 191 | | **beta.37 … 39** | **4** | beta.66 | last release with v4 = beta.39 | |
| 192 | | **beta.40 … 44** | **5** | beta.66 | last release with v5 = beta.44 | |
| 193 | | **beta.45** | **6** | beta.74 | only release with v6 | |
| 194 | | **beta.46 … 59** | **7** | beta.84 | bumped to v7 in beta.46 (PR #477); last v7 on npm = beta.59 (= latest) | |
| 195 | | current branch | 7 | workspace | | |