authorgravatar for julius0216@outlook.comJulius Marminge <julius0216@outlook.com> 2026-09-24 20:38:31-07:00
committergravatar for noreply@github.comGitHub <noreply@github.com> 2026-09-24 20:38:31-07:00
logfc46b8c3dd691ed2e39da28a2407e1b1597f2832
tree7fcda780ca701a5da13c8cd1d360d640d65437c8
parent568c9bc4d0ef0dddf74d813619c9877bd35624d7
signature Signed by PGP key B5690EEEBB952194

ci(relay): add a forced manual relay deploy (#13550)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

2 files changed, 26 insertions(+), 8 deletions(-)

.github/workflows/deploy-relay.yml+12-1
......@@ -4,6 +4,15 @@ on:
44 push:
55 branches:
66 - main
7 # Alchemy does not redeploy the Worker when only a Config value read in its
8 # Init changes (alchemy-run/alchemy#1831), so a changed repository variable
9 # needs a forced deploy to reach production.
10 workflow_dispatch:
11 inputs:
12 force:
13 description: Redeploy every resource, including ones with no detected changes
14 type: boolean
15 default: true
716
817permissions:
918 contents: read
......@@ -17,6 +26,8 @@ concurrency:
1726jobs:
1827 deploy_relay:
1928 name: Deploy production relay
29 # A manual run from another branch would deploy that branch to production.
30 if: github.ref == 'refs/heads/main'
2031 runs-on: blacksmith-8vcpu-ubuntu-2404
2132 timeout-minutes: 15
2233 environment:
......@@ -55,7 +66,7 @@ jobs:
5566 - --filter=t3code-relay...
5667
5768 - name: Deploy production relay stage
58 run: vp run --filter t3code-relay deploy --stage prod --yes --no-input
69 run: vp run --filter t3code-relay deploy --stage prod --yes --no-input ${{ inputs.force && '--force' || '' }}
5970 env:
6071 # The PublishClientConfig action writes the client env here instead
6172 # of the repo-root .env; nothing on the runner reads it.
docs/operations/release.md+14-7
......@@ -140,6 +140,11 @@ Required `production` environment secrets:
140140- `CLERK_SECRET_KEY`
141141- `APNS_PRIVATE_KEY`
142142
143The relay Worker reads these variables and secrets when it is deployed. Alchemy does not redeploy the
144Worker when only one of these values changes ([alchemy-run/alchemy#1831](https://github.com/alchemy-run/alchemy/issues/1831)),
145so a push to `main` without relay code changes leaves the old value in place. After changing one, run
146the **Deploy T3 Connect relay** workflow manually from `main` with **force** checked.
147
143148The account-scoped repository credentials are consumed by Alchemy while provisioning relay stages; they
144149are not bound into the relay Worker. The production deployment uses an Axiom personal access token,
145150so `AXIOM_ORG_ID` must accompany `AXIOM_TOKEN`. The `prod` stage owns the retained PlanetScale
......@@ -163,18 +168,20 @@ because those builds register recovery and replace a deleted tunnel after wake.
1631681. Deploy the relay and migration with cleanup `off`.
1641692. Release the server build and confirm current hosts register recovery. Older hosts stay marked
165170 legacy and are never candidates.
1663. Set `dry-run`, deploy, and read the sweep counters (`scanned`, `wouldDelete`, `skippedLegacy`,
167 `skippedOrphan`, `failed`, `truncated`) across several sweeps.
1713. Set `dry-run`, run a forced relay deploy, and read the sweep counters (`scanned`, `wouldDelete`,
172 `skippedLegacy`, `skippedOrphan`, `failed`, `truncated`) across several sweeps. Each sweep records
173 them, and the active `mode`, as `relay.managed_endpoint_reaper.*` attributes on its
174 `relay.managed_endpoint_reaper.sweep` span in Axiom.
1681754. Run the disposable-host canary below.
1691765. Set `enabled` only after the canary recovers without a server restart.
170177
171178The job runs every five minutes with a five-minute grace period for tunnels that lost their
172179connector, so a candidate is usually removed five to ten minutes after it goes down. Tunnels that
173180never connected wait an hour. One sweep attempts at most 100 deletions, so a backlog takes longer.
174`RELAY_TUNNEL_CLEANUP_MODE` is read at deploy time. Changing it, including turning cleanup off during
175an incident, needs a relay deploy.
181Changing `RELAY_TUNNEL_CLEANUP_MODE`, including turning cleanup off during an incident, needs a forced
182relay deploy. Confirm the new `mode` on the next sweep span.
176183
177To roll back, set cleanup to `off` and deploy the relay before downgrading any host. Keep the
184To roll back, set cleanup to `off` and run a forced relay deploy before downgrading any host. Keep the
178185recovery endpoints deployed while current server builds are in use. The nullable columns can stay.
179186
180187### Disposable-host canary
......@@ -194,8 +201,8 @@ stage, test Cloudflare account, disposable host, and disposable T3 home. Keep pr
194201 and pause it with `kill -STOP <first-pid>`. Wait until Cloudflare reports it down for over five
195202 minutes.
1962035. Confirm dry-run counts the first tunnel in `wouldDelete` and the second in `skippedLegacy`.
1976. Set cleanup `enabled` on the disposable stage and deploy. Confirm in the test Cloudflare account
198 that the first tunnel is deleted and the legacy tunnel still exists.
2046. Set cleanup `enabled` on the disposable stage and deploy it with `--force`. Confirm in the test
205 Cloudflare account that the first tunnel is deleted and the legacy tunnel still exists.
1992067. Resume the first child with `kill -CONT <first-pid>`. Confirm the running server detects the
200207 repeated rejection, requests recovery, and becomes reachable at the same hostname without a
201208 restart.