clover's git

the-knife

1	# dev-stress
2
3	A cross-cloud `alchemy dev` stack that exists to be **rewritten while it is
4	running**.
5
6	Every other dev example asserts that a stack converges.
7	[`test/dev-stress.test.ts`](./test/dev-stress.test.ts) asserts that the dev
8	server *survives*: it launches the real `alchemy dev` CLI as a child process
9	against a throwaway copy of this project, then edits, breaks, moves and
10	churns the project's files and checks — over HTTP, against pinned ports —
11	that the server never dies and never serves stale code.
12
13	## Topology
14
15	```
16	AWS (floci emulator) Cloudflare (workerd + docker)
17	──────────────────── ─────────────────────────────
18	Lambda ApiFunction ◀────────── Worker ApiWorker (path `main`)
19	├ S3 StressBucket cross- Worker EchoWorker (Effect-native `main`)
20	├ DynamoDB Table cloud ├ KV + R2 + Durable Object
21	└ SQS + consumer hop └ Container SandboxContainer
22	Lambda MicroVM ◀────────── Worker MicrovmWorker
23	StressMicrovm mounts (IAM User + AccessKey + assume-role,
24	(image + instances) minted by Alchemy for the Worker)
25	ECS Cluster + 2 Services
26	EcsService (context build)
27	EcsInlineService (inline Dockerfile)
28	EC2 Instance (hosted program)
29	StressEc2Box + VPC network
30	(i-….localhost.floci.io:8798)
31	Website.StaticSite Website.StaticSite (build mode)
32	(dev-command child)
33	```
34
35	`MicrovmWorker` is the headline: a Cloudflare Worker that **mounts an AWS
36	Lambda MicroVM**. A Worker has no AWS execution role, so binding the MicroVM
37	instance operations makes Alchemy mint an IAM User + AccessKey + assume-role
38	Role for it and assume that role at runtime. `GET /roundtrip` boots a VM,
39	waits for it to run, mints a data-plane auth token, drives both protocols the
40	VM speaks (typed RPC and a raw HTTPS route), and terminates it again.
41
42	The cross-cloud hop is real: local `workerd` fetches the Lambda's function
43	URL, which under `alchemy dev` is served by the floci emulator behind a
44	self-signed certificate. It only resolves because `alchemy dev` puts the
45	emulator's CA on `NODE_EXTRA_CA_CERTS` for every process it spawns.
46
47	## The two reload paths
48
49	The suite tells them apart by counting `Plan:` renders in the CLI's output
50	(`DevServer.planCount`), which is exactly the number of exec-child restarts:
51
52	| Edit | Path | `planCount` |
53	| ------------------------------------------------- | ----------------------------------------------- | ----------- |
54	| a file the **stack** imports (`src/echo/marker.ts`) | `bun --watch` re-runs the stack → replan → apply | goes up |
55	| a file only the **bundler** sees (`src/api/marker.ts`) | the Worker provider's watch loop hot-swaps the script | stays flat |
56
57	`src/api-worker.ts` is referenced by path and never imported by
58	`alchemy.run.ts`, which is what puts it on the bundler-only path.
59
60	## What the suite covers
61
62	**Convergence and reload**
63
64	1. **Boot** — every local resource serves; the cross-cloud hop works; AWS
65	identities are the emulator's, not real AWS.
66	2. **The MicroVM** — a Worker boots a VM, drives typed RPC *and* a raw HTTPS
67	route against it, and terminates it.
68	3. **Hot reload, both paths** — plus proof that a user-code restart does
69	*not* bounce sidecar-hosted children (`Command.Dev`'s pid is unchanged).
70	4. **Moved modules** — the importer points at a path that does not exist
71	yet; the file arrives; the new path is watched too.
72
73	**Broken states — the dev server logs and waits, never exits**
74
75	5. A syntax error in a stack-imported module, a module-scope `throw`, and a
76	resource that cannot reconcile (a `strictPort` collision). In all three
77	the CLI survives, unrelated resources keep serving, and the next save
78	recovers.
79
80	**Resource-graph churn — the substance of the suite**
81
82	6. **A Cloudflare Worker** added, renamed onto a new port (a create + a
83	delete), and removed.
84	7. **A whole AWS Lambda** — new module, new import, new `Function`, new S3
85	`Bucket`, two new bindings — added, hot-swapped, and removed.
86	8. **A Queue + its consumer event source + a new Durable Object class**
87	grafted onto a Worker that is already serving (a class migration on a
88	live script), driven produce→consume, then removed again.
89	9. **A second S3 bucket and its bindings** added to the running Lambda and
90	removed, observed through the cross-cloud hop.
91	10. **ECS hot reload, four ways a docker build changes** — a context file,
92	the Dockerfile itself, an env *prop* (no file event — the engine path),
93	and an inline-Dockerfile *prop*. All four must roll the running
94	containers.
95	11. **EC2 hot reload** — the hosted instance runs as a real container in
96	the emulator, serves its bundled program through floci's host-routing
97	mux, and a content edit updates it IN PLACE through the engine
98	(re-plan → bundle re-upload → reboot): same instance id, same address,
99	new code.
100	12. **Cloudflare Container hot reload** — editing the container's program
101	rebuilds the image and restarts the running container; and the
102	container reaches a service on the HOST through an env var written as
103	`http://localhost:…` (the dev runtime rewrites loopback hosts to
104	`host.docker.localhost` — the #1334 database shape).
105	13. **A replacement** — DynamoDB's partition key is immutable, so changing
106	it swaps the table under the running Lambda; then swaps it back.
107	14. **A second MicroVM image** built, bound to the running Worker, booted,
108	and removed — while the first image keeps working.
109	15. **The entire AWS Lambda subsystem** deleted in one edit (Function,
110	bucket, table, queue, event source, and the cross-cloud binding that
111	named it) while Cloudflare keeps serving — then restored.
112	16. **Binding churn** — a changed binding value, a binding that appears and
113	disappears, and a `Command.Dev` config change that must restart the
114	child (new pid).
115
116	**Load and final state**
117
118	17. **Rapid fire** — 25 bundler-path saves and 15 stack-path saves in
119	bursts, asserting convergence on the last write and that the watchers
120	coalesce (measured: 15 saves in ~1s produce a single re-apply).
121	18. **Simultaneous cross-cloud edits**, then a full health re-probe and a
122	clean Ctrl-C shutdown.
123
124	Nothing is skipped: Docker is a hard requirement, and a machine without it
125	fails the suite rather than quietly passing it.
126
127	## What it has found
128
129	Bugs the suite surfaced on its first runs, each now fixed and pinned by a
130	focused regression test closer to the code:
131
132	- **Lambda hot swap died after any engine update.** The dev watch loop
133	enrolled the function at a stable S3 key once, then only `PutObject`ed
134	there. Every engine-driven update (a binding added, an env var changed)
135	re-pointed the function at a content-addressed key, and the watcher's
136	one-time-enrollment memo never re-pointed it back — so the next source
137	edit was silently ignored until the next engine update happened to
138	re-bundle. `FlociFunctionProvider` now re-enrolls whenever the engine has
139	reconciled since the last swap
140	(`test/AWS/Lambda/Function.local.test.ts`).
141	- **A request in flight across a Worker hot swap hung forever.** The dev
142	proxy parked the retryable request in its retry queue and only drained
143	that queue on the *next* PUT or the *next* incoming request. A client
144	waiting on that response never sends another, workerd's hang detector
145	eventually cancelled the stalled proxy call, and the client got nothing.
146	The proxy now re-drives its queue immediately after parking
147	(`packages/cloudflare-runtime/src/core/test/proxy/WorkerProxy.test.ts`).
148	- **Cloudflare Containers never hot reloaded, at all.** Three stacked
149	causes: the local provider's image hash was memoized in the RPC sidecar,
150	whose artifact store outlives every run — so the diff compared the FIRST
151	run's hash forever and reported noop; the worker's restart config only
152	carried the image's stable paths, so even a detected change never
153	restarted the instance; and user Dockerfile/context files are not
154	imported by the stack, so `bun --watch` never replanned for them. Fixed
155	by per-plan cache eviction, the content hash riding the container
156	binding into the worker's hashed config, and a context watcher in the
157	local worker runner (`test/Cloudflare/Container/LocalContainerReload.test.ts`).
158	- **Dev containers couldn't reach services on the host**
159	([#1334](https://github.com/alchemy-run/alchemy/issues/1334)): env
160	values injected into a dev container carry loopback URLs (locally
161	emulated databases, dev servers) that dangle inside the container. The
162	workerd docker proxy now rewrites loopback hosts in URL-shaped env
163	values to `host.docker.localhost`. Docker Desktop maps that through
164	`host-gateway`; native Linux unix-socket-tunnels the port into the
165	sidecar netns so UFW never sees a SYN to the bridge IP.
166	- **ECS tasks never rolled on prop-driven updates.** Restart/roll logic
167	lived only in the file-watch trigger, so a prop change — an inline
168	`dockerfile` edit, a new env var — registered a new task-definition
169	revision (and `updateService`d onto it) while the running containers kept
170	serving the old one until the next source edit. The dev-watch skeleton
171	now has an `onReconciled` hook that fires on BOTH engine-driven and
172	watcher-driven reconciles, and both ECS dev providers roll their tasks
173	from it (`test/AWS/Local/EcsDev.local.test.ts`).
174	- **A MicroVM image never rebuilt on a content-only edit.** The image's
175	diff compared props only; the in-VM program is not a prop, so
176	`alchemy dev` *and* `alchemy deploy` called the edit a noop. The diff now
177	hashes the bundled content (memoized for the run) before the props
178	resolve, the way the Lambda Function diff does
179	(`test/AWS/Lambda/MicrovmImage.local.test.ts`).
180
181	Two dev-server behaviours the suite tolerates rather than asserts against,
182	because they self-heal: the bundler hot-swaps a Worker's new script a beat
183	before the stack re-apply registers the resources it references (a new
184	Durable Object class, a new image ARN), so that window serves
185	`DurableObject 'X' not found` / an undefined binding; and a slow delete
186	(a MicroVM image) is interrupted by the `bun --watch` restart of every
187	subsequent edit, completing once edits pause.
188
189	## Running it
190
191	Requires Docker (floci, the Cloudflare Container, and the MicroVM). No cloud
192	credentials are used or needed — the suite runs under an alchemy profile
193	that exists nowhere on disk, with AWS credentials stripped from the
194	environment and `CI=1` set so the placeholder Cloudflare env credentials are
195	accepted.
196
197	It also requires a **built workspace** (`pnpm build`). The MicroVM binding
198	pulls `@alchemy.run/floci` into the Worker bundle, and the Worker bundler
199	resolves that workspace package through its `import` condition — i.e.
200	`packages/floci/lib`. On an unbuilt tree the MicroVM Worker boots with
201	`No such module "@alchemy.run/floci"`.
202
203	```sh
204	bun test # from examples/dev-stress
205	DEBUG=1 bun test # mirror the CLI's output to stderr
206	NO_DESTROY=1 bun test # keep the scratch project for inspection
207	DEV_STRESS_PORT_BASE=9100 bun test # move the pinned ports
208	```
209
210	The first apply builds a MicroVM image, so a cold run takes a few minutes
211	before the first assertion runs.
212
213	The suite never touches the checked-in tree: it copies this project into
214	`.stress/<stage>/` and mutates the copy. A full CLI log of the run is left
215	at `.stress/<stage>/dev-stress.log`.
216
217	## Running the stack by hand
218
219	```sh
220	bun run dev
221	```
222
223	`src/ports.ts` pins the local addresses (8790–8795 by default, plus fixed
224	8796–8798 for ECS and EC2), so the same URLs work every time.