clover's git

the-knife

1	# AWS Service Bring-Up Process
2
3	This document describes the process for going from zero to full Alchemy coverage for a single AWS service:
4
5	- all canonical resources
6	- all bindings
7	- all event sources
8	- all ergonomic helpers
9	- deterministic audit and test coverage checks
10
11	Use this process whenever adding a brand new AWS service or finishing an incomplete one.
12
13	## Goal
14
15	For a given AWS service, the end state should include:
16
17	1. Every canonical Alchemy resource for that service.
18	2. Every important AWS API operation represented either as:
19	- a binding,
20	- a resource lifecycle provider,
21	- an event source surface,
22	- or an intentional helper abstraction.
23	3. Runtime-specific event-source implementations where applicable.
24	4. End-to-end tests covering the implemented binding and event-source surface.
25	5. Deterministic audit checks that report what is still missing.
26
27	## Source Of Truth
28
29	Start from the distilled spec in:
30
31	- `submodules/distilled/packages/aws/src/services/<service>.ts`
32
33	Never start from ad-hoc memory of the AWS service. The distilled spec is the source of truth for operations.
34
35	## Core Concepts
36
37	Every distilled operation must be classified into one of these buckets:
38
39	### 1. Binding
40
41	Use a binding when the operation is a runtime capability.
42
43	Examples:
44
45	- `GetItem(table)`
46	- `PutItem(table)`
47	- `ListTables()`
48	- `DescribeTable(table)`
49
50	Bindings are:
51
52	- one file per operation
53	- the combined `Binding.Service` form (`interface X extends Binding.Service<X, "id", Shape>` + `const X = Binding.Service<X>("id")`); the deploy-time IAM registration is inlined into the impl layer under `if (!globalThis.__ALCHEMY_RUNTIME__)`, resolving the host via `yield* Binding.host`
54	- usually named `alchemy/src/AWS/<Service>/<Operation>.ts` (callable + types) with the impl layer in `<Operation>Http.ts` (AWS runtime impls call the distilled HTTP API authenticated by the Lambda's IAM role; `Http`, not `Binding` — `Binding` is a Cloudflare native-worker concept)
55
56	### 2. Resource
57
58	Use a resource when the operation set implies lifecycle ownership of infrastructure.
59
60	Examples:
61
62	- `createTable` / `updateTable` / `deleteTable` -> `Table`
63	- `createBucket` / `deleteBucket` -> `Bucket`
64
65	Resources are:
66
67	- canonical Alchemy infrastructure entities
68	- implemented as `Resource` contract + provider in a single file
69
70	### 3. Event Source
71
72	Use an event source when the service can push records/events into a runtime.
73
74	This always has two layers:
75
76	1. Service-level abstraction in `alchemy/src/AWS/<Service>/...`
77	2. Runtime-specific implementation in places like:
78	- `alchemy/src/AWS/Lambda/...`
79	- `alchemy/src/Process/...`
80
81	Examples:
82
83	- `consumeBucketEvents(bucket, handler)`
84	- `consumeQueueMessages(queue, handler)`
85	- `consumeTableChanges(table, handler)` for DynamoDB-style change streams
86
87	### 4. Helper
88
89	Use a helper when multiple raw operations should collapse into a more ergonomic surface.
90
91	Examples:
92
93	- `consumeBucketEvents(bucket, handler)`
94	- `consumeQueueMessages(queue, handler)`
95	- batch or transaction wrappers
96
97	Helpers should not hide missing low-level primitives. Implement the primitives first.
98
99	## Resource Arity
100
101	Classify each binding by resource arity:
102
103	- `0`: service/account scoped
104	- example: `ListTables`
105	- `1`: one resource
106	- example: `GetItem(table)`
107	- `2+`: multiple resources
108	- example: `RestoreTableToPointInTime(fromTable, toTable)`
109	- example: copy, batch, or transaction style operations
110
111	This classification helps decide:
112
113	- binding shape
114	- helper shape
115	- policy shape
116	- whether the operation belongs on a resource or service surface
117
118	Arity should be modeled in terms of canonical resources whenever possible.
119
120	- good: a `2`-arity binding accepts `<From extends Table, To extends Table>`
121	- bad: a `2`-resource operation accepts one `Table` plus a raw `string` target name
122	- only fall back to raw identifiers when there is no real canonical resource to bind against
123	- when there's a missing canonical resource, that might suggest we need to add one
124
125	### Case Study: `ExecuteTransaction`
126
127	Use `ExecuteTransaction` as the reference pattern for bindings that touch `1..*` canonical resources.
128
129	The ambiguity we want to avoid is:
130
131	- bad: `ExecuteTransaction()` with IAM `Resource: ["*"]`
132	- bad: `ExecuteTransaction(tableNames: string[])`
133	- bad: a SID like `AWS.DynamoDB.ExecuteTransaction(2 table(s))` that hides which resources were bound
134
135	The required pattern is:
136
137	- good: `ExecuteTransaction(tableA, tableB, ...)`
138	- good: the binding type requires at least one table
139	- good: the policy enumerates exactly those table ARNs
140	- good: the SID is deterministic and names the participating resources
141
142	Runbook for any `1..*` resource-bound binding:
143
144	1. Model the binding arguments as a non-empty tuple of canonical resources.
145	2. Call `.bind(resourceA, resourceB, ...)`, never `.bind()` with hidden resource discovery.
146	3. Before constructing the SID, sort the resources by `LogicalId` so equivalent calls produce the same binding identity.
147	4. Pass the sorted resource array into the `host.bind` template so the SID renders each resource name explicitly, for example `AWS.DynamoDB.ExecuteTransaction(TableA, TableB)`.
148	5. Build IAM `Resource` from those same sorted resources, for example `sortedTables.map((table) => table.tableArn)`.
149	6. Only use `Resource: ["*"]` if the operation is truly service-scoped or AWS IAM does not support resource-level scoping for that API.
150
151	Reference shape:
152
153	```ts
154	type ExecuteTransactionTables = [Table, ...Table[]];
155
156	const sortedTables = [...tables].sort((a, b) =>
157	a.LogicalId.localeCompare(b.LogicalId),
158	);
159
160	yield *
161	host.bind`Allow(${host}, AWS.DynamoDB.ExecuteTransaction(${sortedTables}))`({
162	policyStatements: [
163	{
164	Effect: "Allow",
165	Action: [
166	"dynamodb:PartiQLSelect",
167	"dynamodb:PartiQLInsert",
168	"dynamodb:PartiQLUpdate",
169	"dynamodb:PartiQLDelete",
170	],
171	Resource: sortedTables.map((table) => table.tableArn),
172	},
173	],
174	});
175	```
176
177	## Full Bring-Up Loop
178
179	Follow this loop until audit is clean or only intentionally deferred items remain.
180
181	### Step 1: Run Audit
182
183	Run:
184
185	```bash
186	pnpm audit:service dynamodb
187	```
188
189	The audit should report:
190
191	- implemented bindings
192	- missing bindings
193	- resource lifecycle ops
194	- event source ops
195	- helper candidates
196	- registration gaps
197	- missing binding tests
198
199	### Step 2: Build The Service Model
200
201	Before coding, explicitly answer:
202
203	1. What are the canonical resources?
204	2. What bindings belong to each resource?
205	3. What service-scoped bindings exist?
206	4. What event-source surfaces exist?
207	5. What helpers should exist?
208	6. Which items are intentionally deferred?
209
210	For DynamoDB, for example:
211
212	- canonical resource: `Table`
213	- folded table-owned surface: local/global secondary indexes live on `Table`, not a standalone `SecondaryIndex` resource
214	- bindings: item/table/admin operations
215	- event source: Kinesis streaming destination / change stream surface
216	- helper: `consumeTableChanges(table, props?, handler)`
217
218	### Step 3: Implement Missing Bindings
219
220	Implement the smallest coherent slice first.
221
222	Good order:
223
224	1. read/admin bindings
225	2. write/update bindings
226	3. transaction/batch bindings
227	4. restore or special-case bindings
228
229	Binding conventions:
230
231	- one file per operation
232	- no auto-marshalling
233	- user passes raw AWS SDK/distilled types
234	- the binding should mostly inject resource identifiers like `TableName`
235	- policies should be explicit and minimal
236	- if an operation is `2+`-arity, the binding should capture all participating resources so the policy can stay least-privilege
237	- never use `Resource: ["*"]` for a resource-bound binding if it can be avoided by passing canonical resources to `.bind(...)`
238	- if an operation touches `1..*` canonical resources, model the binding to accept those resources explicitly so the policy can enumerate only those ARNs
239	- `Resource: ["*"]` is only acceptable when the operation is truly service-scoped or AWS does not support narrower resource-level IAM for that API
240	- do not add IAM `Sid` fields in binding policy statements unless there is a demonstrated AWS requirement for one
241
242	Example:
243
244	- good: `GetItemRequest extends Omit<GetItemInput, "TableName">`
245	- bad: replacing AWS input types with custom marshalled `Record<string, any>`
246
247	### Step 4: Register Everything
248
249	After each binding/resource implementation, update:
250
251	1. `alchemy/src/AWS/<Service>/index.ts`
252	2. `alchemy/src/AWS/Providers.ts`
253
254	If registration is missing, audit should flag it.
255
256	### Step 5: Add Binding Tests Immediately
257
258	Every implemented binding should have a corresponding `describe("<BindingName>")` block in:
259
260	- `alchemy/test/AWS/<Service>/Bindings.test.ts`
261
262	Examples:
263
264	```ts
265	describe("GetItem", () => {
266	test("gets an existing item", ...);
267	test("returns undefined for missing item", ...);
268	});
269	```
270
271	```ts
272	describe("PutItem", () => {
273	test("puts an item into the table", ...);
274	});
275	```
276
277	Deterministic rule:
278
279	- if a binding exists, audit should warn if the matching `describe("<BindingName>")` block is missing
280
281	### Step 6: Use A Real Lambda Fixture
282
283	Use a real Lambda fixture in:
284
285	- `alchemy/test/AWS/<Service>/handler.ts`
286
287	Pattern:
288
289	1. create resource(s)
290	2. bind operations
291	3. expose HTTP endpoints for each tested operation
292	4. use those endpoints from the E2E test
293
294	This keeps tests end-to-end while still giving fine-grained per-binding coverage.
295
296	Layer provisioning rule:
297
298	- when a Lambda fixture provides both composite layers and foundational binding layers, do not put them all in one flat `Layer.mergeAll(...)`
299	- composite layers include event sources, sinks, and higher-level helpers that themselves depend on lower-level bindings
300	- foundational layers include the binding/capability implementations such as `GetItemLive`, `PutObjectLive`, `PublishLive`, `PublishBatchLive`, and similar
301	- use `Effect.provide(Layer.provideMerge(...))` so the foundational layer group is provided to the composite layer group
302	- otherwise `Layer.mergeAll(...)` only unions outputs and requirements, and sibling layers do not satisfy each other's requirements
303
304	Required shape:
305
306	```ts
307	Effect.provide(
308	Layer.provideMerge(
309	Layer
310	.mergeAll
311	// composite services: event sources, sinks, helpers
312	(),
313	Layer
314	.mergeAll
315	// foundational bindings/capabilities they depend on
316	(),
317	),
318	);
319	```
320
321	Example failure mode:
322
323	- `TopicSinkLive` depends on `PublishBatch`
324	- if `TopicSinkLive` and `PublishBatchLive` are only siblings in the same `Layer.mergeAll(...)`, the final Lambda effect still requires `PublishBatch`
325	- grouping them with `Layer.provideMerge(...)` removes that leaked requirement
326
327	### Step 7: Make Setup Observable
328
329	Fixture setup must log clearly:
330
331	- destroying previous resources
332	- deploying fixture
333	- function URL
334	- readiness probe URL
335	- readiness retries
336	- readiness success
337
338	If setup is confusing, add logs before adding retries.
339
340	### Step 8: Keep Readiness Failure Fast
341
342	Do not wait minutes for a function to become ready.
343
344	Current convention:
345
346	- if the function is not ready in about 20 seconds, fail the setup
347
348	Use a short retry budget and log each failed readiness attempt.
349
350	### Step 9: Implement Missing Resource Surface
351
352	Once bindings are in good shape, fill missing resource-level gaps:
353
354	- missing resource providers
355	- placeholder resources
356	- resource update/delete gaps
357	- missing nested infrastructure surfaces such as indexes or replicas
358
359	Important:
360
361	- audit may only recognize canonical resources implied directly by lifecycle operations
362	- you must still inspect the service for real resource-shaped gaps not fully inferred by the script
363
364	Example:
365
366	- a placeholder like `SecondaryIndex.ts` still counts as incomplete coverage until it is either removed or folded into the canonical resource model
367
368	### Step 10: Implement Event Sources
369
370	For stream/notification services:
371
372	1. define a service-level abstraction
373	2. add runtime-specific implementation(s)
374	3. add helper surface
375	4. add tests
376
377	When the event source needs an intermediate canonical resource, the binding should
378	create that resource automatically instead of forcing user code to instantiate it.
379	SNS is the reference case:
380
381	- the public binding is `notifications(topic, handler)`
382	- the Lambda runtime policy creates the `Subscription` resource automatically
383	- any service-to-Lambda invoke permission stays in the runtime policy layer that
384	wires the event source, not in user code
385	- the canonical resource still exists and can be used directly when needed; the
386	helper just creates it on behalf of the user
387
388	For DynamoDB-style changes this likely means:
389
390	1. table-side stream/destination surface
391	2. Lambda runtime integration
392	3. `consumeTableChanges(table, props?, handler)` helper
393	4. E2E coverage
394
395	### Step 11: Implement Helpers
396
397	After low-level primitives exist, add ergonomic helpers for:
398
399	- stream subscriptions
400	- batch operations
401	- transactions
402
403	Helpers should feel native to Alchemy and match established service patterns.
404
405	### Case Study: DynamoDB Streams
406
407	DynamoDB Streams is the reference pattern for mutable event-source configuration that belongs to a canonical resource but still needs binding-based composition.
408
409	Required shape:
410
411	- the canonical resource remains `Table`
412	- `Table` keeps stream state in its attributes, but does not accept `streamSpecification` as a plain input prop
413	- stream enablement is requested through the table binding contract
414	- the public helper is `consumeTableChanges(table, props?, handler)`
415	- the service-level abstraction lives in `alchemy/src/AWS/DynamoDB/Stream.ts`
416	- the Lambda runtime implementation lives in `alchemy/src/AWS/Lambda/TableEventSource.ts`
417
418	Why this pattern exists:
419
420	- stream enablement mutates the table itself
421	- the consumer is another resource, usually a Lambda Function
422	- prop-driven stream configuration makes circular composition awkward
423	- bindings let the consumer request the mutation while `Table` stays the canonical owner of stream state
424
425	Implementation rules:
426
427	1. The helper attaches stream requirements to `Table` through bindings.
428	2. The `Table` provider derives the effective stream configuration from bindings during `create` and `update`.
429	3. Zero stream bindings means the table stream should be disabled.
430	4. Multiple bindings may coexist only when they request the same `StreamViewType`.
431	5. Conflicting `StreamViewType` requests must fail deterministically before AWS calls are made.
432	6. Runtime-specific layers handle IAM, host wiring, and event-source mapping resources; they do not move stream ownership out of `Table`.
433
434	Lambda-first slice:
435
436	- implement `consumeTableChanges(table, props?, handler)` first for Lambda
437	- the Lambda layer binds the table stream requirement, grants stream-read IAM, and creates `AWS.Lambda.EventSourceMapping`
438	- Process or other runtimes can be added later without changing `Table` back to a prop-driven stream model
439
440	### Step 12: Re-Run Tests And Audit
441
442	After every meaningful slice:
443
444	1. run the service-specific E2E tests
445	2. rerun `pnpm audit:service <service>`
446	3. use the updated output to choose the next slice
447
448	Repeat until:
449
450	- no important missing bindings remain
451	- resource surface is complete
452	- event-source surface is complete
453	- tests are in place
454
455	## Deterministic Checks We Want
456
457	The audit should help enforce:
458
459	1. Missing bindings from distilled operations.
460	2. Missing registration in `index.ts`.
461	3. Missing registration in `Providers.ts`.
462	4. Missing `describe("<BindingName>")` blocks in `Bindings.test.ts`.
463	5. Avoidable `Resource: ["*"]` usage in non-zero-arity bindings.
464
465	Over time it should also grow to flag:
466
467	6. placeholder resources with no provider
468	7. missing event-source surface for detected stream operations
469	8. missing helper surface for known event-source patterns
470
471	## AWS-Specific Conventions Learned
472
473	### No Auto-Marshalling
474
475	Bindings do not auto-marshall request/response payloads for DynamoDB-style operations.
476
477	User responsibility:
478
479	- pass raw AWS/distilled input types
480	- marshal/unmarshal attribute values themselves
481
482	### `Output.interpolate`
483
484	Only use `Output.interpolate` when actually composing a string.
485
486	Examples:
487
488	- good: `table.tableArn`
489	- good: `Output.interpolate\`${table.tableArn}/index/\*\``
490	- bad: `Output.interpolate\`${table.tableArn}\``
491
492	### `Effect.orDie`
493
494	For Lambda test fixtures, apply `Effect.orDie` once at the outer request handler layer, not repeatedly per route.
495
496	### Binding Test Structure
497
498	Use one `describe("<BindingName>")` block per binding.
499
500	Inside that block:
501
502	- happy path
503	- relevant unhappy paths
504
505	Do not lump all bindings into one large undifferentiated test block.