clover's git

the-knife

1	# gcp-pubsub-fanout
2
3	Event fan-out on Cloud Run with Pub/Sub push. A public orders API
4	publishes one event per order change to a single topic. Two private
5	consumers subscribe to it independently: one sends confirmation emails,
6	the other records every event in BigQuery. Emails that can never be sent
7	are dead-lettered instead of retried forever.
8
9	```
10	┌─ type = "order.created" ──> Email ──> gs://…/emails/:eventId.json
11	POST /orders ──> Orders ──> OrderEvents │ 5 failed attempts
12	POST /orders/:id/cancel │ v
13	│ DeadOrderEvents ──> DeadOrderEventsInbox
14	└─ every event ──> Analytics ──> BigQuery order_events
15	```
16
17	- `src/resources.ts` — the `OrderEvents` topic, the `DeadOrderEvents`
18	topic and its `DeadOrderEventsInbox` pull subscription, the `Outbox`
19	bucket, and the `Warehouse` dataset with its `order_events` table.
20	- `src/Orders.ts` — public `GCP.Function`, the producer:
21	- `POST /orders` `{ email, total }` publishes `order.created`.
22	- `POST /orders/:id/cancel` `{ email }` publishes `order.cancelled`.
23	- Both return `202 { orderId, eventId }`. The event type is also set as
24	the message's `type` attribute, which is what subscription filters
25	match on.
26	- `src/Email.ts` — private `GCP.Function`. Its subscription filters on
27	`attributes.type = "order.created"`, so Pub/Sub never pushes it any
28	other event type. It writes one "sent email" object per event to the
29	outbox. Mail to a `.invalid` address always fails; after 5 attempts
30	Pub/Sub forwards the event to `DeadOrderEvents`.
31	- `src/Analytics.ts` — private `GCP.Function` with no filter. It inserts a
32	row for every event, using the event id as the BigQuery `insertId` so a
33	redelivery does not duplicate it.
34
35	Adding a third consumer is one more `consumeTopicMessages` on the topic;
36	the producer does not change.
37
38	## Bindings and event sources
39
40	| Host | Binding / event source | IAM granted |
41	| --------- | ----------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
42	| Orders | `GCP.PubSub.WriteTopic(OrderEvents)` | `roles/pubsub.publisher` on the topic, to the host's service account |
43	| Email | `GCP.Storage.WriteBucket(Outbox)` | `roles/storage.objectUser` on the bucket, to the host's service account |
44	| Email | `GCP.PubSub.consumeTopicMessages(OrderEvents, { filter, deadLetter })` | `roles/run.invoker` on Email, to its own service account; `roles/pubsub.publisher` on `DeadOrderEvents` and `roles/pubsub.subscriber` on the push subscription, to the Pub/Sub service agent |
45	| Analytics | `GCP.BigQuery.WriteTable(OrderEventsTable)` | `roles/bigquery.dataEditor` on the table, to the host's service account |
46	| Analytics | `GCP.PubSub.consumeTopicMessages(OrderEvents)` | `roles/run.invoker` on Analytics, to its own service account |
47
48	Both consumers use `GCP.Run.TopicEventSource`, the push implementation.
49	Each call creates its own push subscription on the topic, pointed at
50	`/__alchemy/pubsub/orderevents` on the consuming service and signed with
51	an OIDC token for that service's runtime account. The runtime verifies the
52	token before the handler runs, so the consumers stay private. A handler
53	that fails answers `500`, and Pub/Sub redelivers.
54
55	The dead-letter option:
56
57	```ts
58	yield* GCP.PubSub.consumeTopicMessages(
59	orders,
60	{
61	filter: 'attributes.type = "order.created"',
62	deadLetter: { topic: deadOrders, maxDeliveryAttempts: 5 },
63	},
64	handler,
65	);
66	```
67
68	With `deadLetter` set, the subscription also gets a 10s–600s exponential
69	retry backoff (override it with `retryPolicy`). Without it, Pub/Sub
70	redelivers a failed push almost at once, and all 5 attempts could be spent
71	in the seconds a fresh deploy's `run.invoker` grant takes to propagate,
72	dead-lettering good orders.
73
74	Pub/Sub forwards dead letters as the project's service agent
75	(`service-{projectNumber}@gcp-sa-pubsub.iam.gserviceaccount.com`); the
76	subscription grants it the two roles it needs. A topic with no
77	subscription drops what it receives, which is why the dead-letter topic
78	has `DeadOrderEventsInbox`. Dead letters keep their original attributes
79	and gain `CloudPubSubDeadLetterSourceSubscription`.
80
81	## Deploy
82
83	Requires Docker (the services are built from `main`). Credentials come from your alchemy profile: run `alchemy profile` once and pick GCP (*Service account JSON* for a key file, or *Stored* for an access token or key kept in `~/.alchemy/credentials`, plus a default region), then deploy with `--profile <name>`.
84
85	```sh
86	pnpm deploy --profile <name>
87	```
88
89	```sh
90	curl -X POST "$url/orders" -H 'content-type: application/json' \
91	-d '{"email":"ada@example.com","total":42}'
92	curl -X POST "$url/orders/$orderId/cancel" -H 'content-type: application/json' \
93	-d '{"email":"ada@example.com"}'
94	gcloud storage ls "gs://$bucketName/emails/"
95	gcloud pubsub subscriptions pull "$deadLetterSubscription" --auto-ack
96	```
97
98	## Test
99
100	```sh
101	ALCHEMY_PROFILE=<name> bun test
102	```
103
104	Deploys the stack and checks both push subscriptions (filter, dead-letter
105	policy, the service agent's grants). It then places orders and waits for
106	each one's email object in Cloud Storage and its row in BigQuery; cancels
107	an order and checks the cancellation reaches BigQuery but not the email
108	consumer; and places an order to a `.invalid` address and pulls it from
109	the dead-letter subscription. Finally it destroys the stack and checks the
110	services, subscriptions, topics, bucket, and dataset are gone.
111
112	## Destroy
113
114	```sh
115	pnpm destroy --profile <name>
116	```
117
118	The bucket and dataset use `forceDestroy: true`, so destroy removes their
119	contents too. Drop it for real data.