clover's git

the-knife

1	---
2	title: Observability
3	description: Effect emits OpenTelemetry natively and the exporter is a Layer. Provision the receiving end — datasets, monitors, notifiers, alarms — as resources in the same Stack as the code that emits the signals.
4	---
5
6	Effect already emits traces, metrics, and logs, and every alchemy
7	runtime ships with the OTLP exporter built in — the exporter is a
8	Layer, wired to resources through the binding infrastructure. With
9	alchemy the receiving end is code too: datasets, ingest tokens,
10	monitors, notifiers, and alarms live in the same Stack as the
11	Workers and Functions they observe, so a threshold change is a diff
12	in the PR — reviewable, revertable, answered by `git blame`.
13
14	## OTel everywhere
15
16	Every `Effect.withSpan` opens a span, every `Metric` update records a
17	data point, every `Effect.logInfo` ships a log line — and each fetch,
18	queue message, cron fire, and invocation gets an `http.server`-style
19	root span automatically. Where the signals go is decided by a
20	telemetry Layer provided on the Function/Worker:
21
22	```typescript
23	import * as Alchemy from "alchemy";
24
25	Effect.provide(
26	Layer.mergeAll(
27	Cloudflare.R2.ReadWriteBucketBinding,
28	Alchemy.Telemetry.layerOtlp({
29	url: "https://collector.example.com",
30	headers: { "x-api-key": apiKey },
31	}),
32	),
33	)
34	```
35
36	Any OTLP endpoint works — Axiom, an OTel collector, any
37	OTLP-compatible vendor. Swap the layer, ship somewhere else; the code
38	emitting the signals never changes. Buffered telemetry is flushed
39	when each request's scope closes (via `waitUntil` on Cloudflare, so
40	export never delays a response). See
41	[Telemetry](/infrastructure-as-effects/telemetry) for the full
42	mechanics.
43
44	## Provision the receiving end
45
46	The endpoint the exporter targets is a resource. An
47	[Axiom](/axiom) `Dataset` is declared per OTEL signal and exposes its
48	OTLP endpoints as outputs:
49
50	```typescript
51	// src/observability.ts
52	import * as Axiom from "alchemy/Axiom";
53
54	export const Traces = Axiom.Dataset("Traces", {
55	name: "app-traces",
56	kind: "otel:traces:v1",
57	});
58	export const Logs = Axiom.Dataset("Logs", {
59	name: "app-logs",
60	kind: "otel:logs:v1",
61	});
62	```
63
64	Each dataset's `otelTracesEndpoint` / `otelLogsEndpoint` /
65	`otelMetricsEndpoint` attributes are [Outputs](/infrastructure-as-code/outputs) —
66	references you wire into whatever runs the exporter.
67
68	## Bind the datasets to the runtime
69
70	Mint a least-privilege ingest token, then provide `Axiom.Telemetry`
71	on the Worker — a binding layer that wires the datasets and token in
72	one line:
73
74	```typescript
75	// src/observability.ts
76	export const Ingest = Axiom.ApiToken("Ingest", {
77	name: "prod-ingest",
78	datasetCapabilities: {
79	"app-traces": { ingest: ["create"] },
80	"app-logs": { ingest: ["create"] },
81	},
82	});
83
84	// src/worker.ts — in the Worker's constructor Effect:
85	import { Ingest, Logs, Traces } from "./observability.ts";
86
87	Effect.provide(
88	Layer.mergeAll(
89	Cloudflare.R2.ReadWriteBucketBinding,
90	Axiom.Telemetry({ token: Ingest, traces: Traces, logs: Logs }),
91	),
92	)
93	```
94
95	Building the layer binds each dataset's OTLP endpoint and the
96	token's `Authorization` header onto the Worker — the bearer travels
97	as a `secret_text` binding (see [Secrets](/environments/secrets)) and
98	never appears in plaintext. At runtime the built-in exporter ships
99	each signal to its dataset — no exporter code in the Worker.
100
101	## Export from the platform
102
103	On Cloudflare you can skip the in-process exporter entirely: an
104	`ObservabilityDestination` has Cloudflare push Workers Logs telemetry
105	(traces, logs, or metrics) to an OTLP collector — no code changes in
106	the Worker:
107
108	```typescript
109	// in the Stack's generator
110	const traces = yield* Traces;
111
112	yield* Cloudflare.Workers.ObservabilityDestination("TracesExport", {
113	url: traces.otelTracesEndpoint,
114	headers: { authorization: `Bearer ${process.env.AXIOM_TOKEN}` },
115	logpushDataset: "opentelemetry-traces",
116	});
117	```
118
119	One destination per signal (`opentelemetry-traces`,
120	`opentelemetry-logs`, `opentelemetry-metrics`). The endpoint URL and
121	headers update in place; changing the dataset triggers a replacement.
122
123	## Alerts in code
124
125	Monitors and notifiers are resources too. A `Notifier` is an alert
126	destination; a `Monitor` is a scheduled APL query that fires it:
127
128	```typescript
129	const slack = yield* Axiom.Notifier("ops-slack", {
130	name: "ops-channel",
131	properties: {
132	slack: { slackUrl: process.env.SLACK_WEBHOOK_URL! },
133	},
134	});
135
136	yield* Axiom.Monitor("error-rate", {
137	name: "High error rate",
138	type: "Threshold",
139	aplQuery: `
140	['app-traces']
141	| where status >= 500
142	| summarize count() by bin_auto(_time)
143	`,
144	operator: "Above",
145	threshold: 100,
146	intervalMinutes: 5,
147	rangeMinutes: 5,
148	notifierIds: [slack.id],
149	});
150	```
151
152	Notifiers cover Slack, email, PagerDuty, Opsgenie, Discord, Teams,
153	and custom webhooks; monitors come in `Threshold`, `MatchEvent`, and
154	`AnomalyDetection` flavors. Raising the threshold is a one-line diff,
155	and the same notifier is reused across every monitor in the stack.
156
157	## CloudWatch alarms and dashboards
158
159	The same idea on AWS: alarms reference metrics by name and fire SNS
160	topics, and dashboards are structured documents — all declared next
161	to the resources they observe:
162
163	```typescript
164	import * as AWS from "alchemy/AWS";
165
166	const alerts = yield* AWS.SNS.Topic("Alerts");
167
168	const errors = yield* AWS.CloudWatch.Alarm("HighErrors", {
169	MetricName: "Errors",
170	Namespace: "AWS/Lambda",
171	Statistic: "Sum",
172	Period: 60,
173	EvaluationPeriods: 1,
174	Threshold: 1,
175	ComparisonOperator: "GreaterThanOrEqualToThreshold",
176	AlarmActions: [alerts.topicArn],
177	});
178
179	yield* AWS.CloudWatch.Dashboard("ApiHealth", {
180	DashboardBody: {
181	widgets: [
182	{
183	type: "metric",
184	width: 12,
185	properties: {
186	title: "Lambda errors",
187	view: "timeSeries",
188	stat: "Sum",
189	metrics: [["AWS/Lambda", "Errors", "FunctionName", "api"]],
190	},
191	},
192	{
193	type: "alarm",
194	width: 12,
195	properties: {
196	title: "Alarms",
197	alarms: [errors.alarmArn],
198	},
199	},
200	],
201	},
202	});
203	```
204
205	`AWS.CloudWatch.CompositeAlarm` ANDs/ORs alarm states into a
206	higher-level alert (`AlarmRule: 'ALARM("HighErrors") OR
207	ALARM("HighLatency")'`), and because stacks are per-stage, `prod`,
208	`staging`, and `pr-42` each get their own dashboards and alarms.
209
210	## Where next
211
212	- [Telemetry](/infrastructure-as-effects/telemetry) — how the built-in exporters work across every runtime, and the override API
213	- [Axiom](/axiom) — datasets, tokens, monitors, notifiers, and dashboards as resources
214	- [Axiom observability guide](/cloudflare/observability/axiom-observability) — end-to-end setup for a Cloudflare Worker
215	- [Tutorial Part 6](/cloudflare/tutorial/part-6) — the step-by-step version
216	- [Secrets](/environments/secrets) — how `Redacted` values like ingest tokens are bound