clover's git

the-knife

1	# T3 Connect
2
3	T3 Connect uses Clerk for cloud identity. The relay manages environment links,
4	credentials for reaching environments, and managed tunnel allocations. After
5	bootstrap, clients send application traffic through the environment's tunnel
6	hostname; the relay Worker does not proxy their HTTP or WebSocket sessions.
7	The one exception is automation webhooks: the relay forwards
8	`/v1/hooks/:environmentId/:hookId/:token` to the environment's tunnel so
9	senders get a stable URL. It keeps bodies and tokens out of its traces and
10	leaves token and signature verification to the environment
11	([forwarder](../../infra/relay/src/hooks/HookForwarder.ts)).
12
13	By default the forwarder stores nothing. An environment can opt in to having
14	the relay hold requests while it is offline
15	(`hold_webhooks_while_offline` on its link). Only then does the relay store the
16	raw request, including the hook token in the path, in a Durable Object for
17	that environment, with SQLite storage. The object pushes held requests back
18	through the tunnel from its alarm, oldest first, and backs off while the
19	environment stays away. When the tunnel reconnects, the environment asks the
20	relay to deliver right away. Requests are deleted once the environment
21	answers, after 24 hours, or when no user has the environment linked. The relay
22	still never checks the token; delivery goes through the same environment route.
23	Every forward carries `x-t3-relay-delivery-id`, so a request that reached the
24	environment before a timeout and is delivered again later runs once
25	([inbox object](../../infra/relay/src/hooks/HookInboxObject.ts)).
26
27	A Durable Object, not Postgres or Queues, because held requests are write-once,
28	read-once bodies of up to 1 MiB that need per-environment order, caps, and
29	retry timing. Queues cap messages at 128 KB and cannot hold one environment's
30	requests back while it is away.
31
32	Clerk, deployment, and native authentication setup live in the
33	[Connect setup runbook](../operations/connect-setup.md).
34
35	## The relay is a trusted broker
36
37	An authenticated cloud user still needs an active environment link. The relay
38	asks that environment to mint a one-time bootstrap credential bound to the
39	client's DPoP key. The client exchanges it directly with the environment for an
40	[environment session](./environment-auth.md). The relay never receives that
41	session token, and possessing the bootstrap credential alone does not permit
42	redeeming it without the client's private key.
43
44	Both sides authenticate this exchange. The environment accepts only bounded,
45	replay-guarded relay proofs for its own identity, linked user, and requested
46	operation. Signed environment responses bind the result to the request nonce;
47	mint responses also bind the credential to the client proof key. The relay
48	verifies those bindings before returning a credential. This prevents a different
49	process behind the tunnel from impersonating the linked environment. The checks
50	meet in the
51	[environment link service](../../apps/server/src/cloud/CloudLink.ts) and
52	[relay connector](../../infra/relay/src/environments/EnvironmentConnector.ts).
53
54	The relay holds the signing authority for mint requests. DPoP protects an honest
55	exchange from credential reuse; it does not make a compromised relay signing
56	key harmless. Keep that trust assumption explicit when changing the protocol.
57
58	Managed tunnels expose only a validated loopback HTTP origin. Link proof checks
59	reject forwarded authority headers, and the relay resolves endpoints from its
60	own managed allocations rather than a caller-supplied URL. Health and mint
61	requests must not follow redirects. These restrictions keep endpoint discovery
62	from turning into arbitrary relay egress or exposing another service on the
63	environment host.
64
65	## A link outlives a connector process
66
67	CLI authorization, desired exposure, and a running connector have different
68	lifetimes. Linking can record intent while the server is stopped. Startup
69	reconciles that intent. CLI logout removes the stored cloud credential and
70	disables exposure without uninstalling the environment's background service.
71
72	Managed allocations belong to a user/environment pair. Provisioning checkpoints
73	external tunnel and DNS resources so retries can reconcile partial work. A
74	normal shutdown of a CLI-managed link releases its tunnel to avoid paying for
75	an idle resource, retaining the hostname reservation for the next startup.
76	It also retains the allocation record so the environment remains "offline"
77	rather than becoming "not authorized".
78
79	Two cases must retain the tunnel across shutdown. A link installed through a
80	client has no startup provisioning path and depends on its stored connector
81	token. An update handoff immediately starts a replacement server, and replacing
82	the tunnel would add routing propagation delay to every update. These exceptions
83	belong to [shutdown handling](../../apps/server/src/cloud/CloudLink.ts).
84
85	Release and unlink claim the allocation generation before deleting external
86	resources. A delayed cleanup must not delete a tunnel reused by a concurrent
87	restart or relink. Unlink commits authorization revocation before external
88	teardown, because a database failure must leave the active link usable. Failed
89	teardown retains enough state to retry. See the
90	[managed endpoint lifecycle](../../infra/relay/src/environments/ManagedEndpointProvider.ts).
91
92	## Idle tunnels are reclaimed and recovered
93
94	Cloudflare bills a tunnel whether or not a connector is attached, so a laptop
95	that sleeps with a linked environment leaves a paid tunnel behind. The relay's
96	five-minute maintenance job can reclaim those tunnels. `RELAY_TUNNEL_CLEANUP_MODE`
97	selects `off`, `dry-run`, or `enabled`, with `off` as the default. The mode is
98	read at deploy time, so changing it means a relay deploy, not a variable flip.
99	A candidate is a same-stage tunnel that Cloudflare reports down for at least
100	five minutes, or one that never connected and is at least an hour old. The
101	longer grace for never-connected tunnels covers a pairing still in progress.
102
103	Cleanup deletes only tunnels whose host has registered recovery. Allocations
104	without recovery registration belong to hosts that cannot replace a deleted
105	tunnel and are left alone. Allocations with no recorded tunnel ID, or a
106	different tunnel ID, are skipped because a provision may own them. A tunnel with
107	no allocation row at all is counted as `skippedOrphan` and never deleted: there
108	is no row to lock, so a relink that adopts it by name could race the delete.
109	Clear those by hand. Each sweep is bounded: at most ten list requests, 100 deletions, a
110	two-minute deadline, and an early stop on a Cloudflare rate limit. Each sweep
111	starts one budget further along the candidate list, so a block of deletes that
112	keep failing cannot starve the tunnels listed after them. See the
113	[reaper](../../infra/relay/src/environments/ManagedEndpointReaper.ts).
114
115	A host registers recovery at startup by sending its tunnel ID and loopback
116	origin with a short-lived signature from the environment key. Registration
117	touches Cloudflare only when the local host or port changed, and once per
118	existing allocation on the first registration after the upgrade because the
119	stored origin is empty. First registrations are jittered so an auto-update wave
120	does not hit the relay at once. The host stores a confirmed-origin marker with
121	the connector config, and a later boot starts the connector before registration
122	only when that marker matches the current config and port. If registration
123	cannot reach the relay for ten minutes, the host starts its stored config anyway
124	and keeps registering in the background until it can reconcile the origin.
125	If the connector exits, or `cloudflared` reports repeated tunnel
126	rejections, the host asks the relay for a replacement, at most once every two
127	minutes. The relay
128	provisions under the same allocation, so the hostname and DNS record survive
129	and clients keep their bindings. Every mutation on an allocation bumps its
130	`generation`, and deletion locks the row at the generation it claimed, so a
131	host that reconnects mid-sweep wins.
132
133	## The relay client follows the server's pin
134
135	The host runs `cloudflared` pinned by `CLOUDFLARED_VERSION` in
136	[`relayClient.ts`](../../packages/shared/src/relayClient.ts), and bumping it there
137	(version, URLs, and checksums) is the whole release step. A linked host that
138	starts a server with a new pin keeps its connector up on the newest older managed
139	release, installs the pinned one in the background, then starts a second
140	connector on it for the same tunnel. The old connector stops only after the new
141	one registers, so the swap never takes the host off the relay. If the new one
142	fails to spawn, the old one keeps serving and the install retries later. A host
143	with no relay client installs one first. A `cloudflared` on `PATH` or an explicit `T3CODE_CLOUDFLARED_PATH` is the
144	user's choice and is never replaced, since linking asks before downloading.
145
146	Once the pinned release registers a connection, managed releases older than the
147	pin are deleted except the newest of them. That one, and any newer release, can
148	belong to another server sharing the same T3 home, such as a rollback after a
149	failed update, which could not run again without it.
150
151	The version comes from running `cloudflared version`, not from the folder name.
152	Connectors from before `--no-autoupdate` replaced themselves in place, so a
153	managed folder can hold another release. Such a binary still runs as a fallback
154	while the pin installs. Every binary needs `CLOUDFLARED_MIN_VERSION`, the oldest
155	release that accepts every flag the connector is started with. Raise it whenever
156	a new flag is added.
157
158	## OAuth traps
159
160	Interactive clients and the headless CLI use the same Clerk application but
161	different credentials. The relay accepts both session-template JWTs and CLI
162	OAuth tokens; requiring a JWT template for the CLI would reject valid logins.
163	The CLI is a public OAuth client using PKCE and stores no client secret.
164
165	Loopback CLI authorization starts on the hosted `/connect` page so sign-in
166	completes before entering Clerk's authorize endpoint. Sending a signed-out
167	browser straight to that endpoint loses the authorize parameters during the
168	sign-in redirect. The [shared flow](../../packages/shared/src/connectAuth.ts)
169	preserves PKCE and state for the loopback callback.
170
171	SSH and headless sessions use Clerk's OAuth device authorization grant because
172	the browser cannot ordinarily reach a listener on the remote machine. The CLI
173	polls Clerk's token endpoint directly while the user approves a short code on
174	Clerk's hosted device page; the hosted app plays no part and there is no
175	redirect URI or PKCE. The grant must be enabled on the CLI OAuth application
176	or the device endpoint returns an error before any prompt is shown.