1# Substation
2
3`https://substation.ai.paperclover.net` serves the Substation dashboard and its
4native APIs. Snow OAuth protects the dashboard and requires `infra-admin`.
5Substation's trusted-proxy mode accepts the identity header from the local
6OAuth proxy. Caddy sends `/v1/*` and `/v0/management/*` directly to the hub,
7which checks its own client and management keys.
8
9## Set up accounts
10
111. Open the dashboard and sign in through Snow.
122. Open **Settings → Hub accounts** and add a Claude or Codex account.
133. Follow the provider's sign-in link. For Codex, enter the displayed device
14 code at the provider's page; for Claude, paste the returned code into
15 Substation. Repeat for each account.
164. Check that each account shows its quota and can take requests.
17
18The container includes both providers' login tools. Login scratch directories
19are disposable; the hub saves and refreshes the resulting account files in its
20own store. Production starts empty and imports no CPA accounts or history.
21
22## Connect clients
23
24The client key is saved on Sandwich at
25`~/.config/substation/client-key`, readable only by Clover. SSO mode deliberately
26does not reveal or rotate keys through Substation's dashboard. Nomad variable
27`nomad/jobs/substation` owns the initial `api_key` and `management_key`.
28
29For Claude Code:
30
31```sh
32export SUBSTATION_CLIENT_KEY="$(cat ~/.config/substation/client-key)"
33export ANTHROPIC_BASE_URL=https://substation.ai.paperclover.net
34export ANTHROPIC_AUTH_TOKEN="$SUBSTATION_CLIENT_KEY"
35claude
36```
37
38For Codex, use a separate `CODEX_HOME` to keep its normal sign-in and settings:
39
40```sh
41mkdir -p ~/.codex-substation
42```
43
44Save this as `~/.codex-substation/config.toml`:
45
46```toml
47model = "gpt-6.1-sol"
48model_provider = "substation"
49
50[model_providers.substation]
51name = "Substation"
52base_url = "https://substation.ai.paperclover.net/v1"
53env_key = "SUBSTATION_CLIENT_KEY"
54wire_api = "responses"
55```
56
57Then run:
58
59```sh
60export SUBSTATION_CLIENT_KEY="$(cat ~/.config/substation/client-key)"
61CODEX_HOME="$HOME/.codex-substation" codex
62```
63
64The [official OpenAI configuration documentation](https://learn.chatgpt.com/docs/config-file/config-advanced)
65describes custom providers. Choose a model the connected account supports.
66Substation serves Anthropic Messages and OpenAI Responses; it does not implement
67`/v1/chat/completions`.
68
69T3 Code's hub quota integration uses the same base address and the separate
70management key from `~/.config/substation/management-key` on Sandwich. The
71management API is `/v0/management`; it remains behind its own bearer key. The
72public reverse proxy reaches the hub over loopback, so the hub's remote
73management switch is unnecessary. A client key grants no management access.
74
75## Persistence and deployment
76
77Production state lives under `/srv/prod/substation/data`: `auth` holds provider
78accounts, `hub` holds routing state, `dashboard` holds SQLite history and
79settings, `home` holds the login tools' home, and `run` holds build identities.
80Nomad owns the generated keys and OAuth client credentials. Existing numeric
81service identities remain reserved when a service is retired.
82
83`service/substation/build-source.json` pins a local Jujutsu revision of
84`../substation` and its release version. `service/substation/Dockerfile` builds
85the program and includes pinned Claude and Codex login tools. The exporter
86includes its source manifest as build provenance. Snow Globe receives source and builds locally;
87it receives no GitHub SSH key or repository credentials.
88
89Stage with `python3 tools/deploy.py stage substation`. Previews start with empty
90storage, their own keys, and a separate Snow OAuth client. Updates preserve
91that preview's state. Use disposable accounts and prompts when testing.
92Production uses described, conflict-free main and `python3 tools/deploy.py prod`.
93The in-app updater is unavailable inside the immutable image.
94
95CPA's live dataset, preview, keys, and routing are retired after the replacement
96passes production checks. Restricted off-server recovery copies retain the old
97store; restoring them requires a separate explicit request.
98
99## Request diagnostics
100
101API responses include a Caddy-generated `X-Request-ID`. API access records in
102Caddy's journal contain that ID, trace ID, path, method, protocol, status, bytes,
103duration, declared content length, upstream address, upstream status, and time
104to upstream headers. Other headers, query strings, cookies, and request/response
105bodies are excluded from these access records. The log shipper retains them in
106Victoria Logs.
107
108The `monitor` task checks anonymous rejection, client-key model access, and
109management-key version access through HTTPS every minute. It requires the edge
110request ID and rejects redirects, so an OAuth login page cannot pass as a healthy
111API. It sends no generation requests and logs only check outcomes and IDs.
112
113Inspect `substation` task `monitor` in the service logs, and search Caddy's
114journal for the response's request ID. A 499 with no upstream status can mean
115client cancellation or Caddy's request-body idle timeout; an upstream 499 is
116recorded separately.
117API 499 responses default to `X-Should-Retry: true` so Claude can recover from
118request-body timeouts; an upstream's explicit retry decision takes precedence.
119Caddy-generated API 499/504 errors close HTTP/1 connections and include the
120request ID and retry hint. A timed-out upload can still have queued body bytes;
121reusing that connection lets those bytes become an invalid next request and
122can produce an unlogged 431 before routing. Upstream responses keep their
123retry decisions and connection behavior.
124The packaged [Caddy patch](../nixos/caddy-body-timeout.patch) preserves failed
125read deadlines while the upstream transport closes the body. Otherwise,
126net/http can block draining a small unfinished upload before error handling.
127Compare `bytes_read` with `content_length` to identify incomplete fixed-length
128uploads. A missing length can mean a chunked request; it does not prove completion.
129A 200 followed by a truncated stream needs the edge byte count and the hub's
130stream outcome. A successful probe does not establish that every generation
131stream completed.
132
133`snowglobe-substation-watch.service` follows Caddy with a persisted journal
134cursor and samples TCP counters once per second for the hub and known API
135clients. It survives service restarts and reboots. The latest two minutes are
136retained in `/var/lib/snowglobe/substation-watch/recent.json`; each 499 or 5xx
137retains its request metadata and surrounding TCP progress in a private file
138named by request ID. The directory keeps the latest 100 incidents. No packet
139payloads, request bodies, cookies, or credentials are collected.
140
141Watcher heartbeat and incident summaries appear in its journal and Victoria
142Logs. `upload_complete: false` proves an incomplete fixed-length upload;
143the saved receive/send queues, byte counters, receive windows, and retransmits
144distinguish stalled sending from server backpressure and network loss.
145Incident files also retain matching TCP sequence numbers, acknowledgements,
146SACK ranges, windows and flags. These establish packet loss and which side
147sent FIN/RST first. Only parsed transport fields are retained.
148
149Worktop's `net.paperclover.substation-watch` LaunchAgent records matching
150Substation sockets under `~/.local/state/substation-watch`. Minute files retain
15112 hours of process IDs, CPU time/state, TCP queues, byte counters,
152retransmissions and windows.
153Paired captures showed Worktop leave a SACK-reported hole unrepaired for 75–109
154seconds. `net.paperclover.tcp-rack-off` disables RACK before login while retaining
155SACK recovery; it is a system-wide transport trade, not a Substation setting.
156The root-owned LaunchDaemon is installed from the exported
157`tools/net.paperclover.tcp-rack-off.plist` and should exit zero after each boot.
158The installed script is an exported revision of `tools/substation-client-watch.py`;
159the LaunchAgent follows its `~/.local/share/substation-watch/current.py` link.
160Match UTC timestamps to the server incident before comparing TCP progress.
161An upload with queued bytes and no acknowledgement progress for 10 seconds
162retains the preceding 120 socket samples in a private `stall-<timestamp>.json` file.
163The latest 100 stalls survive rolling-history expiry and watcher restarts.
164Each stalled socket has one archive, updated after the stall and after packet
165ingestion catches up, retaining later retransmissions and closure.
166When an administrator starts a TCP summary capture into `tcp-headers.log`, the
167client watcher retains up to two minutes of matching sequence/ACK/SACK metadata
168with each stall, bounded to 100,000 packets.
169It consumes and truncates that append-only log at 64 MiB without restarting
170tcpdump. Archives mark each truncation; exclude that instant from packet-loss
171conclusions because concurrent writes can be lost during truncation.
172Packet ingestion runs independently of socket sampling. Samples record unread
173capture bytes, active ingestion and errors. Infer missing sender packets only
174after ingestion finishes and the backlog clears, with no capture error or
175truncation gap in that interval. The LaunchAgent uses `ProcessType: Standard`.