| 1 | # Devices |
| 2 | |
| 3 | The environment server owns simulators and emulators the way it owns |
| 4 | terminals: discovery, streaming, and agent access all run there, and every |
| 5 | client reaches them through the environment connection. This is what makes the |
| 6 | Device panel work over Tailscale and T3 Connect, including when an SSH host runs the devices. |
| 7 | |
| 8 | ## Two external tools, one seam |
| 9 | |
| 10 | [expo-device-hub](../../apps/server/src/device/LocalDeviceHost.ts) streams and |
| 11 | [agent-device](../../apps/server/src/device/AgentDeviceShim.ts) drives. Each is |
| 12 | npm-installed at a pinned version into the T3 home after its matching Device |
| 13 | panel consent step. Manual setup installs and starts only expo-device-hub; |
| 14 | agent-device remains absent and stopped until agent access is granted. Both run |
| 15 | with the server's Node; `npx` would make the first `device_open` after a reboot |
| 16 | depend on the registry. The hub is a supervised child rather than an imported |
| 17 | middleware because serve-sim loads private CoreSimulator frameworks through a |
| 18 | native addon, and a crash there must not take the server down. |
| 19 | |
| 20 | Everything platform-specific sits behind |
| 21 | [`DeviceHost`](../../apps/server/src/device/DeviceHost.ts). The service, the |
| 22 | proxy, and the MCP tools only see a hub origin and an agent-device endpoint. |
| 23 | SSH hosts forward both endpoints to server loopback. Every proxied request |
| 24 | also carries the host id; device ids alone are not unique across hosts. |
| 25 | |
| 26 | ## The hub is never exposed |
| 27 | |
| 28 | serve-sim has a shell-exec route whose token is readable from its own |
| 29 | unauthenticated `/api`, and serve-emu's action routes have no auth at all. The |
| 30 | hub binds loopback and the only way in is the |
| 31 | [proxy](../../apps/server/src/device/DeviceHubProxy.ts), which allowlists the |
| 32 | stream, config, and screenshot routes and authenticates every request as an |
| 33 | environment session. `<img>` and `WebSocket` cannot carry headers, so the proxy |
| 34 | authenticates like the `/ws` upgrade: cookie, or a short-lived `wsTicket` that |
| 35 | bearer and DPoP clients mint over authenticated HTTP. The ticket is stripped |
| 36 | before the request reaches the hub. |
| 37 | |
| 38 | Stream responses carry `Cache-Control: no-transform`; the compression |
| 39 | middleware would otherwise buffer an MJPEG body that never ends. In browser dev, |
| 40 | the Vite proxy must forward WebSocket upgrades for `/api`, not only `/ws`. |
| 41 | |
| 42 | ## Device settings never go through the hub |
| 43 | |
| 44 | serve-sim's preview drives its Tools panel by sending shell commands over that |
| 45 | same exec channel. Proxying it, even allowlisted, would hand any environment |
| 46 | session arbitrary command execution on the host, so T3 does not. The |
| 47 | [`device.action`](../../apps/server/src/device/DeviceActions.ts) RPC runs the |
| 48 | underlying `simctl`, `adb`, and serve-sim helper binaries itself through |
| 49 | `DeviceHostReady.run`, one typed action per control, and returns the settings |
| 50 | it reads back. The proxy allowlist grows only with read routes (accessibility |
| 51 | tree, foreground app, event log) and refuses non-GET methods everywhere except |
| 52 | screenshot capture and stream tuning. |
| 53 | |
| 54 | ## Agents drive through the CLI |
| 55 | |
| 56 | The `device_*` toolkit is deliberately four tools: list, open, screenshot, and |
| 57 | close. Driving happens through the `agent-device` CLI, which has the semantic |
| 58 | snapshot model agents need and stays current with its own releases. T3 prepends |
| 59 | a shim directory to the provider's PATH. The CLI installs on the environment |
| 60 | server even when that server cannot run simulators. Hosts start on demand. |
| 61 | |
| 62 | That environment is fixed when the provider subprocess spawns, so |
| 63 | [`prepareMcpSession`](../../apps/server/src/orchestration-v2/ProviderSessionManager.ts) |
| 64 | starts agent-device only when device support and agent access have both been |
| 65 | enabled, the session has the `device` capability, and the machine can run at |
| 66 | least one platform. Starting it later from `device_open` would leave the |
| 67 | already-running agent without the CLI. |
| 68 | |
| 69 | How to drive a device is returned from `device_open`, not kept in an |
| 70 | always-loaded prompt or skill: it costs nothing in threads that never open a |
| 71 | device and cannot drift from the pinned CLI version. The always-on prompt block |
| 72 | is a few lines that point at the tools and prefer them over raw `simctl` and |
| 73 | `adb`, which stay available for what the tools do not cover. |
| 74 | |
| 75 | ## The viewer decodes both vendored protocols |
| 76 | |
| 77 | The hub vendors two streaming servers with different wire formats. iOS video is |
| 78 | an HTTP body of AVCC envelopes decoded with WebCodecs, with input on a separate |
| 79 | binary WebSocket; Android multiplexes SEMU-framed H.264 and JSON gestures over |
| 80 | one WebSocket. [`deviceStream.ts`](../../apps/web/src/components/device/deviceStream.ts) |
| 81 | speaks both so one panel covers both platforms. |
| 82 | |
| 83 | Simulators encode H.264 High 5.1. Hardware decoders on some machines and all |
| 84 | headless browsers reject that profile, and WebCodecs is secure-context only, so |
| 85 | the viewer probes `isConfigSupported` and falls back to the MJPEG endpoint on |
| 86 | iOS. Android has no MJPEG; there the panel reports that it cannot decode. |