clover's git

the-knife

1	# Orchestration V2
2
3	This document set describes the target architecture for the next orchestration model. It is not a patch plan for the current implementation and it intentionally ignores migration/backward compatibility. Those concerns should be handled after the target model is coherent.
4
5	V2 is an orchestrator rewrite, not a rewrite of the whole app domain platform. Existing non-orchestration domains, persistence/migration infrastructure, websocket/RPC infrastructure, and projection streaming semantics should be retained unless V2 exposes a concrete orchestration-specific gap.
6
7	V2 is designed around the real provider behavior observed in the Codex app-server probes, but it is not Codex-specific. Codex is treated as the richest protocol we currently have; weaker providers are adapted into the same model with app-owned ids and explicit capability flags.
8
9	## Documents
10
11	- [Core Graph And Data Model](./core-graph-and-data-model.md)
12	- [Entity IDs And Correlation](./entity-ids-and-correlation.md)
13	- [Feature Lifecycles](./feature-lifecycles.md)
14	- [Thread Lineage And Context Transfer](./thread-lineage-and-context-transfer.md)
15	- [Provider Switching And Context Handoff](./provider-switching-and-context.md)
16	- [Orchestrator MCP Server](./orchestrator-mcp-server.md)
17	- [Provider Capability System](./provider-capability-system.md)
18	- [Testing Strategy](./testing-strategy.md)
19
20	## Primary Goals
21
22	- Preserve provider-native lifecycle fidelity without leaking provider ids into app identity.
23	- Model root turns, subagents, tools, approvals, plans, and checkpoints as one execution graph.
24	- Make root-run completion the only event that completes a user-visible turn.
25	- Support forking from normal threads and completed subagent/provider threads.
26	- Support changing providers between runs as a first-class context handoff.
27	- Model forks, provider handoffs, merge-back, and subagents through shared thread lineage and context transfer primitives.
28	- Support providers with weak or missing ids through deterministic app-owned id allocation.
29	- Make feature behavior capability-driven, not provider-name-driven.
30
31	## Key Invariants
32
33	1. App ids are primary. Provider ids are refs.
34	2. Provider events are never rewritten to look like another provider event.
35	3. Child execution completion never closes the parent run.
36	4. Checkpoints attach to checkpointable execution scopes. Root-run checkpoints advance app run history; child/subagent checkpoints are nested and do not advance the parent run count.
37	5. Rollback is expressed in app run count and reconciled with provider conversation state.
38	6. Every command targets app ids; adapters translate to provider refs at the edge.
39	7. Missing provider capability is represented explicitly and handled by policy.
40	8. Provider switches create explicit context handoff artifacts; they are not hidden prompt hacks.
41	9. Forks record app-level lineage first. Provider-native forks and portable context handoffs are lazy resolution strategies chosen when a run starts.
42	10. Tests should prefer replay-backed integration coverage over mocked unit tests. The only normal substitute in orchestration tests is the provider runtime transport.
43
44	## Conceptual Layers
45
46	```text
47	Native provider protocol
48	-> Rotating raw provider diagnostics
49	-> Provider adapter / normalizer
50	-> V2 orchestration event store using existing persistence patterns
51	-> Runtime execution graph
52	-> Conversation projection
53	-> UI / API views
54	```
55
56	Raw provider diagnostics store what the provider actually sent or received for debugging and replay capture. The durable app state is the normalized orchestration state: events/entities, correlation refs, projections, and command receipts. The runtime graph stores what provider behavior means. The conversation projection stores what users see.
57
58	V2 should integrate with the existing orchestration command/event/projection infrastructure patterns rather than creating an unrelated event system. The V2-specific work is the graph model, provider lifecycle semantics, adapter contracts, normalizers, policies, and projections.
59
60	V2 owns its event schema. The V1 command and event unions are gone; V1 rows survive only as input to the legacy importer (`apps/server/src/orchestration-v2/legacy/`), and project events keep their own application event shape.
61
62	## Minimal Mental Model
63
64	```text
65	AppThread
66	Run 1
67	root ExecutionNode
68	tool ExecutionNode
69	approval ExecutionNode
70	subagent ExecutionNode
71	ProviderThread
72	child root ExecutionNode
73	Run 2
74	root ExecutionNode
75	```
76
77	An app thread is the user-visible conversation. A run is the counted user-visible turn. Execution nodes are the tree of work inside the run. Provider threads are provider-native conversation handles that can be attached to app threads or nested execution nodes.
78
79	Provider switches do not create new app threads. They create or reactivate provider threads and attach context handoff summaries to the next run.
80
81	Forks do create new app threads, but they should not force provider selection at fork time. Forking records thread lineage and a pending source point. The first run on the fork resolves that source point through native provider fork when possible or portable context transfer when needed.
82
83	## Probe-Derived Requirements
84
85	The Codex app-server probes showed several protocol realities that the V2 model must preserve:
86
87	- `thread/status/changed` can become idle before or around completion, but `turn/completed` is the authoritative turn terminal event.
88	- `turn/interrupt` completes as a request first; the interrupted terminal state arrives later through `turn/completed`.
89	- Approval requests are provider-initiated JSON-RPC requests scoped to provider thread, turn, and item.
90	- `thread/rollback` returns an authoritative provider thread snapshot after rollback.
91	- Subagent child `turn/completed` events can occur before the parent/root turn completes.
92	- Child provider turns are real provider turns and must not be remapped onto the parent provider turn id.
93
94	These observations are why V2 separates app runs, provider turns, and execution nodes.
95
96	## Existing Platform Boundaries
97
98	V2 should reuse existing app infrastructure where that infrastructure is not the source of the orchestration bug class:
99
100	- Command dispatch should keep the existing serialized/idempotent command handling and command receipt pattern.
101	- Projection streams should keep the existing snapshot-plus-cursor semantics used by the current API.
102	- Persistence should keep the existing SQLite/migration/repository infrastructure and projection-stream semantics.
103	- Raw provider frames should continue to be diagnostic log data, with bounded retention through rotating logs.
104	- Replay tests should replace only the provider transport/process boundary.
105
106	V2 may add V2-native orchestration events, projection tables, command policies, and provider execution services. It should not add a durable raw-provider-event database or a separate generic event bus unless the existing infrastructure cannot satisfy a documented V2 requirement.
107
108	V2 command dispatch returns the last committed stored-event sequence for accepted commands. Duplicate command ids must return the same receipt-backed result without re-running provider side effects. This is the API-level boundary the frontend uses for reconnect/recovery cursors.
109
110	## Tracked Follow-Up: Durable Effect Outbox
111
112	The current app pipeline uses domain events plus reactors to trigger provider side effects:
113
114	```text
115	command
116	-> domain event(s)
117	-> projection
118	-> reactor observes event
119	-> provider side effect
120	-> provider output ingestion
121	-> domain event(s)
122	```
123
124	This is valid, but it can become hard to trace because the provider side effect is implicit in a live subscription. If V2 needs stronger restart/recovery/debuggability, introduce a durable effect outbox for orchestration side effects:
125
126	```text
127	command
128	-> transaction:
129	append domain event(s)
130	append provider effect request(s)
131	-> projection
132	-> stream to UI
133
134	effect worker
135	-> claim effect request
136	-> call provider
137	-> ingest provider output
138	-> append domain event(s)
139	-> mark effect request completed/failed
140	```
141
142	Example effect request:
143
144	```ts
145	type ProviderEffectRequest = {
146	id: ProviderEffectRequestId;
147	kind:
148	| "provider.turn.start"
149	| "provider.turn.interrupt"
150	| "provider.runtime-request.respond"
151	| "provider.thread.rollback"
152	| "provider.thread.fork";
153	status: "pending" | "running" | "completed" | "failed" | "cancelled";
154	threadId: ThreadId;
155	runId: RunId | null;
156	nodeId: NodeId | null;
157	provider: ProviderKind;
158	payload: unknown;
159	causationEventId: EventId;
160	commandId: CommandId | null;
161	attempts: number;
162	lastError: string | null;
163	};
164	```
165
166	This should be treated as an infrastructure improvement, not a prerequisite for the first V2 slice. The first V2 implementation can use the existing reactor pattern, but it should keep the provider side-effect boundary clear enough that moving to an outbox later does not require redesigning the graph or provider adapters.