authorgravatar for git@paperclover.netclover caruso <git@paperclover.net> 2025-10-15 23:36:01-07:00
committergravatar for git@paperclover.netclover caruso <git@paperclover.net> 2026-04-07 23:19:57-07:00
logdabfbff74b36a6c68f9c37e8d6c5b9ca4e87ace2
treeec3a7e2a5265593516cf5811186685f4bc0ad0c2
parent0c49257d99c7511a2855a13f1ebaa92aabaa900b
signature Signed by SSH key SHA256:xbd+BjjhyBfwk7GVoURf9Yx0gzDerHbvYv7SddNWmAs

WIP: blogs

6 files changed, 548 insertions(+), 1 deletions(-)

lib/subprocess/ffmpeg.ts+1-1
......@@ -21,7 +21,7 @@ export interface SpawnOptions {
2121 *
2222 * ```ts
2323 * await ffmpeg.spawn({
24 * cmd: ["-i", "hello.mov", "-c:v", "libsvtav1", "hello.mp4"],
24 * args: ["-i", "hello.mov", "-c:v", "libsvtav1", "hello.mp4"],
2525 * progress: progress.start("encode hello.mov"),
2626 * });
2727 * ```
src/blog/backend.ts+4
......@@ -1,5 +1,9 @@
11export const app = new Hono();
22
3app.get("/blog/webdev/progress-and-api-design", async (c) => {
4 return assets.serveAsset(c, "/blog/webdev/progress-and-api-design/en", 200);
5});
6
37app.get("/blog/webdev/one-year-next-app-router", async (c) => {
48 return assets.serveAsset(c, "/blog/webdev/one-year-next-app-router/en", 200);
59});
src/blog/drafts/incrementality.mdx created+128
......@@ -0,0 +1,128 @@
1# Incremental Design is why Bun's Dev-Server is Fast
2
3> While I wrote from scratch and maintained on Bun's hot-reloading development
4> server until May 2025, I am not affiliated with Bun or Anthropic. I will be
5> discussing the revision of [`DevServer.zig`] with my latest commit.
6
7[`DevServer.zig`]: https://github.com/oven-sh/bun/blob/67f0c3e016aa479738469adac2b79a1862b88122/src/bake/DevServer.zig
8
9Time and time again, I've learned that with a long running, interactive
10process, the best way to improve it's speed is not hyper-optimizing a routine
11or throwing more threads at something. It's finding ways to reuse work that had
12already been performed. Sure, adding threads and reducing the time of a
13computation helps, but imagine cutting re-compilation time of a hundred
14thousand line codebase in half, versus making it always rebuild in under 16
15milliseconds.
16
17That was the goal with the design of Bun's `DevServer.zig`, the subsystem that
18powered bundling, serving, and rebuilding in the `bun ./index.html` command.
19And when it got highlighted in the 1.3 release, people noted. My favorite
20reaction is during Theo Browne's video on the release with his editor
21configured to save on every keypress. The changes arrived within a frame, two,
22or sometimes even the same frame the changes appear in the editor window.
23
24https://youtu.be/dSIgEJSi0rY?t=1664
25
26This post will go over general strategies and examples of applying incremental
27and resumable design to software, and then we'll also take a deep dive into how
28Bun exactly bundles its code.
29
30TOC
31
32## General Incremental Design
33
34## `sitegen`'s Incremental Builder
35
36## How Bun's DevServer Works
37
38one rule: every file must be compilable in isolation of every other file. with
39this achieved, a thread pool can build each file isolated its own thread, but
40more importantly it means that changing a file will guarantee that all
41unchanged files are used as is.
42
43the devserver was implemented mostly as a single file, one spanning 8.5k lines
44before i left. large files are honestly slept on; they are extremely easy to
45navigate with CMD+F (or `/` in Vim) (and a shame that the code got split up
46into many files for the sake of LLMs)
47
48### the output format
49
50The output format is tightly coupled with the design. Let's take a look at it:
51
52```ts
53((modules, config) => {
54 // *insert HMR runtime*
55})({
56 // Bundled Modules
57 "index.html": [ [ "app.ts", 0 ], [], [], () => {}, false],
58
59 "app.ts": [ [ "name.ts", 1, "name" ], [], [], (hmr) => {
60 var [import_name] = hmr.imports;
61 hmr.updateImport = [(module) => import_name = module];
62
63 console.info(`Hello, ${import_name.name}`);
64 }, false],
65
66 "name.ts": [ [], [ "name" ], [], (hmr) => {
67 hmr.exports = {
68 name: "clover"
69 };
70 }, false],
71}, {
72 main: "index.html",
73 bun: "1.3.0",
74 generation: "7e327d2c",
75 version: "f621f7ac5ecff08a",
76 console: false
77})
78//# sourceMappingURL=/_bun/client/000000007e327d2c.js.map
79```
80
81The foundation of this model is a callback to how Webpack bundles looked, at
82least a decade ago when I was picking apart webpack bundles trying to
83understand what these new magic tools were actually doing.
84
85- webpack inspiration
86- compact representation
87- concatenatable
88- esm is not incremental
89- async modules
90- missing imports
91- star imports
92
93### random fun facts
94
95
96```
97 /// The conversion logic is completely different for format .internal_bake_dev
98 /// For CommonJS, all statements are copied `inside_wrapper_suffix` and this returns.
99 ///
100 /// For ESM, this function populates all three lists:
101 /// 1. outside_wrapper_prefix: all import statements, unmodified.
102 /// 2. inside_wrapper_prefix: a var decl line and a call to `module.retrieve`
103 /// 3. inside_wrapper_suffix: all non-import statements
104 ///
105 /// The imports are rewritten at print time to fit the packed array format
106 /// that the HMR runtime can decode. This encoding is low on JS objects and
107 /// indentation.
108 ///
109 /// 1 ┃ "module/esm": [ [
110 /// ┃ 'module_1', 1, "add",
111 /// ┃ 'module_2', 2, "mul", "div",
112 /// ┃ 'module_3', 0, // bare or import star
113 /// ], [ "default" ], [], (hmr) => {
114 /// 2 ┃ var [module_1, module_2, module_3] = hmr.imports;
115 /// ┃ hmr.onUpdate = [
116 /// ┃ (module) => (module_1 = module),
117 /// ┃ (module) => (module_2 = module),
118 /// ┃ (module) => (module_3 = module),
119 /// ┃ ];
120 ///
121 /// 3 ┃ console.log("my module", module_1.add(1, module_2.mul(2, 3));
122 /// ┃ module.exports = {
123 /// ┃ default: module_3.something(module_2.div),
124 /// ┃ };
125 /// }, false ],
126 /// ----- "is the module async?"
127 fn convertStmtsForChunkForDevServer(
128 ```
src/blog/pages/webdev/progress-and-api-design/demos/sample-scan.ts created+68
......@@ -0,0 +1,68 @@
1import * as progress from "lib/progress.ts";
2// these other libraries will not be focused on that much
3import * as queue from "lib/queue.ts";
4import * as ffmpeg from "lib/subprocess/ffmpeg.ts";
5import * as fs from "node:fs/promises";
6import * as path from "node:path";
7
8export async function mediaScanner(
9 rootDir: string,
10 // accept `Ref` into any function you'd like to trace
11 progress: progress.Ref,
12) {
13 // Automatically called `rootNode.end()` with the `using` syntax.
14 using rootNode = progress.start(`Scan ${rootDir}`, { total: 1 });
15
16 // With a `progress.Node`, changing the status is done with setters
17 rootNode.text = "Scanning..."; // to change the status text
18 rootNode.value = 0; // to change the number of completed items
19 rootNode.total = 1; // to change the number of total items
20
21 const completion = Promise.withResolvers<void>();
22 // `queue.wrap` returns a function that limits concurrency to
23 // the number of cpu cores available (also usable as a priority queue)
24 const recurse = queue.wrap(async (file: string) => {
25 // Progress Nodes can have children with `.start()`
26 // Since it is not given a `total`, this won't have a bar.
27 using fileNode = rootNode.start(file);
28
29 await new Promise((resolve) => setTimeout(resolve, 100)); // demo
30
31 const stat = await fs.stat(file);
32 if (stat.isDirectory()) {
33 const children = await fs.readdir(file);
34
35 // Mutation schedules a re-render, batched reasonably.
36 rootNode.total += children.length;
37
38 for (const child of children) {
39 recurse(path.join(file, child));
40 }
41 return;
42 }
43
44 if (file.endsWith(".mov")) {
45 // in my library, there is also a wrapper for spawning `ffmpeg` with
46 // automatic progress tracking. we'll track this under the file node.
47 await ffmpeg.spawn({
48 args: ["-i", file, "-c:v", "libx264", file.replace(/\.mov/, ".mp4")],
49 progress: fileNode.start("encode h.264 mp4"),
50 });
51 // (the ffmpeg helper calls `.end()` for us.
52 }
53
54 rootNode.value += 1; // increment the progress by one
55
56 if (rootNode.value === rootNode.total) {
57 completion.resolve(); // all done!
58 }
59 });
60
61 recurse(rootDir);
62
63 await completion.promise;
64}
65
66export async function main() {
67 await mediaScanner("C:\\media", progress);
68}
src/blog/pages/webdev/progress-and-api-design/en.mdo created+346
......@@ -0,0 +1,346 @@
1---
2layout: "#src/blog/tags/blog-layout.marko"
3meta:
4 title: Clover's Progress API, Modular Abstractions, and "Good" API Design
5 description: yo we meow these
6 keywords: ["webdev", "software design"]
7 authors: ["clover caruso"]
8 embed:
9 thumbnail: /open-graph/next-js.png
10 canonical: /blog/webdev/progress-and-api-design
11date: "Mar 10th, 2026"
12---
13
14-- A small demo here
15
16Great Libraries begin as small helpers for a specialized use case, then
17extracted into their own projects because they are deemed useful. When a small
18component of one project becomes its own piece of software, it's critical to
19design the abstraction to be simple, understandable and modular.
20
21In this post, we'll take a look thru two of my recent library projects:
22[`@clo/lib/progress.ts`][progress] and [`@clo/react-mutation`]. Both of these
23libraries are written with great care to their API surface. They are also great
24examples since they are mostly written from scratch (Progress depends on
25`node:process`, React Mutation depends on React) and are easy to analyze.
26
27> While this post is going to be focused on web development with TypeScript,
28> the overall ideas translate to any language or tool. For example, a carefully
29> designed `interface` could be represented in C as a pointer table, or in Rust
30> with a `dyn` trait, obviously with proper consideration to the problem.
31
32<table-of-contents />
33
34## Why is a *Progress Bar* Library Exciting?
35
36Everything in Clover Progress is modular. For an introduction into how it's
37actually used, we'll start with instrumenting a little video encoding workflow.
38
39```tsx diff
40+import * as progress from "lib/progress.ts";
41
42 // these other libraries will not be focused on that much
43 import * as ffmpeg from "lib/subprocess/ffmpeg.ts";
44 import * as queue from "lib/queue.ts";
45 import * as fs from "node:fs/promises";
46 import * as path from "node:path";
47
48 export async function mediaScanner(
49 rootDir: string,
50+ // accept `Ref` into any function you'd like to trace
51+ progress: progress.Ref,
52 ) {
53+ // Automatically called `rootNode.end()` with the `using` syntax.
54+ using rootNode = progress.start(`Scan ${rootDir}`, { total: 1 });
55+
56+ // With a `progress.Node`, changing the status is done with setters
57+ rootNode.text = "Scanning..."; // to change the status text
58+ rootNode.value = 0; // to change the number of completed items
59+ rootNode.total = 1; // to change the number of total items
60
61- let active = 1;
62 const completion = Promise.withResolvers<void>();
63
64 // `queue.wrap` returns a function that limits concurrency to
65 // the number of cpu cores available (also usable as a priority queue)
66 const recurse = queue.wrap(async (file: string) => {
67+ // Progress Nodes can have children with `.start()`
68+ // Since it is not given a `total`, this won't have a bar.
69+ using fileNode = rootNode.start(file);
70
71 await new Promise((resolve) => setTimeout(resolve, 100)); // demo
72
73 const stat = await fs.stat(file);
74 if (stat.isDirectory()) {
75 const children = await fs.readdir(file);
76
77+ // Mutation schedules a re-render, batched reasonably.
78+ rootNode.total += children.length;
79- active += children.length;
80
81 for (const child of children) {
82 recurse(path.join(file, child));
83 }
84 return;
85 }
86
87 if (file.endsWith(".mov")) {
88+ // in my library, there is also a wrapper for spawning `ffmpeg` with
89+ // automatic progress tracking. we'll track this under the file node.
90 await ffmpeg.spawn({
91 args: ["-i", file, "-c:v", "libx264", file.replace(".mov", ".mp4"), "-y"],
92+ progress: fileNode.start("encode h.264 mp4"),
93 });
94+ // (the ffmpeg helper calls `.end()` for us.
95 }
96
97+ rootNode.value += 1; // increment the progress by one
98
99+ // You can use the value and total as regular variables.
100+ if (rootNode.value === rootNode.total) {
101- if (--active === 0) {
102 completion.resolve(); // all done!
103 }
104 });
105
106 recurse(rootDir);
107
108 await completion.promise;
109 }
110```
111
112To run it, an existing progress node could be passed, or the global `progress`
113module also satisfies this interface (`Ref` is simply just anything with the
114`start` function).
115
116```ts
117import * as progress from "@clo/lib/progress.ts";
118
119await mediaScanner("/Users/clo/media", progress);
120
121// or compose as a part of a larger program
122export function doTheMediaScanningPart(ref: progress.Ref) {
123 await mediaScanner("/Users/clo/media", ref);
124
125 using _ = ref.start("upload resulting files");
126 // ...
127}
128```
129
130The above example is a very, very abridged version of the [`file-scan.ts`]
131script that powers my website's [file viewer]. But with just this code, we've
132shown a real world use case made better with this simple progress tracing. Watch
133as the logs for each `ffmpeg` process are displayed under their respective node,
134and then when the process finishes each encoding's logs are grouped together.
135This has made it much easier for me to debug errors when they happen, especially
136on long encoding sessions when I was first writing the script.
137
138[`file-scan.ts`]: https://git.paperclover.net/clo/sitegen/src/branch/master/src/file-viewer/bin/file-scan.ts
139[file viewer]: /file
140
141<clover-video
142 file="/2026/progress blog post/01 - file viewer clone.mp4"
143/>
144
145I've still been exploring more uses of Clover Progress, but here are some more
146little demos.
147
148-- TODO: make all of these visual demos with asciinema / real demo
149
150- DB Migrations
151 - At my work, we used an [early version] of Clover Progress to visualize the
152 long-running migrations. Top level `console.log`s don't interfere with the
153 Terminal UI, making it easy to add this to an existing codebase. The
154 estimation algorithm was added upstream after we kept copy-pasting it
155 between migrations.
156- Developer CLIs
157 - The presentation of `value`/`total` can be customized, for example
158 `units: "bytes"`. Progress trees are just amazing for visualizing
159 parallel or multi-threaded work.
160- HTTP Server
161 - In development, separate trees can be used to show different requests, or
162 also identify slow routes by visually spotting them in the log. This example
163 can be tested with the simple HTTP server provided by `@clo/lib/http`.
164- Server${'<->'}Browser IPC
165 - A headless progress instance can be created with `new progress.Root`, and
166 that root can be serialized into a `ReadableStream` to be decoded and
167 rendered in a browser.
168- AI Sub-agents
169 - Thinking traces and complex tool calls from concurrent agents are hard to
170 visualize. This demo isn't really concrete yet, but I'm looking to
171 optionally integrate Clover Progress into a friend's [AI SDK project].
172
173[early version]: https://jsr.io/@clo/console
174
175## The Most Important Aspect of Library Design
176
177Before I dive into concepts , I want to share the most important thing about
178library design: **you MUST drive library decisions from
179real-world testing**, otherwise you have no idea what actually works or not. An
180idea may seem great on paper, but with its hidden flaws only revealled after
181it's done.
182
183As one creates more and more libraries, this becomes less of a concern. I'm
184able to somewhat correctly predict how an API will feel to use (insert joke
185aboout Next.js 16), so I can get pretty far without testing. But even then, the
186feedback provided from testing will always shine light on the gaps, especially
187when *other* people test it.
188
189With my library demo out of the way, let's take a look at some fun patterns I've
190found:
191
192## Trivial Interfaces
193
194`progress` mostly revolves around one interface, `progress.Ref`, a reference
195point for reporting progress. Functions that can report live progress take it as
196a parameter.
197
198```ts
199import * as progress from "@clo/lib/progress.ts";
200import { delay } from "@clo/lib/async.ts";
201
202// (imported as progress.Ref)
203interface Ref {
204 /**
205 * creates a new trackable unit of work as a child of this one.
206 * when given an estimate, a progress bar is rendered.
207 */
208 start(text: string, opts?: StartOptions): Node;
209}
210
211async function doInterestingWork(p: progress.Ref) {
212 const node = p.start("doing some interesting work");
213
214 const subtask1 = node.start("subtask", { total: 10 });
215 const subtask2 = node.start("subtask", { total: 30 });
216
217 for (let i = 0; i < 30; i += 1) {
218 subtask1.value += 1;
219 if (i % 3 === 0) subtask2.value += 1;
220 await delay(100);
221 }
222
223 subtask1.end();
224 subtask2.end();
225
226 node.end();
227}
228```
229
230> To make `progress.Node` easier to use, it aliases `end` to
231> `[Symbol.dispose]`, which can be used with the recently stabilized
232> [`using` syntax][using]. I'll be doing that for the rest of this post.
233
234By taking in the `Ref` parameter (that only declares `start` instead of a full
235`Node`), it makes this function modular to however the caller wants to report
236progress. There are four primary ways to create a valid Ref, and they all flow
237extremely naturally when instrumenting code.
238
239- `progress.Node` implements `start` to create sub-nodes. I could pass
240 `subtask1` to another function to report its progress within the sub-task.
241- The progress module exports `start`, which means the module namespace import
242 satisfies the interface, eg `doInterestingWork(progress)`. In Node.js, tree
243 shaking isn't worth worrying about, but in the browser you could also pass
244 `progress.global` or `{ start: progress.start }` if you really wanted to be sure.
245- A headless progress tree created via `new progress.Root()`, which is explained
246 in the next section.
247- `progress.nullNode` returns a no-op `Node` where the `start` function returns
248 itself, all setters are no-ops. Part of the design of `Node` is that most of the
249 fields are optional, meaning a no-op `Node` implementation can just ignore the
250 existence of all of most of its fields.
251
252## Headless Design
253
254A pattern I love for many reasons is the *headless design* pattern, where
255something connected to global state is built up with. For Clover Progress, it
256is possible to construct a `Root` node that does not render to a screen.
257Instead, it takes in host APIs and acts as an event emitter.
258
259```ts
260const root = new progress.Root({
261 delay, // optionally pass a different timer function
262 now, // optionally pass a different "now" function
263});
264root.on("change", (activeItems: progress.ReadOnlyNode[]) => {
265 console.log(activeItems); // update the screen
266});
267
268// root has a start function, so this works with no edits
269// to the actual codebase.
270mediaScanner("C:\\media", root);
271```
272
273There's actually a second layer to this. The code used to implement TUI
274"widgets", the items in the terminal that persist after console logs, is a more
275advanced version of this. In addition to taking timing APIs, it also takes a
276handle to the terminal output. By providing a system environment, a few
277functions are returned. The most important, `startWidget`, is what the Clover
278Progress TUI is made up on. With this abstraction boundary, the progress code
279does not need to worry about redrawing the screen, but instead just formatting
280the ANSI output.
281
282```ts
283// @clo/lib/log.ts
284export function headlessWidgetHost(env: HeadlessWidgetEnv): HeadlessWidgetHost;
285
286/** {@linkcode widgetHost}'s input takes terminal i/o as well as timing APIs */
287export interface HeadlessWidgetEnv {
288 /** recieves ANSI escape sequences for interactive data */
289 writeInteractive(text: string): void;
290 /** recieves log content (from `writeLine`) */
291 writeOutput(text: string): void;
292 /** monotonic milliseconds */
293 now(): ReturnType<typeof performance.now>;
294 /** after resolving, `now()` should have increased by the delay time */
295 delay: typeof async.delay;
296 /** called often. */
297 getSize(): { columns: number; rows: number };
298}
299
300/** an implementation of an ANSI-based widget host */
301export interface HeadlessWidgetHost {
302 /** see the top-level {@linkcode writeLine} function */
303 writeLine(text: string): void;
304 /** see the top-level {@linkcode getDrawLock} function */
305 getDrawLock(): ts.Dispose;
306 /** see the top-level {@linkcode startWidget} function */
307 startWidget(widget: Widget): ts.Dispose;
308 /** stop all widgets and remove all timers. */
309 cancel(): void;
310 /** generic delay function */
311 delay?: typeof async.delay;
312 /** generic now function */
313 now?: typeof performance.now;
314}
315```
316
317What I love about this is it means my code doesn't have any dependencies,
318except for an *optional* dependency on `node:process` if you use the global
319progress node.
320
321### Testing
322
323-- theyre just beautiful holy shit
324
325### Alternate Platforms
326
327-- talk about browser progress more
328
329### Serialization
330
331-- talk about encodeByteStream
332-- this use case is still being proven
333
334## Good Abstractions Should be Hard to Design
335
336(or, why i don't want to release a billion libraries)
337
338### Overcooked Design
339
340-- It's very easy to get carried away, especially in javascript
341-- Avoid Unreadable Generics
342-- Please do not change the language (cough cough Next)
343
344### Okay so I might have actually overcooked it
345
346...
src/blog/tags/table-of-contents.css+1
......@@ -2,6 +2,7 @@
22 background-color: #0003;
33 border-radius: 8px;
44 padding: 1rem;
5 margin: 1rem 0;
56
67 h2 {
78 text-decoration: none;