#96WIP: feat(lib): Incremental Build Tool API

~clover commented · 6 months ago
Author
Owner
resolves #27 resolves #88 blocked by #90 because i don't want to configure a JSX plugin

been playing with node:vm's experimental modules. it is neat because it gives us a hook into the module system that no other runtime has:

able to reflect what files are pointed to by each file

can adjust how files are processed before loading

can unload/reload modules, with old ones being garbage collected

this commit rebuilds hot.ts and incremental.ts as a new system that only depends on these node builtins. here is a big todo list for progress

[x] module loading incremental.graph.import("src/main.ts");

[x] node.js ESM resolution

[x] plugin system to load extensions like .tsx, .marko, .svelte

[x] commonjs (stub that is not actually incremental, just don't use cjs please)

[ ] tracing the files used in resolution and initialization

[x] module resolution that traces what files it read

[ ] linking modules to each other

[ ] tracing a list of css files from an entrypoint

[ ] allowing arbitrary jobs that cache when re-run

[x] buffer-focused data serialization

[ ] arbitrary dependencies on other Work, other files, and other directory contents

[ ] recomputing work when it should be

[ ] creating a filesystem watcher to detect changes

[ ] reloading parts of the graph

[ ] optional bait: support import.meta.hot since it's so easy

[ ] serializing internal state to disk

[ ] save and revive work entries

[ ] save and revive graph state

[ ] save and revive transpiled files

[ ] save and revive the v8 cache

by implementing @clo/lib/incremental, clover sitegen can:

stop using and patching node.js internals. everything is now fair-game APIs

drop the esbuild powered hot.ts, using real ESM whenever possible

provide a base for more features (issue 4 and 5), which could possibly be solved as their own top-level repos.

but most importantly, it makes the incremental api available for anyone who wants to use it, intead of it being a very hyper specific component to clover sitegen.

this pr also drops msgpackr. since lib cannot have dependencies, data serialization is unified into the core format instead of being a stupid layer on top; the incremental state is a rich sqlite database. this should help with large incremental graphs that contain many megabytes of files.

Imported pull request: incremental → master.
~clover changed the status to Todo · 6 months ago
~clover commented · 6 months ago
Author
Owner
Forgejo: pull request updated.

{"is_force_push":false,"commit_ids":["2477fbe7175925af816af5ec6ffdf0ce49b29904"]}
~clover added label site · 6 months ago
~clover added label lib · 6 months ago
~clover removed label site · 6 months ago

neat idea:

// run.js would use something like this
incremental.reviveAndExecute(import.meta.resolve("./framework/generate.ts"));

incremental.graph.import("./") // untraced?

incremental.graph.transaction() // similar to incr.compile? this is where ref comes in?

// inside of generate.ts
const result = await inc.cachedWork([key], async (io) => {
  // do a task. rules are to not pollute globals
  io.import("...");
});

low key think this should be named something else top level

~clover commented · 6 months ago
Author
Owner
Forgejo: pull request updated.

{"is_force_push":true,"commit_ids":["2477fbe7175925af816af5ec6ffdf0ce49b29904","1e02d3c568cb77b9866f405bf7d4441acb47a4e1"]}
~clover commented · 6 months ago
Author
Owner
Forgejo: title changed.

old_title: WIP: feat(lib): Incremental Module Loader API

new_title: WIP: feat(lib): Incremental Build Tool API
~clover commented · 6 months ago
Author
Owner
Forgejo: pull request updated.

{"is_force_push":true,"commit_ids":["1e02d3c568cb77b9866f405bf7d4441acb47a4e1","26c98740bb3b766cbc3fd88800c499b950d0347a"]}
~clover commented · 4 months ago
Author
Owner
Forgejo: pull request updated.

{"is_force_push":true,"commit_ids":["26c98740bb3b766cbc3fd88800c499b950d0347a","46ce321c04db60bcbd364af1b6d4d3f7e9e064a3"]}
~clover commented · 4 months ago
Author
Owner
Forgejo: pull request updated.

{"is_force_push":true,"commit_ids":["46ce321c04db60bcbd364af1b6d4d3f7e9e064a3","6a4e985a960dc0c1fcf7776082292b717d25a6d8"]}