Skip to content

build: clean stale dist output and add worktree setup script - #922

Open
ugcodrr wants to merge 42 commits into
mainfrom
build/dev-setup-and-clean-dist
Open

ugcodrr wants to merge 42 commits into
mainfrom
build/dev-setup-and-clean-dist

Conversation

@ugcodrr

@ugcodrr ugcodrr commented Oct 6, 2026 •

Copy link
Copy Markdown
Member

Makes the repo safe and fast for many agent threads working in parallel git worktrees, and fixes stale build output that could crash the bots.

  • Stale dist fix: a per-package clean task (clean-dist.ts) deletes outputs whose source was deleted before every build and cache restore, without ever emptying dist during pnpm build:watch. Replaces --delete-dir-on-start.
  • Build cache fixes: corrected globalDependencies, cached the skin-renderer .node binary, made lint and the skin-renderer build react to root .oxlintrc.jsonc and Cargo.toml changes, and stopped installs adding a stray pkg/ entry to pnpm-lock.yaml.
  • One-command worktree setup: setup-worktree.ts links config and textures, checks out the asset submodules at the pinned commits from local repos, installs, restores blurred backgrounds from a shared cache, and builds in ~6s. Safe to run in many worktrees at once.

swc only writes new and changed files, so dist kept compiled files whose
sources were deleted or renamed. The command loader imports every
*.command.js under dist/commands, so an orphaned command either comes
back as a ghost command or crashes startup with an ESM SyntaxError.

- add --delete-dir-on-start to every swc build script
- purge all dist folders before build and build:watch; turbo cache hits
  restore outputs on top of an existing dist without running the build
  script, so orphans survived branch switches
pnpm.lock and tsconfig.json do not exist, so lockfile and base tsconfig
changes never invalidated the build cache.
The napi build writes skin-renderer.<platform>.node next to package.json,
which was not in the build outputs. A cache hit restored pkg/ but not
the binary, so the renderer failed to load at runtime.
wasm-pack writes packages/skin-renderer/pkg/package.json, which the
packages/** glob picks up as a workspace project. After a build, the next
pnpm install added an empty packages/skin-renderer/pkg importer to
pnpm-lock.yaml even with --frozen-lockfile.
Prepares the main clone or a linked git worktree for development with
pnpm scripts setup-worktree:

- links config.json/config.js and the ignored texture pack from the main
  checkout
- checks out asset submodules at the commit the branch pins, as git
  worktrees of the main checkout's submodule repos, so nothing is
  re-cloned
- installs dependencies and builds, reusing turbo's shared worktree cache
- restores blurred private backgrounds from a cache in the git common dir,
  keyed by the backgrounds tree and blur.mjs, instead of re-blurring
  (~30s) in every checkout
@vercel

vercel Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
statsify Ignored Ignored Preview Oct 10, 2026 11:39pm UTC

Purging every dist folder from the root build script meant a filtered
build like pnpm build --filter=@statsify/util emptied the dist of every
app it did not rebuild.

Add a per-package clean task that build depends on, so turbo cleans
exactly the packages it is about to build or restore from cache,
including under turbo watch.
- move existing submodule checkouts to the commit the branch pins, so
  re-running setup after a submodule bump updates the assets; checkouts
  on a branch or with local changes are left alone with a warning
- warn and continue when a pinned submodule commit cannot be fetched,
  matching the behaviour when a submodule cannot be cloned
Root config files outside a package are not task inputs by default, so
editing them neither invalidated cached results nor marked packages as
affected.

- lint, lint:ci: include .oxlintrc.jsonc. A PR that only changed lint
  rules ran 0 lint tasks with --affected in CI, so new violations could
  merge unchecked
- skin-renderer build: include the root Cargo.toml (workspace and
  release profile), so profile changes rebuild the native binary
- fmt: disable caching. The formatter rewrites the files it reads, and
  a cache hit after a .oxfmtrc.jsonc change skipped formatting
git rev-parse --show-toplevel prints forward slashes on Windows while
the common dir goes through path.resolve, so the main checkout never
matched its own root and was treated as a linked worktree.
- start threads in a worktree by default
- skip T3's own submodule init (worktreeSubmodules: none), which would
  clone both asset repos over the network in every worktree; setup-worktree
  checks them out from the main checkout's submodule repos instead
- run setup-worktree when a worktree is created, and make the agent wait
  for it so it never starts in a checkout without deps or builds; node is
  called directly because pnpm deps don't exist yet
- add a Verify script: build, typecheck, lint, test
dist was cleaned twice: each swc build ran --delete-dir-on-start, and the
turbo clean task deleted the whole folder before build. Keep the clean
task, since only it also runs before a cache restore, and drop
--delete-dir-on-start.

clean now deletes only files whose source in src no longer exists,
instead of the whole folder. A full wipe had no dependencies, so under
turbo watch every affected package's dist was emptied immediately and
stayed empty until its dependency chain rebuilt: apps/api/dist/index.js
was missing for ~8s after a one-line change in util. With pruning, no
dist entry file goes missing during watch rebuilds, and orphans are still
removed before both real builds and cache restores.
- blur cache: copy new entries to <key>.tmp-<pid> and rename them into
  place instead of deleting the whole cache folder and copying into it.
  A setup whose key is already stored drops its copy. After storing,
  evict only the entries listed before the rename, skipping .tmp-
  folders, so concurrent setups with different keys can't delete each
  other's entries and the newest entry always survives
- check a restore against the backgrounds folder and regenerate when it
  is incomplete, since an entry can be evicted while it is being copied
- retry a submodule fetch once and recheck the commit, since concurrent
  fetches of the shared submodule repo can fail to lock refs
- make worktree prune best-effort; git locks worktrees while adding
  them, so it can't remove one another setup is creating
- export the cache functions and only run setup when executed directly
- document the steps, concurrency, and node usage in the header
Temporary folders were always skipped during eviction, so one left
behind by a crashed setup stayed forever. Evict .tmp- folders whose
mtime is over an hour old. Copying keeps a live folder's mtime current,
so fresh ones are still skipped.
AGENTS.md covers repo conventions, worktree setup, how to verify changes,
what agents must not do in worktrees, branch and thread rules for T3, and
the main-checkout-only run steps. CLAUDE.md imports it so Claude Code
reads the same file.
Node 24 strips types natively, so the script still runs with plain node
before pnpm install and needs no build step. It only uses erasable
syntax. JSDoc types become annotations, and t3.json, the scripts
package and AGENTS.md point at the .ts file.

clean-dist stays JavaScript: it runs in every build, including deploys
on the self-hosted runner, whose node version isn't pinned.
The deploy server runs node 24.7.0, which strips types natively, so the
clean task can run the .ts file directly like setup-worktree. All 12
swc packages point their clean script at clean-dist.ts.
An entry that was stored incomplete could never be replaced: every
restore failed the completeness check, setup re-blurred, and storing
failed because the key already existed.

- delete an entry that restores incomplete, so it gets replaced
- don't cache blur output that is missing backgrounds. blur.mjs exits 0
  when a worker crashes, e.g. on an ignored .DS_Store in backgrounds,
  which left 304 of 380 backgrounds
- check completeness against the tracked backgrounds (git ls-tree), the
  same files the cache key covers, so ignored files don't count
When fetching assets/public failed, linking the texture pack still
created assets/public/minecraft-textures. On the next run git worktree
add refused the non-empty folder, and the uncaught error stopped setup
before pnpm install, on every rerun.

- only link the texture pack when assets/public is checked out
- catch a failed worktree add, log how to recover, and continue like
  the other submodule failures
linkFromMain created every link as a junction. Windows junctions only
point at directories, so config.js or config.json became an unusable
directory link, and because the call succeeded the copy fallback never
ran. Use a junction for directories and a file symlink for files; when
Windows refuses a file symlink without Developer Mode or admin rights,
the copy fallback runs.
git stores a submodule's repo under .git/modules/<name>, not its path.
They're the same for both asset submodules today, but after a move
(git mv keeps the name) setup would fall back to a full network clone,
or pick the wrong repo. Read names and paths from .gitmodules.
The .d check didn't depend on the extension but ran inside the
SOURCE_EXTENSIONS loop. Return before the loop instead. Same behavior.
The blur cache functions were exported, and setup guarded behind
import.meta.main, only so a throwaway race test could import them. No
test uses them, and importing still ran git at module load.
The key only covered the backgrounds and blur.mjs, but the output also
depends on the installed skia-canvas and stackblur-canvas, which decode,
blur, and encode the images. A bump that changed the output kept
restoring old backgrounds in every checkout. Add their installed
versions to the key, and skip the cache when they can't be read.
…te fails

Two steps could still stop setup before install and build:

- moving an existing submodule checkout to the pinned commit; a broken
  .git link failed every rerun at the same spot. Log how to recover and
  continue with the checkout as is
- writing the optional blur cache; a full disk or unwritable .git
  stopped setup after a successful blur. Log it and continue
The config loader tries config.json first but loads it with a plain
import(), which node 24 rejects for JSON (ERR_IMPORT_ATTRIBUTE_MISSING),
so a config.json breaks every app. Stop linking it into worktrees, and
make AGENTS.md say to use config.js.
AGENTS.md said not to run anything in apps/scripts except
setup-worktree, but every pnpm build runs apps/scripts/src/clean-dist.ts
through the clean task. Say not to run them directly instead.
…rees

When the main checkout had no repo for a submodule (e.g. no access to
the private assets), setup ran git submodule update --init inside the
worktree. That clones into the worktree's own git dir, so every worktree
cloned it again, and a private repo could hang on a credential prompt.
Skip it with a message saying how to set it up in the main checkout.
Setup read the pinned commit from HEAD, so rerunning it in the main
checkout during a staged but uncommitted submodule bump moved the
submodule back to the old commit. Read it from the index, like
git submodule update does. In a fresh worktree both are the same.
A failing pnpm blur (or reading the blur cache key) threw out of the
script before pnpm build, leaving the worktree without dist. The
backgrounds are only needed when an app renders, so log it and continue
to the build like the other optional steps.
git fetch origin only downloads what remote branches reach, so a commit
pinned from a deleted, squash-merged branch never arrived and the
worktree went without that submodule. When fetching branches doesn't
bring the commit in, fetch it by sha like git submodule update does.
Without -z, git quotes and escapes file names with special characters
(café.png came out as "caf\303\251.png"), so they never matched the
files blur.mjs writes. The completeness check would always fail and the
blur cache would never be used.
…se them

As globalDependencies they went into every task hash, so editing either
one also rebuilt the Rust skin-renderer (napi + wasm-pack) in every
worktree, although it uses neither. Make .swcrc an input of build and
test (vitest.shared.ts reads it), and tsconfig.base.json an input of
typecheck and lint. skin-renderer's own turbo.json replaces its build
inputs, so its cache no longer depends on them. pnpm-lock.yaml stays
global on purpose.
Setup runs unattended and T3 makes the agent wait for it, so a fetch
from the private assets repo on a machine without saved credentials
would wait forever for a username. Run fetches with
GIT_TERMINAL_PROMPT=0 so git fails right away and setup continues
without that submodule. Saved credentials and credential helpers still
work.
Import assertions (assert { type: "json" }) were removed in node 22, so
the script failed to parse on node 24 with SyntaxError: Unexpected
identifier 'assert'. Use the with { type: "json" } syntax instead.
setup-worktree.ts and clean-dist.ts were never typechecked: apps/scripts
had no typecheck script, so pnpm typecheck, CI, and Verify skipped them.

- add a typecheck script, so CI's typecheck --affected covers them
- enable erasableSyntaxOnly. node runs them by stripping types and
  throws ERR_UNSUPPORTED_TYPESCRIPT_SYNTAX on syntax like enums, which
  would break every pnpm build since clean-dist runs in it
- only include the .ts scripts. The .js scripts aren't typechecked, and
  including them pulled in workspace packages that use enums
- load node's types explicitly; they had only arrived through the .js
  scripts' imports
- find the main checkout from git worktree list, so it also works when
  its git dir isn't <main>/.git (e.g. --separate-git-dir)
- on Windows, only run pnpm through a shell. cmd.exe split git's path
  arguments that contain spaces
- also stop Git Credential Manager's GUI prompt during fetches
- read .gitmodules with --null, so submodule paths can contain spaces
- when restoring from the blur cache fails but the entry is complete
  (e.g. a full disk), keep the entry instead of evicting it from under
  other setups
swc compiles .mts to .mjs, .cts to .cjs, and .mjs to .js, and copies
.cjs as is with --copy-files. clean-dist assumed every .mjs and .cjs
output came from a .ts-like source, so it would have deleted those
outputs on every build. No package has such files yet.
- replace the cleanup-only opening line, so feature threads don't hold
  back; cleanup rules now apply when the task is cleanup
- move "Do not" to the top and "Running locally" to the end
- threads must commit before ending their turn, and never commit to
  main, push, open PRs, launch threads, or create tags (v* deploys)
- never print config.js, edit skin-renderer, rename schema fields
  (Mongo paths and leaderboard keys come from field names), or guess
  Hypixel API keys
- document the license header, discord-bot import aliases, i18n,
  Discord command limits, commit message style, and when to run fmt
- add packages/discord and base.hypixel-command.ts to the
  single-thread list
- verify with the --affected checks CI runs, flag unchecked rendering,
  and end each thread with a standard report
Match CI, which runs typecheck and lint with --affected. This covers
every package a branch changes plus its dependents, and only runs the
skin-renderer lint, which rewrites files, when Rust code changed.
Link AGENTS.md and CLAUDE.md from the main checkout when it has them,
the same way as config.js, so each developer can keep their own agent
instructions out of the repo and still get them in every worktree.
Agent instructions are personal, so they stay local (in
.git/info/exclude) and setup links them into worktrees.
T3 settings are personal, like the agent instruction files, so t3.json
stays local. setup-worktree still works as a T3 worktree-creation
script or when run by hand.
@ugcodrr
ugcodrr requested a review from jacobk999 October 11, 2026 00:14

This branch was previously deployed

1 inactive (outdated) deployment
Preview — dbc97896 Deployed Oct 10, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant