Skip to content

perf(workspace): batch deps-cache invalidation into one workspace fs scan #10445

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
davidfirst wants to merge 6 commits into
base: master
Choose a base branch
from
+89 −31
Open

perf(workspace): batch deps-cache invalidation into one workspace fs scan #10445

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
5503acb
perf(workspace): batch deps-cache invalidation into one workspace fs …
davidfirst Jun 23, 2026
db0e386
docs: link PR #10445 in component-loading redesign status
davidfirst Jun 23, 2026
db242dc
fix(workspace): harden components mtime index against build failure a…
davidfirst Jun 23, 2026
3b9a34f
fix(workspace): scan only the node_modules footprint the deps cache n…
davidfirst Jun 24, 2026
7e1857b
refactor(workspace): drop the shared mtime index, restrict the per-co…
davidfirst Jun 24, 2026
5bae110
Merge branch 'master' into component-loading-batch-deps-invalidation
davidfirst Jun 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 21 additions & 6 deletions scopes/toolbox/fs/last-modified/last-modified.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,16 +4,31 @@ import fs from 'fs-extra';
import { compact } from 'lodash';

/**
* check recursively all the sub-directories as well
* last-modified mtime of a component's directory structure (recursively), used as the dependency
* fs-cache freshness signal.
*
* the source tree is scanned recursively, but the scan stops at the `node_modules` boundary: the
* dependency auto-detect (whose cached result this guards) never traverses *into* a package, so the
* package internals and the transitive store are irrelevant — recursing `node_modules` followed the
* symlinked store and made this scan ~60x larger. Instead only the `node_modules` dir and its
* `@scope` dirs' mtimes are taken: a direct dep added / removed / version-relinked / componentId-
* changed all go through a relink that rewrites the symlink entry, bumping the containing dir.
*/
async function getLastModifiedDirTimestampMs(rootDir: string): Promise<number> {
const allDirs = await globby(rootDir, {
// source subdirectories, excluding the deep node_modules subtree.
const sourceDirs = await globby(rootDir, {
onlyDirectories: true,
// ignore: ['**/node_modules/**'], // need to think about it more. sometimes we do want to invalidate cache upon node_modules changes inside component dir
// stats: true // todo: consider retrieving the stats from here.
ignore: ['**/node_modules/**'],
});
allDirs.push(rootDir);
return getLastModifiedPathsTimestampMs(allDirs);
sourceDirs.push(rootDir);
// node_modules/@scope dirs (catch a scoped dep changing within an already-existing scope). a *bare*
// `node_modules` glob would recurse the whole symlinked tree, so its own dir mtime is taken via the
// direct stat in getLastModifiedPathsTimestampMs below, not globbed.
const scopeDirs = await globby(`${rootDir}/node_modules/@*`, {
onlyDirectories: true,
followSymbolicLinks: false,
});
Comment on lines +27 to +30

@qodo-free-for-open-source-projects qodo-free-for-open-source-projects Bot Jun 29, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

1. Symlinked scopes not tracked 🐞 Bug ≡ Correctness

getLastModifiedDirTimestampMs() uses globby() with { onlyDirectories: true, followSymbolicLinks:
false } for ${rootDir}/node_modules/@*, which can omit @scope entries that are symbolic links;
changes under those scopes won’t affect the freshness signal and can incorrectly reuse stale
deps-cache.
Agent Prompt 
### Issue description
`getLastModifiedDirTimestampMs()` builds `scopeDirs` via `globby(`${rootDir}/node_modules/@*`, { onlyDirectories: true, followSymbolicLinks: false })`. If an `@scope` folder in `node_modules` is represented as a symlink, it may be excluded from `scopeDirs`, so changes inside that scope (e.g., relinks under `@scope/`) won't bump the computed last-modified timestamp and the deps fs-cache can be treated as fresh incorrectly.

### Issue Context
Elsewhere in the repo, node_modules entries (including scoped ones) are explicitly treated as potentially being either directories or symlinks (and scoped entries are recursed into), so the scan should include symlinked `@scope` roots as well.

### Fix Focus Areas
- scopes/toolbox/fs/last-modified/last-modified.ts[17-32]

### Suggested fix
Replace the `globby(`${rootDir}/node_modules/@*`, ...)` call with a non-recursive directory listing of `${rootDir}/node_modules`:
- `readdir(node_modulesPath, { withFileTypes: true })`
- filter entries whose name starts with `@` and where `dirent.isDirectory()` **or** `dirent.isSymbolicLink()`
- add `${rootDir}/node_modules/<scopeName>` paths to `scopeDirs`

This avoids deep traversal while still capturing symlinked scope directories.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

return getLastModifiedPathsTimestampMs([...sourceDirs, ...scopeDirs, `${rootDir}/node_modules`]);
}

export async function getLastModifiedPathsTimestampMs(paths: string[]): Promise<number> {
Expand Down
93 changes: 68 additions & 25 deletions scopes/workspace/workspace/component-loading-redesign.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Component Loading Redesign

**Status:** Phase 1 shipped; Phase 2 in progress
**Last updated:** 2026-06-15 (code references are against `master` @ `59855b104`; line numbers will drift)
**Last updated:** 2026-06-23 (code references are against `master`; line numbers will drift)

This document is the source of truth for a multi-phase effort to simplify Bit's component-loading
mechanism: fewer caches, a staged (lazy) loading pipeline, a single env/aspect load planner, and a
Expand Down Expand Up @@ -204,11 +204,18 @@ earlier ones teach us).
### Phase 2 — Quick perf wins on existing seams

- [x] Benchmark harness committed + baseline recorded (see §4) — **gate for the rest of the phase**
- [ ] Lazy file contents in `ModelComponent.toConsumerComponent`
- [x] Restrict the deps-cache invalidation scan to the node*modules footprint the cache depends on:
the per-component freshness scan stops at the `node_modules` boundary (auto-detect never traverses
\_into* a package) and only takes each component's `node_modules`/`@scope` dir mtimes — a direct
dep added/removed/relinked bumps them. Cuts warm `bit status` fs syscalls ~37% (74.3k→46.4k);
warm-wall-neutral (I/O-wait), helps cold/CI (see §4.1). Kept per-component (no shared index — it
added complexity for zero syscall benefit and over-scanned single-component commands).
- [ ] `on-load` slot-laziness (`loadDocs: false, loadCompositions: false` for non-UI flows) — the
largest CPU-bound stage (9.2s) and the next **warm-wall** target
- [ ] Lazy file contents in `ModelComponent.toConsumerComponent` (helps `graph`, not `status`)
- [ ] `bit deps usage`: ids + stored deps instead of full load
- [ ] IDE metadata endpoint (`api-for-ide.ts`): S0-S2-level data only
- [ ] `bit remove` / forking: drop full-component loads where only ids/paths are used
- [ ] Default `loadDocs: false, loadCompositions: false` for non-UI flows

### Phase 3 — Cache consolidation

Expand Down Expand Up @@ -270,13 +277,13 @@ Numbers below are aggregate self-time across ~313 components at ~6× concurrency

**`bit status` (warm, ~13s wall):**

| stage | aggregate self-time | ~wall | note |
| ----------------------------------------------------------- | ------------------- | ----- | --------------------------------------------- |
| `legacy-load-deps` | 43s | ~7s | dependency-object materialization (see below) |
| `on-load` (slot handlers: docs, compositions, schema, pkg…) | 10s | ~1.7s | trimmable for non-UI flows |
| `dependency-resolution` (Harmony resolver) | 7.7s | ~1.3s | |
| `execute-load-slot` (own) | 5.8s | ~1s | |
| `consumer-fs-load` (file content reads) | negligible | — | not a `status` cost |
| stage | aggregate self-time | ~wall | note |
| ----------------------------------------------------------- | ------------------- | ----- | ----------------------------------------------------------- |
| `legacy-load-deps` | 43s | ~7s | filesystem traversal in deps-cache invalidation (see below) |
| `on-load` (slot handlers: docs, compositions, schema, pkg…) | 10s | ~1.7s | trimmable for non-UI flows |
| `dependency-resolution` (Harmony resolver) | 7.7s | ~1.3s | |
| `execute-load-slot` (own) | 5.8s | ~1s | |
| `consumer-fs-load` (file content reads) | negligible | — | not a `status` cost |

**`bit graph --json` (warm, ~20s wall; ~7.5s of it is loading):**

Expand All @@ -285,23 +292,38 @@ Numbers below are aggregate self-time across ~313 components at ~6× concurrency
| `consumer-fs-load` (file content reads) | 5.8s | dominant load cost for graph |
| `legacy-load-deps` | 1.2s | graph uses a lighter load path |

**Key conclusions (validated, not hypothesized):**
**Key conclusions (validated by measurement; these _supersede_ the earlier "object materialization"
hypothesis, which deeper sub-step instrumentation disproved):**

- **The dependency FS cache works.** On a warm `bit status`, dep loading is **635 cache hits, 0
misses/recomputes**. `legacy-load-deps` is _not_ re-resolving dependencies.
- **`status`'s dominant cost (~7s wall) is dependency-object _materialization on cache hit_** —
`DependenciesData.deserialize` + reconstructing full `Dependency`/`DependencyList` objects +
`applyOverrides`, for every component, even though `status` reads little of it. This is the
"all-or-nothing, always fully materialize" problem of §1.1 — **structural, not a cache bug.**
Reducing it needs the staged/lazy-loading work (defer dependency-object construction), **not a
Phase-2 quick fix.** Earlier framing of this as "39s" was aggregate-concurrent self-time, not
wall; wall is ~13s.
- **`graph`'s dominant load cost is file-content reads (`consumer-fs-load`, 5.8s)** → this is what
**lazy file contents** (§2.1) targets; validated as a real win for `graph`/scope-side loads, but
it does **not** help `status` (whose file reads are negligible).
- Implication for Phase-2 ordering: lazy file contents helps `graph`; per-command partial loads help
`deps usage`/IDE/`remove`/forking; `loadDocs/loadCompositions: false` trims `status`'s slot work
(~1.7s). The big `status` number is deferred to the staged-loading phase.
- **`legacy-load-deps` is filesystem I/O, not object materialization.** Sub-step timing of the warm
cache-hit path (313 components, aggregate self-time): `statFiles` **22.3s** + `cacheRead` **10.9s**,
versus `deserialize` **9ms**, `applyOverrides` 0.4s, `updateVersions` 0.1s. The cost is the
deps-cache _read + invalidation_ layer paid per component — **not** `DependenciesData.deserialize` /
`Dependency` reconstruction (negligible). `statFiles` was a recursive `globby` per component that
**followed the component's `node_modules` symlink into the shared workspace `node_modules`** (226k
of 230k scanned entries), run 313× per command.
- **Aggregate self-time ≠ wall — sharply.** Restricting the node*modules traversal to the footprint
the cache depends on (the dependency auto-detect never traverses \_into* a package, so only the
component's `node_modules`/`@scope` dir mtimes matter — a dep add/remove/relink bumps them; the deep
tree and transitive store are irrelevant) cut **fs syscalls 74.3k→46.4k per warm `bit status`
(~37%)** — yet a same-state wall A/B moved warm wall by **~0.3s**. The removed work is I/O-\_wait\* that overlaps with CPU on the single JS thread; on
a warm SSD it was never on the critical path (real win on cold/CI/networked filesystems, where it
is). `cacheRead` (10.9s) is likewise I/O-wait, so consolidating it would also be warm-wall-neutral.
(An earlier iteration _ignored_ node_modules entirely — faster still, but a correctness regression:
the cache resolves imports through node_modules, so its structure must invalidate the cache.)
- **The warm-wall bottleneck is CPU-bound, single-threaded JS** — the stages whose self-time ≈ their
wall contribution: **`on-load` slot handlers (9.2s)**, **`dependency-resolution` (7.2s)**,
**`workspace.get` (5s)**. These — not the deps-cache I/O — are what move warm `status` wall.
- **`graph`'s dominant load cost is file-content reads (`consumer-fs-load`, 5.8s)** → the target for
**lazy file contents** (§2.1); a real win for `graph`/scope-side loads, but it does **not** help
`status` (whose file reads are negligible).
- Implication for Phase-2 ordering: the deps-cache invalidation batch ships as a standalone fs/CPU
efficiency win (helps cold/CI, warm-wall-neutral). The next **warm-wall** target is `on-load`
slot-laziness (`loadDocs`/`loadCompositions: false` for non-UI flows) — the largest CPU-bound stage.
Earlier framing of the deps cost as "39s"/"materialization" was both an aggregate-vs-wall and a
cause misread; corrected here.

---

Expand All @@ -310,7 +332,7 @@ Numbers below are aggregate self-time across ~313 components at ~6× concurrency
| Phase | State | OpenSpec change | PRs |
| ----------------------- | ----------- | ------------------------------ | --------------------------------------------------- |
| 1 — Observability | done | `component-load-observability` | [#10418](https://github.com/teambit/bit/pull/10418) |
| 2 — Quick perf wins | in progress | — | — |
| 2 — Quick perf wins | in progress | — | [#10445](https://github.com/teambit/bit/pull/10445) |
| 3 — Cache consolidation | not started | — | — |
| 4 — Staged pipeline | not started | — | — |
| 5 — Env planner | not started | — | — |
Expand Down Expand Up @@ -340,3 +362,24 @@ Numbers below are aggregate self-time across ~313 components at ~6× concurrency
file-content reads (`consumer-fs-load`, 5.8s) → the target for lazy file contents. (Correction: an
earlier "39s" figure was aggregate-concurrent self-time, not wall; warm wall is ~13s.) Direction
for the next Phase-2 PR intentionally left open.
- 2026-06-23 — Deeper sub-step instrumentation **disproved the "object materialization" conclusion**
(see §4.1). The warm `legacy-load-deps` cost is filesystem I/O in the deps-cache invalidation:
`statFiles` 22.3s + `cacheRead` 10.9s aggregate, while `deserialize` is 9ms. `statFiles` was a
per-component recursive `globby` that followed each component's `node_modules` symlink into the
shared `node_modules` (226k/230k scanned entries), run 313× per command. Shipped the first Phase-2
perf change: restrict that per-component scan to the node*modules footprint the cache depends on —
it stops at the `node_modules` boundary and only takes the `node_modules`/`@scope` dir mtimes
(`getLastModifiedDirTimestampMs` in `@teambit/toolbox.fs.last-modified`). Result: **warm `bit status`
fs syscalls 74.3k→46.4k (~37%)**, `readFile` traffic unchanged (no regression, checked against the
bootstrap fs-read e2e metric). **Important correctness lesson:** a first cut simply \_ignored*
node*modules (74.3k→44.8k) — but the deps auto-detect resolves imports \_through* node_modules and
reads each direct dep's `package.json` (`name`/`componentId`), so the cache depends on node_modules
content; ignoring it is a regression. The tree builder stops at the package boundary
(`generate-tree-madge.ts` filter), so only the component's `node_modules`/`@scope` dir mtimes are
needed (a dep add/remove/relink bumps them) — not the deep tree. See [[deps-cache-node-modules-invalidation]].
(A shared command-scoped index was tried and dropped: it added memoization/invalidation complexity
for zero syscall benefit — batching saves no syscalls here — and over-scanned single-component commands.)
Key perf lesson reaffirmed: aggregate self-time ≠ wall — a same-state A/B moved warm wall only ~0.3s
because the cut work was I/O-wait overlapping CPU on the single JS thread (real win on cold/CI/
networked FS). The warm-wall bottleneck is CPU-bound: `on-load` (9.2s), `dependency-resolution`
(7.2s), `workspace.get` (5s) — next warm-wall target is `on-load` slot-laziness.