[lukesandberg]

Draft / prototype, stacked on #98760 (andrewimm/minify-before-chunk). Design doc: approved in the session that produced this PR (summary below).

Summary

Chunk outputs (.js and .js.map) are now composed from their chunk items' cells instead of being persisted as flat bytes.

• turbo-tasks-fs: CellRope, a flat list of inline segments (scaffolding), ResolvedVc<Rope> cell references and nested ResolvedVc<CellRope> references. read() concatenates them without copying. The encoding stores references, not the referenced bytes.
• File::from_cell_rope: a File can carry the CellRope it was read from, in a field ignored by equality, ordering and hashing. Readers see normal bytes. DiskFileSystem::write persists only the CellRope for such files, instead of PersistedFileContent.
  • At capture time, the write effect re-reads the rope (untracked) and checks it against the content hash computed when the write was declared. If a referenced cell changed or can't be read, nothing is captured, and applying takes the existing ApplyError::Retry path, which re-runs the producer. Stale or mixed bytes are never written.
  • File's equality ignores the composition, so a cell can keep an older, equal File whose rope no longer reads its bytes (turbo-tasks keeps the old value when the new one is equal). write therefore only uses the rope if it reads exactly the file's bytes, and otherwise writes flat. Without this guard such a write fails with "gave up applying effects after 4 retries" (covered by a test).
  • The hash is the same one a flat write of the same bytes produces, so moving a path between the two doesn't rewrite unchanged files.
• turbopack-core: CodeCells replaces PersistedCode. The code and each section's source map are their own persisted Rope cells.
• Structure-only composition (second commit): each chunk has a composition task that builds a CodeComposition (the body CellRope plus references to the items' CodeCells) without reading item bytes. A separate reader task (composed_code) builds the flat Code, its hash and the source-map offsets. The output file's rope nests the body (prefix + Nested(body) + suffix) instead of copying its segment list. See below.
• turbopack-ecmascript: module factories are stored as CodeCells. With per-item minification (turbopack: experimental minify-before-chunking feature #98760), each minified factory is its own persisted task (minified_module_factory) instead of an unpersisted intermediate. CodeModuleIdAndPath now holds only references (ResolvedVc<CodeCells> plus the code cell), so it stays equal when only an item's bytes change.
• Browser and Node.js chunk content compose their code and maps this way. Whole-chunk minification is unchanged: it produces new flat code, so it keeps the flat path.

Why the File field instead of a new AssetContent variant: 21 files match AssetContent::File, many with let … else (Next's dev versioned_content_map, next-core emit.rs, version hashing, the snapshot harness). They would have silently treated a composed file as "not a file". With the field, no reader changes.

Update: keep compositions stable across item edits (4e965124f2)

Measuring warm writes on the first commit showed that a dev item edit still rewrote ~5.7 MiB lz4 of composition-task data. The backend persists a task's data (all cells, cell hashes and dependency sets) as one blob, and composed_code held the body rope plus content-derived values (the flat Code, its hash, map cells keyed by chunk byte offset). It re-ran on every item edit. content also stored a second copy of the body's segments.

This commit:

• splits each chunk's composed_code into a structure-only composition task and a reader. The composition only depends on item order, ids, strictness and cell references, so a payload-only edit (even one that changes the item's length) doesn't re-execute it and its body isn't rewritten;
• computes byte offsets (map sections) in the tasks that read the bytes (CodeComposition::build_code / generate_source_map);
• adds CellRopeSegment::Nested so the file rope references the body instead of copying it.

Dev leaf edit (shared module edited, single-snapshot runs, lz4 task data from print_cache_item_size_with_compressed; on-disk = polled snapshot SST bytes):

	base (#98760)	first commit	this commit (3 clean runs)
task data total	44.34 MiB	36.44 MiB	33.04–33.83 MiB
chunk-output tasks	15.54	8.14	5.03
snapshot bytes on disk	38.07 MB	34.29 MB	31.94–32.69 MB

• The composition tasks write no data entries in the edit step (meta-only, ~0.45 MiB).
• content: 1.91 → 0.21 MiB (references plus the comment tail). composed_code: 3.07 → 0.61 MiB.
• Source-map tasks grew 0.76 → 1.37 MiB: they now depend on every item's cells to compute offsets. The map file rope (section headers contain line offsets) is out of scope here.
• Prod is unchanged (±0.04 MiB); composition tasks are meta-only there too.
• Some runs show large whole-graph re-persist events in no-edit/edit steps (also on base); they're excluded from the comparison above. GC logs show tens of thousands of per-request endpoint/version/write tasks collected in warm dev sessions on both binaries, a plausible contributor that isn't attributed precisely yet.

Warning

Unresolved: in one warm next dev restart with this commit's binary, the process aborted with a turbo-tasks panic: task 1128469 (task, MustExist): task is missing in memory or persistent storage (no backtrace). It hasn't recurred in ~16 later attempts with RUST_BACKTRACE=full and GC-collection logging (including 500 ms snapshot intervals to force GC during requests). The first commit had 0 panics in ~6.5 comparable control runs. These counts are too small to say whether this commit makes it more likely. In the control, GC sometimes collected whole chunk subtrees including per-item factory tasks; with this commit it never did. The cause isn't established. A canary GC/restore race is plausible but unproven.

New tests in this commit:

• code_builder::tests::piece_edits_dont_recompose: a length-changing item edit updates bytes and source map (equal to CodeBuilder) without re-executing the composition; the body is unchanged.
• turbopack-core/tests/composition_persistence.rs: GC enabled, persisted, three sessions; direct-body and nested-with-tail files through DiskFileSystem::write; content edit; equal-bytes swap of an item to another task (GC collects the orphan); restore and rewrite of deleted files.
• CellRope: nested reads (tracked and untracked) equal the concatenation, encode as references only, follow inner cell changes; errors surface through nested ropes.

Strict directives (2c38a90a7d)

#98760 removes a strict factory's "use strict" when it places the factory inside a strict wrapper (the mixed IIFE or the all-strict wrapper). Copying the factory's bytes without the directive would defeat referencing its cell, so the removal is deferred to read time instead:

• CellRopeBuilder::push_cell_skipping(cell, skip) references a cell plus an OptionByteRange cell. When the rope is read, the range it holds is left out with turbopack: experimental minify-before-chunking feature #98760's O(1) Rope::skip; None leaves the bytes unchanged, and a range outside the cell fails the read. CellRope stays generic: it doesn't know about directives.
• CodeCells gets a strict_directive_range cell, created by the task that owns the code cell. CodeCompositionBuilder::push_code_cells_without_strict_directive references both, so the composition stays structure-only: it depends on strictness and layout, not on where the directive is. build_code and generate_source_map apply the same removal, so bytes and maps equal the CodeBuilder path.
• Test: skipped_directives_follow_piece_edits_without_recomposing edits the code before a directive (moving it). The output updates, the composition doesn't re-execute, its body stays equal, and every section map is still referenced as a cell.

Failure semantics

• Errors propagate through task outputs, as before. An item error fails composition, then the chunk's content(), then the write. The previous file is left untouched. (In practice, chunk-item codegen errors are already turned into a throwing factory plus an issue.)
• A cell whose task later fails keeps its last successful value (turbo-tasks behavior, pinned by a test). A raw reference therefore need not report its producer's error. Different cells can also change independently, so the rope is not a snapshot. Chunk composition retains task-output dependencies for error propagation; capture checks the complete content hash before writing.

Verification (current tip)

• turbopack snapshot tests: 136/136 with no snapshot changes, so the output is byte-identical to turbopack: experimental minify-before-chunking feature #98760; execution tests: 323/323.
• unit tests across turbo-tasks-fs, turbopack-core (incl. composition_persistence), turbopack-ecmascript, turbopack-browser and turbopack-nodejs: 982 passed.
• Earlier, before the restack: byte-identical bench/nested-deps-app-router-many-pages output vs turbopack: experimental minify-before-chunking feature #98760 for next build (per-item minification) and next dev, cold and warm.
• cargo fmt; clippy -D warnings (--all-targets --no-deps) on the touched crates.

Measurements (first commit)

bench/nested-deps-app-router-many-pages (100 pages), local debug build of next-swc, TURBO_ENGINE_VERSION pinned so the filesystem cache is enabled. Single run per configuration. "Snapshot bytes" are the cache bytes written by snapshot commits in the session. Compaction output is reported separately, because it depends heavily on timing.

Dev (next dev, all pages fetched; then sessions without edits, editing a shared leaf module, and editing one page)

	base (#98760)	first commit	delta
cold cache size	326.2 MB	286.8 MB	−39.5 MB (−12.1%)
shared leaf edit, snapshot bytes	45.5 MB	41.8 MB	−3.7 MB (−8.1%)
shared leaf edit, compaction bytes	0	0	0
page-local edit, snapshot bytes	28.9 MB	29.0 MB	≈ 0
page-local edit, compaction bytes	400.3 MB	71.5 MB	timing-dependent; no savings claim
warm session without edits (2nd)	0	0	0

Prod (next build, turbopackMinifyBeforeChunking on)

	base (#98760)	first commit	delta
cold cache size	170.2 MB	171.5 MB	+1.2 MB (+0.7%)
rebuild without edits, bytes written	17.6 MB	17.6 MB	+0.1%
shared leaf edit, bytes written	22.8 MB	22.8 MB	−0.3%
page edit, bytes written	17.1 MB	17.1 MB	≈ 0

These are polled SST sizes, not block-device I/O: transient files may be missed and metadata/LOG writes aren't included.

• Dev is where the duplication was: chunk .js/.map bytes were persisted again on top of the items' PersistedCode.
• Prod is roughly neutral. The minified factories now persist in their own tasks instead of inside the chunk files. In exchange, a warm build shouldn't need to re-minify unchanged factories; I haven't measured that.
• A tracked read of the rope in DiskFileSystem::write's guard cost ~35 MB of dev cache (dependency edges to every item cell), so the guard reads untracked.

Open questions / follow-ups

• The warm-restart panic above.
• Source-map output: the map file rope still changes with offsets, and its task now carries dependencies on every item's cells.
• Scope-hoisted merged modules, the prod rename cascade and non-chunk warm writes are untouched (they dominate prod edit writes).
• Measure warm-build time in prod, which is what would justify persisting minified factories.
• Extend to CSS chunks and other composed outputs. Rope could decode known static strings like RcStr does (orthogonal).

Design choices

• Flat list, not a tree, with one level of nesting used to extend the body; no stored total length because referenced cells can change.
• read() yields a regular Rope sharing item buffers. The fs effect still captures a temporary Rope before applying a whole-file write; no output-file-write savings are claimed.
• A streaming writer trait wouldn't eliminate that capture requirement and would introduce producer-specific persistence/hashing. It was not chosen for this prototype.

[github-actions]

Tests Passed

Commit: 6a215f6