v2 builder: cabal v2-build slicing, dev shell, plan-nix matching#2504
Draft
hamishmack wants to merge 116 commits into
Draft
v2 builder: cabal v2-build slicing, dev shell, plan-nix matching#2504hamishmack wants to merge 116 commits into
hamishmack wants to merge 116 commits into
Conversation
Adds the v2 component builder alongside v1 and a project-level
`builderVersion` switch that picks between them. Where v1 runs
`Setup.hs configure / build` per component, v2 runs `cabal
v2-build` against a per-slice slicing repo and pre-composed
starting store, registers each new unit under
`$out/store/ghc-<ver>[-inplace]/...`, and emits a plan-entry
diff against plan-nix when something diverges.
Highlights:
* `builder/comp-v2-builder.nix` — per-component slice builder.
Drives a per-slice cabal.project (constraints pinned from
plan-nix's `depends`, `extra-packages:` listing every
transitive dep, per-pkg `flags:` / `ghc-options:` /
`configure-options:` blocks) and a per-slice slicing repo
of source tarballs and a `00-index.tar.gz` so cabal's solver
hits a closed package set. Hands the cabal.project plus
the slice's direct dep slices off to `build-cabal-slice.nix`
to actually run cabal.
* `builder/build-cabal-slice.nix` — generic runner the per-
component builder calls into. Composes the starting store
via `lndir` from every dep slice (walks `pkgsHostTarget`
for direct deps and follows each entry's
`$out/nix-support/transitive-deps` for the rest of the
closure, so transitive dep slices stay out of nix-side
`buildInputs`), runs `cabal v2-build`, captures every newly
installed unit back into `$out/store`, writes the slice's
own `nix-support/transitive-deps` for downstream consumers,
and on `dryRunOnly` emits a per-entry diff between cabal's
planned `plan.json` and plan-nix's expectations.
* `builder/compose-store.nix` — `lndir`-based store
composition with the same direct + transitive-deps walk.
Used by `comp-v2-builder` for the `passthru.store` exposed
on each slice and by `shell-for-v2.nix` for the dev shell's
composed dep store.
* `builder/cabal-install-patches/{prune-unreachable-sublibs*,
skip-installed-revdeps-in-completed}.patch` — three
cabal-install patches the v2 slice solver needs to keep its
plan in sync with plan-nix.
* `builder/shell-for-v2.nix` — v2 dev shell. Resolves each
selected component's direct external `component.depends` to
its library (via `haskellLib.dependToLib`, so sublib
references resolve to their sublib slices), composes those
into a starting cabal store via `composeStore` (the
transitive closure is materialised at build time from each
slice's `nix-support/transitive-deps`), exposes the result
via `~/.cabal/store/<ghc>-inplace/` (the default
`exposePackagesVia = "cabal-store"`) or via a
`GHC_ENVIRONMENT`-wrapped ghc (`exposePackagesVia =
"ghc-pkg"`), and merges `passthru.depSlices` from
`inputsFrom` so cross + native shells both see each other's
slices.
* `builder/{default,hspkg-builder}.nix` — dispatch on
`builderVersion`: the per-component derivation at
`hsPkgs.<pkg>.components.<kind>.<n>` is whichever builder
the project selects (no per-component opt-in).
`hspkg-builder.nix` also resolves `homeDependIds` and the
`packageRefersOwnExe` flag from `config.plan-json-by-id`
(the project-level index) so these per-component lookups
are O(log N) attrset hits instead of linear scans over the
install-plan.
* `modules/{component-driver,project-common,shell}.nix` —
`builderVersion`, `crossTemplateHaskellSupport`,
`exposePackagesVia` options.
* `modules/install-plan/configure-args.nix` — filter
`--ghc-option=-hide-all-packages` out of `ghcOptions` for
the v2 builder (cabal injects it on every Setup configure
call, and round-tripping it through cabal.project would
duplicate it in `pkgHashProgramArgs`); extract
`--configure-option=...` entries into a new dedicated
`package.configureOptions` option (declared in
`modules/package-options.nix`) so the v2 builder can emit
them back into its own cabal.project.
* `modules/install-plan/non-reinstallable.nix` — gate the
ghcjs/wasm `ghci`-related `non-reinstallable` set on
`builderVersion != 2`; under v2 the slice solver follows
cabal's install plan literally.
* `lib/check.nix` — `passthru.isSlice` branch that runs the
slice-built exe directly instead of going through v1's
`setup test` pipeline.
* `lib/default.nix` — `uniqueWithNameKey` prefers
`identifier.unit-id` when present so two `foo-1.2.3` slices
with different inputs partition into different buckets.
* `lib/call-cabal-project-to-nix.nix` — set
`CABAL_INSTALLED_PACKAGE_ID_OS = pkgs.stdenv.buildPlatform.parsed.kernel.name;`
on the plan-to-nix derivation so the patched cabal-install
(#2501) actually fires; without the env var the patch is a
no-op.
* `compiler/ghc/default.nix`, `overlays/{bootstrap,tools}.nix`
— pin GHC, `hscolour`, and the boot tools to
`builderVersion = 1` so v2 churn doesn't trigger GHC
rebuilds while v2 is still settling.
* `overlays/{windows,mingw_w64}.nix` — v2-builder-aware
Windows-cross TH wrapper: `mingw_w64.nix` exposes
`wrapGhc :: ghc -> ghc` and `iservRuntimeLibs`;
`overlays/windows.nix` plumbs the build-platform
`runCommand` / `makeWrapper` / `libffi` through to that
module.
* `overlays/haskell.nix` — project-level shell dispatch
(`shellForV1` / `shellForV2`), `crossTemplateHaskellSupport`
plumbing, and `cabalProjectLocal` refactored so the
Windows-host iserv-proxy linker flags can be additive.
* `test/cabal-sublib-shell/` — new test exercising a sublib
consumer in the v2 dev shell. Other test wiring tweaks
(`test/{th-dlls,th-dlls-minimal,cabal-sublib}/default.nix`,
`test/cabal.project.local`, `test/{setup-deps,
shell-for-setup-deps}/pkg/pkg.cabal`, `test/default.nix`)
to keep things passing under v2.
* `ci.nix` — pin `evalSystem` default to `x86_64-linux` so
cross derivations evaluate consistently.
* `docs/dev/builder-v{1,2}.md` (+ `docs/SUMMARY.md` entry) —
design notes for the two builders.
Performance notes:
* `transitiveTarballs` is exposed as the deduped values of
`depTarballsDeduped`. The un-deduped concat (which sticks
every direct dep's full transitive list onto each consumer
unfiltered) grows exponentially with depth in a typical
Haskell graph (every package shares `base` / `bytestring` /
`text` / ...) and materialises multi-GiB blocks in the
evaluator on cardano-wallet-sized projects.
* `homeDepSliceOf` / `depTransitiveTarballsOf` fall through
to any sublib's slice when a dep package defines only
`library <name>` stanzas (no main library — e.g.
`cardano-wallet-ui:{common,shelley}`).
* `propagated` keeps `libs` Windows-only (matching v1):
cross-target slices land in the v2 dev shell's
`inputsFrom`, and unconditionally propagating `libs` would
drag e.g. `pkgs.liburing` from a Linux-target slice into a
Darwin consumer's `pkgsHostTarget`. Each slice gathers its
own transitive sysLibs (deduped) into `extraBuildInputs`
via `transitiveDepLibs` instead.
A cluster of fixes that together let the v2 dev shell evaluate, build,
and reuse prebuilt slices for cardano-wallet on aarch64-darwin. Most
fixes are independent in spirit but interact at the slice level — they
are bundled here to keep the cardano-wallet shell green in a single
commit.
- Sublib reachability env var (`HASKELLNIX_EXTRA_SUBLIB_SEEDS`)
threaded through both the solver-side and install-plan cabal-install
patches. Slices that target a sublib (or transitively depend on
one) seed the patched reachability walk from `pkgLibDepClosure` so
unrelated sublibs (e.g. `lib:testlib`'s deps) stay pruned.
- Source-repository-package handling in slices. Each
`pkg-src.type = "source-repo"` package gets a per-slice minimal git
repo (`git init -b minimal && git add . && git commit`) wrapped
around `${src}` and emitted as a `source-repository-package` block
in the slice's `cabal.project` — the same shape
`lib/call-cabal-project-to-nix.nix` produces at project level — so
cabal hashes the same source bytes plan-nix saw and the slice's
UnitId matches. Build-tools (`alex`, `happy`, `hsc2hs`, ...)
composed via `transitiveBuildToolSlices` so consumers find them in
the cabal-store instead of rebuilding.
- Plan-nix flag honouring for source-repo packages. Module-level
flag overrides (e.g. cardano-wallet's `flags.release = true`) don't
reach cabal at plan time, so plan-nix records cabal-file defaults.
`flagBlockFor` now reads from plan-json for source-repo pkgs so the
slice's `pkgHashFlagAssignment` matches plan-nix.
- Custom-build packages skip the UnitId check. Plan-nix carries a
single shared `id` per Custom-build package; cabal-install splits
them into per-component UnitIds that can never match. Detected via
missing `component-name` on the plan entry.
- `propagatedBuildInputs` for `depSlices` (was `buildInputs`) so
pkg-config deps and other propagated inputs chain transitively
via stdenv. Splice handles the cross-compilation case for normal
nixpkgs deps; slice $outs themselves don't auto-swap but cardano-
wallet's native shell doesn't trip that.
- `cp -rL` for local-package tarballs to inline out-of-bounds
symlinks like cardano-wallet's `lib/wallet/specifications ->
../../specifications` that cabal-install 3.16+ rejects as
[Cabal-7125] "Unsafe link target in tar archive".
- `$out/store` cleanup at end of installPhase. When the expected
UnitId is known, keep only `<uid>/`, `package.db/<uid>.conf` and
`lib/libHS<uid>-*` — `rm -rf $ghcDir` drops everything else
(including the lndir-composed dep symlink tree, which on a deep
graph was 10k+ entries that fixupPhase + NAR serialisation walked
for nothing). Falls back to `find -type l -delete` when the UnitId
isn't known (source-repo or `style: "local"`). Cache files
(`package.cache{,.lock}`) dropped from every slice — stale on
arrival downstream and a major source of "Keeping existing link"
spam at lndir time.
- `find -type d` instead of `ls -d */` for the unitdirs-before
snapshot. bash expands `*/` into args including OS-prefix
unit-ids (`-clsss-1.8.0.1-...`), and `ls` interprets the leading
`-` as flags. Same root cause as the earlier `grep -qx --` fix.
- v2 shell: walk through local packages transitively, collecting
external deps at every step. Local packages are excluded from the
composed cabal-store so cabal compiles them on demand from the
user's tree; their (transitive) external deps go in. Build-tool
exes that resolve to local packages are similarly excluded.
- v2 shell: `pkgs.gitMinimal` on PATH so `git --version` works
inside the shell. On Darwin the apple-sdk overlay sets
`DEVELOPER_DIR` to the SDK path (no tools), so `/usr/bin/git`'s
xcrun shim fails with "tool 'git' not found"; nix-managed git
bypasses the shim.
- Slice cabal: `--jobs=$NIX_BUILD_CORES` (capped at 4, mirrors v1)
on the `cabal v2-build` invocation, which passes through to
`Setup build -jN` for ghc per-module parallelism. Build-phase
flag, so it stays out of `pkgHashConfigureOptions` and UnitIds
remain stable.
cabal-install 3.16's `setupHsBuildFlags` deliberately leaves
`buildNumJobs = mempty` (TODO: [nice to have] upstream), so neither
`cabal v2-build --jobs=N` nor `--ghc-options=-jN` actually parallelise
per-module compilation:
* `--jobs=N` is package-level (one slice = one package) and is
*not* threaded through to Setup build.
* `--ghc-options=-jN` would land in `pkgHashConfigureOptions` and
fork the slice's UnitId from plan-nix.
`setup-build-num-jobs-env.patch` adds a small read of
`HASKELLNIX_BUILD_NUM_JOBS` at the call site that constructs
BuildFlags, feeding it into `buildNumJobs`. Build-phase flag, so
UnitIds stay stable. build-cabal-slice.nix exports the env var
capped at 4 (mirrors v1's `-j$(($NIX_BUILD_CORES > 4 ? 4 : ...))`),
so `Setup build` invokes `ghc --make -j4` and per-module compilation
parallelises again.
Verified end-to-end: io-classes slice's GHC response file now shows
`-j4` while the captured UnitId still matches plan-nix's
`-clsss-1.8.0.1-14f3d66e`.
Three fixes to `builder/shell-for-v2.nix` so the shell's cabal store matches the user's intuition (project packages built from the working tree, everything else pre-built): 1. Switch the default `packages` from `selectLocalPackages` to `selectProjectPackages`, and rename `pkgIsLocal` → `pkgIsProject` to use `isProject` instead of `isLocal`. `isLocal` covers source-repository-packages too, but SRPs are pinned upstream code the user doesn't iterate on — they belong in the shell's pre-built store, not excluded from it. 2. Filter project-package exes out of `buildToolDrvs`. Without this, any project exe listed as `build-tool-depends` of another project package gets pre-built and dragged into the shell closure, along with all of its lib slices' deps (~25 stray slices in the cardano-wallet shell). Mirrors v1's `removeSelectedInputs` and the existing `ownBuildToolSlices` filter. 3. Dedup the `projectPkgClosure` walk by `identifier.id` (cabal unit-id), not `identifier.name`. v2 plan-nix splits each unit-id component into its own `hsPkgs` entry — components are spread across siblings — so deduping by package name dropped every sibling after the first and lost their deps (e.g. the `unit` test's `x509`/`pem` deps when `test-common` was visited first).
The `--sha256` pin for the head.hackage tar no longer matched upstream — CI was failing with `hash mismatch in fixed-output derivation`. Update to the current sha256 reported by nix.
The previous commit treated every project package as "iterated on from the working tree" and excluded them all from the shell's cabal store. That broke `tests.cabal-sublib-shell.run`, which intentionally selects just `consumer` and expects `provider` (a sibling project package) to be pre-built — so an unpatched cabal inside the shell can reuse provider's lib/sublib instead of rebuilding from source. Mirror v1's `removeSelectedInputs` semantics: only exclude packages in `selectedPackages` itself. With the default `packages = selectProjectPackages` every project pkg is selected (so the cardano-wallet UX is unchanged); with an explicit `packages = ps: [ ps.foo ]`, only `foo` is excluded and sibling project pkgs land in the store. Also drops the project-pkg closure walk — `selectedPackages` already enumerates every unit-id fragment, so a straight `concatMap getAllComponents` covers all components without the walk.
The test runs `runghc conduit-test.hs` inside the v2 shell; that needs `Conduit` visible to plain `ghc`/`runghc`. Default `exposePackagesVia = "cabal-store"` only seeds `~/.cabal/store` (which `cabal v2-build` reads), so plain ghc still misses the setup-deps. Switch to `"ghc-pkg"` so the wrapped ghc stacks the composed package.db via `GHC_ENVIRONMENT`.
The test sets `enableProfiling = true` / `enableLibraryProfiling = true` via haskell.nix modules. Under v1 those translate directly to per-component `--enable-profiling` configure flags, but under v2 (now the default) the slice reads its toggles from plan.json's recorded `configure-args` — and plan.json was generated from cabal.project alone, where neither flag is set. The slice therefore built without `-prof`, and `+RTS -p` failed with "the flag -p requires the program to be built with -prof". Inject `profiling: True` / `library-profiling: True` at project level via `cabalProjectLocal` so plan-nix records `--enable-profiling` / `--enable-library-profiling`, matching the slice's actual build and keeping the slice's UnitId reproducible. The module-level overrides are kept so v1 still builds with profiling too.
Previously `projectConfigPragmas` read configure-args off the
first configured plan entry and emitted them under a single
`package *` block, on the assumption that "the same flags appear
in every configured plan entry's configure-args". That holds for
project-wide toggles (--enable-optimization, --disable-shared,
...) but breaks the moment a user sets something per-package —
e.g. `package cabal-simple\n profiling: True`. The setting only
landed on cabal-simple's plan entry; if the first configured
entry was a transitive dep, the slice's cabal.project never saw
`profiling: True` and the exe built without `-prof`.
Group plan entries by `pkg-name`, take the union of pragmas
across each pkg's units (cabal.project only supports per-package
granularity), then split into:
* `package *` — pragmas common to *every* pkg-name. The slice's
`cabal v2-build` resolves transitive hackage deps fresh, so
these have to land project-wide for the deps' UnitIds to line
up with plan-nix.
* `package <name>` — for each pkg-name with pragmas extending
the baseline (e.g. `package cabal-simple` getting
`profiling: True` from
`packages.cabal-simple.enableProfiling = true` mirrored into
cabal.project).
`tests.cabal-simple-prof.run` now uses the per-package shape that
mirrors haskell.nix's modules:
package *
library-profiling: True
package cabal-simple
profiling: True
When a project has `supportHpack = true` and a local package ships only `package.yaml` (no `.cabal`), v1 ran hpack at the component-build phase to generate the `.cabal`. v2 never ran it, so the v2 source tarball had no `.cabal` and the `v2-exe-repo` pre-build step failed with `tar: <pkg>-<ver>/<pkg>.cabal: Not found in archive`. Thread `cabal-generator` through `hspkg-builder.nix` to `comp-v2-builder.nix` and run hpack inside the staged source dir before tarballing, mirroring v1's `comp-builder.nix:485` invocation. Skipped when the package isn't local (hackage tarballs already contain a generated `.cabal`) or when `cabalFile` is set (e.g. an X-revision from `package-description-override`, which provides its own `.cabal`). Verified against `tests.cabal-hpack.run`.
Re-enables the `broken = isGhcjs && ghc >= 9.6.1` guard that commit 19b9a51 commented out while bringing up v2. The underlying failure is in ghcjs/emscripten wasm-ld ("section too large" linking the `-O`-optimized C-backend objects), which predates v2 and isn't a slicer issue. Skip the test on ghcjs again so CI doesn't fail on a pre-existing upstream limitation.
`packages.HsOpenSSL.ghcOptions` previously held a `-optc=-Wno-incompatible-pointer-types` workaround. v2's slices read configure-args from plan.json; module-level `ghcOptions` never reach plan-nix's plan.json, so under v2 the slice picked up the flag (via comp-v2-builder's per-package ghcOptions block) but plan-nix did not — and the resulting UnitIds diverged (`HsOpnSSL-0.11.7.10-9ebdbccd` expected vs `-ecc34d19` produced). Move the workaround into `test/cabal.project.local` under `package HsOpenSSL ghc-options:` so plan-nix and the slice agree. Drop the now-unneeded `HsOpenSSL` entry from `package-keys` in `test/modules.nix`. Verified against `tests.exe-dlls.build`.
The test runs `runghc conduit-test.hs` against `env.ghc` / `envDefault.ghc`. With v2's default `exposePackagesVia = "cabal-store"` only `~/.cabal/store` is seeded; plain `runghc` doesn't read the cabal store, so it failed with "Could not find module 'Conduit'". Switch the three shells to `"ghc-pkg"` so `env.ghc` is wrapped to stack the composed package.db via `GHC_ENVIRONMENT`. Mirrors the earlier fix in `tests.shell-for-setup-deps`.
The script ran a per-conf `diff -q` and a per-unit `diff -qr` on
every shell entry — for cardano-wallet's ~240 slices that meant
forking hundreds of `diff` invocations and walking each unit's
file tree. Two short-circuits:
* Per-composedStore marker file at
`~/.cabal/store/.haskell-nix-shell-markers/<basename-of-src>`,
written after a successful run. On re-entry the marker is
present and the script exits before the scan. Different
shells have distinct basenames, so alternating shells don't
invalidate each other's markers.
* Readlink fast-path in the scan: confs and lib files are
installed as symlinks to `$src/...`, and unit dirs are lndir
trees of symlinks under `$src/$ghcName/$unitId/`. When the
target is already in the right shape, skip the per-file `diff`
fork. Only when the cheap check fails does the script fall
back to `diff -q` / `diff -qr` to detect genuine conflicts.
Measured on cardano-wallet's shell (~240 slices, fresh CABAL_DIR):
first run 24.5s, second run 0.012s.
Three incremental improvements toward making `lib/cover.nix` work
against v2 lib/test slices. The end-to-end coverage test still
fails because v2's lib slices use a hashed UnitId
(`pkgb-0.1.0.0-<hash>`) while a sibling test slice rebuilds the
lib inplace and emits `.tix` referencing `pkgb-0.1.0.0-inplace`,
so HPC can't match them up. That alignment is a deeper change;
land the supporting plumbing first:
* Add `srcSubDir` / `srcSubDirPath` to v2 slice passthru so
`lib/cover.nix:19`'s `map (l: l.srcSubDirPath) mixLibraries`
no longer hits "attribute missing".
* Library slices now copy each `extra-compilation-artifacts/hpc/<way>/{mix,tix}`
subdir up to `$out/share/hpc/<way>/{mix,tix}/<pkg>-<ver>/`
(the predictable name `lib/cover.nix` and the cover-report
tests expect — v1 does this too in `comp-builder.nix:831`).
* v2's `lib/check.nix` branch copies `<exeName>.tix` from the
test run to `$out/share/hpc/vanilla/tix/<exeName>/`, mirroring
v1's `lib/check.nix:119`.
v1 slices provided `.profiled` as an overlay rebuild with
`enableLibraryProfiling = true`. v2 reads configure-args from
plan.json, so an overlay would emit `--enable-profiling` toggles
that plan-nix doesn't know about and the slice's UnitId would
diverge.
Replace the (missing) attribute with a `throw` that points users
at `cabal.project` / `cabalProjectLocal`:
package <pkgname>
profiling: True
library-profiling: True
so plan-nix records the toggles and the slice's UnitId stays
aligned. Set the `throw` at both the top level and inside
`passthru` because mkDerivation only lifts passthru → top level at
derivation creation time; post-hoc `// { passthru = ... }` doesn't
re-lift.
Adds `docs/dev/profiling.md` with the migration recipe.
Plan-to-nix's `modules/install-plan/configure-args.nix` already
translates `--ghc-option=` and `--configure-option=` entries from
plan.json's per-pkg `configure-args` into haskell.nix module
options. Extend it to also pick up the `--enable-{profiling,
library-profiling, coverage}` toggles so v1's `comp-builder`
(which reads `enableProfiling`, `enableLibraryProfiling`,
`doCoverage` straight off the component) honours
`package <pkg>\n profiling: True` set in cabal.project — without
needing module-level overrides.
v2 already reads these from plan.json directly via its own
`projectConfigPragmas`; the picked-up values are merely
consistent there.
Migrate the `tests.{exe,exe-lib,th}-dlls.{check-,build-}profiled`
variants to a sibling project whose `cabalProjectLocal` enables
profiling — the v2 builder no longer ships `.profiled` as an
overlay rebuild (see `docs/dev/profiling.md`).
Drops the `cabal-slice-` prefix and reorders the slice's `name` attribute from `<pkg>-<ver>-<ctype>-<cname>` to v1's shape `<pkg>-<ctype>-<cname>-<ver>` (mirrors `comp-builder.nix:268`). Same for the auxiliary `check-*` and `store-*` derivations. Callers that hardcoded v1 derivation names — most prominently `tests.coverage.run`, which checks for `tix/pkgb-test-tests-0.1.0.0-check/tests.tix` — keep working without per-test edits when projects flip `builderVersion = 1 → 2`. Also surface the lib slice's `.mix` files under `<pkg>-<ver>-inplace/<Module>.mix` so they line up with the inplace UnitId that test-slice-built `tests.tix` files reference (`Tix [ TixModule "<pkg>-<ver>-inplace/<Module>" … ]`); the .mix content itself is byte-identical between the lib slice's hash-named UnitId build and the test slice's inplace rebuild. Move the coverage-test's `doCoverage` modules into `cabalProjectLocal` (`package <pkg>\n coverage: True`) so v2's plan-nix records `--enable-coverage` and the slice actually produces .mix/.tix output. The module-level overrides are kept for the v1 builder. Verified against `tests.coverage.run`.
v1 (Setup.hs install) for ghcjs preserves the \`bin/<exe>.jsexe/all.js\` directory layout, which is why \`comp-builder.nix:450\` appends \`.jsexe/all.js\` to the exe name for \`isGhcjs && ghc < 9.8\`. v2 builds via \`cabal v2-build\`, whose install step bundles the \`.jsexe/\` contents into a single self-contained \`#!/usr/bin/env node\` script at \`bin/<cname>\` — there is no \`.jsexe/\` directory in the slice's output. Use the bundled-file path so v2's \`exePath\` / \`find\` lookup land on the file cabal actually produced. Verified against \`aarch64-darwin.unstable.ghc967.ghcjs.hello\` (slice surfaced \`bin/hello\` cleanly, no placeholder fallback).
The slice's `$out/dist-newstyle/` carried cabal's source tarballs and the build tree — hundreds of MB to GB per slice for nontrivial projects. Nothing downstream of a successfully-built slice reads it: subsequent slices only pull from `$out/store/`, and the diagnostic `checkAgainstPlan` is a separate derivation with its own dist-newstyle. Trim it at the end of `installPhase`, after `comp-v2-builder.nix`'s test/bench install step (which does need to find the unpinged binary in `dist-newstyle/build/.../<exe>`). Lift `cache/plan.json` to `$out/plan.json` so it stays available for human debugging — the rest of `cache/` is internal. Verified on the ghc967.ghcjs.hello slice: $out shrinks from ~7.9GB to ~1MB, with `bin/`, `store/`, `plan.json`, `unit-ids`, and `nix-support/` retained.
…`/\`comp.env\` v2 slices don't expose v1's per-component \`.shell\` / \`.env\` attributes — \`shell\` is provided at the project level via \`project.shellFor\`, and that mode covers what this test needs (a wrapped \`ghc\` / \`runghc\` that can see the package's deps). \`exposePackagesVia = "ghc-pkg"\` makes \`shell.ghc\` a wrapped GHC stacking the composed package.db via \`GHC_ENVIRONMENT\`, so \`runghc ./Point.hs\` and \`ghc Point.hs\` resolve the deps the same way v1's \`library.env\` did. Works under both v1 and v2 — no per-component shell required.
v2 slices no longer ship `.profiled` as an overlay rebuild — see
`docs/dev/profiling.md`. Migrate the remaining tests that
referenced `…components.<kind>.<name>.profiled`
(`js-template-haskell`, `th-dlls-minimal`, `gi-gtk`) to a sibling
project whose `cabalProjectLocal` enables profiling, matching the
shape already used by `exe-dlls` / `exe-lib-dlls` / `th-dlls`.
Expose each new project's `plan-nix` under a distinct
`ifdInputs` key so the materialised plan-nix expressions are
covered by CI:
plan-nix — non-profiled (default)
plan-nix-ei — externalInterpreter variant (th-dlls /
th-dlls-minimal)
plan-nix-profiled — profiled
plan-nix-profiled-ei — profiled + externalInterpreter
GHC 9.8 added an `-inplace` suffix to its boot-package UnitIds (`base-4.19.2.0-inplace` vs `base-4.18.3.0` on 9.6) and started emitting a `Project Unit Id` field in `ghc --info`. The dummy ghc/ghc-pkg in `lib/call-cabal-project-to-nix.nix` was hardcoding both, so cabal computed UnitIds against the dummy with `-inplace` even on GHC 9.6. When the slice's real ghc 9.6 then returned ids without `-inplace`, the slice's UnitId for any package depending on a boot package (`colour` ↔ `base`) diverged from plan-nix and the slice failed its UnitId check. Verified empirically that `-inplace` appears starting at GHC 9.8: ghc967 → id: base-4.18.3.0 ghc984 → id: base-4.19.2.0-inplace ghc9103 → id: base-4.20.2.0-inplace ghc9124 → id: base-4.21.2.0-inplace ghc9141 → id: base-4.22.0.0-inplace Make both the `Project Unit Id` field and the `-inplace` suffix conditional on `versionAtLeast ghc.version "9.8"`. Verified against `tests.shell-for.env` on ghc967 (which exercises the colour slice that was failing in CI).
Under v1 every component's `.doc` was a sibling Setup haddock
derivation that shared the lib's UnitId. Under cabal v2-build
this no longer holds: `cabal v2-haddock` flips
`elabBuildHaddocks` (and the haddock-html / haddock-hscolour /
... family) on every unit's `ElaboratedConfiguredPackage`, which
all land in `pkgHashConfigInputs`. Calling `cabal v2-haddock`
against a plan that didn't already have `documentation: True`
forks every UnitId in the closure and triggers a from-source
rebuild.
Round-tripping `documentation: True` through plan.json's
`--ghc-option=-haddock` / `configure-args.nix` /
`ghc-options: -haddock` is NOT equivalent: the ghc-option keeps
haddock comments in `.hi` files but doesn't set the haddock-config
booleans, so the slice's `pkgHashConfigInputs` diverges from
plan-nix's and the dep closure's UnitIds fork (observed on
OneTuple in `tests.sublib-docs`: plan-nix `9a847723` vs slice
`f546bd36`).
Approach:
* `comp-v2-builder.nix` detects per-package `--ghc-option=-haddock`
in plan.json (the `documentation: True` signal cabal-install
surfaces) and emits `package <pkg>\n documentation: True\n`
in the slice's cabal.project for every documented package.
`-haddock` is filtered out of the per-pkg `ghc-options:` block
so cabal doesn't see it twice in `pkgHashGhcOptions`.
* Each library slice exposes `.doc`, a sibling derivation that
runs `cabal v2-haddock` against the already-built unit (no
closure rebuild). `.doc` throws with a migration hint when
documentation isn't in the project's plan-json — that's the
only shape where the UnitIds align.
* Doc slices propagate `(map d: d.doc)` for `docEnabled` deps
only, so cross-package hyperlinks resolve. Mixed projects
(some packages docs, others not) keep working because non-doc
deps come in as plain slices.
* `build-cabal-slice.nix` keeps cabal's native unit-dir layout
(`$out/store/<ghc>/<unit-id>/share/doc/html/`) so doc slices
`lndir` into `~/.cabal/store/` as drop-in replacements, and
cross-package hyperlinks (absolute
`file:///nix/store/<doc-slice>/store/<ghc>/<dep-uid>/...`)
resolve back into the slice's own tree. Non-target unit
haddocks are stripped from `$out` to keep each slice lean —
deps' html lives in the deps' own `.doc` slices.
`docs/dev/haddock.md` documents the v2 semantics, the
`documentation: True` requirement, and why there's no
`slice.haddockDir` (local plan-nix UnitIds use `<pkg>-<ver>-inplace`
form while the slice's cabal-store uses cabal's mangled hash form,
so there's no eval-time-stable html path — callers `find` it
under the doc slice).
Verified `tests.sublib-docs.run` builds Lib.html and Slib.html
under `slice.doc`, and OneTuple's UnitId in plan.json matches
plan-nix's recorded id.
…ools
GHC 9.14.1's `armv7a-android` cross GHC ships *both*
`<prefix>deriveConstants` and an unprefixed `deriveConstants` in
its `bin/` (the latter for build-host use). `ghcShim`'s
single-pass loop iterated `${ghc}/bin/*` alphabetically, hit
`armv7a-...-deriveConstants` first, and the case branch
synthesised an unprefixed-fallback alias
`$out/bin/deriveConstants -> <prefix>deriveConstants` (the
`[ -e ]` guard was for that fallback, not the raw link). When
the loop later reached the real `deriveConstants`, the raw
`ln -s "$f" "$out/bin/$base"` crashed with "File exists".
Earlier cross GHCs only shipped `<prefix>deriveConstants`, so
the fallback was the sole producer of `$out/bin/deriveConstants`
and never collided. GHC 9.14.1 added the unprefixed sibling and
exposed the race.
Switch to a two-pass shape: pass 1 links every source bin/ entry
under its own name, pass 2 synthesises unprefixed aliases only
for prefixed names that have no real unprefixed sibling. Real
files always win; aliases only fill genuine gaps.
Same fix in both `builder/build-cabal-slice.nix` and
`builder/shell-for-v2.nix` (which carries an identical shim for
the dev shell's cross-cabal wrapper).
`modules/configuration-nix.nix` used to unconditionally `mkForce`
`packages.ghc.src` to a symlinkJoin of `(configured-src + generated)`
under `compiler/`, redirecting `lib:ghc`'s source to the local GHC
tree *after* plan-nix had been computed against hackage's
`ghc-X.Y.Z.tar.gz`. v1 (Setup.hs) didn't care — it builds whatever
`src` it's given. v2 forks the slice's UnitId from plan-nix's
recorded one (the slice rebuilds from a different source than the
planner saw) and the slice fails its UnitId check.
Replace with an opt-in project-level flag, `useLocalGhcLib`, that
exposes the GHC compiler tree to the planner up-front. Mechanism
differs by project type:
* Cabal projects (modules/cabal-project.nix) inject a
`source-repository-package` block + `allow-boot-library-installs:
True` into `cabalProjectLocal`, with an `inputMap` entry keyed
by `<url>/<ref>` (string-context-stripped) so haskell.nix
short-circuits the `builtins.fetchGit` path. Plan-to-nix and
the v2 slice both see the same wrapped repo, cabal hashes the
same content into `pkg-src-sha256`, and UnitIds align.
* Stack projects (modules/stack-project.nix) re-add the
`packages.ghc.src` override as a contributed module, gated on
`useLocalGhcLib`. Stack-to-nix's input shape doesn't fit the
cabal source-repo path; instead we lean on stack only supporting
v1 for now (`builderVersion = mkForce 1` in stack-project.nix's
config) — v1 doesn't enforce UnitId alignment, so the post-plan
swap is fine.
The option itself lives in `modules/project-common.nix` so both
project types see it without duplication.
`builder/comp-v2-builder.nix` emits `allow-boot-library-installs:
True` in the slice's cabal.project whenever the target or any non-
pre-existing lib dep is on cabal-install's hard-coded
non-reinstallable list (`ghc`, `template-haskell`, `Cabal`,
`Cabal-syntax`, `ghc-prim`, `ghc-bignum`, `ghc-boot`,
`ghc-boot-th`, `ghc-heap`, `base`, `ghci`, `ghc-internal`, `rts`).
Without it the slice's solver rejects the source instance with
"constraint from non-reinstallable package requires installed
instance" once `ghc` is being source-built.
`test/ghc-lib-reinstallable/cabal.nix` and
`test/ghc-lib-reinstallable/stack.nix` opt in via
`useLocalGhcLib = true`. The stack test's resolver is bumped to
`nightly-2026-05-03` (latest known to the pinned stackage.nix at
the time of this commit).
Verified end-to-end:
* `tests.ghc-lib-reinstallable-cabal.run` (v2, ghc9141) — passes
* `tests.ghc-lib-reinstallable-cabal.run` (v1, ghc9141) — passes
* `tests.ghc-lib-reinstallable-stack.run` (v1, ghc9124) — passes
Plan-to-nix's cabal-install records `--disable-executable-static` in `configure-args` for every component when the project hasn't explicitly enabled it. v1's `builder/comp-builder.nix:384` papered over this by injecting `--ghc-option=-optl=-static` (and `-optl=-pthread`) at comp-builder time — outside plan.nix's view. v1 doesn't enforce UnitId alignment with plan.json, so the post-plan flag insertion is harmless there. Under v2 the slice mirrors plan.json's `configure-args` exactly, so without something at the project level the slice links dynamically and the `tests.c-ffi.run` musl assertion (`grep "not a"` on glibc-ldd output) fails. Inject `executable-static: True` into `cabalProjectLocal` when `stdenv.hostPlatform.isMusl`. Plan-to-nix then records `--enable-executable-static`, v2's `comp-v2-builder.nix:projectConfigPragmas` round-trips it through, and cabal links the executable statically. v1 is unaffected — its post-plan `-optl=-static` injection still fires, redundant but harmless. Verified by running `tests.c-ffi.run` on the `x86_64-linux.unstable.ghc9141.musl64` jobset (commit `d027fbe4e` + this patch in a fresh worktree).
`dummy-ghc` is the eval-time stand-in cabal-install runs against
during plan-to-nix. Its `--info` output shapes plan.json's
per-unit `configure-args` (which feed `pkgHashConfigInputs` and
therefore the UnitId). When the dummy reports different
capabilities than the real cross-ghc the slice rebuilds with,
plan-nix and slice compute different UnitIds and the slice's
expected-package check fails.
The "otherwise" branch (Linux / Darwin / native) reports
`Support shared libraries: YES`, `GHC Dynamic: YES`, RTS ways
including the full `_dyn` family, and `Stage: 2`. Real Android
cross-ghc (`aarch64-unknown-linux-android-ghc 9.14.1`) is a
stage-1 NDK build with no `_dyn` ways and no `Support shared
libraries` field at all (its `GHC Dynamic` is `NO`). Falling
through to "otherwise" made plan-nix record `--enable-shared`
for Android packages while the slice's real ghc silently flipped
to `--disable-shared`, forking the UnitId on
`pkgHashSharedLib`.
Add an explicit Android branch. Reference values cross-checked
against `aarch64-unknown-linux-android-ghc --info` on the live
9.14.1 cross GHC derivation; key fields:
* Support dynamic-too: YES (GHC's flag, kept like real ghc)
* GHC Dynamic: NO (was YES — drove --enable-shared)
* RTS ways: no `_dyn` family
* Stage: 1
* No `Support shared libraries` field (cabal interprets
absence as no-shared-libs, matching real ghc)
Verified `tests.extra-hackage` slices for
`x86_64-linux.unstable.ghc9141.aarch64-android-prebuilt` no
longer fail the slice's UnitId-alignment check. (The `.run`
step still fails because qemu-aarch64 can't load Android's
`/system/bin/linker64` to actually exec the binary — that's a
pre-existing runtime issue, not a v2 alignment problem.)
For musl, real and dummy ghc already agree
(`Support dynamic-too: YES, GHC Dynamic: YES`); the
`executable-static: True` injection in
`modules/cabal-project.nix` stays — it's a user-opt-in flag,
not capability-derived from `ghc --info`.
`shellFor` (both v1 and v2) ran into "no C compiler provided for this platform" under cross targets where the host stdenv has no cc — most visibly ghcjs, also wasm and android-prebuilt. Three distinct issues, all fixed here: 1. `mkShell` itself was `pkgs.mkShell` (host-stdenv-bound). `pkgs.bashInteractive`'s eval reads `stdenv.cc.isClang`, and under ghcjs cross there is no cc → throw. Pass `mkShell = pkgs.buildPackages.mkShell` to both shellForV1 and shellForV2 so the shell drv lives on the build platform regardless of the host (the user opens the shell on their build host anyway). 2. `pkgs.runCommand` was building `packageEnv` and `wrappedGhc` (the wrapped GHC the shell exports as `shell.ghc`) against the cross stdenv. Once these landed in the shell's `nativeBuildInputs`, mkShell's dep-splicing eventually re-evaluated the cross stdenv's bash via setup hooks, hitting the same cc.isClang. Switch both to `pkgs.pkgsBuildBuild.runCommand` and `pkgs.pkgsBuildBuild.makeWrapper` so the wrappers themselves are build-platform drvs. (Native builds are unaffected — `pkgsBuildBuild == pkgs` there.) 3. `store = composedStore` was a top-level attr to mkShell. Recent nixpkgs's `mkDerivation` folds unknown top-level attrs into `env` and type-checks each value, and `isDerivation composedStore` forces evaluation of the cross stdenv's bash the same way. Move `store` into `passthru.store` — mkDerivation lifts passthru into the result drv anyway, so `shell.store` continues to resolve. Verified `tests.with-packages.test-shell` now evaluates under both the `aarch64-darwin.unstable.ghc9141.ghcjs` (cross) and `.native` jobsets, and `shell.store` still resolves on both.
Cabal pretty-prints long `id:` hashes onto a separate indented line:
id:
lens-5.3.6-9653fc7a...
The awk pattern `/^id:[[:space:]]/ { print $2 }` only matched the
boot-db single-line form (`id: base-4.22.0.0-inplace`), so every
composed-store conf was silently skipped — the `package-id` lines for
lens, transformers, ... never made it into the env file even though
their `.conf`s were in the package-db, so ghc reported them as
hidden:
>>> Could not load module `Control.Lens'.
>>> It is a member of the hidden package `lens-5.3.6'.
Read the next line when `id:` has no value on the same line.
The cleanup that follows `cabal v2-build` walks the slice's `$out` and
deletes the lndir-composed dep-slice content — those symlinks are
pure overhead (NAR scan, fixupPhase walk, reference scan all stat
them) and downstream consumers compose dep slices directly from
`nix-support/transitive-deps`.
GHC bakes `<this-slice>/store/.../<dep-uid>/lib` into the slice's own
shared libraries' DT_RUNPATH / LC_RPATH so iserv-dyn / dyld can find
each dep's transitive `.so` at TH-eval time. Plain "wipe dep dirs"
breaks this: stdenv's `shrink-rpath` (Linux fixupPhase) sees the
entry's dir is empty / missing and drops it, and on Darwin
install_name_tool wasn't touching them either — at runtime dyld
fails with `No such file or directory`.
Before wiping, walk every ELF / Mach-O file in the slice's own
unit dirs and rewrite its rpath: for each entry pointing into
`$out/store/`, follow a symlink under it via `readlink -f` to find
the dep slice's actual `/nix/store/<dep-slice>/store/.../<dep-uid>/lib`
and substitute that. Linux uses `patchelf --set-rpath`; Darwin uses
`install_name_tool -rpath`. After the rewrite, `shrink-rpath` sees
populated dirs and keeps the entries.
Confirmed:
* x86_64-linux native with-packages.run (paired with the awk fix
in the v2-shell env-file generator)
* x86_64-linux musl64 js-template-haskell.build — previously
failed with `Error loading shared library libHSth-abstraction-…
.so: No such file or directory` because shrink-rpath had
stripped the dep-slice path
* x86_64-linux native cabal-sublib-shell.run
* armv7a-android-prebuilt c-ffi.run (cross)
The darwin rpath rewriter pipes `otool -l` into `awk`. Under `set -euo pipefail`, when otool errors on a non-Mach-O input (static `.a` archives, the cabal-store `.conf` files etc.), pipefail propagates otool's non-zero exit through the awk pipeline, the command substitution's exit status is non-zero, and `set -e` kills installPhase mid-step — silently, since stderr is redirected to /dev/null. Mirror the ELF branch's pattern: `|| return 0` on the assignment so non-Mach-O files just skip the function quietly.
…walk
Two changes that unblock musl64 TH builds:
1. Set `LD_LIBRARY_PATH=<musl-gcc-libs>/lib` on `stdenv.hostPlatform.isMusl`
so iserv-dyn can resolve `libgcc_s.so.1` from `libstdc++.so` at TH-eval
time. Same workaround `comp-builder.nix` has carried for v1; without
it, `musl64.tests.th-dlls.build` fails with `Error loading shared
library libgcc_s.so.1` because musl's `libstdc++.so` only carries
`RUNPATH=<musl libc dir>` (not the gcc lib dir) and musl's dyld only
consults the immediate library's RUNPATH when resolving its deps.
That same `LD_LIBRARY_PATH` leaks into every subprocess we spawn,
including patchelf — a glibc binary built against glibc's
`libgcc_s.so.1`. Letting patchelf dlopen the musl libgcc segfaults
it (which was also the source of the `shrink-rpath` segfaults in
fixupPhase we'd see on musl). Prepend `unset LD_LIBRARY_PATH` to the
rpath rewrite block on musl; nothing in the rest of installPhase or
fixupPhase needs the iserv search path.
2. Restrict the rpath-walk's `find` to ELF / Mach-O candidates — `.so*`,
`.dylib*`, and any file under `bin/` — so the rewrite functions
don't have to defend against `.hi`, `.a`, `cabal-hash.txt`, etc.
The only remaining defence on the elf side is `|| return 0` on
`patchelf --print-rpath`, which the filter doesn't help with: musl exe
slices link statically (`executable-static: True`) and patchelf bails
with `cannot find section '.dynamic'` on those — comment notes it.
Confirmed:
* x86_64-linux musl64 th-dlls.build (previously failing with both
`libgcc_s.so.1` and the dep-resolution dyld error)
* x86_64-linux native with-packages.run
* x86_64-darwin native exe-lib-dlls.build-profiled
|
Sadly a bit late to the party but I didn't know about the installed sub-libraries issue until I published a package using a couple and trying to build it in nix shell 🫠 I'm trying to upstream fix this in haskell/cabal#11788, feedback welcome! |
… native-musl
Two coupled changes that fix musl64 TH targets that invoke external
binaries from compile-time code.
1. Move the `LD_LIBRARY_PATH=<musl-gcc-libs>/lib` workaround (so
iserv-dyn finds `libgcc_s.so.1` transitively from
`libstdc++.so` at TH-eval time) from a derivation-wide env attr
to a `makeWrapper --prefix LD_LIBRARY_PATH` on the ghc shim's
ghc binaries. v1 sets it derivation-wide; that env then leaks
into every glibc subprocess cabal/ghc spawn — `git`, in
particular, segfaults loading the musl libc as glibc's libc
("invalid ELF header"). Scoping to the ghc wrapper means iserv
still inherits it (so TH eval works) but glibc tools cabal
invokes directly (e.g. `git` for `source-repository-package`)
don't see it. Gated on `isNativeMusl` (build-arch == host-arch).
2. iserv-dyn also forks subprocesses for TH callbacks (e.g.
`githash`'s `$$tGitInfoCwd` runs `git rev-parse`), and those
children inherit `LD_LIBRARY_PATH` through the ghc-shim wrap.
Match what `test/githash/default.nix` already does on
`isNativeMusl`: pick the host-platform (musl) `git` instead of
the build-platform (glibc) one — a musl `git` ignores the
musl-gcc lib path harmlessly. Reach through
`pkgs.pkgsHostHost.gitReallyMinimal` to bypass `mkDerivation`'s
`nativeBuildInputs` auto-splicing, which otherwise resolves
`pkgs.gitReallyMinimal` back to the build-platform variant.
`gitReallyMinimal` rather than `gitMinimal`: gitMinimal's musl
cross-build runs the full git test suite (which fails);
gitReallyMinimal skips `doInstallCheck` and is what
`test/githash` already uses. Unified to `gitReallyMinimal` on
the non-musl branch too — smaller closure, same role.
Also tighten `test/githash/default.nix` to gate on
`haskellLib.isNativeMusl` rather than its prior `isMusl && cross`
heuristic.
Confirmed:
* x86_64-linux musl64 githash.run (was failing with
`git: invalid ELF header`)
* x86_64-linux musl64 cabal-source-repo.run
* x86_64-linux musl64 th-dlls.build
* x86_64-linux native with-packages.run
* x86_64-linux armv7a-android-prebuilt c-ffi.run
* x86_64-darwin native exe-lib-dlls.build-profiled
…iant
cabal v2-build prepends `<unit>/bin/` to PATH when invoking ghc for
any package whose `build-tool-depends:` refers to that unit. In our
cross-compilation setup we compose the build-tool's cross-target
`targetSlice` into the slice's cabal-store (so cabal recognises the
unit-id plan-nix recorded as "already installed") — but that means
the binary cabal prepends to PATH is for the cross-target arch.
ghc's `-pgmF hspec-discover` PATH lookup then resolves to the
cross-arch binary and `posix_spawnp` returns ENOEXEC ("Exec format
error") on the build host.
`withProgFlags` already passes
`--configure-option=--with-<tool>=<buildSlice>/bin/<tool>` so Setup
configure picks up the build-platform variant — but cabal v2-build's
own PATH-prepend for ghc preprocessor invocations bypasses that.
After lndir-composing the cabal-store, walk
`<storeDir>/<ghc>/*-e-<name>-*/bin/<name>` for each transitive
build-tool entry and replace the symlink with a copy of
`${buildSlice}/bin/${name}` (the build-platform binary). The
cross-target unit-id directory stays intact (so cabal's solver
still recognises it), but the executable inside is one that
actually runs on the build machine.
Native builds have an empty `buildToolBinOverlays`, so the overlay
block emits an empty string and other slices' drv hashes are
unchanged.
Confirmed:
* x86_64-linux armv7a-android cabal-22.run (was failing with
`hspec-discover: posix_spawnp: invalid argument (Exec format error)`)
* x86_64-linux musl64 githash.run, th-dlls.build
* x86_64-linux native with-packages.run
* x86_64-linux armv7a-android c-ffi.run
…nclude-dirs
cabal hashes per-package `extra-lib-dirs:` and `extra-include-dirs:`
into `pkgHashExtraLibDirs` / `pkgHashExtraIncludeDirs`. If the user
puts them in `cabalProjectLocal` to point cabal at a C library that
lives outside the standard `lib/` layout (e.g. mingw-w64 import libs
under `bin/`), plan-nix records the corresponding `--extra-lib-dirs=`
/ `--extra-include-dirs=` flags in plan.json's `configure-args` — but
v2 wasn't pulling those flags through to the slice's own
`cabal.project`, so the slice's cabal recomputed the package's
unit-id without them, diverged from plan-nix, and downstream
consumers couldn't reuse the slice.
Three pieces:
1. `modules/package-options.nix`: new per-package `extraLibDirs` and
`extraIncludeDirs` options.
2. `modules/install-plan/configure-args.nix`: extract
`--extra-lib-dirs=PATH` and `--extra-include-dirs=PATH` from
plan.json's `configure-args` into the new options.
3. `builder/comp-v2-builder.nix`: emit a `package <pkg>\n
extra-lib-dirs: ...\n extra-include-dirs: ...\n` block per
sliced package in the slice's `cabal.project`.
Also update `test/th-dlls-minimal/default.nix` to demonstrate the
pattern: spell test-clib's lib dirs out in `cabalProjectLocal` so
plan-nix's `extra-lib-dirs:` matches the slice's, AND keep
`components.library.libs = [test-clib]` so haskell.nix's module
system can resolve the `extra-libraries: test-clib` entry in
test-lib.cabal (without it, eval fails with
"The Nixpkgs package set does not contain the package: test-clib").
Confirmed:
* x86_64-linux ucrt64 th-dlls-minimal.build (was failing with
`Cabal-4345 Missing (or bad) C library: test-clib`)
* x86_64-linux musl64 githash.run, th-dlls.build
* x86_64-linux native with-packages.run
* x86_64-linux armv7a-android cabal-22.run, c-ffi.run
`qemu-arm` segfaults running `iserv-proxy-interpreter` when ghc spawns iserv for the annotations TH eval, so the build can't proceed past `Compiling Lib`. The same chain works on `aarch64-android-prebuilt` (qemu-aarch64), so the issue is specific to the 32-bit ARM qemu / iserv interaction rather than something haskell.nix can sidestep. Disable on `isAndroid && isAarch32` — same shape as the existing `th-dlls` guard.
…musl} Cross from x86_64 to aarch64-linux currently breaks the `useLocalGhcLib = true` slicing path. Disable the cabal flavour of the reinstallable-ghc-lib test on both the glibc and musl aarch64 multiplatform targets; native and other cross targets remain enabled.
Switch the project-level `builderVersion` default back to `1` (Setup.hs / comp-builder) so v1 is exercised by all targets that don't explicitly opt in to `builderVersion = 2`. Lets us confirm the branch hasn't regressed v1 while v2 work continues. Projects that want the v2 slicing builder can set `builderVersion = 2` on their project module.
`withHoogle = true` (the default) makes `shellFor` haddock-build
each package's deps so the bundled hoogle DB has docs. On
musl-static the ghc compiler tree ships no `.dyn_hi` files (only
`.hi` and `.p_hi`), and packages like `OneTuple` need
`Language.Haskell.TH.dyn_hi` at haddock-time for their TH eval —
the build fails with:
Exception when reading interface file
.../template-haskell-2.24.0.0-inplace/Language/Haskell/TH.dyn_hi:
does not exist (No such file or directory)
Mirror `test/cabal-simple`: set `withHoogle = !isStatic`. Native
and other non-static targets keep their hoogle bundle.
The test's `find -path '*/share/doc/html/Lib.html'` only matched v2's flat unit-store layout (`share/doc/html/<Module>.html`). v1 puts haddock html under `share/doc/<pkg>/html/<Module>.html`, so the test failed on the musl64 / cross targets that fall back to v1. Drop the `-path` anchor and switch to `find -name 'Lib.html'` (and `*Slib.html` for the sublib). Both layouts are now accepted.
…args `cabal.project` `debug-info: 2` becomes `--enable-debug-info` in each affected unit's `configure-args` in plan.json (cabal-install records the boolean form regardless of the level). v1's comp-builder reads `enableDWARF` per-component to swap in the `.dwarf` GHC variant and pass `-g3` — picking the flag up here lets a project's cabal.project debug-info stanza flow through to v1 without per-component module overrides. Three pieces: 1. `modules/component-options.nix`: surface `enableDWARF` as a per-component option (matches `enableProfiling`). 2. `builder/comp-builder.nix`: read the default from `component.enableDWARF` instead of hard-coding `false`. 3. `modules/install-plan/configure-args.nix`: map `--enable-debug-info` in plan.json to `enableDWARF = true`. While here, rename the `--enable-*` predicate to `hasEnableFlag` and the resulting attrset to `enableFlags` so the helper now covers debug-info too without misleading naming. Confirmed `tests.cabal-simple-debug.run` on native ghc9141 builds again with the project default `builderVersion = 1`.
The doCoverage installPhase tried
`${testExecutable}-tmp/extra-compilation-artifacts` which expands
to `dist/build/<cname>/<cname><exeExt>-tmp/...`. On Windows hosts
`exeExt = ".exe"`, so the path becomes
`dist/build/<cname>/<cname>.exe-tmp/...` — a path cabal never
creates. The fallback `cp -r dist/hpc $out/share` then errors with
"No such file or directory" and the slice fails.
cabal's per-component tmp dir is `dist/build/<cname>/<cname>-tmp`
regardless of platform, so use that path directly (and drop the
broken `${testExecutable}-tmp` case — it was only ever right when
`exeExt == ""`, which the new path also covers).
Confirmed `tests.coverage.run` on ucrt64 builds; native still passes.
This was referenced May 22, 2026
hamishmack
added a commit
that referenced
this pull request
May 23, 2026
…lity
Cabal-install reads `ghc --info` during plan elaboration to decide
per-unit settings (`--enable-shared` vs `--disable-shared`, RTS-way
inclusion, etc.) that feed into the recorded UnitId hash. The
inline `dummy-ghc` script in `lib/call-cabal-project-to-nix.nix`
was a stripped-down stub whose `--info` output didn't match the
real cross GHCs', so plan-nix recorded UnitIds that diverged from
what a downstream cabal v2-build (or any consumer running cabal
against the real compiler) would compute.
Three changes:
1. **`lib/dummy-ghc.nix`** (new): extract the dummy-ghc script and
emit cross-aware `--info` capabilities matching real cross GHCs
for windows, ghcjs, wasm, android, static, native-musl, and
native — `Support dynamic-too`, `Support shared libraries`,
`RTS ways`, `Stage`, `GHC Dynamic`, the iserv-related fields,
etc.
2. **`lib/call-cabal-project-to-nix.nix`**:
* gate the dummy-ghc-pkg `-inplace` suffix on GHC ≥ 9.8 — older
GHCs register pre-existing packages without it, so the dummy
was synthesising the wrong package ids for plan-to-nix
against ghc 8.10 / 9.0 / 9.2 / 9.4 / 9.6.
* pin `CABAL_INSTALLED_PACKAGE_ID_OS` to the build platform's
OS when running plan-to-nix. Without this the patched
cabal-install's installed-package-id format tracks the eval
system; a darwin host evaluating a linux derivation gets the
`VeryShort` form and forks UnitIds from what cabal v2-build
on a linux builder would compute.
3. **`lib/default.nix`**: tighten `uniqueWithNameKey` so
derivations (`.name = "<pkg>-<ver>"`) and module values
(`.identifier.{name,version}`) can't collide on shared name
fragments — partition into `unit-id:` / `id:` / `name:` buckets.
Correctness is unchanged when buckets are correct; this just
avoids the slow `lib.unique` fallback in more cases.
Pulled out of #2504 (`hkm/builder-v2`).
hamishmack
added a commit
that referenced
this pull request
May 23, 2026
…lity (#2509) Cabal-install reads `ghc --info` during plan elaboration to decide per-unit settings (`--enable-shared` vs `--disable-shared`, RTS-way inclusion, etc.) that feed into the recorded UnitId hash. The inline `dummy-ghc` script in `lib/call-cabal-project-to-nix.nix` was a stripped-down stub whose `--info` output didn't match the real cross GHCs', so plan-nix recorded UnitIds that diverged from what a downstream cabal v2-build (or any consumer running cabal against the real compiler) would compute. Three changes: 1. **`lib/dummy-ghc.nix`** (new): extract the dummy-ghc script and emit cross-aware `--info` capabilities matching real cross GHCs for windows, ghcjs, wasm, android, static, native-musl, and native — `Support dynamic-too`, `Support shared libraries`, `RTS ways`, `Stage`, `GHC Dynamic`, the iserv-related fields, etc. 2. **`lib/call-cabal-project-to-nix.nix`**: * gate the dummy-ghc-pkg `-inplace` suffix on GHC ≥ 9.8 — older GHCs register pre-existing packages without it, so the dummy was synthesising the wrong package ids for plan-to-nix against ghc 8.10 / 9.0 / 9.2 / 9.4 / 9.6. * pin `CABAL_INSTALLED_PACKAGE_ID_OS` to the build platform's OS when running plan-to-nix. Without this the patched cabal-install's installed-package-id format tracks the eval system; a darwin host evaluating a linux derivation gets the `VeryShort` form and forks UnitIds from what cabal v2-build on a linux builder would compute. 3. **`lib/default.nix`**: tighten `uniqueWithNameKey` so derivations (`.name = "<pkg>-<ver>"`) and module values (`.identifier.{name,version}`) can't collide on shared name fragments — partition into `unit-id:` / `id:` / `name:` buckets. Correctness is unchanged when buckets are correct; this just avoids the slow `lib.unique` fallback in more cases. Pulled out of #2504 (`hkm/builder-v2`).
hamishmack
added a commit
that referenced
this pull request
May 23, 2026
* Add `useLocalGhcLib` project option Surface what `modules/configuration-nix.nix` used to do unconditionally as an opt-in `useLocalGhcLib` flag, so the `packages.ghc.src` override only fires when a project actually constrains the `ghc` package (e.g. `ghc-lib-reinstallable`). Four pieces: * `modules/project-common.nix`: add the `useLocalGhcLib` option (default `false`). * `modules/configuration-nix.nix`: drop the unconditional `packages.ghc.src` / `packages.ghc.package-description-override` overrides — they're moved into the per-project wiring below. * `modules/stack-project.nix`: under `useLocalGhcLib`, re-apply the `packages.ghc.src` post-plan override. Stack-to-nix can't use the cabal-project route, so this keeps the existing behaviour for stack users who flip the flag. * `modules/cabal-project.nix`: under `useLocalGhcLib`, inject a `source-repository-package` block into `cabalProjectLocal` pointing at the configured-src + generated GHC tree, and add an `inputMap` entry so haskell.nix doesn't try to fetch the URL. Cabal then hashes the wrapped repo's content into `pkg-src-sha256` and installs `lib:ghc` like any other reinstallable dep. Projects that need the previous always-on behaviour now set `useLocalGhcLib = true` on the project module; everyone else gets a smaller plan-nix and avoids the unconditional `configured-src` materialisation. Pulled out of #2504 (`hkm/builder-v2`). * ghc-lib-reinstallable test + changelog: set useLocalGhcLib = true Flips the new opt-in flag in both test variants and adds a changelog entry telling users to do the same when they constrain `lib:ghc`.
hamishmack
added a commit
that referenced
this pull request
May 23, 2026
Surface the static-linking / library-for-ghci / shared-libs flags that comp-builder and `lib/check.nix` already apply at the artefact level into the plan-to-nix step, so plan-nix's recorded configure-args and UnitIds match what cabal v2-build would compute against the real compiler. Four mkIfs added to `modules/cabal-project.nix`: * **musl host** — `package * \n executable-static: True`. comp-builder adds `--ghc-option=-optl=-static` at build time; surfacing the toggle here makes plan-to-nix record `--enable-executable-static`. Build artefacts are unchanged. * **x86_64-darwin host** — `package * \n library-for-ghci: True`. Mirrors what comp-builder passes for `!ghcjs && !wasm && !android` (always true on darwin). * **android host** — `package * \n ghc-options: -optl-static -optl-ldl` (plus `-optl-no-pie` on aarch32). Mirrors `lib/check.nix`'s test-exe setupBuildFlags re-wrap. * **wasm GHC ≥ 9.12** — `package * \n shared: True`. Wasm's RTS linker only loads `.so` files; `--disable-shared` (cabal's default given the real compiler's reported capabilities) would force a `.a`-only install that TH-eval can't load. All four directives sit at `mkBefore` priority so a project's own `cabalProjectLocal` overrides them. Cache impact: plan-nix hashes change for affected platforms. Changelog entry added in `changelog.md` describing the cache-bust and the opt-out path. Pulled out of #2504 (`hkm/builder-v2`).
hamishmack
added a commit
that referenced
this pull request
May 23, 2026
…cal defaults
Two related changes to `modules/cabal-project.nix`:
**1. `cabalProjectLocal` / `cabalProjectFreeze` no longer auto-load
from disk.** The options were typed `nullOr lines` with
`readIfExists`-based defaults that did IFD lookups for
`cabal.project.local` / `cabal.project.freeze` in the project
source. Internal projects (hadrian, ghc-extra-projects) explicitly
set them to `null` just to suppress those reads, and the nullability
also prevented `mkBefore` directives from merging cleanly.
Switch the types to plain `lines` (default `""`) — projects that
relied on the implicit `readFile` behaviour now do it explicitly:
cabalProjectLocal = builtins.readFile ./cabal.project.local;
Internal callers that set `null` for IFD-avoidance lose the
explicit assignments — the new default is already IFD-free.
**2. Platform-conditional defaults are now injected into every
cabal project's `cabalProjectLocal`.** Four `mkIf` blocks added:
* **musl host** — `package * \n executable-static: True`.
comp-builder adds `--ghc-option=-optl=-static` at build time;
surfacing the toggle here makes plan-to-nix record
`--enable-executable-static`. Build artefacts are unchanged.
* **x86_64-darwin host** — `package * \n library-for-ghci: True`.
Mirrors what comp-builder passes for `!ghcjs && !wasm && !android`
(always true on darwin).
* **android host** —
`package * \n ghc-options: -optl-static -optl-ldl` (plus
`-optl-no-pie` on aarch32). Mirrors `lib/check.nix`'s
test-exe `setupBuildFlags` re-wrap.
* **wasm GHC ≥ 9.12** — `package * \n shared: True`. Wasm's RTS
linker only loads `.so` files; `--disable-shared` (cabal's
default given the real compiler's reported capabilities) would
force a `.a`-only install that TH-eval can't load.
All four sit at `mkBefore` priority so a project's own
`cabalProjectLocal` overrides them.
Cache impact: plan-nix hashes change for affected platforms.
Internal GHC builds verified byte-identical (drvPath
`na9chax1gj2n6jx3d6v86v6fdah9r9fd-ghc-9.14.1.drv` matches master).
Changelog entry added describing both changes.
Pulled out of #2504 (`hkm/builder-v2`).
hamishmack
added a commit
that referenced
this pull request
May 23, 2026
…cal defaults (#2511) Two related changes to `modules/cabal-project.nix`: **1. `cabalProjectLocal` / `cabalProjectFreeze` no longer auto-load from disk.** The options were typed `nullOr lines` with `readIfExists`-based defaults that did IFD lookups for `cabal.project.local` / `cabal.project.freeze` in the project source. Internal projects (hadrian, ghc-extra-projects) explicitly set them to `null` just to suppress those reads, and the nullability also prevented `mkBefore` directives from merging cleanly. Switch the types to plain `lines` (default `""`) — projects that relied on the implicit `readFile` behaviour now do it explicitly: cabalProjectLocal = builtins.readFile ./cabal.project.local; Internal callers that set `null` for IFD-avoidance lose the explicit assignments — the new default is already IFD-free. **2. Platform-conditional defaults are now injected into every cabal project's `cabalProjectLocal`.** Four `mkIf` blocks added: * **musl host** — `package * \n executable-static: True`. comp-builder adds `--ghc-option=-optl=-static` at build time; surfacing the toggle here makes plan-to-nix record `--enable-executable-static`. Build artefacts are unchanged. * **x86_64-darwin host** — `package * \n library-for-ghci: True`. Mirrors what comp-builder passes for `!ghcjs && !wasm && !android` (always true on darwin). * **android host** — `package * \n ghc-options: -optl-static -optl-ldl` (plus `-optl-no-pie` on aarch32). Mirrors `lib/check.nix`'s test-exe `setupBuildFlags` re-wrap. * **wasm GHC ≥ 9.12** — `package * \n shared: True`. Wasm's RTS linker only loads `.so` files; `--disable-shared` (cabal's default given the real compiler's reported capabilities) would force a `.a`-only install that TH-eval can't load. All four sit at `mkBefore` priority so a project's own `cabalProjectLocal` overrides them. Cache impact: plan-nix hashes change for affected platforms. Internal GHC builds verified byte-identical (drvPath `na9chax1gj2n6jx3d6v86v6fdah9r9fd-ghc-9.14.1.drv` matches master). Changelog entry added describing both changes. Pulled out of #2504 (`hkm/builder-v2`).
hamishmack
added a commit
that referenced
this pull request
May 24, 2026
Bake `--optimistic-linking` into iserv-proxy / iserv-proxy-interpreter at link time via `-with-rtsopts`. `GHC/Linker/Executable.hs` emits this into the generated `main.c` as `__conf.rts_opts`, which `setupRtsFlags` processes with `RtsOptsAll` — bypassing the `OPTION_UNSAFE` gate that `+RTS --optimistic-linking -RTS` on the command line is subject to. Makes the runtime linker tolerate undefined symbols when loading object files at TH-eval time; splices that don't actually reference the missing symbol then resolve fine instead of aborting the load. `-rtsopts=all` is kept so wrapper scripts / GHCRTS can still override at invocation. `--optimistic-linking` is only available in GHC's RTS from 9.14 onwards; gated on the Nix side since cabal.project doesn't allow `if` inside a `package` stanza. Refactor `cabalProjectLocal` for the iserv-proxy project from `//`-chained `optionalAttrs` blocks to a single string built via `optionalString` concatenation so the new directive can compose with the existing aarch64+<9.8 threaded gate without the last `//` clobbering it. Also drop the previous `allow-newer: *:base, *:bytestring` block on GHC > 9.10. The bounds in iserv-proxy.cabal (`base < 5`, `bytestring < 0.13`, etc.) already accommodate current GHCs; if a transitive package needs relaxation we'll re-add a more targeted allow-newer once we know which one. Pulled out of #2504 (`hkm/builder-v2`).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the v2 component builder alongside v1 and a project-level
builderVersionswitch that picks between them. Where v1 runsSetup.hs configure / buildper component, v2 runscabal v2-buildagainst a per-slice slicing repo and pre-composed starting store, registers each new unit under$out/store/ghc-<ver>[-inplace]/..., and emits a plan-entry diff against plan-nix when something diverges.Highlights:
builder/comp-v2-builder.nix— per-component slice builder. Drives a per-slice cabal.project (constraints pinned from plan-nix'sdepends,extra-packages:listing every transitive dep, per-pkgflags:/ghc-options:/configure-options:blocks) and a per-slice slicing repo of source tarballs and a00-index.tar.gzso cabal's solver hits a closed package set. Hands the cabal.project plus the slice's direct dep slices off tobuild-cabal-slice.nixto actually run cabal.builder/build-cabal-slice.nix— generic runner the per- component builder calls into. Composes the starting store vialndirfrom every dep slice (walkspkgsHostTargetfor direct deps and follows each entry's$out/nix-support/transitive-depsfor the rest of the closure, so transitive dep slices stay out of nix-sidebuildInputs), runscabal v2-build, captures every newly installed unit back into$out/store, writes the slice's ownnix-support/transitive-depsfor downstream consumers, and ondryRunOnlyemits a per-entry diff between cabal's plannedplan.jsonand plan-nix's expectations.builder/compose-store.nix—lndir-based store composition with the same direct + transitive-deps walk. Used bycomp-v2-builderfor thepassthru.storeexposed on each slice and byshell-for-v2.nixfor the dev shell's composed dep store.builder/cabal-install-patches/{prune-unreachable-sublibs*, skip-installed-revdeps-in-completed}.patch— three cabal-install patches the v2 slice solver needs to keep its plan in sync with plan-nix.builder/shell-for-v2.nix— v2 dev shell. Resolves each selected component's direct externalcomponent.dependsto its library (viahaskellLib.dependToLib, so sublib references resolve to their sublib slices), composes those into a starting cabal store viacomposeStore(the transitive closure is materialised at build time from each slice'snix-support/transitive-deps), exposes the result via~/.cabal/store/<ghc>-inplace/(the defaultexposePackagesVia = "cabal-store") or via aGHC_ENVIRONMENT-wrapped ghc (exposePackagesVia = "ghc-pkg"), and mergespassthru.depSlicesfrominputsFromso cross + native shells both see each other's slices.builder/{default,hspkg-builder}.nix— dispatch onbuilderVersion: the per-component derivation athsPkgs.<pkg>.components.<kind>.<n>is whichever builder the project selects (no per-component opt-in).hspkg-builder.nixalso resolveshomeDependIdsand thepackageRefersOwnExeflag fromconfig.plan-json-by-id(the project-level index) so these per-component lookups are O(log N) attrset hits instead of linear scans over the install-plan.modules/{component-driver,project-common,shell}.nix—builderVersion,crossTemplateHaskellSupport,exposePackagesViaoptions.modules/install-plan/configure-args.nix— filter--ghc-option=-hide-all-packagesout ofghcOptionsfor the v2 builder (cabal injects it on every Setup configure call, and round-tripping it through cabal.project would duplicate it inpkgHashProgramArgs); extract--configure-option=...entries into a new dedicatedpackage.configureOptionsoption (declared inmodules/package-options.nix) so the v2 builder can emit them back into its own cabal.project.modules/install-plan/non-reinstallable.nix— gate the ghcjs/wasmghci-relatednon-reinstallableset onbuilderVersion != 2; under v2 the slice solver follows cabal's install plan literally.lib/check.nix—passthru.isSlicebranch that runs the slice-built exe directly instead of going through v1'ssetup testpipeline.lib/default.nix—uniqueWithNameKeyprefersidentifier.unit-idwhen present so twofoo-1.2.3slices with different inputs partition into different buckets.lib/call-cabal-project-to-nix.nix— setCABAL_INSTALLED_PACKAGE_ID_OS = pkgs.stdenv.buildPlatform.parsed.kernel.name;on the plan-to-nix derivation so the patched cabal-install (nix-tools: apply cabal-install unit-id-OS-override patch to regular b… #2501) actually fires; without the env var the patch is a no-op.compiler/ghc/default.nix,overlays/{bootstrap,tools}.nix— pin GHC,hscolour, and the boot tools tobuilderVersion = 1so v2 churn doesn't trigger GHC rebuilds while v2 is still settling.overlays/{windows,mingw_w64}.nix— v2-builder-aware Windows-cross TH wrapper:mingw_w64.nixexposeswrapGhc :: ghc -> ghcandiservRuntimeLibs;overlays/windows.nixplumbs the build-platformrunCommand/makeWrapper/libffithrough to that module.overlays/haskell.nix— project-level shell dispatch (shellForV1/shellForV2),crossTemplateHaskellSupportplumbing, andcabalProjectLocalrefactored so the Windows-host iserv-proxy linker flags can be additive.test/cabal-sublib-shell/— new test exercising a sublib consumer in the v2 dev shell. Other test wiring tweaks (test/{th-dlls,th-dlls-minimal,cabal-sublib}/default.nix,test/cabal.project.local,test/{setup-deps, shell-for-setup-deps}/pkg/pkg.cabal,test/default.nix) to keep things passing under v2.ci.nix— pinevalSystemdefault tox86_64-linuxso cross derivations evaluate consistently.docs/dev/builder-v{1,2}.md(+docs/SUMMARY.mdentry) — design notes for the two builders.Performance notes:
transitiveTarballsis exposed as the deduped values ofdepTarballsDeduped. The un-deduped concat (which sticks every direct dep's full transitive list onto each consumer unfiltered) grows exponentially with depth in a typical Haskell graph (every package sharesbase/bytestring/text/ ...) and materialises multi-GiB blocks in the evaluator on cardano-wallet-sized projects.homeDepSliceOf/depTransitiveTarballsOffall through to any sublib's slice when a dep package defines onlylibrary <name>stanzas (no main library — e.g.cardano-wallet-ui:{common,shelley}).propagatedkeepslibsWindows-only (matching v1): cross-target slices land in the v2 dev shell'sinputsFrom, and unconditionally propagatinglibswould drag e.g.pkgs.liburingfrom a Linux-target slice into a Darwin consumer'spkgsHostTarget. Each slice gathers its own transitive sysLibs (deduped) intoextraBuildInputsviatransitiveDepLibsinstead.