Skip to content

feat(server): non-interactive bundle attestation for /v1/bundle#1891

Open
lockwobr wants to merge 2 commits into
mainfrom
feat/server-bundle-attestation
Open

feat(server): non-interactive bundle attestation for /v1/bundle#1891
lockwobr wants to merge 2 commits into
mainfrom
feat/server-bundle-attestation

Conversation

@lockwobr

Copy link
Copy Markdown
Contributor

Summary

POST /v1/bundle?attest=true now returns a signed bundle using a non-interactive, server-owned identity (KMS or private-Sigstore keyless), with tool provenance from the aicrd binary attestation shipped in the image. The server never signs with request-supplied identity.

Motivation / Context

The /v1/bundle endpoint never attested or signed bundles: the handler built BundleOptions with no Attester, so a CI pipeline or platform bundling via the API could not get a signed artifact back. This closes the server-side gap in the supply-chain epic. The CLI's default signing path is interactive browser OIDC, a non-starter for a daemon, so this adds non-interactive server signing.

Fixes: #1150
Related: #1149

Type of Change

  • New feature (non-breaking change that adds functionality)
  • Build/CI/tooling

Component(s) Affected

  • API server (cmd/aicrd, pkg/server)
  • Bundlers (pkg/bundler, pkg/component/*)
  • Core libraries (pkg/errors, pkg/k8s) — pkg/client/v1, pkg/defaults
  • Docs/examples (docs/, examples/)
  • Other: .goreleaser.yaml, new server-kms-e2e workflow, chainsaw suite

Implementation Notes

Trust boundary: the server signs as itself. Signing identity comes entirely from server env config, never from the request; ?attest=true is only a boolean trigger. Request-supplied / on-behalf-of-caller OIDC is explicitly out of scope.

Two signing modes (mutually exclusive, fail-fast validated at startup):

  • KMS: AICR_SIGNING_KEY (awskms:// / gcpkms:// / azurekms:// / hashivault://).
  • Keyless against a private Sigstore: AICR_FULCIO_URL + a token source (AICR_IDENTITY_TOKEN_FILE, read fresh per request, or GHA ambient). Shared: AICR_REKOR_URL, AICR_TLOG_UPLOAD (KMS-only), AICR_SIGNING_CONFIG_PATH.

Tool provenance (the aicrd binary attestation): the release build (.goreleaser.yaml) now runs cosign attest-blob on the aicrd binary and ships the result per-architecture in the ko image via KO_DATA_PATH. The server verifies its own attestation once at startup, caches it, and embeds it in each signed bundle. It fails to start if signing is enabled but the attestation is missing or invalid. The attestation path resolves via AICR_BINARY_ATTESTATION_FILE, then KO_DATA_PATH/GOARCH, then next-to-binary; the identity pin is overridable (still org-pinned) via AICR_BINARY_ATTESTATION_IDENTITY_REGEXP, mirroring the CLI's --certificate-identity-regexp.

Additive, CLI-safe: the bundler gains WithVerifiedBinaryAttestation and the client facade gains BundleOptions.BinaryAttestation. With neither set (the CLI path), behavior is byte-identical: no attest is inferred from Attester != nil, and the discover-and-verify-per-run path is unchanged.

Testing

make test    # -race, all packages
make lint    # golangci-lint + yamllint
go test -tags integration ./pkg/server/ -run TestServerSigning   # vs real OpenBAO KMS
  • make test green on the changed packages; make lint 0 issues; golangci-lint (incl. -tags integration) 0 issues.
  • Go integration test (//go:build integration, skips without VAULT_ADDR) verified live against a real OpenBAO Transit KMS: bundle returned signed, bundleAttested=true.
  • New chainsaw suite tests/chainsaw/signing/server-bundle-attestation-vault/ smoke mode passed live (server rejects attest=true when unconfigured with 400; unsigned bundle still returned). The full attested path runs in the new server-kms-e2e.yaml workflow (needs GitHub OIDC to attest the binary), the same CI-only limitation as the existing signing e2es.
  • The integration test is build-tagged out of make test, and the chainsaw suite has no chainsaw-test.yaml, so neither is auto-run by make test / make e2e / make qualify; they do not affect local dev.

Note: tests/releasepolicy fails on local macOS due to a pre-existing environment gap (GNU timeout absent in publish-homebrew.sh), unrelated to this change; it fails identically on main and passes in CI. Tracked for a separate portability fix.

Risk Assessment

  • Medium — Touches the shared bundler signing path, but strictly additively (CLI path byte-identical), and server signing is opt-in and off unless configured.

Rollout notes: Server signing is disabled by default (no env config → capability off, current behavior preserved). Turning it on in a real deployment depends on the aicrd binary attestation being present in the image, which this PR wires into the release build. No migration.

Checklist

  • Tests pass locally (make test with -race) — modulo the pre-existing environmental releasepolicy gap noted above
  • Linter passes (make lint)
  • I did not skip/disable tests to make CI green
  • I added/updated tests for new functionality
  • I updated docs if user-facing behavior changed
  • Changes follow existing patterns in the codebase
  • Commits are cryptographically signed (git commit -S)

@lockwobr
lockwobr requested review from a team as code owners July 23, 2026 16:02
@lockwobr lockwobr added the theme/supply-chain SLSA, SBOM, Sigstore, and provenance verification label Jul 23, 2026
@lockwobr lockwobr self-assigned this Jul 23, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Recipe evidence check

No leaf overlays affected by this PR.

This gate is warning-only and never blocks merge.

@github-actions

github-actions Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Coverage Report ✅

Metric Value
Coverage 80.7%
Threshold 80%
Status Pass
Coverage Badge
![Coverage](https://img.shields.io/badge/coverage-80.7%25-brightgreen)

Merging this branch changes the coverage (2 decrease, 2 increase)

Impacted Packages Coverage Δ 🤖
github.com/NVIDIA/aicr/pkg/bundler 81.08% (+1.20%) 👍
github.com/NVIDIA/aicr/pkg/bundler/verifier 75.50% (-0.08%) 👎
github.com/NVIDIA/aicr/pkg/client/v1 77.54% (+0.18%) 👍
github.com/NVIDIA/aicr/pkg/defaults 100.00% (ø)
github.com/NVIDIA/aicr/pkg/server 78.66% (-0.53%) 👎

Coverage by file

Changed files (no unit tests)

Changed File Coverage Δ Total Covered Missed 🤖
github.com/NVIDIA/aicr/pkg/bundler/bundler.go 81.81% (+1.55%) 786 (+11) 643 (+21) 143 (-10) 👍
github.com/NVIDIA/aicr/pkg/bundler/verifier/verifier.go 74.00% (-0.07%) 300 (+3) 222 (+2) 78 (+1) 👎
github.com/NVIDIA/aicr/pkg/client/v1/bundle.go 76.19% (+2.42%) 63 (+2) 48 (+3) 15 (-1) 👍
github.com/NVIDIA/aicr/pkg/defaults/timeouts.go 0.00% (ø) 0 0 0
github.com/NVIDIA/aicr/pkg/server/bundle_handler.go 75.56% (+17.49%) 90 (+28) 68 (+32) 22 (-4) 🎉
github.com/NVIDIA/aicr/pkg/server/serve.go 0.00% (ø) 30 (+7) 0 30 (+7)
github.com/NVIDIA/aicr/pkg/server/signing.go 71.11% (+71.11%) 90 (+90) 64 (+64) 26 (+26) 🌟

Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code.

@coderabbitai

coderabbitai Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Adds server-side bundle attestation through POST /v1/bundle?attest=true, supporting KMS and keyless signing modes. Signing configuration is validated at startup, and the server verifies and caches its own binary attestation for embedding in signed bundles. Bundler and client APIs accept verified attestation bytes. GoReleaser generates binary attestations, while new CI and OpenBAO-based tests validate signed and unsigned server flows.

Estimated code review effort: 5 (Critical) | ~120 minutes

Suggested labels: theme/ci-dx

Suggested reviewers: mchmarny

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: non-interactive bundle attestation in the server API.
Description check ✅ Passed The description matches the changeset and explains the server-side attestation feature, signing modes, and supporting updates.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/server-bundle-attestation

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/server/bundle_handler.go`:
- Around line 203-225: Add an authorization check before the attestation setup
in the attestRequested branch, requiring an authorized principal or approved
recipe/source policy before constructing the attester or signing the bundle.
Reject unauthorized requests with the existing error-response mechanism, and
preserve the current signing flow only for requests that pass this authorization
check.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: b3245382-984d-4413-8951-497bb1cc31c6

📥 Commits

Reviewing files that changed from the base of the PR and between 091edf8 and 8f45f19.

📒 Files selected for processing (19)
  • .github/workflows/server-kms-e2e.yaml
  • .gitignore
  • .goreleaser.yaml
  • api/aicr/v1/server.yaml
  • docs/contributor/api-server.md
  • docs/user/api-reference.md
  • pkg/bundler/bundler.go
  • pkg/bundler/bundler_test.go
  • pkg/client/v1/bundle.go
  • pkg/client/v1/bundle_test.go
  • pkg/defaults/timeouts.go
  • pkg/server/bundle_handler.go
  • pkg/server/bundle_handler_test.go
  • pkg/server/serve.go
  • pkg/server/signing.go
  • pkg/server/signing_integration_test.go
  • pkg/server/signing_test.go
  • tests/chainsaw/signing/server-bundle-attestation-vault/README.md
  • tests/chainsaw/signing/server-bundle-attestation-vault/run.sh

Comment thread pkg/server/bundle_handler.go
@lockwobr
lockwobr force-pushed the feat/server-bundle-attestation branch from 8f45f19 to 78c7d7e Compare July 23, 2026 16:12

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/server/serve.go`:
- Around line 111-114: Update the parseSigningConfig error path in serve to
return the existing err directly instead of wrapping it with errors.Wrap.
Preserve the structured error code and message produced by parseSigningConfig.

In `@pkg/server/signing.go`:
- Around line 66-80: Move the parseTLogUpload call out of the unconditional
start of parseSigningConfig and into the KMS signing branch guarded by the
existing hasKMS check. Preserve error propagation when KMS mode is enabled,
while ignoring malformed or stray AICR_TLOG_UPLOAD values when no signing mode
is configured or keyless mode is selected.
- Around line 221-274: Update loadBinaryAttestation to use
aicrerrors.PropagateOrWrap when handling errors returned by verify(ctx),
preserving existing structured error codes while defaulting uncoded failures to
ErrCodeInternal with the current context message.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 451ed38b-bd88-4855-bbb8-266fa12b404a

📥 Commits

Reviewing files that changed from the base of the PR and between 8f45f19 and 78c7d7e.

📒 Files selected for processing (19)
  • .github/workflows/server-kms-e2e.yaml
  • .gitignore
  • .goreleaser.yaml
  • api/aicr/v1/server.yaml
  • docs/contributor/api-server.md
  • docs/user/api-reference.md
  • pkg/bundler/bundler.go
  • pkg/bundler/bundler_test.go
  • pkg/client/v1/bundle.go
  • pkg/client/v1/bundle_test.go
  • pkg/defaults/timeouts.go
  • pkg/server/bundle_handler.go
  • pkg/server/bundle_handler_test.go
  • pkg/server/serve.go
  • pkg/server/signing.go
  • pkg/server/signing_integration_test.go
  • pkg/server/signing_test.go
  • tests/chainsaw/signing/server-bundle-attestation-vault/README.md
  • tests/chainsaw/signing/server-bundle-attestation-vault/run.sh

Comment thread pkg/server/serve.go
Comment thread pkg/server/signing.go
Comment thread pkg/server/signing.go
@lockwobr
lockwobr force-pushed the feat/server-bundle-attestation branch 2 times, most recently from c621912 to 20a021c Compare July 23, 2026 16:45

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.goreleaser.yaml:
- Around line 112-137: Extract the duplicated cosign attest-blob retry and
backoff loop from the aicr and aicrd post-build hooks into a shared script such
as hack/cosign-attest-blob.sh that accepts the bundle path and required signing
inputs. Update both hooks to invoke the shared script while preserving their
distinct bundle destinations and aicrd kodata staging behavior.

In `@pkg/server/signing.go`:
- Around line 227-262: Update defaultBinaryAttestationVerifier to read
attestPath once into a bounded buffer, then reuse that exact buffer for both
verifier.VerifyBinaryAttestation and the returned cached data. Remove the
separate os.Open/io.ReadAll path after verification while preserving the
existing size-limit and error handling behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: c24126c7-7fde-4ba2-ae7c-a548451ff56e

📥 Commits

Reviewing files that changed from the base of the PR and between c621912 and 20a021c.

📒 Files selected for processing (20)
  • .github/workflows/server-kms-e2e.yaml
  • .gitignore
  • .goreleaser.yaml
  • THIRD_PARTY_NOTICES.md
  • api/aicr/v1/server.yaml
  • docs/contributor/api-server.md
  • docs/user/api-reference.md
  • pkg/bundler/bundler.go
  • pkg/bundler/bundler_test.go
  • pkg/client/v1/bundle.go
  • pkg/client/v1/bundle_test.go
  • pkg/defaults/timeouts.go
  • pkg/server/bundle_handler.go
  • pkg/server/bundle_handler_test.go
  • pkg/server/serve.go
  • pkg/server/signing.go
  • pkg/server/signing_integration_test.go
  • pkg/server/signing_test.go
  • tests/chainsaw/signing/server-bundle-attestation-vault/README.md
  • tests/chainsaw/signing/server-bundle-attestation-vault/run.sh

Comment thread .goreleaser.yaml
Comment thread pkg/server/signing.go
@lockwobr
lockwobr force-pushed the feat/server-bundle-attestation branch from 20a021c to 9ff88b0 Compare July 23, 2026 17:10
@lockwobr
lockwobr enabled auto-merge (squash) July 23, 2026 17:31

@mchmarny mchmarny left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CHANGES REQUESTED — 2 merge blockers. Reviewed server signing, binary-provenance staging, handler and client wiring, and the KMS E2E at the exact head. Focused race tests, affected-package lint, shell and workflow validation all passed, and the current live E2E executed full mode; the two fail-open paths identified inline still need correction.

Comment thread pkg/server/signing.go
Comment thread .github/workflows/server-kms-e2e.yaml
Server-side signing for POST /v1/bundle so a bundle can be returned
signed via a non-interactive identity, closing the server gap in the
supply-chain epic (#1149).

- Server signs as itself from env config, never with request-supplied
  identity. Two modes, fail-fast validated at startup: KMS
  (AICR_SIGNING_KEY) and keyless against a private Sigstore
  (AICR_FULCIO_URL + token source). ?attest=true opts a request in.
- Tool provenance: the aicrd binary attestation is produced by the
  release build (goreleaser cosign attest-blob) and shipped per-arch in
  the image via ko kodata; the server verifies it once at startup,
  caches it, and embeds it per bundle. Path resolves via
  AICR_BINARY_ATTESTATION_FILE, KO_DATA_PATH/GOARCH, or next-to-binary;
  identity pin overridable (org-pinned) via
  AICR_BINARY_ATTESTATION_IDENTITY_REGEXP.
- Additive bundler seam (WithVerifiedBinaryAttestation) + client facade
  (BundleOptions.BinaryAttestation); CLI path byte-identical.
- E2E: Go integration test (real OpenBAO KMS, verified live) + chainsaw
  server-signing suite + server-kms-e2e CI workflow.
- OpenAPI + user/contributor docs updated.
- Regenerate THIRD_PARTY_NOTICES.md, which was left stale on main by the
  direct-push go dep upgrade (091edf8) that skipped `make notices`;
  folded in here to unblock the notices-freshness gate.

Fixes #1150
@lockwobr
lockwobr force-pushed the feat/server-bundle-attestation branch from 8592c7d to 5e0d8a9 Compare July 23, 2026 22:26
@lockwobr
lockwobr requested a review from mchmarny July 23, 2026 22:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(server): bundle attestation for /v1/bundle (non-interactive signing)

2 participants