Add additive corporate-CA support: APM_EXTRA_CA_BUNDLE (npm NODE_EXTRA_CA_CERTS parity)

[danielmeppiel]

Context

Fast-follow from the #2005 / #2004 OS-trust-store work. A red-team review (enterprise-proxy + package-manager-precedent panels) flagged that APM has no additive CA path: REQUESTS_CA_BUNDLE / CURL_CA_BUNDLE replace the trust set (and skip truststore injection entirely). Many enterprises hand developers a single corporate-root PEM but do not permit OS trust-store edits, so they need "OS roots plus this one extra CA", not "only this CA".

This is the standard npm model (NODE_EXTRA_CA_CERTS, additive) and Node's --use-system-ca (system + bundled + extra). APM lacks the equivalent.

Proposal

Add an additive APM_EXTRA_CA_BUNDLE (and optionally APM_EXTRA_CA_DIR) that is layered on top of the OS trust store rather than replacing it:

• Implement via a scoped SSLContext that first loads the OS/certifi defaults, then load_verify_locations(APM_EXTRA_CA_BUNDLE) — OS/certifi defaults remain intact.
• Propagate to apm run child runtimes through the same build_child_tls_env() seam introduced in feat(tls): verify against the OS trust store by default (closes #2004) #2005 (export the extra bundle so Python and Node children both honor it, e.g. NODE_EXTRA_CA_CERTS for Node runtimes).
• Precedence must be documented explicitly alongside the existing REQUESTS_CA_BUNDLE (replace) and APM_DISABLE_TRUSTSTORE (opt-out) knobs.

Acceptance

• An enterprise PEM set via APM_EXTRA_CA_BUNDLE is trusted in addition to the OS store for both apm install and apm run.
• e2e test: private-CA server trusted via APM_EXTRA_CA_BUNDLE while public HTTPS (OS/certifi roots) still verifies.

Deferred deliberately from #2005 to avoid half-shipping (Node-only) the additive path.

[github-actions]

Triage decision

accept

Proposed labels

theme/security
area/enterprise
area/docs-site
type/feature
status/accepted
priority/high

Milestone

null -- No open milestone matching the v0.24.x release series was found. This is a priority/high enterprise feature; maintainer should assign to the appropriate next-minor milestone.

Suggested next action

Draft the implementation plan for the SSLContext builder and child-runtime propagation strategy (especially how Python child processes via requests will honor the extra CA without the REQUESTS_CA_BUNDLE replacement footgun) before opening a PR.

Suggested issue comment

This is accepted -- the REQUESTS_CA_BUNDLE footgun (replaces the trust set, breaks public HTTPS for
corporate CA users) is a real enterprise blocker, and NODE_EXTRA_CA_CERTS parity is the right model.
The additive APM_EXTRA_CA_BUNDLE approach is the correct design.

A few implementation notes worth settling before the PR opens:

1. **Parent process**: SSLContext with load_verify_locations(APM_EXTRA_CA_BUNDLE) on top of OS/certifi
   defaults is the right Python primitive.
2. **Node children via apm run**: NODE_EXTRA_CA_CERTS export via build_child_tls_env() is
   straightforward.
3. **Python children via apm run**: `requests` does not use SSLContext directly -- the cleanest
   propagation is writing a merged PEM bundle to a temp file and setting REQUESTS_CA_BUNDLE to it,
   which avoids the replacement footgun for children too. This detail is worth capturing in the PR
   description.

The acceptance criteria (e2e test: private-CA server trusted via APM_EXTRA_CA_BUNDLE while public
HTTPS on OS/certifi roots still verifies) are concrete and testable. Documenting the precedence
(APM_EXTRA_CA_BUNDLE additive / REQUESTS_CA_BUNDLE replace / APM_DISABLE_TRUSTSTORE opt-out) in
enterprise/security.md is a required part of the implementing PR.

Per-lens notes (collapsed)

DevX UX Expert -- User-Need Reviewer

The current state -- REQUESTS_CA_BUNDLE replaces the trust set -- is a genuine enterprise UX failure:
developers must choose between trusting their corporate CA and trusting the public internet. APM_EXTRA_CA_BUNDLE
resolves this by layering on top of OS/certifi defaults, matching the NODE_EXTRA_CA_CERTS mental model
that Node.js ecosystem developers already know. The naming is clear, the behavior is predictable, and
the acceptance criteria (e2e test) are concrete. Documenting the precedence table (additive vs replace
vs opt-out) is essential -- enterprise developers need to reason about the behavior without trial and
error.

Supply Chain Security Expert -- Risk-Surface Reviewer

This touches the TLS trust surface. Key security notes:
(1) The additive approach is correct and safe -- load_verify_locations() extends, not replaces, the
trust set. No weakening of existing verification.
(2) APM_EXTRA_CA_BUNDLE path must be validated at startup: file must exist and be a readable PEM
bundle; no path traversal should be possible.
(3) Child runtime propagation requires care: exporting NODE_EXTRA_CA_CERTS for Node children is safe
(additive by design). For Python children using requests, propagating a merged PEM bundle via
REQUESTS_CA_BUNDLE avoids the replacement footgun but introduces a temp-file lifecycle question.
(4) Precedence documentation is a security-auditor requirement: enterprises need to understand which
knob wins when multiple CA env vars are set.
theme/security applies (TLS trust configuration). area/enterprise is the primary area.

OSS Growth Hacker -- Contributor-Tone Reviewer

Not activated -- danielmeppiel is the site admin and project maintainer; no first-time contributor tone
adjustment is needed for this issue.

Python Architect -- Architecture Reviewer

Implementation requires: (1) A new TLS utility (likely in utils/tls.py or extending existing TLS
helpers) that builds an SSLContext loading OS/certifi defaults + APM_EXTRA_CA_BUNDLE. (2) All HTTP
requests.Session creation points must use the custom SSLContext (via requests.adapters.HTTPAdapter
with a custom ssl_context). (3) build_child_tls_env() (introduced in #2005) extended to export
NODE_EXTRA_CA_CERTS for Node children. (4) For Python children that use requests, the cleanest
propagation is writing a merged PEM bundle to a temp file and setting REQUESTS_CA_BUNDLE -- this
avoids the replacement footgun but adds a temp-file lifecycle concern. The issue is correctly scoped
and the architectural seam (#2005) exists. No status/needs-design needed for the happy path;
the Python-child propagation detail is worth capturing in the PR description before coding.

Doc Writer -- Documentation Reviewer

The implementing PR needs to update docs/src/content/docs/enterprise/security.md to include:
(1) APM_EXTRA_CA_BUNDLE in the environment variable reference alongside REQUESTS_CA_BUNDLE and
APM_DISABLE_TRUSTSTORE, with a clear precedence table (additive vs replace vs opt-out).
(2) A note on how APM_EXTRA_CA_BUNDLE propagates to child runtimes via apm run (Node children:
NODE_EXTRA_CA_CERTS; Python children: merged bundle). area/docs-site is warranted as a label since
the security model documentation requires a substantive addition. The suggested comment wording
correctly flags the Python-child propagation detail as something to capture before the PR opens.

APM CEO -- Triage Arbiter

Well-scoped enterprise feature from the maintainer, filling a real gap: REQUESTS_CA_BUNDLE-as-replace
is a known footgun that blocks enterprise adoption for corporate CA customers. NODE_EXTRA_CA_CERTS
parity is a competitive-positioning win (APM matches Node.js behavior on a table-stakes enterprise
requirement). The theme/security label is correct (TLS trust configuration). area/enterprise is
the primary area; area/docs-site rides because the security model documentation requires a
substantive update. priority/high reflects the enterprise-blocker status. The Python-child
propagation detail (how to avoid the REQUESTS_CA_BUNDLE replacement footgun for children) is worth
settling in an implementation note before the PR opens -- but it is not a design blocker for accepting
the issue. Decision: accept. Milestone: null -- no open milestone for v0.24.x; maintainer assigns.

{
  "decision": "accept",
  "decision_detail": "",
  "theme": "theme/security",
  "areas": ["area/enterprise", "area/docs-site"],
  "type": "type/feature",
  "status": "status/accepted",
  "priority": "priority/high",
  "preserved_labels": [],
  "milestone": null,
  "next_action": "Draft the implementation plan for the SSLContext builder and Python-child propagation strategy (merged PEM temp bundle vs REQUESTS_CA_BUNDLE) before opening a PR.",
  "comment_markdown": "This is accepted -- the REQUESTS_CA_BUNDLE footgun (replaces the trust set, breaks public HTTPS for corporate CA users) is a real enterprise blocker, and NODE_EXTRA_CA_CERTS parity is the right model. The additive APM_EXTRA_CA_BUNDLE approach is the correct design.\n\nA few implementation notes worth settling before the PR opens:\n\n1. Parent process: SSLContext with load_verify_locations(APM_EXTRA_CA_BUNDLE) on top of OS/certifi defaults is the right Python primitive.\n2. Node children via apm run: NODE_EXTRA_CA_CERTS export via build_child_tls_env() is straightforward.\n3. Python children via apm run: requests does not use SSLContext directly -- the cleanest propagation is writing a merged PEM bundle to a temp file and setting REQUESTS_CA_BUNDLE to it, which avoids the replacement footgun for children too.\n\nThe acceptance criteria (e2e test: private-CA server trusted via APM_EXTRA_CA_BUNDLE while public HTTPS on OS/certifi roots still verifies) are concrete and testable. Documenting the precedence (APM_EXTRA_CA_BUNDLE additive / REQUESTS_CA_BUNDLE replace / APM_DISABLE_TRUSTSTORE opt-out) in enterprise/security.md is a required part of the implementing PR."
}

---

Triage status: agentic proposal pending human ratification.
Silence is approval. Maintainers can:

• Override any label or milestone above by editing it directly --
  human edits are authoritative and will not be reverted on
  subsequent runs.
• Re-trigger triage by applying the status/needs-triage label, or
  by removing status/triaged to enroll the issue in the next
  daily sweep.

Posted by the Triage Panel workflow. See .apm/skills/apm-triage-panel for the panel skill.

Generated by Triage Panel · 129.2 AIC · ⌖ 22.4 AIC · ⊞ 9.2K · ◷

[TameTheGame]

Implementation plan before opening the PR:

1. Keep src/apm_cli/core/tls_trust.py as the single trust-policy owner. Validate a non-empty APM_EXTRA_CA_BUNDLE as a readable, regular, parseable PEM before any HTTP module is imported. Invalid input will fail closed with an actionable error; verification is never disabled.
2. Preserve explicit precedence: APM_DISABLE_TRUSTSTORE > REQUESTS_CA_BUNDLE / CURL_CA_BUNDLE (replacement) > APM_EXTRA_CA_BUNDLE (additive) > OS trust store > certifi fallback.
3. For the parent process, inject truststore as today, then install a scoped truststore.SSLContext subclass whose constructor loads the validated extra PEM. Wire that context at the existing process-wide SSL/urllib3 seam and feature-detect Requests' optional preloaded context. This covers direct Requests calls, existing Sessions, custom adapters, and proxy pools without creating a second HTTP-session authority.
4. For Python child runtimes, extend the shipped, dependency-free .pth bootstrap with the same additive context behavior. This deliberately avoids the proposed merged-certifi temp file: a merged PEM cannot portably enumerate and retain Windows/macOS native roots, while the bootstrap can preserve native trust plus the extra CA without a temp-file lifecycle.
5. Extend build_child_tls_env() so apm run and managed runtime subprocesses receive the normalized extra path as NODE_EXTRA_CA_CERTS, while preserving an explicit non-empty NODE_EXTRA_CA_CERTS. Apply that seam to both apm run subprocess branches. Rust/Codex keeps its runtime-owned trust configuration; the PR will not claim otherwise.
6. Add hermetic coverage with independent loopback CAs: default trust remains valid, the extra private CA becomes valid only in additive mode, replacement variables remain replacement, disable remains highest precedence, malformed/missing/directory paths fail closed, Node env propagation is non-destructive, and a foreign Python venv verifies through the shipped bootstrap.
7. Update the enterprise security model, SSL troubleshooting page, environment-variable reference, docs scope assertions, and Unreleased changelog entry.

I also checked adjacent PR #2602. It touches the same TLS owner for diagnostics and already advertises this precedence, so I will rebase before opening and fold its diagnostic/isolation behavior in if it lands first.

[TameTheGame]

Implementation-plan amendment after adversarial review:

• I changed the child strategy to APM-owned per-process snapshots under ~/.apm/tls: an extra-only PEM for Node and a certifi+extra PEM for ordinary Requests children. This makes generic apm run Python children additive without depending on the managed bootstrap and makes source mutation after validation irrelevant. Managed Python runtimes still install the bootstrap so native trust can be retained when truststore is available.
• The corrected precedence is REQUESTS_CA_BUNDLE > CURL_CA_BUNDLE > APM_DISABLE_TRUSTSTORE > APM_EXTRA_CA_BUNDLE > OS trust > certifi fallback. The disable switch suppresses APM-managed OS/additive propagation but intentionally does not unset an explicit replacement bundle.
• If truststore injection is unavailable, the parent merged fallback is deliberately Requests-scoped. Parent stdlib urllib paths keep their normal fallback because they do not consume Requests bundle variables.

The PR now exercises private and public roots together, child and grandchild propagation, native Node override preservation, source-bundle mutation after validation, and exact rollback after failed TLS publication.