Update release with new fabrica-based services; remove old services

[travisbcotton]

Pull Request Template

Thank you for your contribution! Please ensure the following before submitting:

Checklist

• My code follows the style guidelines of this project
• I have added/updated comments where needed
• I have added tests that prove my fix is effective or my feature works
• I have run make test (or equivalent) locally and all tests pass
• DCO Sign-off: All commits are signed off (git commit -s) with my real name and email
• REUSE Compliance:
  • Each new/modified source file has SPDX copyright and license headers
  • Any non-commentable files include a <filename>.license sidecar
  • All referenced licenses are present in the LICENSES/ directory

Description

Please include a summary of the change and which issue is fixed.
Also include relevant motivation and context.

Fixes #(issue)

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update

---

For more info, see Contributing Guidelines.

[davidallendj]

Just a couple of other notes before merging. We need to update the *.container files to use the most up-to-date version of our services including:

1. SMD after this PR is merged.
2. Tokensmith to v0.3.0 or later
3. Boot-service after creating a release
4. Metadata-service after creating a release

We also need to update systemd/targets/openchami.target to require the new services as well.

[davidallendj]

Another note...we're going to update the CoreDHCP config in /etc/openchami/configs/coredhcp.yaml to reflect the change from this PR if we upgrade to the latest version.

Here's snippet of the tutorial config should look like after the changes:

    - coresmd: |
        svc_base_uri=https://demo.openchami.cluster:8443 
        ipxe_base_uri=http://172.16.0.254:8081 
        ca_cert=/root_ca/root_ca.crt 
        cache_valid=30s 
        lease_time=1h 
        single_port=false
    - bootloop: |
        lease_file=/tmp/coredhcp.db 
        script_path=default 
        lease_time=5m 
        ipv4_start=172.16.0.200 
        ipv4_end=172.16.0.250

[davidallendj]

A couple of changes:

1. I think OPAAL_URL can be removed
2. I think JWKS_URL should be updated to use the tokensmith JWKS endpoint. In the tutorial, it will be something like http://tokensmith:8080/.well-known/jwks.json.
3. The same change needs to be made to SMD_JWKS_URL as well.

[davidallendj]

We'll need /etc/openchami/configs/haproxy.cfg to be updated to remove the old service routes and add the new ones for tokensmith, boot-service, and metadata-service.

[synackd]

We'll have to note these major changes in the release notes once this is merged. We'll want to bump the minor version on the tag.

[davidallendj]

Should we provide a /etc/openchami/configs/boot-service.yaml here alongside the /etc/openchami/configs/tokensmith.json? I think it should go in systemd/configs/boot-service.yaml here to be copied in the appropriate location.

Edit: Just to add, here's the default boot-service config.yaml:

systemd/configs/boot-service.yaml

# SPDX-FileCopyrightText: 2025 OpenCHAMI Contributors
#
# SPDX-License-Identifier: MIT

# OpenCHAMI Boot Service Configuration Example
#
# This is a comprehensive example configuration file for the OpenCHAMI boot service.
# To use this configuration:
#   1. Copy this file to config.yaml: cp config.example.yaml config.yaml
#   2. Customize the settings below for your environment
#   3. Remove or comment out sections you don't need
#
# Configuration precedence (highest to lowest):
#   1. Command-line flags
#   2. Environment variables (e.g., BOOT_SERVICE_PORT=8082)
#   3. Configuration file (config.yaml)
#   4. Default values

# =============================================================================
# SERVER CONFIGURATION
# =============================================================================

# HTTP server settings
port: 8082                    # Port to listen on
host: "0.0.0.0"              # Interface to bind to (0.0.0.0 for all interfaces)
read_timeout: 30             # HTTP read timeout in seconds
write_timeout: 30            # HTTP write timeout in seconds
idle_timeout: 120            # HTTP idle timeout in seconds

# =============================================================================
# STORAGE CONFIGURATION
# =============================================================================

# Data storage settings
data_dir: "./data"           # Directory for storing boot configurations
storage_type: "file"         # Storage backend: "file", "database" (future)

# Database settings (when storage_type: "database")
# database:
#   driver: "postgres"
#   host: "localhost"
#   port: 5432
#   name: "boot_service"
#   user: "boot_user"
#   password: "boot_password"
#   ssl_mode: "require"
#   max_connections: 25
#   connection_timeout: 30

# =============================================================================
# FEATURE TOGGLES
# =============================================================================

# Authentication
enable_auth: false           # Enable TokenSmith JWT authentication
                            # Set to true for production environments

# Metrics and monitoring
enable_metrics: true         # Enable Prometheus metrics endpoint
metrics_port: 9092          # Port for metrics endpoint (/metrics)

# API compatibility
enable_legacy_api: true     # Enable legacy BSS-compatible endpoints
                           # Disable to force use of new API only

# =============================================================================
# AUTHENTICATION CONFIGURATION (when enable_auth: true)
# =============================================================================

auth:
  # Core authentication settings
  enabled: false             # Must match enable_auth above

  # JWT validation method (choose one):

  # Option 1: JWKS URL (recommended for production)
  jwks_url: "https://auth.openchami.org/.well-known/jwks.json"
  jwks_refresh_interval: "1h"  # How often to refresh JWKS cache

  # Option 2: Static RSA public key (for development/testing)
  # jwt_public_key: |
  #   -----BEGIN PUBLIC KEY-----
  #   MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA...
  #   -----END PUBLIC KEY-----

  # JWT validation options
  jwt_issuer: "https://auth.openchami.org"     # Expected token issuer
  jwt_audience: "boot-service"                  # Expected token audience
  validate_expiration: true                     # Check token expiration
  validate_issuer: true                        # Validate issuer claim
  validate_audience: true                      # Validate audience claim

  # Authorization requirements
  required_claims: ["sub", "iss", "aud"]      # Required JWT claims
  required_scopes: ["boot:read"]              # Required OAuth2 scopes

  # Development/testing options (never use in production)
  allow_empty_token: false    # Allow requests without tokens
  non_enforcing: false       # Log auth failures but don't block requests

# =============================================================================
# HARDWARE STATE MANAGER INTEGRATION
# =============================================================================

# HSM (Hardware State Manager) settings
hsm_url: "http://localhost:27779"  # URL of the HSM service
                                   # Set to your HSM endpoint

# TokenSmith-backed HSM service authentication
# When both hsm_url and tokensmith_url are configured, boot-service exchanges a
# bootstrap token for short-lived service tokens and adds them to HSM requests.
# Standardized env vars: TOKENSMITH_URL, TOKENSMITH_BOOTSTRAP_TOKEN,
# TOKENSMITH_TARGET_SERVICE, TOKENSMITH_SCOPES, TOKENSMITH_REFRESH_SKEW_SEC
tokensmith_url: "http://localhost:8080"
tokensmith_target_service: "hsm"
tokensmith_scopes: "hsm:read"
tokensmith_refresh_skew_sec: 120
# tokensmith_bootstrap_token: "<bootstrap-jwt>"  # Prefer env var for secrets
# Environment fallback: TOKENSMITH_BOOTSTRAP_TOKEN

# HSM authentication (when HSM requires auth)
# hsm_auth:
#   type: "service_token"      # Authentication type for HSM
#   service_name: "boot-service"
#   token_endpoint: "http://tokensmith:8080/token"

# =============================================================================
# EXTERNAL SERVICES
# =============================================================================

# TokenSmith authentication service (when enable_auth: true)
tokensmith:
  url: "http://localhost:8080"                    # TokenSmith service URL
  timeout: 30                                    # Request timeout in seconds

  # Service-to-service authentication
  service_auth:
    enabled: false                               # Enable service tokens
    service_name: "boot-service"                 # This service's identifier
    token_endpoint: "/token"                     # Token endpoint path

# BSS (Boot Script Service) integration
bss:
  enabled: false                                 # Enable BSS integration
  url: "http://localhost:27778"                 # BSS service URL
  timeout: 30                                   # Request timeout in seconds

# =============================================================================
# LOGGING AND MONITORING
# =============================================================================

# Logging configuration
logging:
  level: "info"               # Log level: debug, info, warn, error
  format: "json"             # Log format: json, text
  output: "stdout"           # Log output: stdout, stderr, file
  # file: "/var/log/boot-service.log"  # Log file (when output: file)

# Health check configuration
health:
  enabled: true              # Enable health check endpoint
  endpoint: "/health"        # Health check URL path
  timeout: 5                # Health check timeout in seconds

# =============================================================================
# PERFORMANCE AND SCALING
# =============================================================================

# Request limits
limits:
  max_request_size: "10MB"   # Maximum request body size
  max_concurrent: 100        # Maximum concurrent requests
  rate_limit: 1000          # Requests per minute per IP

# Caching (future feature)
# cache:
#   enabled: false
#   type: "memory"           # Cache type: memory, redis
#   ttl: "5m"               # Cache TTL
#   max_size: "100MB"       # Maximum cache size

# =============================================================================
# DEVELOPMENT AND TESTING
# =============================================================================

# Development mode settings
development:
  enabled: false             # Enable development mode
  cors_enabled: true        # Enable CORS for browser testing
  cors_origins: ["*"]       # Allowed CORS origins
  debug_endpoints: false    # Enable debug/diagnostic endpoints
  mock_services: false      # Use mock external services

# =============================================================================
# DEPLOYMENT ENVIRONMENT EXAMPLES
# =============================================================================

# Uncomment and modify one of these sections for your deployment environment:

# --- Development Environment ---
# enable_auth: false
# enable_metrics: true
# logging:
#   level: "debug"
# development:
#   enabled: true
#   debug_endpoints: true

# --- Production Environment ---
# enable_auth: true
# enable_metrics: true
# auth:
#   enabled: true
#   jwks_url: "https://auth.openchami.org/.well-known/jwks.json"
#   jwt_issuer: "https://auth.openchami.org"
#   jwt_audience: "boot-service"
#   required_scopes: ["boot:read"]
# logging:
#   level: "info"
#   format: "json"

# --- Kubernetes/Container Environment ---
# port: 8080
# host: "0.0.0.0"
# data_dir: "/data"
# auth:
#   jwks_url: "http://tokensmith:8080/.well-known/jwks.json"
#   jwt_issuer: "openchami-tokensmith"
#   jwt_audience: "openchami-cluster"
# hsm_url: "http://smd:27779"
# logging:
#   format: "json"
#   output: "stdout"

[synackd]

Another note...we're going to update the CoreDHCP config in /etc/openchami/configs/coredhcp.yaml to reflect the change from this PR if we upgrade to the latest version.

Here's snippet of the tutorial config should look like after the changes:

    - coresmd: |
        svc_base_uri=https://demo.openchami.cluster:8443 
        ipxe_base_uri=http://172.16.0.254:8081 
        ca_cert=/root_ca/root_ca.crt 
        cache_valid=30s 
        lease_time=1h 
        single_port=false
    - bootloop: |
        lease_file=/tmp/coredhcp.db 
        script_path=default 
        lease_time=5m 
        ipv4_start=172.16.0.200 
        ipv4_end=172.16.0.250

We may want to add default hostname rules since the default if none is to prefix with unknown-. Maybe something like:

rule=type:Node,hostname:n{04d}
rule=type:NodeBMC,hostname:{id}

The above will make the node hostnames be like n0001 and make the BMC hostnames be their xname.

[synackd]

Initial code review without testing this yet.

[synackd]

Testing now. I get:

sed: can't read /etc/containers/systemd/opaal.container: No such file or directory

when running the openchami-certificate-update script.

If getting rid of hydra, we'll want to remove references to it, e.g. in

release/scripts/openchami_profile.sh

Line 27 in d77457c

${CONTAINER_CMD:-docker} exec hydra hydra create client \

We can probably just get rid of those functions.

[synackd]

Rebased to incorporate openchami.target changes on main.

[erl-hpe]

Capturing this here so it is public and not just part of a DM thread...

Sometime around the time of the rebase, the following commits were lost on this branch:

commit eaee9ff67a947fe66147067ac1f4ecf14ef98018 (HEAD -> trcotton/tokensmith-container, origin/trcotton/tokensmith-container)
Author: Devon Bautista <17506592+synackd@users.noreply.github.com>
Date:   Wed Jun 17 08:33:06 2026 -0600

    bump metadata-service to v0.1.2

    Enable YAML data resource unmarshalling.

    Signed-off-by: Devon Bautista <17506592+synackd@users.noreply.github.com>

commit 2536ba1f6b100b5404a79b630258cd1197cc11eb
Author: Devon Bautista <17506592+synackd@users.noreply.github.com>
Date:   Wed Jun 17 08:32:37 2026 -0600

    bump boot-service to v0.1.7

    Enable YAML data resource unmarshalling.

    Signed-off-by: Devon Bautista <17506592+synackd@users.noreply.github.com>

commit 75322b3b2f4e29167011c1e9ed5672d2751b3630
Author: Devon Bautista <17506592+synackd@users.noreply.github.com>
Date:   Wed Jun 17 02:25:08 2026 -0600

    metadata-service: add CAP_NET_ADMIN and TUN device for WireGuard tunnels

    Signed-off-by: Devon Bautista <17506592+synackd@users.noreply.github.com>

This means, among other things, that metadata-service regressed from v0.1.2 to v0.1.1 in the branch, which causes metadata service to stop working.

[davidallendj]

I think we can take these out of the openchami.env.

# Environemnt Variables
URLS_SELF_ISSUER=https://${SYSTEM_URL}/
URLS_SELF_PUBLIC=https://${SYSTEM_URL}/
URLS_LOGIN=https://${SYSTEM_URL}/login
URLS_CONSENT=https://${SYSTEM_URL}/consent
URLS_LOGOUT=https://${SYSTEM_URL}/logout

[erl-hpe]

I am getting a failure between acme-register and step-ca that appears to prevent acme-deploy from running and, therefore prevents haproxy from starting. On my VM based deployment on my laptop (MacBook) I see this about 10% to 20% of the time. On my vTDS deployment, running in a Virtual Node (VM hosted by a GCP instance) I am seeing it about 95% of the time. I am putting in a long-ish writeup here of what I see that probably belongs in an issue somewhere, but might be useful here. If someone wants to tell me where to post the issue, that would be great. This will, basically, block deployment on vTDS and, hence, the integration testing of OpenCHAMI release for PR 50, which is why I am putting this in a comment here. I do not see this on the main branch (which is not surprising).

What I see in the logs is that step-ca starts up and appears to initialize step-ca successfully, arriving at "your CA is ready to go!" After this, step-ca receives a GET request from 10.89.3.3 (which is acme-register) on the path acme/acme/directory and sends back this (successful: 200) response:

{
  "newNonce": "https://step-ca:9000/acme/acme/new-nonce",
  "newAccount": "https://step-ca:9000/acme/acme/new-account",
  "newOrder": "https://step-ca:9000/acme/acme/new-order",
  "revokeCert": "https://step-ca:9000/acme/acme/revoke-cert",
  "keyChange": "https://step-ca:9000/acme/acme/key-change"
}

Then acme-register notes that is is is using https://step-ca:9000/acme/acme/directory for its CA, creates a key, and registers an account. This appears to entail a successful (200) HEAD request on /acme/acme/new-nonce and a successful POST (201) on /acme/acme/new-account both handled by step-ca and then acme-register does a bunch of stuff, culminating in two POSTs to step-ca. These are both successful (201 and 200 respectively). The first is to /acme/acme/new-order and gets the response:

{
  "id": "QIsZSee09Zu9omAt7xpczKTLV20VccxR",
  "status": "pending",
  "expires": "2026-07-11T15:33:13Z",
  "identifiers": [
    {
      "type": "dns",
      "value": "demo.openchami.cluster"
    }
  ],
  "notBefore": "2026-07-10T15:32:13Z",
  "notAfter": "2026-07-11T15:33:13Z",
  "authorizations": [
    "https://step-ca:9000/acme/acme/authz/N9wmNUnEIKpjrb8m1FCJsLuN7QKHBhc2"
  ],
  "finalize": "https://step-ca:9000/acme/acme/order/QIsZSee09Zu9omAt7xpczKTLV20VccxR/finalize"
}

the second is on /acme/acme/authz/N9wmNUnEIKpjrb8m1FCJsLuN7QKHBhc2 and receives the response:

{
  "identifier": {
    "type": "dns",
    "value": "demo.openchami.cluster"
  },
  "status": "pending",
  "challenges": [
    {
      "type": "dns-01",
      "status": "pending",
      "token": "UD9SFLdN7PA2gW6YlPsc7Uer1M478vRk",
      "url": "https://step-ca:9000/acme/acme/challenge/N9wmNUnEIKpjrb8m1FCJsLuN7QKHBhc2/zxsfydTkb4F4D0zD3VmEdjgtetGJ14Kc"
    },
    {
      "type": "http-01",
      "status": "pending",
      "token": "UD9SFLdN7PA2gW6YlPsc7Uer1M478vRk",
      "url": "https://step-ca:9000/acme/acme/challenge/N9wmNUnEIKpjrb8m1FCJsLuN7QKHBhc2/VJ96qO73KqZGqhUGXaneELZgIDgEUpTT"
    },
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "token": "UD9SFLdN7PA2gW6YlPsc7Uer1M478vRk",
      "url": "https://step-ca:9000/acme/acme/challenge/N9wmNUnEIKpjrb8m1FCJsLuN7QKHBhc2/diacl4ojMJeqntZVJSYZ2TSSwwUIemGW"
    }
  ],
  "wildcard": false,
  "expires": "2026-07-11T15:33:13Z"
} 

After that acme-register logs the following:

• Getting webroot for domain='demo.openchami.cluster'

• Verifying: demo.openchami.cluster

• Standalone mode server
  At this point, step-ca starts reporting TLS handshake errors from 10.89.3.3 (acme-register) and acme-register reports:

• demo.openchami.cluster: Unknown status: . Verification error: (with suggestions about debugging acme-register)

• Please refer to https://curl.haxx.se/libcurl/c/libcurl-errors.html for error code: 77

• Could not get nonce, let's try again.

The last continuing in a loop until acme-register gives up and fails with Giving up sending to CA server after 20 retries.

When the acme-register service finally terminates with a failure, systemd logs Dependency failed for The acme-deploy container. and acme-deploy logs Job acme-deploy.service/start failed with result 'dependency'.

After this, acme-register restarts and completely succeeds, printing out its certs and the whole bit, but acme-deploy never tries to restart, so it never starts and haproxy which depends on it also never starts.

[erl-hpe]

I have found a fix for the acme-register->acme-deploy->haproxy dependency chain that works and makes the chain resilient to failures in any of the three services. It uses Upholds and PartOf in the following sequence:

• acme-register: Upholds=acme-deploy.service
• acme-deploy: PartOf=openchami.taget acme-register.service and Upholds=haproxy.service
• haproxy: PartOf=openchami.target acme-deploy.service

This overrides the dependency failed state of acme-deploy and haproxy that happens when acme-register fails on its first startup and causes acme-deploy to be started when acme-register finally succeeds and haproxy to be started when acme-deploy finally succeeds.

[davidallendj]

I have found a fix for the acme-register->acme-deploy->haproxy dependency chain that works and makes the chain resilient to failures in any of the three services. It uses Upholds and PartOf in the following sequence:

• acme-register: Upholds=acme-deploy.service
• acme-deploy: PartOf=openchami.taget acme-register.service and Upholds=haproxy.service
• haproxy: PartOf=openchami.target acme-deploy.service

This overrides the dependency failed state of acme-deploy and haproxy that happens when acme-register fails on its first startup and causes acme-deploy to be started when acme-register finally succeeds and haproxy to be started when acme-deploy finally succeeds.

I just pushed these changes to the branch for the containers. They seemed to have been working when I tested on my JS2 instance.

[synackd]

Capturing this here so it is public and not just part of a DM thread...

Sometime around the time of the rebase, the following commits were lost on this branch:

commit eaee9ff67a947fe66147067ac1f4ecf14ef98018 (HEAD -> trcotton/tokensmith-container, origin/trcotton/tokensmith-container)
Author: Devon Bautista <17506592+synackd@users.noreply.github.com>
Date:   Wed Jun 17 08:33:06 2026 -0600

    bump metadata-service to v0.1.2

    Enable YAML data resource unmarshalling.

    Signed-off-by: Devon Bautista <17506592+synackd@users.noreply.github.com>

commit 2536ba1f6b100b5404a79b630258cd1197cc11eb
Author: Devon Bautista <17506592+synackd@users.noreply.github.com>
Date:   Wed Jun 17 08:32:37 2026 -0600

    bump boot-service to v0.1.7

    Enable YAML data resource unmarshalling.

    Signed-off-by: Devon Bautista <17506592+synackd@users.noreply.github.com>

commit 75322b3b2f4e29167011c1e9ed5672d2751b3630
Author: Devon Bautista <17506592+synackd@users.noreply.github.com>
Date:   Wed Jun 17 02:25:08 2026 -0600

    metadata-service: add CAP_NET_ADMIN and TUN device for WireGuard tunnels

    Signed-off-by: Devon Bautista <17506592+synackd@users.noreply.github.com>

This means, among other things, that metadata-service regressed from v0.1.2 to v0.1.1 in the branch, which causes metadata service to stop working.

Thanks, Eric. I've re-added these commits.

[synackd]

I think we can take these out of the openchami.env.

# Environemnt Variables
URLS_SELF_ISSUER=https://${SYSTEM_URL}/
URLS_SELF_PUBLIC=https://${SYSTEM_URL}/
URLS_LOGIN=https://${SYSTEM_URL}/login
URLS_CONSENT=https://${SYSTEM_URL}/consent
URLS_LOGOUT=https://${SYSTEM_URL}/logout

#61 cleans this up in a more idiomatic way. Once that gets merged, we can rebase this on that.

[davidallendj]

#61 cleans this up in a more idiomatic way. Once that gets merged, we can rebase this on that.

So #61 is going to be merged before this one? There's going to be quite a bit of clean up if you already have a deployed testing environment for this PR.

[synackd]

#61 cleans this up in a more idiomatic way. Once that gets merged, we can rebase this on that.

So #61 is going to be merged before this one? There's going to be quite a bit of clean up if you already have a deployed testing environment for this PR.

That was my original plan so that the legacy services would get the new systemd paths as well as the new ones in this PR, and so that we could work out the initial kinks of that PR and this one in parallel.

[davidallendj]

That was my original plan so that the legacy services would get the new systemd paths as well as the new ones in this PR, and so that we could work out the initial kinks of that PR and this one in parallel.

I am more focused on getting the new services merged into the release after testing what we already got here. I don't really plan on testing the other PR and would prefer not having to change my current testing environment to finish testing especially since the PR is already massive.

[synackd]

That was my original plan so that the legacy services would get the new systemd paths as well as the new ones in this PR, and so that we could work out the initial kinks of that PR and this one in parallel.

I am more focused on getting the new services merged into the release after testing what we already got here. I don't really plan on testing the other PR and would prefer not having to change my current testing environment to finish testing especially since the PR is already massive.

I've already tested that PR and am working with @seantronsen to finalize it if you're worried about stretching yourself thin.

That said, while it would be nice to have a version of the release that uses the proper systemd paths for the old services, I don't think it's strictly necessary. I would be fine waiting on merging 61 after this gets merged and working out the kinks there, but I wouldn't want to tag a release until the new systemd paths are being used.

[erl-hpe]

As of yesterday afternoon, my deployment tool was able, using the modifications to its approximation of the tutorial procedure that I have had to make to it, to deploy this PR onto both my local VMs and, using vTDS cluster running under GCP. In this comment I will summarize what I have had to do to get this to work, to help in figuring out how much divergence there is still remaining from the original tutorial. It looks like all of these are unavoidable due to the nature of the changes in the PR and probably known, but I think it is worthwhile to capture them here:

• Tokensmith Issuer and SMD Auth Issuer in /etc/openchami/config/openchami.env need to be the same, and, at least in my experience need to both be changed to http://tokensmith:8080
• coredhcp takes named parameters instead of positional parameters and cannot handle positional parameters
• The ochami commands to set up boot parameters use the boot service instead of bss
• The ochami commands to set up cloud-init metadata use metadata sub-commands instead of cloud-init sub-commands
• The minimum ochami version required for use with the PR is v0.9.0
• The metadata-service expects its metadata to be wrapped in a structure with a metadata and a spec section whereas the cloud-init service assumed that everything presented to it was (effectively) part of the spec section and there was no metadata or spec section.

In addition to the above, I was forced when using metadata service, to abandon using embedded Jinja2 templating in the group template for SSH keys and write them explicitly into the group template itself. I believe there is work underway to restore the templating and I will test that when it becomes available (presumably in a newer version of metadata-service).