chore(pipe): add kind install + upgrade test for Helm charts #1833
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,87 @@ | ||
| #!/usr/bin/env bash | ||
| # install-test.sh — install + upgrade a chart into the current kube context. | ||
| # Core value over `helm template`: the API server validates every manifest | ||
| # server-side (rejects invalid/immutable/admission-failing objects), and the | ||
| # upgrade path catches immutable-field breaks. Does NOT wait for app pods to | ||
| # become Ready (images are private; readiness is a separate, creds-gated concern). | ||
| # | ||
| # Usage: install-test.sh <chart-dir> [values-file] | ||
| set -uo pipefail | ||
|
|
||
| CHART_DIR="$1" | ||
| CHART="$(basename "$CHART_DIR")" | ||
| # Values: explicit arg wins; else the render-gate's vetted sample values, if any. | ||
| VALUES_FILE="${2:-}" | ||
| [[ -z "$VALUES_FILE" && -f ".github/configs/helm-render-values/${CHART}.yaml" ]] \ | ||
| && VALUES_FILE=".github/configs/helm-render-values/${CHART}.yaml" | ||
| NS="it-${CHART}" | ||
| NSB="it-${CHART}-base" # isolated namespace for the origin/main -> PR baseline upgrade | ||
| REL="$CHART" | ||
| TIMEOUT="${IT_TIMEOUT:-180s}" | ||
| VARGS=() | ||
| [[ -n "$VALUES_FILE" && -f "$VALUES_FILE" ]] && { VARGS=(-f "$VALUES_FILE"); echo " values: $VALUES_FILE"; } | ||
| # --no-hooks: hook Jobs (migrations/bootstrap) pull private images and need real | ||
| # backing services, so they never complete in a credential-less CI cluster. We | ||
| # validate the steady-state manifests server-side; hook Jobs are out of scope here. | ||
| HOOKS=(--no-hooks) | ||
|
|
||
| # Library charts are not installable. | ||
| if grep -qiE '^type:[[:space:]]*library([[:space:]]|$)' "$CHART_DIR/Chart.yaml"; then | ||
| echo "::notice::$CHART is a library chart — skipping install test." | ||
| exit 0 | ||
| fi | ||
|
|
||
| fail() { echo "::error::[$CHART] $1"; kubectl get events -n "$NS" --sort-by=.lastTimestamp 2>/dev/null | tail -15; cleanup; exit 1; } | ||
| cleanup() { | ||
| helm uninstall "$REL" -n "$NS" >/dev/null 2>&1 || true | ||
| helm uninstall "${REL}-base" -n "$NSB" >/dev/null 2>&1 || true | ||
| kubectl delete ns "$NS" "$NSB" --wait=false >/dev/null 2>&1 || true | ||
| } | ||
|
|
||
| cleanup # idempotent: clear any stale release/namespace from a prior aborted run | ||
|
|
||
| echo "===== [$CHART] dependency build =====" | ||
| helm dependency build "$CHART_DIR" >/dev/null 2>&1 || fail "helm dependency build failed" | ||
|
|
||
| # ---- 1. Fresh install of the PR chart (server-side manifest validation) ---- | ||
| echo "===== [$CHART] install (PR) =====" | ||
| if ! helm install "$REL" "$CHART_DIR" ${VARGS[@]+"${VARGS[@]}"} "${HOOKS[@]}" -n "$NS" --create-namespace --timeout "$TIMEOUT" 2>&1; then | ||
| fail "helm install failed (invalid manifest / admission / hook)" | ||
| fi | ||
| [[ "$(helm status "$REL" -n "$NS" -o json | grep -o '"status":"[a-z]*"' | head -1)" == '"status":"deployed"' ]] \ | ||
| || fail "release not in deployed state" | ||
| n=$(kubectl get all -n "$NS" --no-headers 2>/dev/null | wc -l | tr -d ' ') | ||
| echo " created $n objects" | ||
| [[ "$n" -gt 0 ]] || fail "install produced no objects" | ||
|
|
||
| # ---- 2. Upgrade the PR chart in place (upgrade code path + hook re-run) ---- | ||
| echo "===== [$CHART] upgrade (PR -> PR, benign change) =====" | ||
| helm upgrade "$REL" "$CHART_DIR" ${VARGS[@]+"${VARGS[@]}"} "${HOOKS[@]}" -n "$NS" --timeout "$TIMEOUT" \ | ||
| --set-string podAnnotations.helm-install-test="$(date +%s 2>/dev/null || echo x)" 2>&1 \ | ||
| | grep -qE 'STATUS: deployed' || fail "in-place upgrade failed" | ||
|
|
||
| # Free the PR release before the baseline test so only ONE release is ever | ||
| # installed at a time — two full installs of a subchart-heavy chart exhaust a | ||
| # single-node kind cluster's memory and make the baseline flaky. | ||
| helm uninstall "$REL" -n "$NS" >/dev/null 2>&1 || true | ||
| kubectl delete ns "$NS" --wait=false >/dev/null 2>&1 || true | ||
|
|
||
| # ---- 3. Real upgrade path: base (origin/main) -> PR, when the chart exists on main ---- | ||
| if git cat-file -e "origin/main:charts/${CHART}/Chart.yaml" 2>/dev/null; then | ||
| echo "===== [$CHART] upgrade (origin/main -> PR) =====" | ||
| BASE_DIR="$(mktemp -d)/$CHART"; mkdir -p "$BASE_DIR" | ||
| git archive "origin/main" "charts/${CHART}" | tar -x --strip-components=2 -C "$BASE_DIR" 2>/dev/null | ||
| helm dependency build "$BASE_DIR" >/dev/null 2>&1 || echo " (base dep build failed — skipping baseline upgrade)" | ||
| if helm install "${REL}-base" "$BASE_DIR" ${VARGS[@]+"${VARGS[@]}"} "${HOOKS[@]}" -n "$NSB" --create-namespace --timeout "$TIMEOUT" >/dev/null 2>&1; then | ||
| helm upgrade "${REL}-base" "$CHART_DIR" ${VARGS[@]+"${VARGS[@]}"} "${HOOKS[@]}" -n "$NSB" --timeout "$TIMEOUT" 2>&1 \ | ||
| | grep -qE 'STATUS: deployed' || fail "upgrade from origin/main failed (immutable-field break?)" | ||
| echo " origin/main -> PR upgrade OK" | ||
| else | ||
| echo " (baseline install failed — likely unrelated to this PR; skipping)" | ||
|
Comment on lines
+98
to
+106
There was a problem hiding this comment. 🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -e
printf '%s\n' '--- .github/scripts/install-test.sh ---'
cat -n .github/scripts/install-test.sh | sed -n '1,110p'
printf '%s\n' '--- related references ---'
rg -n -C 3 'VARGS|origin/main|baseline|git archive|helm dependency build|helm install|helm upgrade' .github/scripts .github 2>/dev/null | head -n 240Repository: LerianStudio/helm Length of output: 24975 🏁 Script executed: #!/bin/bash
set -e
printf '%s\n' '--- workflow invocation ---'
rg -n -C 5 'install-test\.sh|helm-render-values|values-file|values' .github/workflows .github/scripts 2>/dev/null | head -n 220
printf '%s\n' '--- chart and render-value files ---'
git ls-files 'charts/*/Chart.yaml' '.github/configs/helm-render-values/*' | head -n 160
printf '%s\n' '--- origin/main chart tree for a representative comparison ---'
git ls-tree --name-only origin/main:charts 2>/dev/null | head -n 80
printf '%s\n' '--- relevant branch/ref state ---'
git branch --show-current
git show-ref --verify --quiet refs/remotes/origin/main && git rev-parse --short refs/remotes/origin/main || trueRepository: LerianStudio/helm Length of output: 19700 🏁 Script executed: #!/bin/bash
set -e
python3 - <<'PY'
from pathlib import Path
p = Path(".github/scripts/install-test.sh")
text = p.read_text()
section = text.split("# ---- 3. Real upgrade path", 1)[1].split('echo "===== [$CHART] OK ====="', 1)[0]
assert 'helm dependency build "$BASE_DIR" >/dev/null 2>&1 || echo' in section
assert 'if helm install "${REL}-base" "$BASE_DIR"' in section
assert ' else\n echo " (baseline install failed — likely unrelated to this PR; skipping)"' in section
assert ' echo " (new chart — not on origin/main; skipping baseline upgrade)"' in section
print("source checks: baseline dependency failure is non-fatal; baseline install failure is a skip; chart absence is the separate skip path")
PY
tmp_script="$(mktemp)"
cat >"$tmp_script" <<'BASH'
set -uo pipefail
CHART=demo
CHART_DIR=charts/demo
REL=demo
NSB=it-demo-base
TIMEOUT=180s
VARGS=(-f .github/configs/helm-render-values/demo.yaml)
HOOKS=(--no-hooks)
fail() { printf '%s\n' "FAIL: $1"; exit 1; }
git() {
if [[ "$1" == cat-file ]]; then return 0; fi
return 1
}
helm() {
case "$2" in
dependency) return 1 ;;
install) return 1 ;;
upgrade) printf '%s\n' 'STATUS: deployed'; return 0 ;;
esac
}
if git cat-file -e "origin/main:charts/${CHART}/Chart.yaml" 2>/dev/null; then
BASE_DIR="$(mktemp -d)/$CHART"; mkdir -p "$BASE_DIR"
helm dependency build "$BASE_DIR" >/dev/null 2>&1 || echo " (base dep build failed — skipping baseline upgrade)"
if helm install "${REL}-base" "$BASE_DIR" ${VARGS[@]+"${VARGS[@]}"} "${HOOKS[@]}" -n "$NSB" --create-namespace --timeout "$TIMEOUT" >/dev/null 2>&1; then
helm upgrade "${REL}-base" "$CHART_DIR" ${VARGS[@]+"${VARGS[@]}"} "${HOOKS[@]}" -n "$NSB" --timeout "$TIMEOUT" 2>&1 \
| grep -qE 'STATUS: deployed' || fail "upgrade from origin/main failed (immutable-field break?)"
echo " origin/main -> PR upgrade OK"
else
echo " (baseline install failed — likely unrelated to this PR; skipping)"
fi
else
echo " (new chart — not on origin/main; skipping baseline upgrade)"
fi
echo "===== [$CHART] OK ====="
BASH
output="$(bash "$tmp_script")"
rm -f "$tmp_script"
printf '%s\n' '--- mocked baseline-failure execution ---'
printf '%s\n' "$output"
grep -q 'baseline install failed' <<<"$output"
grep -q '===== \[demo\] OK =====' <<<"$output"
! grep -q 'origin/main -> PR upgrade OK' <<<"$output"
printf '%s\n' 'behavior check: baseline install failure reaches success output without running the upgrade'Repository: LerianStudio/helm Length of output: 288 🏁 Script executed: #!/bin/bash
set -e
tmp_script="$(mktemp)"
cat >"$tmp_script" <<'BASH'
set -uo pipefail
CHART=demo
CHART_DIR=charts/demo
REL=demo
NSB=it-demo-base
TIMEOUT=180s
VARGS=(-f .github/configs/helm-render-values/demo.yaml)
HOOKS=(--no-hooks)
fail() { printf '%s\n' "FAIL: $1"; exit 1; }
git() {
if [[ "$1" == cat-file ]]; then return 0; fi
return 1
}
helm() {
case "$1" in
dependency) return 1 ;;
install) return 1 ;;
upgrade) printf '%s\n' 'STATUS: deployed'; return 0 ;;
*) return 1 ;;
esac
}
if git cat-file -e "origin/main:charts/${CHART}/Chart.yaml" 2>/dev/null; then
BASE_DIR="$(mktemp -d)/$CHART"; mkdir -p "$BASE_DIR"
helm dependency build "$BASE_DIR" >/dev/null 2>&1 || echo " (base dep build failed — skipping baseline upgrade)"
if helm install "${REL}-base" "$BASE_DIR" ${VARGS[@]+"${VARGS[@]}"} "${HOOKS[@]}" -n "$NSB" --create-namespace --timeout "$TIMEOUT" >/dev/null 2>&1; then
helm upgrade "${REL}-base" "$CHART_DIR" ${VARGS[@]+"${VARGS[@]}"} "${HOOKS[@]}" -n "$NSB" --timeout "$TIMEOUT" 2>&1 \
| grep -qE 'STATUS: deployed' || fail "upgrade from origin/main failed (immutable-field break?)"
echo " origin/main -> PR upgrade OK"
else
echo " (baseline install failed — likely unrelated to this PR; skipping)"
fi
else
echo " (new chart — not on origin/main; skipping baseline upgrade)"
fi
echo "===== [$CHART] OK ====="
BASH
output="$(bash "$tmp_script")"
status=$?
rm -f "$tmp_script"
printf '%s\n' '--- corrected mocked baseline-failure execution ---'
printf '%s\n' "$output"
test "$status" -eq 0
grep -q 'base dep build failed' <<<"$output"
grep -q 'baseline install failed' <<<"$output"
grep -q '===== \[demo\] OK =====' <<<"$output"
! grep -q 'origin/main -> PR upgrade OK' <<<"$output"
printf '%s\n' 'behavior check: baseline install failure reaches success output without running the upgrade'Repository: LerianStudio/helm Length of output: 443 Fail when the baseline upgrade does not run. Lines 74–80 ignore baseline dependency-build failures and treat every baseline-install failure as a successful skip. Use values from 🤖 Prompt for AI Agents |
||
| fi | ||
| else | ||
| echo " (new chart — not on origin/main; skipping baseline upgrade)" | ||
| fi | ||
|
|
||
| echo "===== [$CHART] OK =====" | ||
| cleanup | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,107 @@ | ||
| name: Helm Install Test | ||
|
|
||
| # Installs each changed chart into a throwaway kind cluster and exercises the | ||
| # upgrade path. The value over the render-gate (which only runs `helm template`) | ||
| # is that the real API server VALIDATES every manifest server-side — it rejects | ||
| # invalid field values, bad API versions, and immutable-field changes on upgrade | ||
| # that `helm template` renders happily. Hook Jobs (migrations/bootstrap) are | ||
| # skipped (--no-hooks in the script): they pull private images and need real | ||
| # backing services, so they cannot complete in a credential-less CI cluster. | ||
| # App pods are NOT waited on for the same reason — steady-state manifest | ||
| # validation + the upgrade path are what this gate proves. | ||
|
|
||
| on: | ||
| pull_request: | ||
| paths: | ||
| - "charts/**" | ||
| - ".github/configs/helm-render-values/**" | ||
| - ".github/scripts/install-test.sh" | ||
| - ".github/workflows/helm-install-test.yml" | ||
| workflow_dispatch: {} | ||
|
|
||
| concurrency: | ||
| group: helm-install-test-${{ github.ref }} | ||
| cancel-in-progress: true | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| install-test: | ||
| name: Install + upgrade (kind) | ||
| runs-on: blacksmith-4vcpu-ubuntu-2404 | ||
| timeout-minutes: 30 # a full --all sweep (workflow/lib change) installs every app chart sequentially | ||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 | ||
| with: | ||
| fetch-depth: 0 # full history so the script can `git archive origin/main` for the baseline upgrade | ||
|
|
||
| - name: Fetch base branch | ||
| run: git fetch --no-tags --depth=1 origin main | ||
|
|
||
| - name: Setup Helm | ||
| uses: azure/setup-helm@1a275c3b69536ee54be43f2070a358922e12c8d4 # v4 | ||
|
|
||
| - name: Create kind cluster | ||
| uses: helm/kind-action@a1b0e391336a6ee6713a0583f8c6240d70863de3 # v1.10.0 | ||
| with: | ||
| cluster_name: chart-install-test | ||
|
|
||
| - name: Determine chart scope | ||
| id: scope | ||
| shell: bash | ||
| env: | ||
| EVENT_NAME: ${{ github.event_name }} | ||
| BASE_SHA: ${{ github.event.pull_request.base.sha }} | ||
| HEAD_SHA: ${{ github.sha }} | ||
| run: | | ||
| set -euo pipefail | ||
| all_app_charts() { | ||
| for d in charts/*/; do | ||
| c="$(basename "$d")" | ||
| grep -qiE '^type:[[:space:]]*library([[:space:]]|$)' "charts/$c/Chart.yaml" 2>/dev/null && continue | ||
| printf '%s ' "$c" | ||
| done | ||
| } | ||
| if [[ "$EVENT_NAME" != "pull_request" ]]; then | ||
| echo "charts=$(all_app_charts)" >> "$GITHUB_OUTPUT"; exit 0 | ||
| fi | ||
| changed="$(git diff --name-only "$BASE_SHA" "$HEAD_SHA")" | ||
| # An infra change (script/workflow/shared library/fixtures) can affect every | ||
| # chart, so test them all; otherwise only the charts whose files changed. | ||
| if grep -Eq '^(\.github/scripts/install-test\.sh|\.github/workflows/helm-install-test\.yml|charts/lerian-common/)' <<< "$changed"; then | ||
| echo "charts=$(all_app_charts)" >> "$GITHUB_OUTPUT"; exit 0 | ||
| fi | ||
| scope="" | ||
| for c in $(awk -F/ '/^charts\/[^/]+\// {print $2}' <<< "$changed" | sort -u); do | ||
| [[ -d "charts/$c" ]] || continue # skip deleted charts | ||
| grep -qiE '^type:[[:space:]]*library([[:space:]]|$)' "charts/$c/Chart.yaml" && continue | ||
| scope="${scope:+$scope }$c" | ||
| done | ||
| echo "charts=$scope" >> "$GITHUB_OUTPUT" | ||
|
Comment on lines
+77
to
+82
There was a problem hiding this comment. 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win Select charts for render-values changes. A pull request that changes only Map render-values filenames to their chart, or include render-values changes in the full-chart scope. 🤖 Prompt for AI Agents |
||
|
|
||
| - name: Install + upgrade each chart | ||
| shell: bash | ||
| env: | ||
| CHARTS: ${{ steps.scope.outputs.charts }} | ||
| run: | | ||
| set -uo pipefail | ||
| if [[ -z "${CHARTS// }" ]]; then | ||
| echo "No installable charts in scope — nothing to test."; exit 0 | ||
| fi | ||
| echo "Charts in scope: $CHARTS" | ||
| fails="" | ||
| for c in $CHARTS; do | ||
| echo "::group::install-test $c" | ||
| if bash .github/scripts/install-test.sh "charts/$c"; then | ||
| echo "::endgroup::" | ||
| else | ||
| echo "::endgroup::"; fails="${fails:+$fails }$c" | ||
| fi | ||
| done | ||
| if [[ -n "$fails" ]]; then | ||
| echo "::error::Install/upgrade test failed for: $fails" | ||
| exit 1 | ||
| fi | ||
| echo "All charts in scope installed and upgraded cleanly." | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
Repository: LerianStudio/helm
Length of output: 4049
🏁 Script executed:
Repository: LerianStudio/helm
Length of output: 21927
🏁 Script executed:
Repository: LerianStudio/helm
Length of output: 330
🌐 Web query:
kubectl delete --wait=false behavior Helm uninstall --wait default official documentation💡 Result:
The behavior of
kubectl deleteandhelm uninstallregarding the--waitflag differs in its default state and mechanism. kubectl delete The--waitflag defaults to true [1][2]. When enabled,kubectl deletewaits for all resources specified to be removed from the API (including waiting for finalizers to complete) before returning [1][2]. To prevent this blocking behavior and return immediately after the delete request is acknowledged by the API, you must explicitly pass--wait=false[3][2]. helm uninstall The--waitflag forhelm uninstallbehaves differently [4][5]. Unlikekubectl, the default behavior when the--waitflag is omitted is not to wait for full resource deletion; it instead relies on its internal "hookOnly" strategy [4][5]. When--waitis explicitly provided, it triggers a "watcher" strategy that monitors resources until they are ready or deleted, up to the defined--timeout[4][5]. Crucially, because Helm defaults to "background" cascade strategy, simply adding--waitmay not be sufficient to ensure all resources—especially those with finalizers—are fully cleaned up before the command returns [4][5]. Official Helm documentation recommends using the flag combination--cascade foreground --waitto ensure resources with finalizers are fully deleted before the command exits [4][5]. If a user encounters issues where resources persist after an uninstall, this combination is the standard workaround [6].Citations:
Wait for Helm and namespace deletion before the next install.
Add
--wait --cascade=foreground --timeout "$TIMEOUT"to bothhelm uninstallcommands, and remove--wait=falsefrom both namespace deletions. Otherwise, the baseline install and the next chart can run while previous workloads still consume kind-node capacity, causing flaky or out-of-memory test runs.🤖 Prompt for AI Agents