Repository navigation
kinder: add native workflow groups and an extended test suite #3334
+1,533
−169
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| - name: ci-kubernetes-e2e-kubeadm-kinder-extended-{{ dashVer .KubernetesVersion }} | ||
| cluster: eks-prow-build-cluster | ||
| interval: 12h | ||
| decorate: true | ||
| labels: | ||
| preset-dind-enabled: "true" | ||
| preset-kind-volume-mounts: "true" | ||
| annotations: | ||
| testgrid-dashboards: sig-cluster-lifecycle-kubeadm | ||
| testgrid-tab-name: kubeadm-kinder-extended-{{ dashVer .KubernetesVersion }} | ||
| testgrid-alert-email: sig-cluster-lifecycle-kubeadm-alerts@kubernetes.io | ||
| description: "OWNER: sig-cluster-lifecycle (kinder); Runs independent kubeadm feature and regression workflows and reports all scenario failures" | ||
| testgrid-num-columns-recent: "20" | ||
| testgrid-num-failures-to-alert: "4" | ||
| testgrid-alert-stale-results-hours: "48" | ||
| decoration_config: | ||
| timeout: 180m | ||
| extra_refs: | ||
| - org: kubernetes | ||
| repo: kubernetes | ||
| base_ref: {{ branchFor .KubernetesVersion }} | ||
| path_alias: k8s.io/kubernetes | ||
| - org: kubernetes | ||
| repo: kubeadm | ||
| base_ref: main | ||
| path_alias: k8s.io/kubeadm | ||
| spec: | ||
| containers: | ||
| - image: gcr.io/k8s-staging-test-infra/kubekins-e2e:{{ .TestInfraImage }}-{{ imageVer .KubernetesVersion }} | ||
| command: | ||
| - runner.sh | ||
| - "../kubeadm/kinder/ci/kinder-run.sh" | ||
| args: | ||
| - {{ .WorkflowFile }} | ||
| securityContext: | ||
| privileged: true | ||
| resources: | ||
| limits: | ||
| memory: "9000Mi" | ||
| cpu: 2000m | ||
| requests: | ||
| memory: "9000Mi" | ||
| cpu: 2000m |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| - name: ci-kubernetes-e2e-kubeadm-kinder-upgrade-etcd-rollback-{{ dashVer .KubernetesVersion }} | ||
| cluster: eks-prow-build-cluster | ||
| interval: {{ .JobInterval }} | ||
| decorate: true | ||
| labels: | ||
| preset-dind-enabled: "true" | ||
| preset-kind-volume-mounts: "true" | ||
| annotations: | ||
| testgrid-dashboards: sig-cluster-lifecycle-kubeadm | ||
| testgrid-tab-name: kubeadm-kinder-upgrade-etcd-rollback-{{ dashVer .KubernetesVersion }} | ||
| testgrid-alert-email: sig-cluster-lifecycle-kubeadm-alerts@kubernetes.io | ||
| description: "OWNER: sig-cluster-lifecycle (kinder); Verifies etcd data rollback after a failed kubeadm upgrade" | ||
| testgrid-num-columns-recent: "20" | ||
| {{ .AlertAnnotations }} | ||
| decoration_config: | ||
| timeout: 60m | ||
| extra_refs: | ||
| - org: kubernetes | ||
| repo: kubernetes | ||
| base_ref: {{ branchFor .KubernetesVersion }} | ||
| path_alias: k8s.io/kubernetes | ||
| - org: kubernetes | ||
| repo: kubeadm | ||
| base_ref: main | ||
| path_alias: k8s.io/kubeadm | ||
| spec: | ||
| containers: | ||
| - image: gcr.io/k8s-staging-test-infra/kubekins-e2e:{{ .TestInfraImage }}-{{ imageVer .KubernetesVersion }} | ||
| command: | ||
| - runner.sh | ||
| - "../kubeadm/kinder/ci/kinder-run.sh" | ||
| args: | ||
| - {{ .WorkflowFile }} | ||
| securityContext: | ||
| privileged: true | ||
| resources: | ||
| limits: | ||
| memory: "9000Mi" | ||
| cpu: 2000m | ||
| requests: | ||
| memory: "9000Mi" | ||
| cpu: 2000m |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| version: 1 | ||
| summary: | | ||
| Run independent kubeadm feature and regression workflows using native Kinder workflow groups. | ||
| Cover functionality beyond regular upgrade and downgrade testing. | ||
| A failed scenario is reported without skipping the remaining scenarios. | ||
| See https://github.com/kubernetes/kubeadm/issues/3331. | ||
| workflows: | ||
| - name: upgrade-etcd-rollback | ||
| workflow: upgrade-etcd-rollback-{{ .KubernetesVersion }}.yaml | ||
| timeout: 70m | ||
| - name: upgrade-no-addon-config-maps | ||
| workflow: upgrade-no-addon-config-maps-{{ .KubernetesVersion }}.yaml | ||
| timeout: 70m | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,253 @@ | ||
| # IMPORTANT! This workflow is imported by upgrade-etcd-rollback-* workflows. | ||
| version: 1 | ||
| summary: | | ||
| Verify etcd data rollback after a failed kubeadm upgrade. | ||
| Regression test for https://github.com/kubernetes/kubeadm/issues/3330. | ||
| vars: | ||
| kubernetesVersion: ci/latest | ||
| baseImage: kindest/base:v20260214-ea8e5717 | ||
| image: kindest/node:etcd-rollback | ||
| clusterName: kinder-upgrade-etcd-rollback | ||
| kubeadmVerbosity: 6 | ||
| tasks: | ||
| - name: pull-base-image | ||
| cmd: docker | ||
| args: | ||
| - pull | ||
| - '{{ .vars.baseImage }}' | ||
| - name: add-kubernetes-version | ||
| cmd: kinder | ||
| args: | ||
| - build | ||
| - node-image-variant | ||
| - --base-image={{ .vars.baseImage }} | ||
| - --image={{ .vars.image }} | ||
| - --with-init-artifacts={{ .vars.kubernetesVersion }} | ||
| - --loglevel=debug | ||
| timeout: 15m | ||
| - name: create-cluster | ||
| cmd: kinder | ||
| args: | ||
| - create | ||
| - cluster | ||
| - --name={{ .vars.clusterName }} | ||
| - --image={{ .vars.image }} | ||
| - --control-plane-nodes=1 | ||
| - --worker-nodes=0 | ||
| - --loglevel=debug | ||
| timeout: 5m | ||
| - name: init | ||
| cmd: kinder | ||
| args: | ||
| - do | ||
| - kubeadm-init | ||
| - --name={{ .vars.clusterName }} | ||
| - --kubeadm-verbosity={{ .vars.kubeadmVerbosity }} | ||
| - --loglevel=debug | ||
| timeout: 5m | ||
| - name: prepare-rollback-test | ||
| description: Save an API object and the original etcd manifest/data marker, then prepare a failing etcd Pod patch. | ||
| cmd: docker | ||
| args: | ||
| - exec | ||
| - '{{ .vars.clusterName }}-control-plane-1' | ||
| - bash | ||
| - -ceu | ||
| - | | ||
| set -euo pipefail | ||
| export KUBECONFIG=/etc/kubernetes/admin.conf | ||
| state=/kinder/rollback | ||
| data=/var/lib/etcd | ||
| mkdir -p "${state}/patches" | ||
|
|
||
| # Pull the fixture before taking etcd down; image pull failure must fail the job. | ||
| crictl pull registry.k8s.io/e2e-test-images/busybox:1.37.0-2 | ||
| kubectl create configmap rollback-test --from-literal=value=before-upgrade | ||
| kubectl get configmap rollback-test -o jsonpath='{.metadata.uid}:{.data.value}' > "${state}/object-before" | ||
| cp /etc/kubernetes/manifests/etcd.yaml "${state}/etcd-before.yaml" | ||
| printf 'before-upgrade\n' > "${state}/marker-before" | ||
| cp "${state}/marker-before" "${data}/rollback-marker" | ||
| touch "${state}/armed" | ||
|
|
||
| # Kubeadm installs this patch after backup. Kubelet terminates the original | ||
| # static Pod before starting its replacement, so the init container owns the | ||
| # data directory exclusively. Preserve all DB/WAL bytes rather than truncating | ||
| # a database file; the regular file at "member" makes old etcd require restore. | ||
| cat > "${state}/patches/etcd+strategic.yaml" <<'PATCH' | ||
|
Comment on lines
+72
to
+76
Member
There was a problem hiding this comment. could we avoid an init container and static pod patches? i am trying to avoid complicated bash in kinder, although we do have a few tasks with complicated bash already... if we have to do some complicated state, maybe we could do it from kinder's go code... |
||
| spec: | ||
| initContainers: | ||
| - name: fail-etcd-upgrade | ||
| image: registry.k8s.io/e2e-test-images/busybox:1.37.0-2 | ||
| command: | ||
| - /bin/sh | ||
| - -ec | ||
| - | | ||
| test -f /rollback-state/armed | ||
| test ! -e /rollback-state/fault-injected | ||
| cmp /rollback-state/marker-before /var/lib/etcd/rollback-marker | ||
| test -d /var/lib/etcd/member | ||
| test ! -e /var/lib/etcd/member.failed-upgrade | ||
| mv /var/lib/etcd/member /var/lib/etcd/member.failed-upgrade | ||
| printf 'injected upgrade failure\n' > /var/lib/etcd/member | ||
| printf 'after-backup\n' > /var/lib/etcd/rollback-marker | ||
| printf 'created-after-backup\n' > /var/lib/etcd/rollback-extra | ||
| printf 'injected\n' > /rollback-state/fault-injected | ||
| exit 1 | ||
| volumeMounts: | ||
| - name: etcd-data | ||
| mountPath: /var/lib/etcd | ||
| - name: rollback-state | ||
| mountPath: /rollback-state | ||
| volumes: | ||
| - name: rollback-state | ||
| hostPath: | ||
| path: /kinder/rollback | ||
| type: Directory | ||
| PATCH | ||
|
|
||
| version=$(kubeadm version -o short) | ||
| printf '%s\n' "${version}" > "${state}/version" | ||
| cat > "${state}/upgrade.yaml" <<CONFIG | ||
| apiVersion: kubeadm.k8s.io/v1beta4 | ||
| kind: UpgradeConfiguration | ||
| apply: | ||
| kubernetesVersion: ${version} | ||
| allowExperimentalUpgrades: true | ||
| allowRCUpgrades: true | ||
| forceUpgrade: true | ||
| etcdUpgrade: true | ||
| certificateRenewal: false | ||
| patches: | ||
| directory: ${state}/patches | ||
| timeouts: | ||
| upgradeManifests: 2m | ||
| # Bound deliberately unhealthy etcd checks, including kubeadm's ten retries. | ||
| etcdAPICall: 5s | ||
| CONFIG | ||
| timeout: 5m | ||
| - name: fail-etcd-upgrade | ||
| description: Run the real kubeadm upgrade and require both a nonzero exit code and completed fault injection. | ||
| cmd: docker | ||
| args: | ||
| - exec | ||
| - '{{ .vars.clusterName }}-control-plane-1' | ||
| - bash | ||
| - -ceu | ||
| - | | ||
| set -euo pipefail | ||
| export KUBECONFIG=/etc/kubernetes/admin.conf | ||
| state=/kinder/rollback | ||
| data=/var/lib/etcd | ||
| result=0 | ||
| kubeadm upgrade apply --config "${state}/upgrade.yaml" --v=6 > "${state}/upgrade.log" 2>&1 || result=$? | ||
| printf '%s\n' "${result}" > "${state}/upgrade-exit-code" | ||
| cat "${state}/upgrade.log" | ||
| if [[ "${result}" -eq 0 ]]; then | ||
| echo 'FAIL: upgrade unexpectedly succeeded' >&2 | ||
| exit 1 | ||
| fi | ||
| # This acknowledgment is outside the data backup and is written only after | ||
| # injection completes. A preflight error or an image-pull failure cannot pass. | ||
| if [[ ! -s "${state}/fault-injected" ]]; then | ||
| echo 'FAIL: upgrade failed before the fault injection completed' >&2 | ||
| exit 1 | ||
| fi | ||
| timeout: 15m | ||
| - name: verify-etcd-data-rollback | ||
| description: Require the original manifest/data marker and removal of files created after backup. | ||
| cmd: docker | ||
| args: | ||
| - exec | ||
| - '{{ .vars.clusterName }}-control-plane-1' | ||
| - bash | ||
| - -ceu | ||
| - | | ||
| set -euo pipefail | ||
| export KUBECONFIG=/etc/kubernetes/admin.conf | ||
| state=/kinder/rollback | ||
| data=/var/lib/etcd | ||
| failed=0 | ||
| if ! cmp -s "${state}/etcd-before.yaml" /etc/kubernetes/manifests/etcd.yaml; then | ||
| echo 'FAIL: original etcd manifest was not restored' >&2 | ||
| failed=1 | ||
| fi | ||
| if ! cmp -s "${state}/marker-before" "${data}/rollback-marker"; then | ||
| echo 'FAIL: original etcd data was not restored' >&2 | ||
| failed=1 | ||
| fi | ||
| if [[ ! -d "${data}/member" || -e "${data}/member.failed-upgrade" || -e "${data}/rollback-extra" ]]; then | ||
| echo 'FAIL: failed-upgrade files survived rollback' >&2 | ||
| failed=1 | ||
| fi | ||
| if [[ -n "$(find "${data}" -mindepth 1 -maxdepth 1 -name 'kubeadm-backup-etcd-*')" ]]; then | ||
| echo 'FAIL: backup was nested inside the live etcd directory' >&2 | ||
| failed=1 | ||
| fi | ||
| # Fail on a broken restore without adding a second API-readiness timeout. | ||
| [[ "${failed}" -eq 0 ]] || exit 1 | ||
| timeout: 1m | ||
| - name: verify-api-recovery | ||
| description: Wait for API recovery and verify that the original ConfigMap UID and value survived rollback. | ||
| cmd: docker | ||
| args: | ||
| - exec | ||
| - '{{ .vars.clusterName }}-control-plane-1' | ||
| - bash | ||
| - -ceu | ||
| - | | ||
| set -euo pipefail | ||
| export KUBECONFIG=/etc/kubernetes/admin.conf | ||
| state=/kinder/rollback | ||
| data=/var/lib/etcd | ||
| deadline=$((SECONDS + 180)) | ||
| until kubectl --request-timeout=5s get --raw=/readyz > /dev/null 2>&1; do | ||
| if (( SECONDS >= deadline )); then | ||
| echo 'FAIL: API server did not recover after rollback' >&2 | ||
| exit 1 | ||
| fi | ||
| sleep 2 | ||
| done | ||
| kubectl --request-timeout=10s get configmap rollback-test -o jsonpath='{.metadata.uid}:{.data.value}' > "${state}/object-after" | ||
| cmp "${state}/object-before" "${state}/object-after" | ||
| timeout: 4m | ||
| - name: collect-rollback-diagnostics | ||
| cmd: docker | ||
| args: | ||
| - exec | ||
| - '{{ .vars.clusterName }}-control-plane-1' | ||
| - bash | ||
| - -ceu | ||
| - | | ||
| mkdir -p /kinder/rollback | ||
| find /var/lib/etcd -maxdepth 6 -type f -printf '%p %s bytes\n' > /kinder/rollback/data-files.txt | ||
| find /var/lib/etcd -maxdepth 1 -printf '%f %y\n' > /kinder/rollback/data-entries.txt | ||
| crictl ps -a > /kinder/rollback/containers.txt | ||
| force: true | ||
| ignoreError: true | ||
| - name: export-rollback-diagnostics | ||
| cmd: docker | ||
| args: | ||
| - cp | ||
| - '{{ .vars.clusterName }}-control-plane-1:/kinder/rollback' | ||
| - '{{ .env.ARTIFACTS }}/etcd-rollback' | ||
| force: true | ||
| ignoreError: true | ||
|
Comment on lines
+213
to
+234
Member
There was a problem hiding this comment. i am not so sure why these are two separate tasks. also do we really care about diagonostics? |
||
| - name: get-logs | ||
| cmd: kinder | ||
| args: | ||
| - export | ||
| - logs | ||
| - --name={{ .vars.clusterName }} | ||
| - --loglevel=debug | ||
| - '{{ .env.ARTIFACTS }}' | ||
| force: true | ||
| timeout: 5m | ||
| ignoreError: true | ||
| - name: delete | ||
| cmd: kinder | ||
| args: | ||
| - delete | ||
| - cluster | ||
| - --name={{ .vars.clusterName }} | ||
| - --loglevel=debug | ||
| force: true | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we really want to nest workflows within workflows? the mapping of one prow job -> one kinder workflow has been flexible enough for us. sure it can be annoying to have to update these in 2 repositories, but we have a generator for that purpose.
TBH, i am not so sure about this. additionally it introduces running
upgrade-no-addon-config-mapsin a second location, which duplicates testing the same aspect.could we simply introduce the dedicated upgrade-etcd-rollback workflow + its tasks?