Skip to content

Automate PMM HA installation on Linode LKE cluster #1129

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
shrutipradhan-percona wants to merge 2 commits into
base: main
Choose a base branch
from
+221 −0
Open

Automate PMM HA installation on Linode LKE cluster #1129

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
76d5a9a
Add files via upload
shrutipradhan-percona Aug 3, 2026
27a6c82
Enhance PMM installation script with password generation
shrutipradhan-percona Aug 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
211 changes: 211 additions & 0 deletions k8s/createLKE_install_PMM_HA.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,211 @@
#!/bin/bash

set -euo pipefail

#############################################
# Configuration
#############################################

CLUSTER_LABEL="pmm-ha-shruti-install-ha-3aug"
REGION="ap-west" # Change if required
K8S_VERSION="1.36"
NODE_TYPE="g6-standard-4"
NODE_COUNT=7

#KUBECONFIG_FILE="$HOME/.kube/pmm-ha-lke-config"

#############################################
# Prerequisite Checks
#############################################

echo "Checking prerequisites..."

for cmd in linode-cli jq kubectl base64; do
if ! command -v "$cmd" >/dev/null 2>&1; then
echo "ERROR: '$cmd' is not installed."
exit 1
fi
done
Comment on lines +23 to +28

@coderabbitai coderabbitai Bot Aug 3, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Validate helm before creating cloud resources.

The script invokes helm at Line 143. The prerequisite loop does not validate it. If helm is absent, the script creates the LKE cluster and then fails during installation.

Proposed fix 
-for cmd in linode-cli jq kubectl base64; do
+for cmd in linode-cli jq kubectl base64 helm; do
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
for cmd in linode-cli jq kubectl base64; do
if ! command -v "$cmd" >/dev/null 2>&1; then
echo "ERROR: '$cmd' is not installed."
exit 1
fi
done
for cmd in linode-cli jq kubectl base64 helm; do
if ! command -v "$cmd" >/dev/null 2>&1; then
echo "ERROR: '$cmd' is not installed."
exit 1
fi
done
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@k8s/createLKE_install_PMM_HA.sh` around lines 23 - 28, Update the
prerequisite command loop in createLKE_install_PMM_HA.sh to include helm
alongside linode-cli, jq, kubectl, and base64, ensuring helm is validated before
any cloud resources are created.


echo "All required tools found."
echo

#############################################
# Create Cluster
#############################################
echo "Creating Kubernetes cluster..."

linode-cli lke cluster-create \
--label "$CLUSTER_LABEL" \
--region "$REGION" \
--k8s_version "$K8S_VERSION" \
--node_pools.type "$NODE_TYPE" \
--node_pools.count "$NODE_COUNT"

echo
echo "Cluster creation request submitted."
echo

#############################################
# Retrieve Cluster ID
#############################################

echo "It takes some time to Create cluster and add nodes. Please wait..."
echo "Retrieving Cluster ID..."
sleep 120

CLUSTER_ID=""
while [ -z "$CLUSTER_ID" ] || [ "$CLUSTER_ID" = "null" ]; do
CLUSTER_ID=$(linode-cli lke clusters-list --json | jq -r --arg label "$CLUSTER_LABEL" '.[] | select(.label | contains($label)) | .id')

if [ -z "$CLUSTER_ID" ] || [ "$CLUSTER_ID" = "null" ]; then
echo "Waiting for cluster to appear..."
sleep 10
CLUSTER_ID="" # Reset to ensure the loop expression evaluates correctly next turn
fi
done
Comment on lines +58 to +66

@coderabbitai coderabbitai Bot Aug 3, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Add deadlines to polling loops.

Both loops can run forever when the provider reports a persistent failure. The script then never returns control or reports a useful failure.

  • k8s/createLKE_install_PMM_HA.sh#L58-L66: enforce a deadline while waiting for the created cluster to appear.
  • k8s/createLKE_install_PMM_HA.sh#L200-L200: enforce a deadline while waiting for the LoadBalancer ingress address.
📍 Affects 1 file
  • k8s/createLKE_install_PMM_HA.sh#L58-L66 (this comment)
  • k8s/createLKE_install_PMM_HA.sh#L200-L200
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@k8s/createLKE_install_PMM_HA.sh` around lines 58 - 66, Bound both polling
loops in k8s/createLKE_install_PMM_HA.sh: the cluster lookup loop around lines
58-66 and the LoadBalancer ingress wait around line 200. Add a clear deadline or
timeout counter to each, and when exceeded, report a useful failure and exit
instead of continuing indefinitely.


echo "Cluster ID: $CLUSTER_ID"

#############################################
# Wait Until Ready
#############################################

echo "Waiting for cluster to become READY..."

[ $(linode-cli lke pools-list "$CLUSTER_ID" --json | jq '.[] | .nodes[] | select(.status == "ready")' | jq -s 'length') -eq $NODE_COUNT ] && echo "Cluster is READY"
sleep 90
Comment on lines +74 to +77

@coderabbitai coderabbitai Bot Aug 3, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Poll node readiness until a bounded deadline.

Line 76 performs one readiness check. The script continues after Line 77 even when fewer than NODE_COUNT nodes are ready. The kubeconfig and later Helm operations can then fail against a cluster that is still provisioning.

Replace the fixed sleep with a retry loop that exits with an error at a defined deadline.

🧰 Tools
🪛 Shellcheck (0.11.0)

[warning] 76-76: Quote this to prevent word splitting.

(SC2046)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@k8s/createLKE_install_PMM_HA.sh` around lines 74 - 77, Replace the single
readiness check and fixed sleep after “Waiting for cluster to become READY...”
with a polling loop that repeatedly evaluates the ready-node count against
NODE_COUNT, sleeps between attempts, and proceeds only when all nodes are ready.
Add a defined timeout/deadline; when it expires, print an error and exit nonzero
before kubeconfig or Helm operations continue.

#############################################
# Download & Export kubeconfig
#############################################
# Create the directory if it doesn't exist
unset KUBECONFIG
rm -f ~/.kube/config
Comment thread
coderabbitai[bot] marked this conversation as resolved.
mkdir -p /tmp/HA-linode

# Download kubeconfig
linode-cli lke kubeconfig-view "$CLUSTER_ID" --json \
| jq -r '.[0].kubeconfig' \
| base64 --decode > /tmp/HA-linode/kubeconfig.yaml

# Export KUBECONFIG
export KUBECONFIG=/tmp/HA-linode/kubeconfig.yaml
#cp /Users/shruti/Scripts/HA-linode/kubeconfig.yaml ~/.kube/config
#chmod 600 ~/.kube/config
echo "KUBECONFIG exported: $KUBECONFIG"

if [ -f /tmp/HA-linode/kubeconfig.yaml ]; then
export KUBECONFIG=/tmp/HA-linode/kubeconfig.yaml
echo "Kubeconfig downloaded successfully."
else
echo "ERROR: Failed to download kubeconfig."
exit 1
fi
Comment on lines +84 to +101

@coderabbitai coderabbitai Bot Aug 3, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Use private, unique temporary directories.

Both diagnostic paths are predictable shared paths under /tmp. Another local user can pre-create these paths or symlinks. The kubeconfig path can expose or overwrite credential data.

  • k8s/createLKE_install_PMM_HA.sh#L84-L103: create a unique mktemp -d directory with restrictive permissions and store the kubeconfig inside it.
  • k8s/createLKE_install_PMM_HA.sh#L208-L211: create a separate unique mktemp -d directory for diagnostics and write all artifacts there.
🧰 Tools
🪛 ast-grep (0.45.0)

[warning] 88-88: Writing to or reading from a hardcoded, predictable path under /tmp is vulnerable to symlink and TOCTOU attacks: a local attacker can pre-create the file (or a symlink pointing elsewhere) and hijack or corrupt the contents. Generate a unique, unpredictable temporary file with mktemp instead, e.g. tmpfile="$(mktemp)" (or mktemp -d for directories) and reference "$tmpfile".
Context: /tmp/HA-linode/kubeconfig.yaml
Note: [CWE-377] Insecure Temporary File.

(predictable-tmp-file-bash)

[warning] 96-96: Writing to or reading from a hardcoded, predictable path under /tmp is vulnerable to symlink and TOCTOU attacks: a local attacker can pre-create the file (or a symlink pointing elsewhere) and hijack or corrupt the contents. Generate a unique, unpredictable temporary file with mktemp instead, e.g. tmpfile="$(mktemp)" (or mktemp -d for directories) and reference "$tmpfile".
Context: /tmp/HA-linode/kubeconfig.yaml
Note: [CWE-377] Insecure Temporary File.

(predictable-tmp-file-bash)

📍 Affects 1 file
  • k8s/createLKE_install_PMM_HA.sh#L84-L103 (this comment)
  • k8s/createLKE_install_PMM_HA.sh#L208-L211
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@k8s/createLKE_install_PMM_HA.sh` around lines 84 - 103, Replace the
predictable /tmp/HA-linode directory in the kubeconfig setup with a uniquely
created, restrictive mktemp -d directory and store kubeconfig within it,
updating KUBECONFIG and validation accordingly. Also update the diagnostics
block at k8s/createLKE_install_PMM_HA.sh lines 208-211 to use a separate unique
mktemp -d directory for all diagnostic artifacts.

Source: Linters/SAST tools

#############################################
# Verify Cluster
#############################################

echo "Current Context:"
kubectl config current-context

echo
echo "Worker Nodes:"
kubectl get nodes

echo
echo "System Pods:"
kubectl get pods -A

#############################################
# Finished
#############################################

echo
echo "======================================="
echo "Linode Kubernetes Cluster is Ready!"
echo "Cluster ID : $CLUSTER_ID"
echo "Kubeconfig : $KUBECONFIG"
echo "======================================="
echo
echo "Cluster is ready for PMM HA Installation using Helm."

sleep 60

######################################
#Install PMM HA dependencies
######################################
# 1. Create PMM Namespace on LKE

kubectl create namespace pmm

#2. Install PMM HA dependencies - Install operators

helm repo add percona https://percona.github.io/percona-helm-charts/ --force-update
helm repo update


helm install pmm-operators percona/pmm-ha-dependencies --namespace pmm

# Wait for all operators to be ready (typically 2-3 minutes)
kubectl wait --for=condition=ready pod \
-l app.kubernetes.io/name=victoria-metrics-operator \
-n pmm --timeout=300s
kubectl wait --for=condition=ready pod \
-l app.kubernetes.io/name=altinity-clickhouse-operator \
-n pmm --timeout=300s
kubectl wait --for=condition=ready pod \
-l app.kubernetes.io/name=pg-operator \
-n pmm --timeout=300s
echo "PMM dependencies installed successfully!"
# check pods
kubectl get pods -n pmm

# Create secret

kubectl create secret generic pmm-secret \
--from-literal=PMM_ADMIN_PASSWORD="admin" \
--from-literal=PMM_CLICKHOUSE_USER="clickhouse_pmm" \
--from-literal=PMM_CLICKHOUSE_PASSWORD="clickhouse-password" \
--from-literal=VMAGENT_remoteWrite_basicAuth_username="victoriametrics_pmm" \
--from-literal=VMAGENT_remoteWrite_basicAuth_password="vm-password" \
--from-literal=PG_PASSWORD="postgres-password" \
--from-literal=GF_PASSWORD="grafana-password" \
--namespace pmm
Comment thread
coderabbitai[bot] marked this conversation as resolved.
echo " PMM Secrets created successfully!"
sleep 60


#####################################
#Install PMM HA
#####################################

helm install pmm-ha percona/pmm-ha --namespace pmm

# Wait for deployment to complete

kubectl wait --for=condition=ready \
$(kubectl get pods -n pmm -o name | grep pmm-ha-haproxy) \
-n pmm --timeout=15m

kubectl get pods -n pmm
echo " HAPPY HELMING!!!!"

#####################################
# External access - LOAD BALANCER
#####################################
kubectl patch svc pmm-ha-haproxy \
-n pmm \
-p '{"spec":{"type":"LoadBalancer"}}'

while true; do EXTERNAL_IP=$(kubectl get svc pmm-ha-haproxy -n pmm -o jsonpath='{.status.loadBalancer.ingress[0].ip}'); if [[ -n "$EXTERNAL_IP" ]]; then break; fi; echo "Waiting for LoadBalancer..."; sleep 15; done
echo "External IP: ${EXTERNAL_IP}"
echo "PMM HA is available at:"
echo "https://${EXTERNAL_IP}"

###################################
#Get Events and logs
####################################
mkdir -p /tmp/helm-debug
kubectl get pods -n pmm -o wide > /tmp/helm-debug/pods.txt || true
kubectl get events -n pmm --sort-by=.metadata.creationTimestamp > /tmp/helm-debug/events.txt || true
Comment on lines +206 to +210

@coderabbitai coderabbitai Bot Aug 4, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

# Get file context and check the full section around lines 206-210
wc -l k8s/createLKE_install_PMM_HA.sh
head -220 k8s/createLKE_install_PMM_HA.sh | tail -40

Repository: percona/pmm-qa

Length of output: 1538

🏁 Script executed:

# Search for any existing kubectl logs usage in the file
rg "kubectl logs" k8s/createLKE_install_PMM_HA.sh

Repository: percona/pmm-qa

Length of output: 152

🏁 Script executed:

# Check if there's any kubectl logs collection elsewhere or similar debugging patterns
rg "logs|describe" k8s/createLKE_install_PMM_HA.sh | head -20

Repository: percona/pmm-qa

Length of output: 228

Collect container logs, not only pod metadata.

The section header states "Cluster Summary and pod logs", but kubectl get pods -o wide provides only pod status and details. It does not collect container logs. The inline comment "Pod logs and events at /tmp/helm-debug" is inaccurate—only metadata and events are saved. Add kubectl logs commands for PMM pods to the /tmp/helm-debug directory. Handle multiple containers within pods using the --all-containers=true flag or by specifying container names explicitly. Use || true to preserve script resilience if log collection fails.

🧰 Tools
🪛 ast-grep (0.45.0)

[warning] 208-208: Writing to or reading from a hardcoded, predictable path under /tmp is vulnerable to symlink and TOCTOU attacks: a local attacker can pre-create the file (or a symlink pointing elsewhere) and hijack or corrupt the contents. Generate a unique, unpredictable temporary file with mktemp instead, e.g. tmpfile="$(mktemp)" (or mktemp -d for directories) and reference "$tmpfile".
Context: /tmp/helm-debug/pods.txt
Note: [CWE-377] Insecure Temporary File.

(predictable-tmp-file-bash)

[warning] 209-209: Writing to or reading from a hardcoded, predictable path under /tmp is vulnerable to symlink and TOCTOU attacks: a local attacker can pre-create the file (or a symlink pointing elsewhere) and hijack or corrupt the contents. Generate a unique, unpredictable temporary file with mktemp instead, e.g. tmpfile="$(mktemp)" (or mktemp -d for directories) and reference "$tmpfile".
Context: /tmp/helm-debug/events.txt
Note: [CWE-377] Insecure Temporary File.

(predictable-tmp-file-bash)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@k8s/createLKE_install_PMM_HA.sh` around lines 206 - 210, The current kubectl
get pods and kubectl get events commands collect only pod metadata and events,
not actual container logs as indicated by the section header. Add kubectl logs
commands that capture the actual container output from PMM pods to the
/tmp/helm-debug directory. Use the --all-containers=true flag to handle pods
with multiple containers, and apply the || true pattern to each logs command to
maintain the script's resilience when log collection encounters errors.

echo "Pod logs and events at /tmp/helm-debug"
Loading