-
Notifications
You must be signed in to change notification settings - Fork 15
Automate PMM HA installation on Linode LKE cluster #1129
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,221 @@ | ||||||||||||||||||||||||||
| #!/bin/bash | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| set -euo pipefail | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| ############################################# | ||||||||||||||||||||||||||
| # Configuration | ||||||||||||||||||||||||||
| ############################################# | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| CLUSTER_LABEL="pmm-ha-shruti-install-ha-3aug" | ||||||||||||||||||||||||||
| REGION="ap-west" # Change if required | ||||||||||||||||||||||||||
| K8S_VERSION="1.36" | ||||||||||||||||||||||||||
| NODE_TYPE="g6-standard-4" | ||||||||||||||||||||||||||
| NODE_COUNT=7 | ||||||||||||||||||||||||||
| PMM_PW=$1 #PMM password | ||||||||||||||||||||||||||
| #KUBECONFIG_FILE="$HOME/.kube/pmm-ha-lke-config" | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| ############################################# | ||||||||||||||||||||||||||
| # Prerequisite Checks | ||||||||||||||||||||||||||
| ############################################# | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| echo "Checking prerequisites..." | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| for cmd in linode-cli jq kubectl base64; do | ||||||||||||||||||||||||||
| if ! command -v "$cmd" >/dev/null 2>&1; then | ||||||||||||||||||||||||||
| echo "ERROR: '$cmd' is not installed." | ||||||||||||||||||||||||||
| exit 1 | ||||||||||||||||||||||||||
| fi | ||||||||||||||||||||||||||
| done | ||||||||||||||||||||||||||
|
Comment on lines
+23
to
+28
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win Validate The script invokes Proposed fix-for cmd in linode-cli jq kubectl base64; do
+for cmd in linode-cli jq kubectl base64 helm; do📝 Committable suggestion
Suggested change
🤖 Prompt for AI Agents |
||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| echo "All required tools found." | ||||||||||||||||||||||||||
| echo | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| ############################################# | ||||||||||||||||||||||||||
| # Create Cluster | ||||||||||||||||||||||||||
| ############################################# | ||||||||||||||||||||||||||
| echo "Creating Kubernetes cluster..." | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| linode-cli lke cluster-create \ | ||||||||||||||||||||||||||
| --label "$CLUSTER_LABEL" \ | ||||||||||||||||||||||||||
| --region "$REGION" \ | ||||||||||||||||||||||||||
| --k8s_version "$K8S_VERSION" \ | ||||||||||||||||||||||||||
| --node_pools.type "$NODE_TYPE" \ | ||||||||||||||||||||||||||
| --node_pools.count "$NODE_COUNT" | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| echo | ||||||||||||||||||||||||||
| echo "Cluster creation request submitted." | ||||||||||||||||||||||||||
| echo | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| ############################################# | ||||||||||||||||||||||||||
| # Retrieve Cluster ID | ||||||||||||||||||||||||||
| ############################################# | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| echo "It takes some time to Create cluster and add nodes. Please wait..." | ||||||||||||||||||||||||||
| echo "Retrieving Cluster ID..." | ||||||||||||||||||||||||||
| sleep 120 | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| CLUSTER_ID="" | ||||||||||||||||||||||||||
| while [ -z "$CLUSTER_ID" ] || [ "$CLUSTER_ID" = "null" ]; do | ||||||||||||||||||||||||||
| CLUSTER_ID=$(linode-cli lke clusters-list --json | jq -r --arg label "$CLUSTER_LABEL" '.[] | select(.label | contains($label)) | .id') | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| if [ -z "$CLUSTER_ID" ] || [ "$CLUSTER_ID" = "null" ]; then | ||||||||||||||||||||||||||
| echo "Waiting for cluster to appear..." | ||||||||||||||||||||||||||
| sleep 10 | ||||||||||||||||||||||||||
| CLUSTER_ID="" # Reset to ensure the loop expression evaluates correctly next turn | ||||||||||||||||||||||||||
| fi | ||||||||||||||||||||||||||
| done | ||||||||||||||||||||||||||
|
Comment on lines
+58
to
+66
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win Add deadlines to polling loops. Both loops can run forever when the provider reports a persistent failure. The script then never returns control or reports a useful failure.
📍 Affects 1 file
🤖 Prompt for AI Agents |
||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| echo "Cluster ID: $CLUSTER_ID" | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| ############################################# | ||||||||||||||||||||||||||
| # Wait Until Ready | ||||||||||||||||||||||||||
| ############################################# | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| echo "Waiting for cluster to become READY..." | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| [ $(linode-cli lke pools-list "$CLUSTER_ID" --json | jq '.[] | .nodes[] | select(.status == "ready")' | jq -s 'length') -eq $NODE_COUNT ] && echo "Cluster is READY" | ||||||||||||||||||||||||||
| sleep 90 | ||||||||||||||||||||||||||
|
Comment on lines
+74
to
+77
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win Poll node readiness until a bounded deadline. Line 76 performs one readiness check. The script continues after Line 77 even when fewer than Replace the fixed sleep with a retry loop that exits with an error at a defined deadline. 🧰 Tools🪛 Shellcheck (0.11.0)[warning] 76-76: Quote this to prevent word splitting. (SC2046) 🤖 Prompt for AI Agents |
||||||||||||||||||||||||||
| ############################################# | ||||||||||||||||||||||||||
| # Download & Export kubeconfig | ||||||||||||||||||||||||||
| ############################################# | ||||||||||||||||||||||||||
| # Create the directory if it doesn't exist | ||||||||||||||||||||||||||
| unset KUBECONFIG | ||||||||||||||||||||||||||
| rm -f ~/.kube/config | ||||||||||||||||||||||||||
|
coderabbitai[bot] marked this conversation as resolved.
|
||||||||||||||||||||||||||
| mkdir -p /tmp/HA-linode | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| # Download kubeconfig | ||||||||||||||||||||||||||
| linode-cli lke kubeconfig-view "$CLUSTER_ID" --json \ | ||||||||||||||||||||||||||
| | jq -r '.[0].kubeconfig' \ | ||||||||||||||||||||||||||
| | base64 --decode > /tmp/HA-linode/kubeconfig.yaml | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| # Export KUBECONFIG | ||||||||||||||||||||||||||
| export KUBECONFIG=/tmp/HA-linode/kubeconfig.yaml | ||||||||||||||||||||||||||
| echo "KUBECONFIG exported: $KUBECONFIG" | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| if [ -f /tmp/HA-linode/kubeconfig.yaml ]; then | ||||||||||||||||||||||||||
| export KUBECONFIG=/tmp/HA-linode/kubeconfig.yaml | ||||||||||||||||||||||||||
| echo "Kubeconfig downloaded successfully." | ||||||||||||||||||||||||||
| else | ||||||||||||||||||||||||||
| echo "ERROR: Failed to download kubeconfig." | ||||||||||||||||||||||||||
| exit 1 | ||||||||||||||||||||||||||
| fi | ||||||||||||||||||||||||||
|
Comment on lines
+84
to
+101
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔒 Security & Privacy | 🟠 Major | ⚡ Quick win Use private, unique temporary directories. Both diagnostic paths are predictable shared paths under
🧰 Tools🪛 ast-grep (0.45.0)[warning] 88-88: Writing to or reading from a hardcoded, predictable path under /tmp is vulnerable to symlink and TOCTOU attacks: a local attacker can pre-create the file (or a symlink pointing elsewhere) and hijack or corrupt the contents. Generate a unique, unpredictable temporary file with mktemp instead, e.g. (predictable-tmp-file-bash) [warning] 96-96: Writing to or reading from a hardcoded, predictable path under /tmp is vulnerable to symlink and TOCTOU attacks: a local attacker can pre-create the file (or a symlink pointing elsewhere) and hijack or corrupt the contents. Generate a unique, unpredictable temporary file with mktemp instead, e.g. (predictable-tmp-file-bash) 📍 Affects 1 file
🤖 Prompt for AI AgentsSource: Linters/SAST tools |
||||||||||||||||||||||||||
| ############################################# | ||||||||||||||||||||||||||
| # Verify Cluster | ||||||||||||||||||||||||||
| ############################################# | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| echo "Current Context:" | ||||||||||||||||||||||||||
| kubectl config current-context | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| echo | ||||||||||||||||||||||||||
| echo "Worker Nodes:" | ||||||||||||||||||||||||||
| kubectl get nodes | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| echo | ||||||||||||||||||||||||||
| echo "System Pods:" | ||||||||||||||||||||||||||
| kubectl get pods -A | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| ############################################# | ||||||||||||||||||||||||||
| # Finished | ||||||||||||||||||||||||||
| ############################################# | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| echo "=======================================" | ||||||||||||||||||||||||||
| echo "Linode Kubernetes Cluster is Ready!" | ||||||||||||||||||||||||||
| echo "Cluster ID : $CLUSTER_ID" | ||||||||||||||||||||||||||
| echo "Kubeconfig : $KUBECONFIG" | ||||||||||||||||||||||||||
| echo "=======================================" | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| echo "Cluster is ready for PMM HA Installation using Helm." | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| sleep 60 | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| ###################################### | ||||||||||||||||||||||||||
| #Install PMM HA dependencies | ||||||||||||||||||||||||||
| ###################################### | ||||||||||||||||||||||||||
| # 1. Create PMM Namespace on LKE | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| kubectl create namespace pmm | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| #2. Install PMM HA dependencies - Install operators | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| helm repo add percona https://percona.github.io/percona-helm-charts/ --force-update | ||||||||||||||||||||||||||
| helm repo update | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| helm install pmm-operators percona/pmm-ha-dependencies --namespace pmm | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| # Wait for all operators to be ready (typically 2-3 minutes) | ||||||||||||||||||||||||||
| kubectl wait --for=condition=ready pod \ | ||||||||||||||||||||||||||
| -l app.kubernetes.io/name=victoria-metrics-operator \ | ||||||||||||||||||||||||||
| -n pmm --timeout=300s | ||||||||||||||||||||||||||
| kubectl wait --for=condition=ready pod \ | ||||||||||||||||||||||||||
| -l app.kubernetes.io/name=altinity-clickhouse-operator \ | ||||||||||||||||||||||||||
| -n pmm --timeout=300s | ||||||||||||||||||||||||||
| kubectl wait --for=condition=ready pod \ | ||||||||||||||||||||||||||
| -l app.kubernetes.io/name=pg-operator \ | ||||||||||||||||||||||||||
| -n pmm --timeout=300s | ||||||||||||||||||||||||||
| echo "PMM dependencies installed successfully!" | ||||||||||||||||||||||||||
| # check pods | ||||||||||||||||||||||||||
| kubectl get pods -n pmm | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| # Create secret | ||||||||||||||||||||||||||
| PG_PW=$(openssl rand -base64 24 | tr -dc 'a-zA-Z0-9' | head -c 24) | ||||||||||||||||||||||||||
| GF_PW=$(openssl rand -base64 24 | tr -dc 'a-zA-Z0-9' | head -c 24) | ||||||||||||||||||||||||||
| CH_PW=$(openssl rand -base64 24 | tr -dc 'a-zA-Z0-9' | head -c 24) | ||||||||||||||||||||||||||
| VM_PW=$(openssl rand -base64 24 | tr -dc 'a-zA-Z0-9' | head -c 24) | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| kubectl create secret generic pmm-secret \ | ||||||||||||||||||||||||||
| --from-literal=PMM_ADMIN_PASSWORD="$PMM_PW" \ | ||||||||||||||||||||||||||
| --from-literal=PMM_CLICKHOUSE_USER="clickhouse_pmm" \ | ||||||||||||||||||||||||||
| --from-literal=PMM_CLICKHOUSE_PASSWORD="$CH_PW" \ | ||||||||||||||||||||||||||
| --from-literal=VMAGENT_remoteWrite_basicAuth_username="victoriametrics_pmm" \ | ||||||||||||||||||||||||||
| --from-literal=VMAGENT_remoteWrite_basicAuth_password="$VM_PW" \ | ||||||||||||||||||||||||||
| --from-literal=PG_PASSWORD="$PG_PW" \ | ||||||||||||||||||||||||||
| --from-literal=GF_PASSWORD="$GF_PW" \ | ||||||||||||||||||||||||||
| --namespace pmm | ||||||||||||||||||||||||||
|
coderabbitai[bot] marked this conversation as resolved.
|
||||||||||||||||||||||||||
| echo " PMM Secrets created successfully!" | ||||||||||||||||||||||||||
| sleep 45 | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| ##################################### | ||||||||||||||||||||||||||
| #Install PMM HA | ||||||||||||||||||||||||||
| ##################################### | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| helm install pmm-ha percona/pmm-ha --namespace pmm | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| # Wait for deployment to complete | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| kubectl wait --for=condition=ready \ | ||||||||||||||||||||||||||
| $(kubectl get pods -n pmm -o name | grep pmm-ha-haproxy) \ | ||||||||||||||||||||||||||
| -n pmm --timeout=15m | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| kubectl get pods -n pmm | ||||||||||||||||||||||||||
| echo " HAPPY HELMING!!!!" | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| ##################################### | ||||||||||||||||||||||||||
| # External access - LOAD BALANCER | ||||||||||||||||||||||||||
| ##################################### | ||||||||||||||||||||||||||
| kubectl patch svc pmm-ha-haproxy \ | ||||||||||||||||||||||||||
| -n pmm \ | ||||||||||||||||||||||||||
| -p '{"spec":{"type":"LoadBalancer"}}' | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| while true; do EXTERNAL_IP=$(kubectl get svc pmm-ha-haproxy -n pmm -o jsonpath='{.status.loadBalancer.ingress[0].ip}'); if [[ -n "$EXTERNAL_IP" ]]; then break; fi; echo "Waiting for LoadBalancer..."; sleep 15; done | ||||||||||||||||||||||||||
| echo "External IP: ${EXTERNAL_IP}" | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| ################################### | ||||||||||||||||||||||||||
| #Cluster Summary and pod logs | ||||||||||||||||||||||||||
| #################################### | ||||||||||||||||||||||||||
| mkdir -p /tmp/helm-debug | ||||||||||||||||||||||||||
| kubectl get pods -n pmm -o wide > /tmp/helm-debug/pods.txt || true | ||||||||||||||||||||||||||
| kubectl get events -n pmm --sort-by=.metadata.creationTimestamp > /tmp/helm-debug/events.txt || true | ||||||||||||||||||||||||||
|
Comment on lines
+206
to
+210
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: # Get file context and check the full section around lines 206-210
wc -l k8s/createLKE_install_PMM_HA.sh
head -220 k8s/createLKE_install_PMM_HA.sh | tail -40Repository: percona/pmm-qa Length of output: 1538 🏁 Script executed: # Search for any existing kubectl logs usage in the file
rg "kubectl logs" k8s/createLKE_install_PMM_HA.shRepository: percona/pmm-qa Length of output: 152 🏁 Script executed: # Check if there's any kubectl logs collection elsewhere or similar debugging patterns
rg "logs|describe" k8s/createLKE_install_PMM_HA.sh | head -20Repository: percona/pmm-qa Length of output: 228 Collect container logs, not only pod metadata. The section header states "Cluster Summary and pod logs", but 🧰 Tools🪛 ast-grep (0.45.0)[warning] 208-208: Writing to or reading from a hardcoded, predictable path under /tmp is vulnerable to symlink and TOCTOU attacks: a local attacker can pre-create the file (or a symlink pointing elsewhere) and hijack or corrupt the contents. Generate a unique, unpredictable temporary file with mktemp instead, e.g. (predictable-tmp-file-bash) [warning] 209-209: Writing to or reading from a hardcoded, predictable path under /tmp is vulnerable to symlink and TOCTOU attacks: a local attacker can pre-create the file (or a symlink pointing elsewhere) and hijack or corrupt the contents. Generate a unique, unpredictable temporary file with mktemp instead, e.g. (predictable-tmp-file-bash) 🤖 Prompt for AI Agents |
||||||||||||||||||||||||||
| echo "Pod logs and events at /tmp/helm-debug" | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| echo " CLUSTER SUMMARY---------" | ||||||||||||||||||||||||||
| echo "External IP: ${EXTERNAL_IP}" | ||||||||||||||||||||||||||
| echo "PMM HA is available at:" | ||||||||||||||||||||||||||
| echo "https://${EXTERNAL_IP}" | ||||||||||||||||||||||||||
| echo "PMM PAssword : $PMM_PW" | ||||||||||||||||||||||||||
| echo "Clickhouse password: $CH_PW" | ||||||||||||||||||||||||||
| echo "Postgres password: $PG_PW" | ||||||||||||||||||||||||||
| echo "Grafana password: $GF_PW" | ||||||||||||||||||||||||||
| echo "VM password: $VM_PW" | ||||||||||||||||||||||||||
|
Comment on lines
+217
to
+221
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔒 Security & Privacy | 🟠 Major | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: cat -n k8s/createLKE_install_PMM_HA.sh | sed -n '215,225p'Repository: percona/pmm-qa Length of output: 419 🏁 Script executed: # Check if these are the only places where passwords are echoed in the script
rg -n "echo.*PW" k8s/createLKE_install_PMM_HA.shRepository: percona/pmm-qa Length of output: 329 🏁 Script executed: # Verify the password variables are generated or set in the script
rg -n "PMM_PW|CH_PW|PG_PW|GF_PW|VM_PW" k8s/createLKE_install_PMM_HA.sh | head -20Repository: percona/pmm-qa Length of output: 907 🏁 Script executed: # Check if there's any conditional logic around printing these credentials
sed -n '210,225p' k8s/createLKE_install_PMM_HA.shRepository: percona/pmm-qa Length of output: 586 🏁 Script executed: # Check the full script structure to understand if the echo is necessary
# Look for any flags or parameters that might control output
rg -n "#!/bin|set -" k8s/createLKE_install_PMM_HA.sh | head -5Repository: percona/pmm-qa Length of output: 186 🏁 Script executed: # Check if there's any documentation or comments explaining why credentials are printed
sed -n '200,230p' k8s/createLKE_install_PMM_HA.shRepository: percona/pmm-qa Length of output: 1034 Sensitive Data Exposure (CWE-532): Insertion of Sensitive Information into Log File Reachability: Internal Remove credential output from stdout. Lines 217–221 print passwords for PMM, ClickHouse, Postgres, Grafana, and VM to stdout. These credentials are correctly stored in Kubernetes secrets but must not be echoed where CI logs, shell history, or terminal captures retain them. Remove these echo lines. If an operator summary is needed, write only the endpoint and deployment status. 🤖 Prompt for AI Agents |
||||||||||||||||||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Validate the PMM password argument before creating cloud resources.
With
set -u, a missing argument causes an unhelpful shell error at Line 14. An empty argument reaches the secret creation step and writes an emptyPMM_ADMIN_PASSWORD. Require exactly one non-empty argument and print a usage error before creating the cluster.Proposed fix
📝 Committable suggestion
🤖 Prompt for AI Agents