-
Notifications
You must be signed in to change notification settings - Fork 1
Add livid-bot k3s migration tooling #11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -27,5 +27,6 @@ livid-bot | |
| .mise.toml | ||
| .jj | ||
| logs/ | ||
| deploy/k3s/output/ | ||
|
|
||
| .claude/settings.local.json | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,71 @@ | ||
| # livid-bot on k3s | ||
|
|
||
| `livid-bot`의 Docker Compose 런타임을 k3s로 옮기기 위한 Helm chart와 | ||
| 데이터 보존형 컷오버 스크립트 모음이다. | ||
|
|
||
| ## Layout | ||
|
|
||
| - `charts/livid-bot`: PostgreSQL + Discord bot Helm chart | ||
| - `bin/apply-secrets.sh`: `.env` 기반 Kubernetes Secret 반영 | ||
| - `bin/deploy.sh`: Helm 배포 | ||
| - `bin/verify-migration.sh`: Docker 원본 DB와 k3s DB row count 비교 | ||
| - `bin/cutover.sh`: dump/restore + 로그 PVC 이관 + bot scale-up | ||
|
|
||
| ## Runtime Contract | ||
|
|
||
| - Namespace: `livid-bot` | ||
| - Helm release: `livid-bot` | ||
| - Secret names: | ||
| - `livid-bot-db-credentials` | ||
| - `livid-bot-bot-secrets` | ||
| - PVC names: | ||
| - `pgdata-livid-bot-db-0` | ||
| - `livid-bot-logs` | ||
|
|
||
| ## First Deploy | ||
|
|
||
| Prerequisite: `quant` 저장소의 `deploy/k3s/setup/04-bootstrap-cluster.sh`를 먼저 실행해 | ||
| `livid-bot` namespace와 `livid-bot-app` ServiceAccount를 만들어 둔다. | ||
|
|
||
| ```bash | ||
| cd /Users/haril/projects/livid-bot | ||
| ./deploy/k3s/bin/apply-secrets.sh | ||
| ./deploy/k3s/bin/deploy.sh | ||
| ``` | ||
|
|
||
| ## Data-Preserving Cutover | ||
|
|
||
| 컷오버는 짧은 점검창을 전제로 한다. `bot` 컨테이너를 멈춘 뒤 최종 | ||
| `pg_dump`를 수행하고, 복구 가능한 Docker 원본을 그대로 남겨 둔다. | ||
|
|
||
| ```bash | ||
| cd /Users/haril/projects/livid-bot | ||
| ./deploy/k3s/bin/cutover.sh | ||
| ``` | ||
|
|
||
| worktree에서 실행할 때는 원본 Docker Compose 체크아웃 경로를 지정한다. | ||
|
|
||
| ```bash | ||
| SOURCE_PROJECT_DIR=/Users/haril/projects/livid-bot ./deploy/k3s/bin/cutover.sh | ||
| ``` | ||
|
|
||
| 실행 내용: | ||
|
|
||
| 1. Secrets 적용 | ||
| 2. Helm 배포 (`bot.replicas=0`) | ||
| 3. Docker 원본 DB pre-cutover dump | ||
| 4. Docker bot 정지 후 final dump | ||
| 5. k3s PostgreSQL restore | ||
| 6. `./logs`를 PVC로 복사 | ||
| 7. row count / `schema_migrations` 검증 | ||
| 8. `bot.replicas=1`로 scale-up | ||
|
|
||
| ## Rollback | ||
|
|
||
| 원본 Docker DB 볼륨과 `./logs`는 컷오버 후에도 유지된다. | ||
|
|
||
| ```bash | ||
| cd /Users/haril/projects/livid-bot | ||
| helm uninstall livid-bot -n livid-bot | ||
| docker compose up -d db bot | ||
| ``` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| #!/usr/bin/env bash | ||
| set -euo pipefail | ||
|
|
||
| SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" | ||
| PROJECT_DIR="$(cd "${SCRIPT_DIR}/../../.." && pwd)" | ||
| SOURCE_PROJECT_DIR="${SOURCE_PROJECT_DIR:-${PROJECT_DIR}}" | ||
| ENV_FILE="${ENV_FILE:-${SOURCE_PROJECT_DIR}/.env}" | ||
|
|
||
| RELEASE_NAME="${RELEASE_NAME:-livid-bot}" | ||
| NAMESPACE="${NAMESPACE:-livid-bot}" | ||
|
|
||
| if [[ -f "${ENV_FILE}" ]]; then | ||
| set -a | ||
| # shellcheck disable=SC1090 | ||
| source "${ENV_FILE}" | ||
| set +a | ||
| fi | ||
|
|
||
| : "${DISCORD_BOT_TOKEN:?DISCORD_BOT_TOKEN is required}" | ||
| : "${DISCORD_APPLICATION_ID:?DISCORD_APPLICATION_ID is required}" | ||
| : "${DISCORD_GUILD_ID:?DISCORD_GUILD_ID is required}" | ||
|
|
||
| POSTGRES_USER="${POSTGRES_USER:-livid}" | ||
| POSTGRES_PASSWORD="${POSTGRES_PASSWORD:-livid}" | ||
| POSTGRES_DB="${POSTGRES_DB:-livid}" | ||
|
|
||
| kubectl create secret generic "${RELEASE_NAME}-db-credentials" \ | ||
| --namespace "${NAMESPACE}" \ | ||
| --from-literal=POSTGRES_USER="${POSTGRES_USER}" \ | ||
| --from-literal=POSTGRES_PASSWORD="${POSTGRES_PASSWORD}" \ | ||
| --from-literal=POSTGRES_DB="${POSTGRES_DB}" \ | ||
| --dry-run=client \ | ||
| -o yaml | kubectl apply -n "${NAMESPACE}" -f - | ||
|
|
||
| kubectl create secret generic "${RELEASE_NAME}-bot-secrets" \ | ||
| --namespace "${NAMESPACE}" \ | ||
| --from-literal=DISCORD_BOT_TOKEN="${DISCORD_BOT_TOKEN}" \ | ||
| --from-literal=DISCORD_APPLICATION_ID="${DISCORD_APPLICATION_ID}" \ | ||
| --from-literal=DISCORD_GUILD_ID="${DISCORD_GUILD_ID}" \ | ||
| --dry-run=client \ | ||
| -o yaml | kubectl apply -n "${NAMESPACE}" -f - |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,106 @@ | ||
| #!/usr/bin/env bash | ||
| set -euo pipefail | ||
|
|
||
| SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" | ||
| PROJECT_DIR="$(cd "${SCRIPT_DIR}/../../.." && pwd)" | ||
| SOURCE_PROJECT_DIR="${SOURCE_PROJECT_DIR:-${PROJECT_DIR}}" | ||
| OUTPUT_DIR="${PROJECT_DIR}/deploy/k3s/output" | ||
|
|
||
| RELEASE_NAME="${RELEASE_NAME:-livid-bot}" | ||
| NAMESPACE="${NAMESPACE:-livid-bot}" | ||
| SOURCE_POSTGRES_USER="${SOURCE_POSTGRES_USER:-livid}" | ||
| SOURCE_POSTGRES_PASSWORD="${SOURCE_POSTGRES_PASSWORD:-livid}" | ||
| SOURCE_POSTGRES_DB="${SOURCE_POSTGRES_DB:-livid}" | ||
| TIMESTAMP="$(date +%Y%m%d-%H%M%S)" | ||
|
|
||
| PRE_DUMP="${OUTPUT_DIR}/${TIMESTAMP}-pre-cutover.dump" | ||
| FINAL_DUMP="${OUTPUT_DIR}/${TIMESTAMP}-final.dump" | ||
| HELPER_POD="${RELEASE_NAME}-logs-sync" | ||
|
|
||
| mkdir -p "${OUTPUT_DIR}" | ||
|
|
||
| target_secret_key() { | ||
| local key="$1" | ||
| kubectl get secret "${RELEASE_NAME}-db-credentials" \ | ||
| -n "${NAMESPACE}" \ | ||
| -o "jsonpath={.data.${key}}" | python3 -c 'import base64, sys; print(base64.b64decode(sys.stdin.read()).decode(), end="")' | ||
| } | ||
|
|
||
| cleanup_helper() { | ||
| kubectl delete pod "${HELPER_POD}" -n "${NAMESPACE}" --ignore-not-found >/dev/null 2>&1 || true | ||
| } | ||
|
|
||
| trap cleanup_helper EXIT | ||
|
|
||
| "${SCRIPT_DIR}/apply-secrets.sh" | ||
| "${SCRIPT_DIR}/deploy.sh" --set bot.replicas=0 | ||
|
|
||
| kubectl rollout status statefulset/"${RELEASE_NAME}-db" -n "${NAMESPACE}" --timeout=180s | ||
|
|
||
| ( | ||
| cd "${SOURCE_PROJECT_DIR}" | ||
| docker compose exec -T \ | ||
| -e PGPASSWORD="${SOURCE_POSTGRES_PASSWORD}" \ | ||
| db \ | ||
| pg_dump -U "${SOURCE_POSTGRES_USER}" -d "${SOURCE_POSTGRES_DB}" -Fc | ||
| ) > "${PRE_DUMP}" | ||
|
|
||
| ( | ||
| cd "${SOURCE_PROJECT_DIR}" | ||
| docker compose stop bot | ||
| ) | ||
|
|
||
| ( | ||
| cd "${SOURCE_PROJECT_DIR}" | ||
| docker compose exec -T \ | ||
| -e PGPASSWORD="${SOURCE_POSTGRES_PASSWORD}" \ | ||
| db \ | ||
| pg_dump -U "${SOURCE_POSTGRES_USER}" -d "${SOURCE_POSTGRES_DB}" -Fc | ||
| ) > "${FINAL_DUMP}" | ||
|
|
||
| TARGET_POSTGRES_USER="$(target_secret_key POSTGRES_USER)" | ||
| TARGET_POSTGRES_PASSWORD="$(target_secret_key POSTGRES_PASSWORD)" | ||
| TARGET_POSTGRES_DB="$(target_secret_key POSTGRES_DB)" | ||
|
|
||
| kubectl exec -i -n "${NAMESPACE}" "${RELEASE_NAME}-db-0" -- env \ | ||
| PGPASSWORD="${TARGET_POSTGRES_PASSWORD}" \ | ||
| pg_restore \ | ||
| --clean \ | ||
| --if-exists \ | ||
| --no-owner \ | ||
| --no-privileges \ | ||
| -U "${TARGET_POSTGRES_USER}" \ | ||
| -d "${TARGET_POSTGRES_DB}" < "${FINAL_DUMP}" | ||
|
|
||
| kubectl apply -n "${NAMESPACE}" -f - <<EOF | ||
| apiVersion: v1 | ||
| kind: Pod | ||
| metadata: | ||
| name: ${HELPER_POD} | ||
| spec: | ||
| restartPolicy: Never | ||
| containers: | ||
| - name: sync | ||
| image: busybox:1.36 | ||
| command: ["sh", "-ec", "sleep 3600"] | ||
| volumeMounts: | ||
| - name: logs | ||
| mountPath: /logs | ||
| volumes: | ||
| - name: logs | ||
| persistentVolumeClaim: | ||
| claimName: ${RELEASE_NAME}-logs | ||
| EOF | ||
|
|
||
| kubectl wait --for=condition=Ready pod/"${HELPER_POD}" -n "${NAMESPACE}" --timeout=120s | ||
| if [[ -d "${SOURCE_PROJECT_DIR}/logs" ]]; then | ||
| kubectl cp "${SOURCE_PROJECT_DIR}/logs/." "${NAMESPACE}/${HELPER_POD}:/logs" | ||
| fi | ||
|
|
||
| "${SCRIPT_DIR}/verify-migration.sh" | ||
| "${SCRIPT_DIR}/deploy.sh" --set bot.replicas=1 | ||
| kubectl rollout status deployment/"${RELEASE_NAME}" -n "${NAMESPACE}" --timeout=180s | ||
|
|
||
| echo "Cutover completed." | ||
| echo "Pre-cutover dump: ${PRE_DUMP}" | ||
| echo "Final dump: ${FINAL_DUMP}" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| #!/usr/bin/env bash | ||
| set -euo pipefail | ||
|
|
||
| SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" | ||
| PROJECT_DIR="$(cd "${SCRIPT_DIR}/../../.." && pwd)" | ||
|
|
||
| RELEASE_NAME="${RELEASE_NAME:-livid-bot}" | ||
| NAMESPACE="${NAMESPACE:-livid-bot}" | ||
| SERVICE_ACCOUNT_NAME="${SERVICE_ACCOUNT_NAME:-livid-bot-app}" | ||
|
|
||
| if ! kubectl get serviceaccount "${SERVICE_ACCOUNT_NAME}" -n "${NAMESPACE}" >/dev/null 2>&1; then | ||
| echo "Missing ${SERVICE_ACCOUNT_NAME} in namespace ${NAMESPACE}." >&2 | ||
| echo "Run quant/deploy/k3s/setup/04-bootstrap-cluster.sh first." >&2 | ||
| exit 1 | ||
| fi | ||
|
|
||
| helm upgrade --install \ | ||
| "${RELEASE_NAME}" \ | ||
| "${PROJECT_DIR}/deploy/k3s/charts/livid-bot" \ | ||
| --namespace "${NAMESPACE}" \ | ||
| --create-namespace \ | ||
| "$@" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,69 @@ | ||
| #!/usr/bin/env bash | ||
| set -euo pipefail | ||
|
|
||
| SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" | ||
| PROJECT_DIR="$(cd "${SCRIPT_DIR}/../../.." && pwd)" | ||
| SOURCE_PROJECT_DIR="${SOURCE_PROJECT_DIR:-${PROJECT_DIR}}" | ||
|
|
||
| RELEASE_NAME="${RELEASE_NAME:-livid-bot}" | ||
| NAMESPACE="${NAMESPACE:-livid-bot}" | ||
| SOURCE_POSTGRES_USER="${SOURCE_POSTGRES_USER:-livid}" | ||
| SOURCE_POSTGRES_PASSWORD="${SOURCE_POSTGRES_PASSWORD:-livid}" | ||
| SOURCE_POSTGRES_DB="${SOURCE_POSTGRES_DB:-livid}" | ||
| TARGET_POD="${TARGET_POD:-${RELEASE_NAME}-db-0}" | ||
|
|
||
| target_secret_key() { | ||
| local key="$1" | ||
| kubectl get secret "${RELEASE_NAME}-db-credentials" \ | ||
| -n "${NAMESPACE}" \ | ||
| -o "jsonpath={.data.${key}}" | python3 -c 'import base64, sys; print(base64.b64decode(sys.stdin.read()).decode(), end="")' | ||
| } | ||
|
|
||
| source_psql() { | ||
| local sql="$1" | ||
| ( | ||
| cd "${SOURCE_PROJECT_DIR}" | ||
| docker compose exec -T \ | ||
| -e PGPASSWORD="${SOURCE_POSTGRES_PASSWORD}" \ | ||
| db \ | ||
| psql -U "${SOURCE_POSTGRES_USER}" -d "${SOURCE_POSTGRES_DB}" -At -c "${sql}" | ||
| ) | ||
| } | ||
|
|
||
| target_psql() { | ||
| local sql="$1" | ||
| kubectl exec -n "${NAMESPACE}" "${TARGET_POD}" -- env \ | ||
| PGPASSWORD="${TARGET_POSTGRES_PASSWORD}" \ | ||
| psql -U "${TARGET_POSTGRES_USER}" -d "${TARGET_POSTGRES_DB}" -At -c "${sql}" | ||
| } | ||
|
|
||
| TARGET_POSTGRES_USER="$(target_secret_key POSTGRES_USER)" | ||
| TARGET_POSTGRES_PASSWORD="$(target_secret_key POSTGRES_PASSWORD)" | ||
| TARGET_POSTGRES_DB="$(target_secret_key POSTGRES_DB)" | ||
|
|
||
| mapfile -t tables < <(source_psql "SELECT tablename FROM pg_tables WHERE schemaname = 'public' ORDER BY tablename;") | ||
|
|
||
| if [[ "${#tables[@]}" -eq 0 ]]; then | ||
| echo "No public tables found in source database." >&2 | ||
| exit 1 | ||
| fi | ||
|
|
||
| for table in "${tables[@]}"; do | ||
| source_count="$(source_psql "SELECT COUNT(*) FROM \"${table}\";")" | ||
| target_count="$(target_psql "SELECT COUNT(*) FROM \"${table}\";")" | ||
| printf '%-32s source=%s target=%s\n' "${table}" "${source_count}" "${target_count}" | ||
| if [[ "${source_count}" != "${target_count}" ]]; then | ||
| echo "Count mismatch detected for ${table}." >&2 | ||
| exit 1 | ||
| fi | ||
| done | ||
|
|
||
| source_migrations="$(source_psql "SELECT filename FROM schema_migrations ORDER BY filename;")" | ||
| target_migrations="$(target_psql "SELECT filename FROM schema_migrations ORDER BY filename;")" | ||
|
|
||
| if [[ "${source_migrations}" != "${target_migrations}" ]]; then | ||
| echo "schema_migrations mismatch detected." >&2 | ||
| exit 1 | ||
| fi | ||
|
|
||
| echo "Migration verification passed." |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The deployment script hardcodes
"${PROJECT_DIR}/deploy/k3s/charts/livid-bot", but this commit does not add that chart directory (repo search only finds references in docs/scripts). In this state,helm upgrade --installfails immediately with a missing chart path, so bothdeploy.shandcutover.shcannot run in any environment.Useful? React with 👍 / 👎.