Repository navigation
fix(clickhouse): wire up prod log-TTL config that was never mounted - #254
Open
MostafaMoradii wants to merge 1 commit into
Open
MostafaMoradii wants to merge 1 commit into
MostafaMoradii wants to merge 1 commit into
Conversation
The zz-eveys-prod.xml / zz-eveys-async-insert.xml config (written for the 2026-08-14 unbounded-system-log incident) existed only as loose files on the prod host and was never referenced by docker-compose.yml, so ClickHouse kept running with stock trace-level, TTL-less logging. query_log/metric_log/part_log/processors_profile_log regrew to ~3.7GB over the following month, and merging/reconciling that backlog OOM- crashed the container, pinning all 4 cores. Commit the config files and mount them so the fix actually takes effect and can't drift again.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
deploy/compose/clickhouse/config.d/zz-eveys-prod.xmlandusers.d/zz-eveys-async-insert.xmlwere written after the 2026-08-14 unbounded-system-log incident, but only ever existed as loose files on the prod host —docker-compose.ymlnever mounted them, so ClickHouse kept running with stock trace-level, TTL-less logging.query_log/metric_log/part_log/processors_profile_logregrew to ~3.7GB over the following month (vs. ~619MB of actual business data incp_ocpp_frames). Reconciling that backlog on merge pinned all 4 cores at ~380% CPU and OOM-crashed the container.docker-compose.yml, so the fix actually takes effect and can't silently drift out of deployment again.Remediation already applied directly on the prod host (config mounted, stale log-table backlog cleared, container healthy — CPU 380%→7%, load average 11.5→1.1). This PR is to get that fix into version control instead of living only as a manual server-side patch.
Test plan
docker compose -f deploy/compose/docker-compose.yml up -d clickhouseon a clean checkout mounts both files without error