Add optional DUCKDB_MEMORY_LIMIT applied at connect - #130
Merged
Merged
Conversation
Uncapped, DuckDB budgets ~80% of the cgroup limit and competes with the Arrow pending buffer for RSS during the partitioned INSERT — at large FLUSH_SIZE the sum can OOM the pod (charts#14038 saw exactly this shape at 1GB Arrow buffers). With a cap, DuckDB spills to the temp dir beside the on-disk database file (the /tmp emptyDir in k8s) instead of growing RSS. Opt-in via DUCKDB_MEMORY_LIMIT (e.g. "6GB"); unset or empty leaves the default behavior. The value passes through the existing _sanitize_setting_value guard. Companion to the events-nrt FLUSH_SIZE increase in charts; the chart sets it per-consumer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uncapped, DuckDB budgets ~80% of the cgroup limit and competes with the Arrow pending buffer for RSS during the partitioned INSERT — the charts#14038 OOM shape (1GB Arrow buffers OOMKilled the events consumers at 5940Mi). When
DUCKDB_MEMORY_LIMITis set (e.g."6GB"),connect()appliesSET memory_limitthrough the existing_sanitize_setting_valueguard, so DuckDB spills to the temp dir beside the on-disk database file (the/tmpemptyDir in k8s) instead of growing RSS. Spilling requires a file-backedDUCKLAKE_CONNECTION— the chart uses/tmp/duck.db; the code comment documents the:memory:caveat.Unset or empty leaves behavior unchanged (a chart rendering
value: ""is treated as unset — tested). Older images ignore the env var harmlessly.Companion to the pending charts change raising events-nrt
FLUSH_SIZEto 1.5GiB with a 12Gi limit; the chart will setduckdbMemoryLimit: "6GB"per-consumer there. Follows #129 (both were flagged by the adversarial review of that charts change).Verification
just lint/just fmt-checkclean; 692 unit, 65 integration, 4 e2e pass locally (cherry-picked from the reviewed branch; unit suite re-run post-pick).