Skip to content

Add swap_config to node-pool spec#6780

Draft
ajaysundark wants to merge 1 commit into
GoogleCloudPlatform:masterfrom
ajaysundark:swap-as-nodepool-resource-reference
Draft

Add swap_config to node-pool spec#6780
ajaysundark wants to merge 1 commit into
GoogleCloudPlatform:masterfrom
ajaysundark:swap-as-nodepool-resource-reference

Conversation

@ajaysundark

@ajaysundark ajaysundark commented Jun 23, 2026

Copy link
Copy Markdown

This makes Linux swap a node-pool property instead of requiring benchmarks to configure it at runtime. It adds a cloud-neutral swap_config to the node-pool spec.

Changes:

  • container_spec.py: Add SwapConfigSpec (enabled, encrypted, backing_store, size_percent/size_gb, behavior) and wire it up.
  • GKE: Render linuxConfig.swapConfig via --system-config-from-file. Merges with any existing gke_node_system_config. GKE manages the kubelet configuration.
  • EKS: Render an AL2023 nodeadm NodeConfig on overrideBootstrapCommand, attach the io2 swap device via eksctl additionalVolumes, and add mkswap/swapon to preBootstrapCommands.

Verified the rendered output, but it has not been run against a live cluster yet.

@ajaysundark
Add swap_config to node-pool spec (GKE + EKS)
28a5ec6 
This makes Linux swap a node-pool property instead of requiring benchmarks to configure it at runtime. It adds a cloud-neutral swap_config to the node-pool spec.

Changes:
- container_spec.py: Add SwapConfigSpec (enabled, encrypted, backing_store, size_percent/size_gb, behavior) and wire it up.
- GKE: Render linuxConfig.swapConfig via --system-config-from-file. Merges with any existing gke_node_system_config. GKE manages the kubelet configuration.
- EKS: Render an AL2023 nodeadm NodeConfig on overrideBootstrapCommand, attach the io2 swap device via eksctl additionalVolumes, and add mkswap/swapon to preBootstrapCommands.

Verified the field names against the GKE LinuxNodeConfig API and the eksctl/AL2023 schema. The rendered output looks correct, but it has not been run against a live cluster yet.
@ajaysundark ajaysundark marked this pull request as draft June 23, 2026 17:38
@ajaysundark

ajaysundark commented Jun 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Author

cc @hubatish Do you have any feedbacks on the 'swap config surface'? For GKE, existing config surface is enough as the already supported system-config file is what's used for swap configuration as well.
If it makes it easier for EKS or other providers, this may be useful to have a dedicated wiring for the swap configuration details.

@DevVegeta DevVegeta mentioned this pull request Jun 25, 2026
Open
DevVegeta added a commit to ashishsuneja/PerfKitBenchmarker that referenced this pull request Jun 25, 2026
@DevVegeta
fix(swap_encryption/pr2): duplicate FLAGS, cgroup hack, COS ref, EKS …
6afbfe1 
…stub

- Remove duplicate FLAGS = flags.FLAGS
- Remove unused import re
- DaemonSet: lean (fio/cryptsetup/mdadm/sysstat/nvme-cli only)
- Remove cgroup v2 memory.swap.max hack (r3457928855)
- Add _configure_eks_kubelet_swap() stub for nodeadm (PR GoogleCloudPlatform#6780)
- Fix COS_CONTAINERD -> UBUNTU_CONTAINERD (r3472549985)
- swapConfig auto-enables memorySwapBehavior=LimitedSwap (r3472513706)
DevVegeta added a commit to ashishsuneja/PerfKitBenchmarker that referenced this pull request Jun 25, 2026
@DevVegeta
fix(swap_encryption/pr3): duplicate FLAGS, unused import, COS ref, EK…
2e940d8 
…S stub

- Remove duplicate FLAGS = flags.FLAGS
- Remove unused import re
- Add _configure_eks_kubelet_swap() stub (deferred to PR GoogleCloudPlatform#6780)
- Fix COS_CONTAINERD -> UBUNTU_CONTAINERD (r3472549985)
- 7-job fio suite retained
- pyink --line-length=80 pass
DevVegeta added a commit to ashishsuneja/PerfKitBenchmarker that referenced this pull request Jun 25, 2026
@DevVegeta
fix(swap_encryption/pr4): slim DaemonSet, duplicate FLAGS, COS ref, E…
3162242 
…KS stub

- Remove duplicate FLAGS = flags.FLAGS
- Add _configure_eks_kubelet_swap() stub (deferred to PR GoogleCloudPlatform#6780)
- DaemonSet: remove workload tools; separate PKB pods (r3457826290)
- Remove cgroup v2 unlock hack (r3457928855)
- Pre-fetch kernel source for Phase 3b
- Fix COS_CONTAINERD -> UBUNTU_CONTAINERD (r3472549985)
DevVegeta added a commit to ashishsuneja/PerfKitBenchmarker that referenced this pull request Jun 25, 2026
@DevVegeta
fix(swap_encryption/pr5): EKS stub, DaemonSet alignment, COS ref fix
84bd9d7 
- Add _configure_eks_kubelet_swap() stub (deferred to PR GoogleCloudPlatform#6780)
- DaemonSet: aligned with PR4 slim (no redis/memtier/esrally/opensearch)
- Kernel source pre-fetch retained for Phase 3b kernel build
- Fix COS_CONTAINERD -> UBUNTU_CONTAINERD (r3472549985)
- re import retained (used in _parse_redis_benchmark)
DevVegeta added a commit to ashishsuneja/PerfKitBenchmarker that referenced this pull request Jun 29, 2026
@DevVegeta
refactor(swap_encryption/pr2): thin benchmark using BaseResource pattern
22511de 
- Prepare() uses SwapNodePool + SwapDaemonSet from spec.resources
- Cleanup() is empty - PKB framework auto-deletes spec.resources
- All _pod_exec(pod, ...) calls replaced with daemonset.PodExec(...)
- DaemonSet: lean (fio/cryptsetup/mdadm/sysstat/nvme-cli only)
- Remove cgroup v2 memory.swap.max hack (r3457928855)
- Add _configure_eks_kubelet_swap() stub for nodeadm (PR GoogleCloudPlatform#6780)
- Fix COS_CONTAINERD -> UBUNTU_CONTAINERD (r3472549985)
- swapConfig auto-enables memorySwapBehavior=LimitedSwap (r3472513706)
DevVegeta added a commit to ashishsuneja/PerfKitBenchmarker that referenced this pull request Jun 29, 2026
@DevVegeta
refactor(swap_encryption/pr3): thin benchmark using BaseResource pattern
c3c5ec9 
- Prepare() uses SwapNodePool + SwapDaemonSet from spec.resources
- Cleanup() is empty - PKB framework auto-deletes spec.resources
- All _pod_exec(pod, ...) calls replaced with daemonset.PodExec(...)
- Add _configure_eks_kubelet_swap() stub (deferred to PR GoogleCloudPlatform#6780)
- Fix COS_CONTAINERD -> UBUNTU_CONTAINERD (r3472549985)
- 7-job fio suite retained
- pyink --line-length=80 pass
DevVegeta added a commit to ashishsuneja/PerfKitBenchmarker that referenced this pull request Jun 29, 2026
@DevVegeta
refactor(swap_encryption/pr5): thin benchmark + remove duplicate phas…
62ebc94 
…e logic

- Prepare() uses SwapNodePool + SwapDaemonSet from spec.resources
- Cleanup() is empty - PKB framework auto-deletes spec.resources
- All _pod_exec(pod, ...) calls replaced with daemonset.PodExec(...)
- Remove Phase 3a (Redis/memtier): duplicated kubernetes_redis_memtier_benchmark;
  run that benchmark on the swap-enabled cluster instead
- Remove Phase 3c (OpenSearch): fragile in-pod JVM launch; use esrally_benchmark
  on the swap-enabled cluster instead
- Keep Phase 3b (kernel build under cgroup memory cap): unique to this benchmark;
  kernel_compile_benchmark.py is VM-based and cannot run in a K8s pod
- Add _configure_eks_kubelet_swap() stub (deferred to PR GoogleCloudPlatform#6780)
- Fix COS_CONTAINERD -> UBUNTU_CONTAINERD (r3472549985)
DevVegeta added a commit to ashishsuneja/PerfKitBenchmarker that referenced this pull request Jun 29, 2026
@DevVegeta
refactor(swap_encryption/pr1): correct PKB structure - swap_config as…
dfb3f23 
… NodepoolSpec field

BREAKING: replaces SwapNodePool (standalone nodepool lifecycle) with the
correct PKB pattern: swap configuration declared in BENCHMARK_CONFIG and
applied by the existing GKE cluster creation flow.

New files:
- resources/container_service/swap_config.py
  - GkeSwapConfig(BaseResource): WriteLinuxConfigYaml(), ValidHyperdiskThroughput()
  - EksSwapConfig(BaseResource): stub for nodeadm config (deferred to PR GoogleCloudPlatform#6780)

Core framework changes:
- configs/container_spec.py: add SwapConfigSpec(BaseSpec) + _SwapConfigDecoder
  + swap_config field on NodepoolSpec
- resources/container_service/container.py: add swap_config attr to BaseNodePoolConfig
- resources/container_service/container_cluster.py: propagate swap_config in
  _InitializeNodePool() (mirrors sandbox_config pattern)
- providers/gcp/google_kubernetes_engine.py: _AddNodeParamsToCmd() reads
  nodepool_config.swap_config - applies --system-config-from-file,
  UBUNTU_CONTAINERD, --no-enable-autorepair, boot-disk-provisioned-iops/throughput

Thin benchmark:
- BENCHMARK_CONFIG declares benchmark nodepool with swap_config (no separate
  nodepool create needed - GKE cluster creation handles it)
- Prepare(): deploy SwapDaemonSet + delete default-pool
- Run(): verify swap_active + swap_encrypted; report samples
- Cleanup(): empty (PKB auto-deletes spec.resources)

Addresses Ajay reviews:
- r3457826290: swap as base resource plugged into GKE cluster creation flow
- r3457877984: linuxConfig.swapConfig via --system-config-from-file (GkeSwapConfig)
- r3457928855: removed memory.swap.max hack
- r3457964593: UBUNTU_CONTAINERD set per-nodepool in _AddNodeParamsToCmd
- r3472513706: swapConfig auto-enables memorySwapBehavior=LimitedSwap
- r3472549985: UBUNTU_CONTAINERD required for dm-crypt
DevVegeta added a commit to ashishsuneja/PerfKitBenchmarker that referenced this pull request Jun 29, 2026
@DevVegeta
refactor(swap_encryption/pr2): thin benchmark using BaseResource pattern
fe41aa0 
- Prepare() uses SwapNodePool + SwapDaemonSet from spec.resources
- Cleanup() is empty - PKB framework auto-deletes spec.resources
- All _pod_exec(pod, ...) calls replaced with daemonset.PodExec(...)
- DaemonSet: lean (fio/cryptsetup/mdadm/sysstat/nvme-cli only)
- Remove cgroup v2 memory.swap.max hack (r3457928855)
- Add _configure_eks_kubelet_swap() stub for nodeadm (PR GoogleCloudPlatform#6780)
- Fix COS_CONTAINERD -> UBUNTU_CONTAINERD (r3472549985)
- swapConfig auto-enables memorySwapBehavior=LimitedSwap (r3472513706)
DevVegeta added a commit to ashishsuneja/PerfKitBenchmarker that referenced this pull request Jun 29, 2026
@DevVegeta
refactor(swap_encryption/pr3): thin benchmark using BaseResource pattern
d658074 
- Prepare() uses SwapNodePool + SwapDaemonSet from spec.resources
- Cleanup() is empty - PKB framework auto-deletes spec.resources
- All _pod_exec(pod, ...) calls replaced with daemonset.PodExec(...)
- Add _configure_eks_kubelet_swap() stub (deferred to PR GoogleCloudPlatform#6780)
- Fix COS_CONTAINERD -> UBUNTU_CONTAINERD (r3472549985)
- 7-job fio suite retained
- pyink --line-length=80 pass
DevVegeta added a commit to ashishsuneja/PerfKitBenchmarker that referenced this pull request Jun 29, 2026
@DevVegeta
refactor(swap_encryption/pr5): thin benchmark + remove duplicate phas…
531662f 
…e logic

- Prepare() uses SwapNodePool + SwapDaemonSet from spec.resources
- Cleanup() is empty - PKB framework auto-deletes spec.resources
- All _pod_exec(pod, ...) calls replaced with daemonset.PodExec(...)
- Remove Phase 3a (Redis/memtier): duplicated kubernetes_redis_memtier_benchmark;
  run that benchmark on the swap-enabled cluster instead
- Remove Phase 3c (OpenSearch): fragile in-pod JVM launch; use esrally_benchmark
  on the swap-enabled cluster instead
- Keep Phase 3b (kernel build under cgroup memory cap): unique to this benchmark;
  kernel_compile_benchmark.py is VM-based and cannot run in a K8s pod
- Add _configure_eks_kubelet_swap() stub (deferred to PR GoogleCloudPlatform#6780)
- Fix COS_CONTAINERD -> UBUNTU_CONTAINERD (r3472549985)
DevVegeta added a commit to ashishsuneja/PerfKitBenchmarker that referenced this pull request Jun 29, 2026
@DevVegeta
refactor(swap_encryption/pr1): correct PKB structure - swap_config as…
d9b5615 
… NodepoolSpec field

BREAKING: replaces SwapNodePool (standalone nodepool lifecycle) with the
correct PKB pattern: swap configuration declared in BENCHMARK_CONFIG and
applied by the existing GKE cluster creation flow.

New files:
- resources/container_service/swap_config.py
  - GkeSwapConfig(BaseResource): WriteLinuxConfigYaml(), ValidHyperdiskThroughput()
  - EksSwapConfig(BaseResource): stub for nodeadm config (deferred to PR GoogleCloudPlatform#6780)

Core framework changes:
- configs/container_spec.py: add SwapConfigSpec(BaseSpec) + _SwapConfigDecoder
  + swap_config field on NodepoolSpec
- resources/container_service/container.py: add swap_config attr to BaseNodePoolConfig
- resources/container_service/container_cluster.py: propagate swap_config in
  _InitializeNodePool() (mirrors sandbox_config pattern)
- providers/gcp/google_kubernetes_engine.py: _AddNodeParamsToCmd() reads
  nodepool_config.swap_config - applies --system-config-from-file,
  UBUNTU_CONTAINERD, --no-enable-autorepair, boot-disk-provisioned-iops/throughput

Thin benchmark:
- BENCHMARK_CONFIG declares benchmark nodepool with swap_config (no separate
  nodepool create needed - GKE cluster creation handles it)
- Prepare(): deploy SwapDaemonSet + delete default-pool
- Run(): verify swap_active + swap_encrypted; report samples
- Cleanup(): empty (PKB auto-deletes spec.resources)

Addresses Ajay reviews:
- r3457826290: swap as base resource plugged into GKE cluster creation flow
- r3457877984: linuxConfig.swapConfig via --system-config-from-file (GkeSwapConfig)
- r3457928855: removed memory.swap.max hack
- r3457964593: UBUNTU_CONTAINERD set per-nodepool in _AddNodeParamsToCmd
- r3472513706: swapConfig auto-enables memorySwapBehavior=LimitedSwap
- r3472549985: UBUNTU_CONTAINERD required for dm-crypt
DevVegeta added a commit to ashishsuneja/PerfKitBenchmarker that referenced this pull request Jun 29, 2026
@DevVegeta
refactor(swap_encryption/pr5): thin benchmark + Phase 3b kernel build
ab074b3 
Inherits PR1 framework changes (swap_config as NodepoolSpec field).

- Prepare(): deploy SwapDaemonSet + _delete_default_pool(cluster)
- Phase 3b: kernel build slowdown ratio under cgroup memory cap
- Remove Phase 3a/3c (Redis/OpenSearch): use existing PKB benchmarks
- Add _configure_eks_kubelet_swap() stub (deferred to PR GoogleCloudPlatform#6780)
@DevVegeta DevVegeta mentioned this pull request Jun 29, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ajaysundark