slurm: batch the queries and parallelize the writes in the GPU power/clock helpers

[100milliongold]

Problem

set_gpu_power_levels.sh and set_gpu_clocks.sh call nvidia-smi once per GPU to read the target value and once more to apply it, all serially. Both nvidia-smi -pl and nvidia-smi -ac take roughly a second per GPU, so on an 8-GPU node the two helpers together add about 8 s to the prolog of every job for which 50-exclusive-gpu runs. srun surfaces it as:

srun: Prolog hung on node <node>

Observed on DGX OS 7.5.0 (8× B300), Slurm 26.05.1.

Fix

• Read the values for all GPUs in a single --query-gpu call (the per-GPU -i loop was only needed because the value was read one at a time).
• Apply them in parallel, collecting each child's exit status so a failure on any GPU still fails the script.

The same values are written to the same GPUs; only the number of nvidia-smi invocations and their concurrency change. The default branch of set_gpu_clocks.sh already operated on all GPUs at once and is untouched.

Verification

Not yet timed with the patched scripts on hardware — the system where this was found has 50-exclusive-gpu removed from prolog.d (an 8-GPU node shared between jobs should not have every job reset limits and clocks on all GPUs). The serial cost is reproducible there: prolog took 6–8 s per job while the script was in place. Marked as draft for that reason; happy to run a timed before/after if that would help.

Related

The reason every job ran 50-exclusive-gpu in the first place is a separate defect in the exclusive-job detection, addressed in #1391.

[dholt]

The batched queries at set_gpu_power_levels.sh:21 and set_gpu_clocks.sh:13-14 run in process substitutions, whose failures are invisible to readarray and set -e. If a query fails with empty output, the loop is empty and the helper exits 0; partial output can update only some GPUs and also return success. Capture each query through a construct whose status can be checked, validate that all required per-GPU rows are present and aligned, and only then launch writes. Please cover empty, partial, and nonzero query results as required by the changed-path evidence.

---

Automated triage review (agent-generated on the maintainer's behalf; a human maintainer decides merges).