fix(scaling): jitter first tick of scale loop to avoid thundering herd

[ggarb]

Problem

KEDA spawns one scale loop per ScalableObject (ScaledObject / ScaledJob). Each scale loop uses time.NewTimer(pollingInterval) with no per-object offset (pkg/scaling/scale_handler.go). That means objects whose scale loops are
spawned in a short window — by a bulk creation, a single reconcile pass after an operator restart, or a batch API call — end up polling on the same pollingInterval boundary forever.

For external-metric triggers (Prometheus, custom metrics, etc.) this produces a periodic thundering herd against the upstream metric source and against any client-side infrastructure (DNS, TCP accept queues, per-scaler http.Transport
connection pools) in the path. Under light load the effect is invisible; under even moderate per-cluster scale it becomes a correctness problem.

Reproduction

Measured on a 1k ScaledObject Prometheus-trigger load test (bulk-created by kube-burner in a ~33 s window), KEDA 2.19 release image, default pollingInterval: 30s, default KEDA_HTTP_DEFAULT_TIMEOUT=3000:

• ~50 % of scaler polls fail with context deadline exceeded (Client.Timeout exceeded while awaiting headers).
• The Prometheus server is idle on the other end: ~29 mCPU, and 100 % of the queries it received completed in <100 ms per its own histogram.
• KEDA logs ~2500 errors/min; Prom records only a handful of cancelled requests in the same window. Several hundred requests per minute never reach Prom at all.
• Operator CPU profile shows ~20 % of cycles in net.(*Resolver).goLookupIPCNAMEOrder — DNS re-resolution for every cold connection.
• Error pattern is distinctly bursty (~500 errors in a 6 s window, ~10 s apart) — aligned tick timers, not tail latency. Raising the timeout to 10 s does not help; timeouts move to 10 s and dial tcp: connection refused errors start
  appearing too.

Fix

Insert a deterministic per-object offset fnv64a(UID) % pollingInterval before the first tick of each scale loop. Subsequent ticks are unchanged. Keyed off the object's UID so the phase is stable across operator restarts — otherwise
every leader-election flip or pod crash would re-align all loops.

Helper lives in pkg/scaling/jitter.go. Call site is a small select against the jitter timer and ctx.Done() before the existing for loop in startScaleLoop.

Validation

Same cluster, same workload, with this patch. Prometheus' --web.max-connections was raised above its default 512 for these runs — with KEDA's current per-scaler http.Transport (one keepalive per scaler), N > ~500 scalers exceeds
Prom's default connection cap regardless of this fix; once polls are no longer bursting, the connection count stabilizes above the default and Prom starts rejecting new connections. That's a separate efficiency concern (shared
Transport across scalers of the same type) that can be addressed in a follow-up.

1k Prom-trigger:

• 0 scaler-poll errors sustained over 3+ minutes (was ~2500/min)
• Operator CPU: 83 → 70 mCPU
• Operator DNS share of CPU: 20 % → 1.3 % (warm keepalives no longer evicted by bursts)

10k Prom-trigger (same patch):

• 10 000 / 10 000 ScaledObjects, 0 operator restarts
• 316 282 Prom queries served, 99.9997 % under 100 ms, 0 scaler-poll errors across ~20 min of wall clock
• Operator 333 mCPU / 1163 MiB — ~33 mCPU and ~111 MiB per 1 k ScaledObjects, matching the type: cpu sizing baseline within 10 %

Tests

Added pkg/scaling/jitter_test.go covering:

• Offset is in [0, pollingInterval) across 1000 synthetic UIDs
• Same UID produces the same offset across repeated calls (stable phase across restarts)
• Zero or negative interval → 0
• Empty UID → 0
• Distribution across 10 000 synthetic UIDs is approximately uniform per-decile (loose binomial tolerance)

go test ./pkg/... passes on this branch.

Notes

• Backward-compatible: steady-state polling cadence is unchanged. Only the first tick per scale loop is delayed by up to pollingInterval.
• Applies to both ScaledObject and ScaledJob (shared startScaleLoop).
• Does not affect push-scaler paths (startPushScalers is unchanged).

Checklist

• Tests have been added
• Changelog has been updated (Fixes → General)
• Commits are signed (DCO)
• New scaler — N/A
• Schema regen — N/A (no scaler schema changes)
• Helm chart PR — N/A (no deployment manifest changes)
• Docs PR — N/A (behavior is transparent to users)

[snyk-io]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[github-actions]

Thank you for your contribution! 🙏

Please understand that we will do our best to review your PR and give you feedback as soon as possible, but please bear with us if it takes a little longer as expected.

While you are waiting, make sure to:

• Add an entry in our changelog in alphabetical order and link related issue
• Update the documentation, if needed
• Add unit & e2e tests for your changes
• GitHub checks are passing
• Is the DCO check failing? Here is how you can fix DCO issues

Once the initial tests are successful, a KEDA member will ensure that the e2e tests are run. Once the e2e tests have been successfully completed, the PR may be merged at a later date. Please be patient.

Learn more about our contribution guide.

[JorTurFer]

I like the improvement! The only concern I have is that some specific edge cases can use realy long polling intervals. Does it make sense to set a max window? Something like

min(pollingInterval, 1 min)

If the polling interval is short, it will work, but for long polling intervals, the request will be delayed as max 1 min

[rickbrouwer]

I still need to take a good look at the whole thing, but my first question is what is the reason to put this in its own file rather than inline in scale_handler.go?