[fix][R²] agent-sandbox proxy: raise CPU limit to 1 so it can successfully boot.

[drewmacneil]

Impact

The R² agent-sandbox proxy is in permanent CrashLoopBackOff on internal balloon instances, which breaks R² entirely and surfaces to users as a misleading "reached its maximum agent sandbox capacity" error. Observed on cli-4-40, blaker-r2, kcheng-r2, and jvecchio-r2; root-caused and fix-verified live on cli-4-40.

Where this could reach beyond balloons. The 500m cap is a chart default and nothing overrides it — the internal-onprem balloon values and the Terragrunt MSH client values set only proxy.backendDomainSuffixes (and ingress/secret names), so any install with the agent sandbox enabled renders the same limit. Confirmed by rendering brilliant retool-prod, which picks up the identical 500m proxy limit.

That is shared configuration, not an observed failure: we have not seen a customer instance crashloop, and whether one does depends on how its node CPU lands against the ~60s liveness budget. Balloons cleared it by ~4s or lost by ~3s, so the margin is thin either way — the exposure is real even though the reports so far are internal.

Reviewers: this raises a default for every self-hosted install, not just ours. That breadth is deliberate — the cap is wrong everywhere and an internal-only override would leave the same trap in the shipped chart — but please confirm you're comfortable with it.

Fix

rr.agentSandbox.proxy.resources.limits.cpu: 500m → '1', matching agent_executor/helm/agent-executor/values.yaml in the monorepo (the source chart for this component, which already specifies '1').

Applied to both values.yaml and charts/retool/values.yaml — the repo keeps them byte-identical and CI enforces it (.github/workflows/values-yaml-synced.yaml). Chart version bumped 6.11.17 → 6.11.18 per check-version-increment: true in .github/ct.yaml.

Why

The proxy needs ~26 CPU-seconds to bind port 3019. Capped at 500m that is ~64s wall-clock, just past this chart's ~60s liveness budget (initialDelaySeconds: 10 + failureThreshold: 5 × periodSeconds: 10, with no startupProbe in templates/deployment_agent_sandbox.yaml), so kubelet kills it ~3s before it comes up — cleanly, exit 0 — then loses the same race on every retry. Boot was ~56s across three balloon pods spanning three image versions, three nodes, and six days, i.e. CPU-capped rather than load-dependent.

With no ready proxy endpoint, every backend→sandbox call (/sandbox/<id>/run-agent, /run-default-snapshot) fails ECONNREFUSED. Those failures mark sandboxes terminal, the controller deletes the jobs, the prewarm pool churns instead of filling, and assign falls through to on-demand creation — which times out at 30s against a ~67s cold start and returns 503 no capacity available. Hence a capacity message for what is really a proxy outage.

Verified live before opening this: patching cli-4-40's proxy to limits.cpu: 1 cut boot from 63.6s to 26.4s, the pod went 1/1 with 0 restarts, its endpoint moved from notReadyAddresses to addresses, run-agent started returning 200, and controller churn dropped from ~60 events per 6 minutes to 1. That live patch reverts on the balloon's next helm upgrade, which is why it needs to land here.

Verification

helm template against a balloon (cli-4-40) and an MSH customer (brilliant retool-prod), before vs. after, with the chart version held constant to isolate the change: exactly one changed line in each, cpu: 500m → cpu: "1", inside the *-agent-sandbox-proxy Deployment. The *-agent-sandbox-controller Deployment (byte-identical resources block) and the *-agent-sandbox-job-template ConfigMap are unchanged, as are all other Deployments and StatefulSets. helm lint passes.

Deliberately not included

Scoped to the single field verified live. Three related items left for reviewers:

1. controller.resources.limits.cpu is the same 500m and booted in 55.6s against the same 60s budget — it survives on ~4s of margin and is the same bug one unlucky restart away. A dead controller is worse than a dead proxy: no /assign at all. Strong candidate for a follow-up.
2. proxy.replicaCount: 1 → 2, matching the monorepo. More correct — one replica losing the boot race takes R² down for the whole instance — but it doubles proxy pod count on every self-hosted deployment, unlike a limit raise on a pod already requesting 250m.
3. A startupProbe on proxy and controller would remove this failure class rather than widening the margin. Tracked separately in the monorepo chart, which has none either.

[drewmacneil]

• [fix][R²] agent-sandbox proxy: raise CPU limit to 1 so it can successfully boot. #362 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge because the new CPU quantity is valid, reaches the intended proxy Deployment, and remains synchronized across both values files.

The change narrowly adjusts the intended proxy resource limit and chart version without altering dependencies, workload routing, or security boundaries, and no actionable failure remains.

Important Files Changed

Filename	Overview
charts/retool/Chart.yaml	Increments the chart patch version to publish the updated default; no dependency change requires a lockfile update.
charts/retool/values.yaml	Raises the authoritative proxy CPU limit default to one core, which renders directly into the proxy container resources.
values.yaml	Keeps the repository-level customer values copy synchronized with the deployable chart defaults.

_{Reviews (1): Last reviewed commit: "[fix][R²] `agent-sandbox` proxy: raise C..." | Re-trigger Greptile}

[lukefoster11]

This seems reasonable 👍

[drewmacneil]

Merge activity

• Aug 13, 12:30 AM UTC: A user started a stack merge that includes this pull request via Graphite.
• Aug 13, 12:30 AM UTC: @drewmacneil merged this pull request with Graphite.