Skip to content

fix(swarm): report failed service rollouts - #5776

Open
mjafartp wants to merge 6 commits into
Dokploy:canaryfrom
mjafartp:fix/swarm-report-failed-rollouts
Open

mjafartp wants to merge 6 commits into
Dokploy:canaryfrom
mjafartp:fix/swarm-report-failed-rollouts

Conversation

@mjafartp

@mjafartp mjafartp commented Oct 25, 2026 •

Copy link
Copy Markdown

Summary

Rebase of #5218 by @bestmaa onto current canary, with the original commits and authorship preserved, plus three follow-up commits. Fixes #3987.

Today mechanizeDockerContainer calls service.update and the deployment is marked done immediately. With Dokploy's default UpdateConfig (start-first, FailureAction: rollback), a new task that dies inside Swarm's monitor window is rolled back to the previous spec, the old image keeps running, and the deployment log still shows success.

Commits

  1. fix(swarm): report failed service rollouts and fix(swarm): correlate rollout status updates — the two commits from fix(swarm): report failed service rollouts #5218, cherry-picked. The only conflict was in builders/index.ts, where canary added the service-level label carry-over from Docker-image apps: any deploy (webhook or API) rewrites the Swarm service spec and drops the app's traefik.* labels → all domains 404 #5453; both behaviours are kept (Labels: inspect.Spec?.Labels ?? {} is passed to the new service.update call).
  2. test(swarm): keep label carry-over tests after rebase — ports canary's two label tests onto the PR's mocks, and resets the wait mock between tests so a rejected wait cannot leak into later tests.
  3. fix(swarm): skip the rollout wait when no task was running — Swarm's updater exits early when no existing task needs replacing and never writes UpdateStatus. A deploy of a stopped or scaled-to-zero service therefore waited until the timeout and was reported as failed although the new tasks started. The tasks with desired state running are listed before the update; if there are none, the update is applied without waiting. A listing error falls back to waiting.
  4. fix(swarm): allow a rollout ten minutes before timing out — the derived budget ignores image pull time, container start and the health-check start period, so the two-minute floor reported slow but healthy rollouts as failed while Swarm was still converging.

Known limitation, unchanged from the original PR: a service whose tasks stay pending because no node can place them waits until the timeout.

Tests

  • swarm-update (10), mechanizeDockerContainer (14, including the three new cases for the no-running-task path), application.command, preview-build-server, provider-status: pass. Full suite: 1511 passed. tsc --noEmit for server and dokploy: pass. Biome: clean.

RetriggerView in GreptileConfidence Score: 4/5

The PR appears safe to merge, with a non-blocking issue in how it explains some automatic rollbacks.

Summary

The PR waits for existing Swarm service rollouts before marking deployments successful.

  • Deployments wait for Swarm to finish updating existing services.
  • Stopped services and job services skip rollout waits.

Acknowledged behavior:

  • mjafartp intentionally skips waiting when no previous task needs replacing because Swarm may not write UpdateStatus.
  • mjafartp acknowledges that tasks with no eligible node wait until timeout.

Reviews (1) · Last reviewed commit: "fix(swarm): allow a rollout ten minutes ..." · Reviewed by Greptile

bestmaa and others added 5 commits October 25, 2026 22:57
Port the service label tests onto the shared inspect and update mocks, and
reset the rollout wait mock between tests so a rejection set by one test
does not leak into the next.
Swarm's updater exits early when no existing task needs replacing and never
writes UpdateStatus, so deploying a stopped or scaled-to-zero service waited
until the timeout and reported a working deploy as failed.
The derived budget ignores image pull time, container start and the
health-check start period, so a two-minute floor reported slow but healthy
rollouts as failed while Swarm was still converging.
@mjafartp
mjafartp requested a review from Siumauricio as a code owner October 25, 2026 17:41
Comment on lines +176 to +184
const isExpectedRollback =
state?.startsWith("rollback_") &&
Boolean(
await getLatestTaskFailure(
docker,
service.id,
options.expectedForceUpdate,
),
);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Rollback shows the wrong cause

If the first poll sees an automatic rollback and listTasks fails, isExpectedRollback becomes false. Rollback restores the old ForceUpdate, so the function reports “superseded by another operation” instead of the rollback. Users still see a failed deployment, but get the wrong explanation.

Keep task lookup optional for error details rather than using its result to decide whether the rollback belongs to this deployment.

A rollback restores the previous ForceUpdate, so it was only matched to
this update through a failed task. When the lookup failed or the new task
exited cleanly, the rollback was reported as a superseding operation.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Swarm deployment rollback not reflected in Dokploy UI (deployment appears successful even when tasks fail healthcheck)

2 participants