fix(renderer): clear the stuck daemon updating state after restart by namikmesic · Pull Request #38 · namikmesic/puck

namikmesic · 2026-09-30T12:02:35Z

Intent

After the user clicks the daemon update in the environment window, the top bar shows a "Daemon updating" chip and never clears it. Observed on 2026-09-30: the update was requested at 10:28:36 UTC, and two hours later the chip still read "Daemon updating" while the environment was otherwise working.

Evidence from the logs:

App log: instance.daemon-upgrade {"mode":"now"} at 10:28:36 and nothing after it about the upgrade.
Runner log: a control channel opened at 10:28:36; the attach channel closed with reason daemon-closed at 10:28:36 and reopened at 10:28:37; the control channel closed with reason app-closed at 10:29:36, exactly 60 seconds later.
Container log: daemon.upgrade {"mode":"now"} and daemon.exit {"code":75} at 10:28:36, then daemon.start and daemon.listening on the new version 0.1.0+d48a3a42570d and daemon.ready at 10:28:37, then client.attached {"protocol":1,"replay":14}.

So the daemon actually upgraded and restarted within a second. The app is what stays stuck. In src/renderer/instance-store.ts, upgrading is set by the daemon.upgrading event and cleared only when a snapshot is applied, and the reattach after the restart replayed 14 events rather than resyncing with a snapshot, so nothing cleared it. Later periodic resync attachments in the container log did not clear it either, so check whether they belong to the app at all, and whether the app's own daemon-version display (state.daemon) also stays stale after the restart.

Expected: once the daemon is back on the new version, the chip disappears, the top bar shows the new daemon version, and if the restart never completes the user sees an error instead of an endless "updating".

What Changed

The periodic resync attachments in the container log belong to the runner's GitHub token pump (src/puck-runner/daemon-link.ts), whose connections do not forward those snapshots to the app's renderer.

Attach welcomes now include the daemon version, build, and replay head, and the renderer applies them without replacing the projection or fetching another snapshot.
A changed version or build clears the in-progress update and refreshes the displayed daemon; the same build keeps a drain pending across reattach while its original turns are still running, ignores upgrading events replayed through the completed head, and sets an error if the restart does not finish within two minutes.
The top bar replaces "Daemon updating" with "Daemon update failed" and a banner that offers Retry update while the environment is attached, or Reconnect when the attach view already offers it.

Risk Assessment

✅ Low: A new-build welcome or snapshot clears the updating chip and refreshes the daemon version, a restart that never finishes becomes an error after two minutes, and a same-build reattach keeps a drain pending only while its original turns are still running.

Testing

I launched the isolated Puck window and drove the top bar through the reported update, a replay onto the new daemon, a two-minute stall, and the drain reconnect cases. The updating chip cleared to 0.1.0+d48a3a42570d, a stall showed Daemon update failed with Retry update, and a drain stayed pending across a short drop and a long same-build replay until its original turns finished.

Live validation: ✅ go - 8 of 8 scenarios driven live against the product

Scenario	Result	Live	Evidence
Click Daemon update and choose Now; after the daemon comes back on a new build by replaying events, the updating chip disappears and the details show the new version	✅ pass	live	~/.no-mistakes/evidence/01M3S4B93NMVHBA3MHMEEF3YJC/03-chip-cleared-new-version.png
A replayed upgrading event from the build that just restarted does not bring the updating chip back	✅ pass	live	~/.no-mistakes/evidence/01M3S4B93NMVHBA3MHMEEF3YJC/03-chip-cleared-new-version.png
If the restart never finishes, the top bar shows Daemon update failed with Retry update and no Reconnect while the environment is attached	✅ pass	live	~/.no-mistakes/evidence/01M3S4B93NMVHBA3MHMEEF3YJC/04-update-failed-retry.png
A same-build welcome during an immediate update leaves Daemon updating up, and the two-minute deadline then shows the failure	✅ pass	live	~/.no-mistakes/evidence/01M3S4B93NMVHBA3MHMEEF3YJC/06-same-build-welcome-keeps-updating.png
When the runner is unreachable, a failed daemon update offers Reconnect and does not add Retry update	✅ pass	live	~/.no-mistakes/evidence/01M3S4B93NMVHBA3MHMEEF3YJC/07-unreachable-reconnect.png
A one-second disconnect during a drain does not fail the update after more than two minutes while the same build is back and turns are still running, and the new build then clears the chip	✅ pass	live	~/.no-mistakes/evidence/01M3S4B93NMVHBA3MHMEEF3YJC/08-drain-survives-brief-reconnect.png
After that drain's original turns end, a restart that never arrives shows the failure even though a new turn has started	✅ pass	live	~/.no-mistakes/evidence/01M3S4B93NMVHBA3MHMEEF3YJC/10-drain-restart-timeout.png
A drain that stays disconnected for two minutes stays failed, a same-build replay without a snapshot restores Daemon updating, and the deadline fires after the original turns end	✅ pass	live	~/.no-mistakes/evidence/01M3S4B93NMVHBA3MHMEEF3YJC/12-same-build-replay-restores-drain.png

Evidence: Top-bar text after each step

After replay: chips empty, Daemon 0.1.0+d48a3a42570d (d48a3a4). Stall while attached: Daemon update failed, buttons [Retry update]. Unreachable: buttons [Reconnect]. Drain after a one-second drop and two minutes: Daemon updating, title Updating once running turns finish. Long disconnect: Reconnecting plus Daemon update failed, no buttons; same-build welcome restores Daemon updating; two minutes after the original turns end: Daemon update failed, Retry update.

{
  "patched": 7,
  "pageErrors": [],
  "observations": [
    {
      "name": "dialog",
      "ui": {
        "chips": "Daemon update",
        "chipTitles": [
          {
            "text": "Daemon update",
            "title": "This Puck carries a newer environment daemon"
          }
        ],
        "bannerHidden": true,
        "banner": "",
        "bannerPhase": "",
        "bannerButtons": [],
        "dialogHidden": false,
        "dialogText": "Update the environment daemon\n\nThis Puck carries a newer daemon. Updating restarts it; conversations and work items continue from where they are.\n\nCancel\nNow\nAfter running turns finish",
        "facts": "",
        "word": "Ready"
      }
    },
    {
      "name": "updating",
      "seq": 1001,
      "ui": {
        "chips": "Daemon updating",
        "chipTitles": [
          {
            "text": "Daemon updating",
            "title": "Updating now"
          }
        ],
        "bannerHidden": true,
        "banner": "",
        "bannerPhase": "",
        "bannerButtons": [],
        "dialogHidden": true,
        "dialogText": "",
        "facts": "",
        "word": "Ready"
      }
    },
    {
      "name": "cleared",
      "ui": {
        "chips": "",
        "chipTitles": [],
        "bannerHidden": true,
        "banner": "",
        "bannerPhase": "",
        "bannerButtons": [],
        "dialogHidden": true,
        "dialogText": "",
        "facts": "Environment\nacme-launch\nRunner\nbuild-box\nStatus\nReady\nDefinition\nv1.4.0 (a1b2c3d)\nDaemon\n0.1.0+d48a3a42570d (d48a3a4)",
        "word": "Ready"
      }
    },
    {
      "name": "timeout-attached",
      "ui": {
        "chips": "Daemon update failed",
        "chipTitles": [
          {
            "text": "Daemon update failed",
            "title": "The daemon update did not complete within two minutes."
          }
        ],
        "bannerHidden": false,
        "banner": "The daemon update did not complete within two minutes.\nRetry update",
        "bannerPhase": "daemon-update-failed",
        "bannerButtons": [
          "Retry update"
        ],
        "dialogHidden": true,
        "dialogText": "",
        "facts": "",
        "word": "Ready"
      }
    },
    {
      "name": "retry-dialog",
      "ui": {
        "chips": "Daemon update failed",
        "chipTitles": [
          {
            "text": "Daemon update failed",
            "title": "The daemon update did not complete within two minutes."
          }
        ],
        "bannerHidden": false,
        "banner": "The daemon update did not complete within two minutes.\nRetry update",
        "bannerPhase": "daemon-update-failed",
        "bannerButtons": [
          "Retry update"
        ],
        "dialogHidden": false,
        "dialogText": "Update the environment daemon\n\nThis Puck carries a newer daemon. Updating restarts it; conversations and work items continue from where they are.\n\nCancel\nNow\nAfter running turns finish",
        "facts": "",
        "word": "Ready"
      }
    },
    {
      "name": "same-build-keeps-chip",
      "ui": {
        "chips": "Daemon updating",
        "chipTitles": [
          {
            "text": "Daemon updating",
            "title": "Updating now"
          }
        ],
        "bannerHidden": true,
        "banner": "",
        "bannerPhase": "",
        "bannerButtons": [],
        "dialogHidden": true,
        "dialogText": "",
        "facts": "",
        "word": "Ready"
      }
    },
    {
      "name": "same-build-then-timeout",
      "ui": {
        "chips": "Daemon update failed",
        "chipTitles": [
          {
            "text": "Daemon update failed",
            "title": "The daemon update did not complete within two minutes."
          }
        ],
        "bannerHidden": false,
        "banner": "The daemon update did not complete within two minutes.\nRetry update",
        "bannerPhase": "daemon-update-failed",
        "bannerButtons": [
          "Retry update"
        ],
        "dialogHidden": true,
        "dialogText": "",
        "facts": "",
        "word": "Ready"
      }
    },
    {
      "name": "unreachable",
      "ui": {
        "chips": "Daemon update failed",
        "chipTitles": [
          {
            "text": "Daemon update failed",
            "title": "The daemon update did not complete within two minutes."
          }
        ],
        "bannerHidden": false,
        "banner": "The daemon update did not complete within two minutes.\nReconnect",
        "bannerPhase": "daemon-update-failed",
        "bannerButtons": [
          "Reconnect"
        ],
        "dialogHidden": true,
        "dialogText": "",
        "facts": "",
        "word": "Unreachable"
      }
    },
    {
      "name": "drain-survives-reconnect",
      "turns": 4,
      "ui": {
        "chips": "Daemon updating",
        "chipTitles": [
          {
            "text": "Daemon updating",
            "title": "Updating once running turns finish"
          }
        ],
        "bannerHidden": true,
        "banner": "",
        "bannerPhase": "",
        "bannerButtons": [],
        "dialogHidden": true,
        "dialogText": "",
        "facts": "",
        "word": "Ready"
      }
    },
    {
      "name": "drain-clears-on-new-build",
      "ui": {
        "chips": "",
        "chipTitles": [],
        "bannerHidden": true,
        "banner": "",
        "bannerPhase": "",
        "bannerButtons": [],
        "dialogHidden": true,
        "dialogText": "",
        "facts": "Environment\nacme-launch\nRunner\nbuild-box\nStatus\nReady\nDefinition\nv1.4.0 (a1b2c3d)\nDaemon\n0.1.0+d48a3a42570d (d48a3a4)",
        "word": "Ready"
      }
    },
    {
      "name": "drain-still-updating-before-turns-end",
      "ui": {
        "chips": "Daemon updating",
        "chipTitles": [
          {
            "text": "Daemon updating",
            "title": "Updating once running turns finish"
          }
        ],
        "bannerHidden": true,
        "banner": "",
        "bannerPhase": "",
        "bannerButtons": [],
        "dialogHidden": true,
        "dialogText": "",
        "facts": "",
        "word": "Ready"
      }
    },
    {
      "name": "drain-restart-still-waiting",
      "ui": {
        "chips": "Daemon updating",
        "chipTitles": [
          {
            "text": "Daemon updating",
            "title": "Updating once running turns finish"
          }
        ],
        "bannerHidden": true,
        "banner": "",
        "bannerPhase": "",
        "bannerButtons": [],
        "dialogHidden": true,
        "dialogText": "",
        "facts": "",
        "word": "Ready"
      }
    },
    {
      "name": "drain-restart-timeout",
      "ui": {
        "chips": "Daemon update failed",
        "chipTitles": [
          {
            "text": "Daemon update failed",
            "title": "The daemon update did not complete within two minutes."
          }
        ],
        "bannerHidden": false,
        "banner": "The daemon update did not complete within two minutes.\nRetry update",
        "bannerPhase": "daemon-update-failed",
        "bannerButtons": [
          "Retry update"
        ],
        "dialogHidden": true,
        "dialogText": "",
        "facts": "",
        "word": "Ready"
      }
    },
    {
      "name": "disconnected-failed",
      "ui": {
        "chips": "Reconnecting\nDaemon update failed",
        "chipTitles": [
          {
            "text": "Reconnecting",
            "title": ""
          },
          {
            "text": "Daemon update failed",
            "title": "The daemon update did not complete within two minutes."
          }
        ],
        "bannerHidden": false,
        "banner": "The daemon update did not complete within two minutes.",
        "bannerPhase": "daemon-update-failed",
        "bannerButtons": [],
        "dialogHidden": true,
        "dialogText": "",
        "facts": "",
        "word": "Ready"
      }
    },
    {
      "name": "stays-failed-while-disconnected",
      "ui": {
        "chips": "Reconnecting\nDaemon update failed",
        "chipTitles": [
          {
            "text": "Reconnecting",
            "title": ""
          },
          {
            "text": "Daemon update failed",
            "title": "The daemon update did not complete within two minutes."
          }
        ],
        "bannerHidden": false,
        "banner": "The daemon update did not complete within two minutes.",
        "bannerPhase": "daemon-update-failed",
        "bannerButtons": [],
        "dialogHidden": true,
        "dialogText": "",
        "facts": "",
        "word": "Ready"
      }
    },
    {
      "name": "restored-drain",
      "turns": 4,
      "ui": {
        "chips": "Daemon updating",
        "chipTitles": [
          {
            "text": "Daemon updating",
            "title": "Updating once running turns finish"
          }
        ],
        "bannerHidden": true,
        "banner": "",
        "bannerPhase": "daemon-update-failed",
        "bannerButtons": [],
        "dialogHidden": true,
        "dialogText": "",
        "facts": "",
        "word": "Ready"
      }
    },
    {
      "name": "restored-deadline-waiting",
      "ui": {
        "chips": "Daemon updating",
        "chipTitles": [
          {
            "text": "Daemon updating",
            "title": "Updating once running turns finish"
          }
        ],
        "bannerHidden": true,
        "banner": "",
        "bannerPhase": "daemon-update-failed",
        "bannerButtons": [],
        "dialogHidden": true,
        "dialogText": "",
        "facts": "",
        "word": "Ready"
      }
    },
    {
      "name": "restored-deadline-fired",
      "ui": {
        "chips": "Daemon update failed",
        "chipTitles": [
          {
            "text": "Daemon update failed",
            "title": "The daemon update did not complete within two minutes."
          }
        ],
        "bannerHidden": false,
        "banner": "The daemon update did not complete within two minutes.\nRetry update",
        "bannerPhase": "daemon-update-failed",
        "bannerButtons": [
          "Retry update"
        ],
        "dialogHidden": true,
        "dialogText": "",
        "facts": "",
        "word": "Ready"
      }
    }
  ]
}

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 2 issues found → auto-fixed (2) ✅

⚠️ src/renderer/instance-store.ts:401 - A drain that stays disconnected for the two-minute deadline is labeled failed after the app reattaches, even though the same build is back and the original turns are still running. watchUpgrade (line 175) arms the deadline on any non-attached upsert. When it fires while still disconnected, the callback at lines 175-180 clears upgrading and sets upgradeError but leaves upgradePhase as draining and drainTurns intact. Coming back is a replay, not a snapshot: instance-sync.ts:153 only reads a snapshot when the projection is empty, and instance-sync.ts:160 then calls applyWelcome. A same-build welcome (lines 403-408) updates the daemon and calls watchUpgrade, and because upgrading is already null it returns without undoing the failure. The restore at lines 387-392 runs only inside applySnapshot, so it never sees this reattach. The chip and banner stay on "Daemon update failed" for the rest of the drain, and Retry (topbar.ts:323) calls daemon.upgrade while that process is still in upgrading and is rejected. Concrete sequence: drain with turn T in flight, the attach stays down for two minutes (the turn is still running on that daemon), then the channel returns within the event log so the welcome is a replay of the same build and T is still in inflight. Restore that drain on the same-build welcome with the same condition as lines 387-392 (phase still draining, original turns still in flight, status not stopping), so a later turn.end or a real restart can arm a fresh deadline. Leave the error in place when the daemon stays disconnected.
⚠️ src/renderer/topbar.ts:331 - The daemon-update-failed banner always adds a Reconnect button (view.retry || upgradeError), including when the environment is already attached and Retry is the failure action. No intent requirement needs that control: the required outcome is that a restart which never finishes is visible as an error, and the attach view already shows Reconnect when the connection itself can be retried. Recommend removing the || upgradeError branch so Reconnect stays on the existing attach-view rule.

🔧 Fix applied.
1 warning still open:

⚠️ src/renderer/instance-store.ts:401 - A drain that stays disconnected for the two-minute deadline is labeled failed after the app reattaches, even though the same build is back and the original turns are still running. watchUpgrade (line 175) arms the deadline on any non-attached upsert. When it fires while still disconnected, the callback at lines 175-180 clears upgrading and sets upgradeError but leaves upgradePhase as draining and drainTurns intact. Coming back is a replay, not a snapshot: instance-sync.ts:153 only reads a snapshot when the projection is empty, and instance-sync.ts:160 then calls applyWelcome. A same-build welcome (lines 403-408) updates the daemon and calls watchUpgrade, and because upgrading is already null it returns without undoing the failure. The restore at lines 387-392 runs only inside applySnapshot, so it never sees this reattach. The chip and banner stay on "Daemon update failed" for the rest of the drain, and Retry (topbar.ts:323) calls daemon.upgrade while that process is still in upgrading and is rejected. Concrete sequence: drain with turn T in flight, the attach stays down for two minutes (the turn is still running on that daemon), then the channel returns within the event log so the welcome is a replay of the same build and T is still in inflight. Restore that drain on the same-build welcome with the same condition as lines 387-392 (phase still draining, original turns still in flight, status not stopping), so a later turn.end or a real restart can arm a fresh deadline. Leave the error in place when the daemon stays disconnected.

🔧 Fix applied.
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

Live validation: ✅ go - 8 of 8 scenarios driven live against the product

Scenario	Result	Live	Evidence
Click Daemon update and choose Now; after the daemon comes back on a new build by replaying events, the updating chip disappears and the details show the new version	✅ pass	live	~/.no-mistakes/evidence/01M3S4B93NMVHBA3MHMEEF3YJC/03-chip-cleared-new-version.png
A replayed upgrading event from the build that just restarted does not bring the updating chip back	✅ pass	live	~/.no-mistakes/evidence/01M3S4B93NMVHBA3MHMEEF3YJC/03-chip-cleared-new-version.png
If the restart never finishes, the top bar shows Daemon update failed with Retry update and no Reconnect while the environment is attached	✅ pass	live	~/.no-mistakes/evidence/01M3S4B93NMVHBA3MHMEEF3YJC/04-update-failed-retry.png
A same-build welcome during an immediate update leaves Daemon updating up, and the two-minute deadline then shows the failure	✅ pass	live	~/.no-mistakes/evidence/01M3S4B93NMVHBA3MHMEEF3YJC/06-same-build-welcome-keeps-updating.png
When the runner is unreachable, a failed daemon update offers Reconnect and does not add Retry update	✅ pass	live	~/.no-mistakes/evidence/01M3S4B93NMVHBA3MHMEEF3YJC/07-unreachable-reconnect.png
A one-second disconnect during a drain does not fail the update after more than two minutes while the same build is back and turns are still running, and the new build then clears the chip	✅ pass	live	~/.no-mistakes/evidence/01M3S4B93NMVHBA3MHMEEF3YJC/08-drain-survives-brief-reconnect.png
After that drain's original turns end, a restart that never arrives shows the failure even though a new turn has started	✅ pass	live	~/.no-mistakes/evidence/01M3S4B93NMVHBA3MHMEEF3YJC/10-drain-restart-timeout.png
A drain that stays disconnected for two minutes stays failed, a same-build replay without a snapshot restores Daemon updating, and the deadline fires after the original turns end	✅ pass	live	~/.no-mistakes/evidence/01M3S4B93NMVHBA3MHMEEF3YJC/12-same-build-replay-restores-drain.png

node ~/.no-mistakes/evidence/01M3S4B93NMVHBA3MHMEEF3YJC/drive-daemon-update.mjs against the isolated app (PUCK_FIXTURE=full, CDP on the environment window)
Clicked Daemon update, chose Now, then delivered a new-build welcome and a 14-event replay including an old upgrading event
Advanced the window clock past two minutes for an immediate update that never returned, then clicked Retry update
Sent a same-build welcome during an immediate update, then advanced two minutes
Failed an update while the runner was unreachable and checked the banner actions
Drained with four live turns, dropped the attach for one second, returned on the same build, advanced more than two minutes, then welcomed the new build
Ended those original turns, started a new turn, and advanced two minutes
Left a drain disconnected for two minutes, confirmed the failure stayed, then restored it with a same-build welcome and no snapshot and advanced two minutes after the original turns ended

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

Namik Mesic added 7 commits September 30, 2026 12:54

Clear daemon update state on replay reattachment

9eae5c4

no-mistakes(review): Keep drain updates pending across brief reconnects

02c0309

no-mistakes(document): Document daemon update completion and failure

55418ab

Track the original drain through daemon update retries

4d42f14

no-mistakes(review): Show Reconnect only when the attach view offers it

6081332

no-mistakes(review): Restore a running drain on same-build replay

e42c6f4

no-mistakes(document): Correct the drain deadline comment

60e10e5

namikmesic changed the title ~~fix(renderer): clear the daemon updating chip after restart and time out a stuck update~~ fix(renderer): clear the stuck daemon updating state after restart Sep 30, 2026

namikmesic merged commit 6efad7f into main Sep 30, 2026
4 checks passed

Navigation Menu

Repository navigation

fix(renderer): clear the stuck daemon updating state after restart - #38

fix(renderer): clear the stuck daemon updating state after restart#38
namikmesic merged 7 commits into
mainfrom
fm/clear-the-daemon-updating-state-after-th-95

namikmesic commented Sep 30, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

namikmesic commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Intent

What Changed

Risk Assessment

Testing

Pipeline

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

namikmesic commented Sep 30, 2026 •

edited

Loading