Skip to content

metrics: fix reboot-required race condition - #170

Open
elisaado wants to merge 1 commit into
nlewo:mainfrom
elisaado:main
Open

metrics: fix reboot-required race condition#170
elisaado wants to merge 1 commit into
nlewo:mainfrom
elisaado:main

Conversation

@elisaado

Copy link
Copy Markdown
Contributor

I found a race condition in the reboot required metric.

It was caused by:

  • the manager not sending "reboot required" over the broker upon startup
    • (I think. I am not sure comin watch does report the reboot required correctly, but I verified /both/ changes were necessary so I believe this is the case)
  • the metrics subscribing to the broker too late, missing the "reboot required" event

@nlewo

nlewo commented Jun 26, 2026

Copy link
Copy Markdown
Owner

I am not sure comin watch does report the reboot required correctly

Currently, when a client connect to the stream, the server first start by sending the manager state. This is used by the comin watch command to initialize its state. So, when a reboot is required from previous runs, the client doesn't get to be notifed.

I would prefer to only send notifications when the event actually occurs (i already think about renaming the event ManagerState to something such as Started).

Let me try to propose another approach.
I think the issue is that the manager is initialized in the manager.Run function.
What do you think about moving the manager.NeedToReboot initialization from the manager.Run function to the manager.New function?
In this case, the command run could manually set the prometheus need_to_reboot metric, before running the manager.

@elisaado

Copy link
Copy Markdown
Contributor Author

sounds good, will get to working on it!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants