core: BackupFailed alert clears itself even though the backup is still broken

[gsanchietti]

Component and version

ns8-core, node/bin/run-backup. Reproduced on current ns8-stable on a fresh single-node install.

Steps to reproduce

1. Set up a backup with a schedule on any node.
2. Manually mark it as failed, then run it again:

   printf 'node_backup_status{id="N",name="BKrepro"} 0\n' > /run/node_exporter/backupN.prom
   runagent -m node run-backup --backup=N &
   while sleep 1; do grep -o 'node_backup_status.*' /run/node_exporter/backupN.prom; done

3. Separately, simulate a reboot by clearing /run (it's memory-only, so this is exactly what a reboot does) and restarting the backup timer:

   rm -f /run/node_exporter/backup*.prom
   systemctl restart backup-timers.service
   cat /run/node_exporter/backupN.prom

Expected behavior

The backup status shown to Prometheus should always reflect the last known real result. If a backup is genuinely broken, any alert based on it should keep firing until a run actually succeeds.

Actual behavior

• Every time a backup runs, its status file is reset to "unknown" the moment the run starts, before we know the outcome. This makes the status flip failed → unknown → (result) on every run, so a "backup failed" alert clears and re-fires every cycle even though the backup is still broken.
• After a reboot, the status file is gone and nothing rebuilds it. The backup status disappears from monitoring entirely until the next run finishes, silently clearing any active alert.

This is what happened in production: a backup kept failing (retention cleanup was hitting a locked repository), but the alert kept clearing and re-firing on every run and every reboot, so nobody got paged for the real, ongoing failure.

Suggested fix or workaround

• Only reset the status to "unknown" if no previous status exists yet, instead of every run.
• Add a small step that rebuilds the status file from the backup history already stored in Redis, run right after a reboot / timer restart, so a missing file doesn't mean "everything's fine."

Relevant logs or output

Reproduced live:

run starts: node_backup_status ... 0
mid-run:    node_backup_status ... -1   <- should still say 0, backup hasn't recovered
run ends:   node_backup_status ... 1

After simulated reboot:

$ cat /run/node_exporter/backup2.prom
cat: /run/node_exporter/backup2.prom: No such file or directory

Not covered by this issue: the actual retention lock-contention failure that triggered the original incident, and a matching alert-rule tweak needed in ns8-metrics once this is fixed — both worth separate issues.