Skip to content

logs: plain wording and event severity for customer-visible lines - #152

Merged
paulocsanz merged 2 commits into
mainfrom
pcs/customer-facing-log-copy
Oct 9, 2026
Merged

paulocsanz merged 2 commits into
mainfrom
pcs/customer-facing-log-copy

Conversation

@paulocsanz

@paulocsanz paulocsanz commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

What the user now reads

Copy-only pass over the lines that land in a service's deploy logs and the README. No behaviour changes.

Telemetry events in the log (common/src/telemetry.rs)

  • Every telemetry event was logged at INFO (Self-heal: giving up ..., Boot refused ..., ... (write outage)), followed by INFO telemetry sent. Each event now logs at its own severity: ERROR for giving up, boot refused, write outage, process died, bootstrap failed, and component errors; WARN for degraded-but-recovering states (reinit triggered, stand-down, archive drift, etcd data cleared, ...); INFO for the routine ones.
  • telemetry sent / telemetry rejected / telemetry send failed → debug, and the transport error no longer carries the endpoint URL.

Event messages

  • Self-heal: giving up on {node} after {n} attempts (last: {reason}); manual intervention required → Self-heal: giving up on {node} after {n} attempts (last: {reason}). This node needs attention: contact support.
  • Self-heal watcher on {node} standing down: a major upgrade owns this volume (marker phase: {}, age: {}) — if no upgrade workflow is active, the marker is stale and must be removed to restore self-healing → Self-heal on {node} is paused while a major version upgrade owns this volume (marker phase: {}, age: {}). If no upgrade is running, delete .railway-major-upgrade.json at the volume root and redeploy to restore self-healing.
  • DCS unavailable - {node} demoted, cluster {scope} has no leader (write outage) → Cluster {scope} has no primary: {node} sees no healthy primary (write outage)
  • Boot refused on {node} by the major-upgrade guard: {reason} → Boot refused on {node}: {reason}
  • re-clones from leader {} → re-clones from primary {} (two events)
  • {component} error in {context}: {error} was garbled when callers passed sentences as the context. The four callers now pass a short context (reseed marker removal, rest api address, standalone boot, standalone PITR sidecar, standalone PITR setup) and keep the sentence in the error.

Stale upgrade marker (bin/patroni_runner.rs)

  • failed to remove the reseed marker {when}; the self-heal watcher stands down while it exists — if no upgrade is running, remove it manually to restore self-healing → failed to remove the reseed marker {when}. Self-healing on this node is paused while it exists. If no upgrade is running, delete .railway-major-upgrade.json at the volume root and redeploy.
  • self-heal: major upgrade in progress on this volume — standing down → self-heal: paused while a major version upgrade owns this volume

Startup timeout warning (patroni/monitoring.rs)

  • max_startup_timeout is shorter than the WAL archive-stall confirmation dwell — the Stalled-branch reinit can fire ... → PATRONI_MAX_STARTUP_TIMEOUT is shorter than WAL_ARCHIVE_STALL_CONFIRM_SECONDS, so a stalled replica can be reinitialized sooner than intended. Raise PATRONI_MAX_STARTUP_TIMEOUT or lower WAL_ARCHIVE_STALL_CONFIRM_SECONDS.

Archive fetch breaker (pgbackrest-archive-get-wrapper.sh)

  • archive endpoint unreachable for {n} consecutive invocations (last rc={rc}) — connectivity breaker tripped, exiting 126 so recovery crash-loops instead of waiting on a dead archive forever → archive unreachable for {n} consecutive attempts (last rc={rc}); stopping so the restore retries

Vocabulary in customer-visible lines: leader → primary (haproxy/src/monitoring.rs, patroni/self_heal.rs, patroni/backup_watcher.rs, bin/on_role_change.rs master promotion → promotion to primary), member → node (patroni/slot_recovery.rs), DCS → etcd (patroni/reconcile.rs, patroni/backup_watcher.rs).

Supervisors (health_server/mod.rs, patroni/backup_watcher.rs)

  • health server bind/serve failed — respawning in 5s (and siblings) → health server failed; restarting in 5s, the same wording the other HA images use.

README

  • README: internal component names replaced with "Railway" throughout.

Behaviour

Copy-only. TelemetryEvent::severity() is new and only picks the log macro. The e2e grep in test/e2e-ha.sh follows the new watcher string.

Verification

  • cargo fmt --all --check clean, cargo clippy --workspace --all-targets (no new warnings in touched files), cargo test --workspace: 485 passed, 0 failed; bash test/unit-pgbackrest-wrappers.sh: 25 passed, 0 failed (its grep follows the new breaker string).

paulocsanz and others added 2 commits October 9, 2026 02:19
Telemetry events log at their own severity, and delivery outcomes log
at debug without the endpoint URL. The self-heal give-up, the stale
upgrade marker, the startup timeout warning, the archive fetch breaker,
and the component error contexts say what happens to the node and what
to do next in plain words. leader, member, and DCS become primary,
node, and etcd in customer-visible lines. Supervisor lines match the
other HA images. The README names Railway instead of its components.
The e2e greps that still looked for the previous wording of lines this
branch rewrote now match the new text. Test-only change.
@paulocsanz
paulocsanz merged commit c85f0b1 into main Oct 9, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant