Repository navigation
logs: plain wording and event severity for customer-visible lines - #152
Merged
Merged
Conversation
Telemetry events log at their own severity, and delivery outcomes log at debug without the endpoint URL. The self-heal give-up, the stale upgrade marker, the startup timeout warning, the archive fetch breaker, and the component error contexts say what happens to the node and what to do next in plain words. leader, member, and DCS become primary, node, and etcd in customer-visible lines. Supervisor lines match the other HA images. The README names Railway instead of its components.
The e2e greps that still looked for the previous wording of lines this branch rewrote now match the new text. Test-only change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What the user now reads
Copy-only pass over the lines that land in a service's deploy logs and the README. No behaviour changes.
Telemetry events in the log (
common/src/telemetry.rs)INFO(Self-heal: giving up ...,Boot refused ...,... (write outage)), followed byINFO telemetry sent. Each event now logs at its own severity:ERRORfor giving up, boot refused, write outage, process died, bootstrap failed, and component errors;WARNfor degraded-but-recovering states (reinit triggered, stand-down, archive drift, etcd data cleared, ...);INFOfor the routine ones.telemetry sent/telemetry rejected/telemetry send failed→debug, and the transport error no longer carries the endpoint URL.Event messages
Self-heal: giving up on {node} after {n} attempts (last: {reason}); manual intervention required→Self-heal: giving up on {node} after {n} attempts (last: {reason}). This node needs attention: contact support.Self-heal watcher on {node} standing down: a major upgrade owns this volume (marker phase: {}, age: {}) — if no upgrade workflow is active, the marker is stale and must be removed to restore self-healing→Self-heal on {node} is paused while a major version upgrade owns this volume (marker phase: {}, age: {}). If no upgrade is running, delete .railway-major-upgrade.json at the volume root and redeploy to restore self-healing.DCS unavailable - {node} demoted, cluster {scope} has no leader (write outage)→Cluster {scope} has no primary: {node} sees no healthy primary (write outage)Boot refused on {node} by the major-upgrade guard: {reason}→Boot refused on {node}: {reason}re-clones from leader {}→re-clones from primary {}(two events){component} error in {context}: {error}was garbled when callers passed sentences as the context. The four callers now pass a short context (reseed marker removal,rest api address,standalone boot,standalone PITR sidecar,standalone PITR setup) and keep the sentence in the error.Stale upgrade marker (
bin/patroni_runner.rs)failed to remove the reseed marker {when}; the self-heal watcher stands down while it exists — if no upgrade is running, remove it manually to restore self-healing→failed to remove the reseed marker {when}. Self-healing on this node is paused while it exists. If no upgrade is running, delete .railway-major-upgrade.json at the volume root and redeploy.self-heal: major upgrade in progress on this volume — standing down→self-heal: paused while a major version upgrade owns this volumeStartup timeout warning (
patroni/monitoring.rs)max_startup_timeout is shorter than the WAL archive-stall confirmation dwell — the Stalled-branch reinit can fire ...→PATRONI_MAX_STARTUP_TIMEOUT is shorter than WAL_ARCHIVE_STALL_CONFIRM_SECONDS, so a stalled replica can be reinitialized sooner than intended. Raise PATRONI_MAX_STARTUP_TIMEOUT or lower WAL_ARCHIVE_STALL_CONFIRM_SECONDS.Archive fetch breaker (
pgbackrest-archive-get-wrapper.sh)archive endpoint unreachable for {n} consecutive invocations (last rc={rc}) — connectivity breaker tripped, exiting 126 so recovery crash-loops instead of waiting on a dead archive forever→archive unreachable for {n} consecutive attempts (last rc={rc}); stopping so the restore retriesVocabulary in customer-visible lines: leader → primary (
haproxy/src/monitoring.rs,patroni/self_heal.rs,patroni/backup_watcher.rs,bin/on_role_change.rsmaster promotion→promotion to primary), member → node (patroni/slot_recovery.rs), DCS → etcd (patroni/reconcile.rs,patroni/backup_watcher.rs).Supervisors (
health_server/mod.rs,patroni/backup_watcher.rs)health server bind/serve failed — respawning in 5s(and siblings) →health server failed; restarting in 5s, the same wording the other HA images use.README
Behaviour
Copy-only.
TelemetryEvent::severity()is new and only picks the log macro. The e2e grep intest/e2e-ha.shfollows the new watcher string.Verification
cargo fmt --all --checkclean,cargo clippy --workspace --all-targets(no new warnings in touched files),cargo test --workspace: 485 passed, 0 failed;bash test/unit-pgbackrest-wrappers.sh: 25 passed, 0 failed (its grep follows the new breaker string).