fix(etcd): identify the member by peer URL, refuse a duplicate ETCD_NAME, stop the EOF log spew

[paulocsanz]

What happened

On onyx-staging/staging, the service etcd-3 runs with ETCD_NAME=etcd-2. Its logs show Adding self as learner node=etcd-2, Joining as learner cluster=etcd-1=...,etcd-2=http://etcd-3.railway.internal:2380, and etcd's published local member ... Name:etcd-2 ... local-member-id a1331e7952d5ee8a. The real etcd-2 service also publishes Name:etcd-2 (member eae04cfdcb9f0b03). So two members share a name, and their peer URLs differ.

The entrypoint found "self" in member list by name (is_learner, get_my_member_id). With two members named etcd-2, both lookups matched the voting etcd-2, so promote_self logged "Already a voting member", the monitor wrote the bootstrap marker, and the node has stayed a learner since 2026-09-05. At boot on 2026-10-06 the raft config showed 2 voters and 1 learner (this node). That morning's join logs show it re-adding itself as a learner about 20 times in 30 minutes. remove_stale_self also matched name == my_name || peer_url == my_peer_url and removed the first match, which with a shared name can be the other member's entry.

The cluster had three members but only two voters, so it had no fault tolerance. On 2026-10-13 11:42Z one voter (etcd-2, placed cross-region) timed out and ReadIndex failed. Patroni lost the leader lock, and its etcd client rotated to the learner. The learner rejects watches with etcdserver: rpc not supported for learner, logged about 1.8M times a day per Postgres member. The Postgres members have been replicas with no primary for more than 27 hours. Two more clusters of the same customer (onyx-prod/prod, myso-staging) show the same learner error.

Separately, the etcd service logs 2026/10/13 11:42:27 ERROR: Failed to send request: EOF twice every ~50 ms: about 7M lines a day on one member and about 20M across the three clusters. This line comes from etcd itself, not from this repo. etcd's gRPC gateway (ETCD_ENABLE_GRPC_GATEWAY=true, the HTTP/JSON API Patroni's etcd3 client uses) handles the bidirectional Watch and LeaseKeepAlive streams in api/etcdserverpb/gw/rpc.pb.gw.go (v3.6.6, request_Watch_Watch_0 / request_Lease_LeaseKeepAlive_0). When stream.Send fails, it calls grpclog.Errorf("Failed to send request: %v", err). etcd points grpclog's default logger at stderr (grpclog.NewLoggerV2(io.Discard, os.Stderr, os.Stderr) in server/embed/config_logging.go), which explains the Go log timestamp and the ERROR: prefix. Each time the server ends a stream the client is still writing to, the gateway logs one line. Patroni reopening rejected watches against the learner produces this rate. The env var GRPC_GO_LOG_SEVERITY_LEVEL cannot silence it because etcd installs its own grpclog logger.

What changed

• Identity. find_self locates this node's entry by the member ID that etcdctl member add --learner printed, when the node added itself in this run. Otherwise it uses the advertised peer URL from ETCD_INITIAL_ADVERTISE_PEER_URLS. etcd refuses a duplicate peer URL, so the peer URL is unique. Name matching remains only as a fallback when no peer URL is configured. If a peer URL is configured but absent from the list, the node is not a member, whatever other member shares its name. promote_self, remove_stale_self, the "already a member" check in add_self_to_cluster, the cluster string, and move_leader_away (which excluded self by name) all use this lookup now. Each log line records which key matched.
• Name collision refused. Before a node adds itself, on both the follower join and the leader's recovery join, it checks the member list. If a voting member already holds ETCD_NAME at a different peer URL, the node sends a ComponentError and exits without retrying (a configuration error no retry fixes). The message reads: ETCD_NAME is etcd-2 but the voting member etcd-2 already exists at http://etcd-2.railway.internal:2380; this service advertises http://etcd-3.railway.internal:2380. ... To recover: set this service's ETCD_NAME to its own name (etcd-3) and redeploy. It does not auto-rename. A running member that already shares its name (the onyx case) is not killed. It logs and reports a one-time warning with the same fix and keeps working, because it now identifies itself by peer URL.
• Promotion. On every start, whatever the bootstrap path, the monitor confirms that this node's own entry is a voting member and promotes it if it is still a learner. A restarted learner with its data dir and marker present now retries promotion. That restart is what repairs the existing onyx learners once they run this image. On a learner join, the bootstrap marker is written only after the node's own entry is confirmed voting. A fresh join still exits at the 15-minute cap so its unmarked data dir is wiped and rejoined. A restarted member never exits for this: after the cap it retries once a minute and reports EtcdPromotionFailed once.
• EOF spew. The entrypoint already tees etcd's stderr, so it now throttles an allowlist of exactly these two gateway messages: ERROR: Failed to send request: and ERROR: Failed to terminate client stream: . Each distinct message is written at most once a minute, followed by a line with the number of suppressed repeats. Every other line passes through untouched. The trigger itself (Patroni hammering a learner) goes away with the promotion fix.

How it was tested

cargo test --locked -p etcd-entrypoint -p common: etcd-entrypoint 48 passed, common 14 passed, 0 failed. New tests:

• self found by peer URL when two members share a name (the onyx-staging member list), and the real etcd-2 still finds its own entry
• the member ID from member add wins; a stale ID falls back to the peer URL
• a configured but absent peer URL is not matched by name
• name fallback only when no peer URL is configured
• unstarted member (empty name) found by peer URL
• collision refused with the exact message and fix; a learner sharing the name and a node rejoining under its own name/URL are not collisions
• member add output parsing (ID and ETCD_INITIAL_CLUSTER)
• marker gating, and retry/exit policy for a fresh join versus a restart
• gateway line recognition (allowlist only) and the once-per-minute throttle with a drop count

cargo fmt --all --check passes. cargo clippy -p etcd-entrypoint --all-targets reports one warning, for_kv_map in get_voting_member_endpoint, which also exists on main and is untouched here. These changes were not exercised against a live cluster.