fix(#6267): a peer address that identifies nobody says so, and a membership test stops poisoning its own teardown

docs/release-26.9.1.md

@@ -2235,6 +2235,53 @@ nameless `NegativeArraySizeException`.
 
 [#6216](https://github.com/ArcadeData/arcadedb/issues/6216)
 
+## A peer address that identifies nobody says so, instead of waiting for an operation to refuse (#6267)
+
+Five follow-ups from #6221 / #6226. Four are visible to an operator; the rest are test hygiene.
+
+**A withheld peer-to-peer endpoint is now reported.** `getUnambiguousPeerHttpAddress` and its HTTPS twin refuse
+an address two peers both resolve to by returning `null` (#6202), and every caller then decided for itself
+whether to say anything - so the refusal was visible only where one happened to log it, and invisible everywhere
+else. Neither existing warning covered it: the derive warnings fire whenever an address is derived **at all**,
+which is also what a perfectly healthy homogeneous Kubernetes StatefulSet does, so they cannot distinguish
+"deriving, and fine" from "deriving, and two peers just collapsed onto one address". There is now a one-time
+WARNING per protocol - modelled on the `warnAmbiguousRouting` of #6183, but for the peer-to-peer endpoints rather
+than the client routing tables - naming the peers that could not be told apart, the address they share, and the
+`host:{raft:..,http:..}` field to declare in `arcadedb.ha.serverList`. HTTP and HTTPS have separate latches: a
+cluster that declares distinct `http` ports and shares an `https` one must still hear about the second.
+
+**Observable change in `GET /api/v1/cluster`.** Each peer entry now carries `httpAddress` and, only when it is
+not the peer's alone, `httpAddressAmbiguous: true`. Before this, the status endpoint and the Studio HA panel
+displayed a plausible address for every peer with nothing to say that it named none of them, and an operator
+found out when a snapshot resync or a cluster verify refused to dial. A correctly declared cluster carries
+neither field's flag, so nothing changes for one. Studio renders the flag as a warning line on the node's card.
+
+**The presence matrix dialled the address nothing had checked.** `GET /api/v1/cluster?presence=true` asks every
+peer which databases it holds and attributes the answer to that peer - the same unattended dial the verify
+endpoint was making before #6221, with the same failure: on a cluster whose peers collapse onto one derived
+address, every peer was queried on the leader's own endpoint and reported the leader's database list as its own,
+so the matrix showed every database present on every node. It resolves through `PeerDialAddress` now, so a peer
+it cannot identify is reported in `unreachable` - with the reason logged - rather than answered for by whoever
+picked up.
+
+`RaftClusterStatusExporter.exportClusterStatus()`, a second cluster-status JSON builder that nothing has called,
+was removed rather than taught about any of this: the live endpoint is `GetClusterHandler`, and an unreachable
+second view of one cluster is how two views drift apart.
+
+**Test-only, in the HA lane.** `BaseRaftHATest.RESYNC_RETRY_TIMEOUT_MS` drops from 120 s to 30 s. #6226 added an
+instrument rather than guessing, and it has now reported: across nine full `ha-integration-tests` runs (235 tests
+each) not one wait exceeded the 10 s report threshold, and the slowest of the ten classes that use those helpers
+took 53 s wall-clock for the whole class, cluster startup and teardown included. 30 s is what the rest of that
+class already treats as long enough for a cluster to do anything it is going to do - `waitForReplicationIsCompleted`,
+`waitAllReplicasAreConnected` and the leader-election wait all use it - so the one budget with no measurement
+behind it was also the only one four times larger than its siblings. The report threshold drops to 5 s with it, to
+keep the same resolution for the next cut. `DynamicMembershipTest` no longer leaves its own teardown holding a
+peer it evicted to a replica's contract: the base class now waits for, and compares, exactly the servers
+`getServerToCheck()` names, which turned a 30 s-per-evicted-server timeout and a `DatabaseAreNotIdentical` charged
+to `endTest` into neither. Seven `await().until(() -> findLeaderIndex() >= 0)` wrappers that #6226 made redundant
+are gone, and three copies of "only the servers still running" collapse into one helper.
+
+[#6267](https://github.com/ArcadeData/arcadedb/issues/6267)
 ## The same iteration-knob guard, applied to the fourteen `algo.*` procedures #6216 left out of scope (#6264)
 
 [#6216](https://github.com/ArcadeData/arcadedb/issues/6216) established what an iteration-shaped knob needs -

ha-raft/src/main/java/com/arcadedb/server/ha/raft/GetClusterHandler.java

@@ -117,12 +117,29 @@ public ExecutionResponse execute(final HttpServerExchange exchange, final Server
     // idle cluster. Previously no latency figure was exposed in this JSON at all.
     final Map<String, RaftHAServer.ReplicationLatency> replicationLatencies = raftHAServer.getReplicationLatencies();
 
+    // Each peer's HTTP endpoint, plus whether that endpoint identifies that peer and no other. With no 'http'
+    // port declared in arcadedb.ha.serverList a peer's endpoint is derived as its Raft host plus THIS node's
+    // port, so on a cluster whose nodes differ by port every peer collapses onto one address. Reporting the
+    // address alone would show an operator a plausible endpoint per peer with nothing to say that it names
+    // none of them, and they would find out only when a resync or a verify refuses to dial (issue #6267).
+    // Resolved for the whole group in one pass: the question is about the group, since an address identifies a
+    // peer only if no other peer resolves to it, so asking per peer would resolve the group once per peer.
+    final Map<RaftPeerId, RaftHAServer.PeerHttpEndpoint> httpEndpoints = raftHAServer.getPeerHttpEndpoints();
+
     final JSONArray peers = new JSONArray();
     for (final RaftPeer peer : raftHAServer.getRaftGroup().getPeers()) {
       final JSONObject peerJson = new JSONObject();
       final String peerId = peer.getId().toString();
       peerJson.put("id", peerId);
       peerJson.put("address", peer.getAddress());
+      // Both fields are written only when they have something to say: a peer whose endpoint cannot be resolved
+      // carries neither, and a correctly declared cluster carries no flag.
+      final RaftHAServer.PeerHttpEndpoint httpEndpoint = httpEndpoints.get(peer.getId());
+      if (httpEndpoint != null) {
+        peerJson.put("httpAddress", httpEndpoint.address());
+        if (httpEndpoint.ambiguous())
+          peerJson.put("httpAddressAmbiguous", true);
+      }
 
       final boolean peerIsLeader = leaderId != null && peer.getId().equals(leaderId);
       peerJson.put("role", peerIsLeader ? "LEADER" : "FOLLOWER");
@@ -250,15 +267,21 @@ private JSONObject buildPresenceMatrix(final RaftHAServer raftHAServer, final Ra
           if (!dbName.startsWith(ArcadeDBServer.RESERVED_DATABASE_PREFIX))
             dbNames.add(dbName);
       } else {
-        final String httpAddr = raftHAServer.getPeerHttpAddress(peerId);
-        if (httpAddr == null) {
+        // The guarded address, not the best-effort one (issue #6267). This fan-out attributes whatever comes back
+        // to peerIdStr, so an address that resolves to the wrong node - or to this one - fills the matrix with a
+        // reassuring answer nobody asked for: on a cluster whose peers collapse onto one derived address, every
+        // peer would report the local node's databases and the matrix would show them present everywhere. Same
+        // guard the resync and verify paths use, so the three cannot drift apart.
+        final PeerDialAddress dial = PeerDialAddress.resolve(raftHAServer, peerId, "peer");
+        if (dial.refused()) {
+          LogManager.instance().log(this, Level.WARNING,
+              "Presence matrix: not querying peer '%s': %s", peerIdStr, dial.refusal());
           unreachable.add(peerIdStr);
           continue;
         }
-        final String httpsAddr = raftHAServer.getPeerHttpsAddress(peerId);
         try {
           final List<LeaderDatabaseQuery.DatabaseInfo> infos =
-              LeaderDatabaseQuery.fetch(httpAddr, httpsAddr, clusterToken, timeoutMs, server);
+              LeaderDatabaseQuery.fetch(dial.httpAddress(), dial.httpsAddress(), clusterToken, timeoutMs, server);
           for (final LeaderDatabaseQuery.DatabaseInfo info : infos)
             dbNames.add(info.name());
         } catch (final InterruptedException e) {

ha-raft/src/main/java/com/arcadedb/server/ha/raft/RaftClusterStatusExporter.java

@@ -19,9 +19,6 @@
 package com.arcadedb.server.ha.raft;
 
 import com.arcadedb.log.LogManager;
-import com.arcadedb.server.ArcadeDBServer;
-import com.arcadedb.serializer.json.JSONArray;
-import com.arcadedb.serializer.json.JSONObject;
 import org.apache.ratis.protocol.RaftPeer;
 import org.apache.ratis.protocol.RaftPeerId;
 
@@ -34,8 +31,12 @@
 import java.util.logging.Level;
 
 /**
- * Exports cluster status as JSON, prints cluster configuration tables, and manages
- * the replication lag monitor.
+ * Prints the cluster configuration table and manages the replication lag monitor.
+ * <p>
+ * It used to also build a cluster-status JSON, which nothing called: the live status endpoint is
+ * {@link GetClusterHandler}, which assembles its own. A second, unreachable builder of the same document is how
+ * two views of one cluster drift apart - and it did, since the reachable one reports the peer-address ambiguity
+ * of issue #6267 and this one never would have - so it was removed rather than kept in step by hand.
  */
 class RaftClusterStatusExporter {
 
@@ -52,110 +53,6 @@ class RaftClusterStatusExporter {
     this.clusterMonitor = clusterMonitor;
   }
 
-  // -- Status Export --
-
-  JSONObject exportClusterStatus() {
-    final var haJSON = new JSONObject();
-
-    haJSON.put("protocol", "ratis");
-    haJSON.put("clusterName", haServer.getClusterName());
-    haJSON.put("leader", haServer.getLeaderName());
-    // Raw role vs. ability to serve: a freshly elected leader rejects writes until it has committed its
-    // current-term no-op (issue #5453). Both come from one snapshot so the pair is never contradictory.
-    final RaftHAServer.LeadershipState leadership = haServer.getLeadershipState();
-    haJSON.put("isLeader", leadership.leader());
-    haJSON.put("leaderReady", leadership.leaderReady());
-    haJSON.put("localPeerId", haServer.getLocalPeerId().toString());
-    haJSON.put("configuredServers", haServer.getConfiguredServers());
-    haJSON.put("quorum", haServer.getQuorum().name());
-    haJSON.put("currentTerm", haServer.getCurrentTerm());
-    haJSON.put("commitIndex", haServer.getCommitIndex());
-    haJSON.put("lastAppliedIndex", haServer.getLastAppliedIndex());
-
-    // Peer list with replication state (follower indices available only on leader)
-    final var followerStates = haServer.getFollowerStates();
-    final var replicationLatencies = haServer.getReplicationLatencies();
-    final var peers = new JSONArray();
-    final RaftPeerId leaderId = haServer.getLeaderId();
-    for (final RaftPeer peer : haServer.getLivePeers()) {
-      final var peerJSON = new JSONObject();
-      final String peerId = peer.getId().toString();
-      peerJSON.put("id", peerId);
-      peerJSON.put("address", peer.getAddress());
-      peerJSON.put("httpAddress", haServer.getPeerHttpAddress(peer.getId()));
-      peerJSON.put("isLocal", peer.getId().equals(haServer.getLocalPeerId()));
-      peerJSON.put("role", leaderId != null && peer.getId().equals(leaderId) ? "LEADER" : "FOLLOWER");
-
-      for (final var fs : followerStates)
-        if (peerId.equals(fs.get("peerId"))) {
-          peerJSON.put("matchIndex", fs.get("matchIndex"));
-          peerJSON.put("nextIndex", fs.get("nextIndex"));
-          // Time since the leader last heard from this follower (issue #5314): the honest meaning of the
-          // value the CLUSTER CONFIGURATION table used to mislabel as "LATENCY".
-          peerJSON.put("lastContactMs", fs.get("lastRpcElapsedMs"));
-          // Real measured appendEntries/heartbeat round-trip latency (issue #5314), load-independent.
-          final RaftHAServer.ReplicationLatency rtt = replicationLatencies.get(peerId);
-          if (rtt != null) {
-            peerJSON.put("replicationRttMs", rtt.meanMs());
-            peerJSON.put("replicationRttP99Ms", rtt.p99Ms());
-          }
-          if (clusterMonitor != null) {
-            final var lags = clusterMonitor.getReplicaLags();
-            final Long lag = lags.get(peerId);
-            if (lag != null)
-              peerJSON.put("lagging", lag > clusterMonitor.getLagWarningThreshold()
-                  && clusterMonitor.getLagWarningThreshold() > 0);
-            // Studio renders this as a colored badge in the cluster view, so a STALLED follower
-            // jumps out at the operator without having to compare numbers in their head.
-            peerJSON.put("replicaStatus", clusterMonitor.getReplicaStatus(peerId).name());
-          }
-          break;
-        }
-
-      peers.put(peerJSON);
-    }
-    haJSON.put("peers", peers);
-
-    // Database list
-    final var databases = new JSONArray();
-    final var stateMachineForBaseline = haServer.getStateMachine();
-    for (final String dbName : haServer.getServer().getDatabaseNames()) {
-      // Never expose reserved internal databases (e.g. the Raft control directory '.raft').
-      if (ArcadeDBServer.isReservedDatabaseName(dbName))
-        continue;
-      final var databaseJSON = new JSONObject();
-      databaseJSON.put("name", dbName);
-      databaseJSON.put("quorum", haServer.getQuorum().name());
-
-      // Surface the bootstrap baseline applied via BOOTSTRAP_FINGERPRINT_ENTRY (#4147 phase 7).
-      // Null when no bootstrap entry has been committed for this database yet, which is the
-      // normal case for clusters that pre-date #4147 or that never engaged the bootstrap path.
-      final var baseline = stateMachineForBaseline != null
-          ? stateMachineForBaseline.getBootstrapBaseline(dbName) : null;
-      if (baseline != null) {
-        databaseJSON.put("bootstrapLastTxId", baseline.lastTxId());
-        databaseJSON.put("bootstrapFingerprint", baseline.fingerprint());
-      }
-      databases.put(databaseJSON);
-    }
-    haJSON.put("databases", databases);
-
-    // Metrics
-    final var stateMachine = haServer.getStateMachine();
-    final var metricsJSON = new JSONObject();
-    metricsJSON.put("electionCount", stateMachine.getElectionCount());
-    metricsJSON.put("lastElectionTime", stateMachine.getLastElectionTime());
-    metricsJSON.put("startTime", stateMachine.getStartTime());
-    metricsJSON.put("lagWarningThreshold", clusterMonitor.getLagWarningThreshold());
-    haJSON.put("metrics", metricsJSON);
-
-    // Required by RemoteHttpComponent for cluster configuration
-    haJSON.put("leaderAddress", haServer.getLeaderHttpAddress());
-    haJSON.put("replicaAddresses", haServer.getReplicaAddresses());
-
-    return haJSON;
-  }
-
   // -- Cluster Configuration Printing --
 
   /**