apollo_node: restart a node that fails to start instead of failing the run

[asaf-sw]

Restarts a node that fails to start, instead of failing the whole run on what is usually a transient port collision.

This is a safety net and worth reviewing as one: #14964 already removed the two causes behind the failures we actually observed, so holding this until we see whether AddrInUse recurs is a reasonable call.

---

Detailed Summary for AI Bots

Problem

spawn_run_node turns any node exit into a panic that fails the run:

Failed to start MonitoringEndpoint: Os { code: 98, kind: AddrInUse }
Node NodeRunner { node_index: 3, node_execution_id: "hybrid_mempool" } stopped unexpectedly

A port that passed the allocator's probe can be taken before the node binds it, losing the run to something that would have worked on a second attempt.

Change

spawn_run_node restarts a node that exits within 30 seconds of being spawned, up to 3 attempts with a 2 second backoff.

• Any early exit is retried, not AddrInUse specifically. The error appears only in the child's own output, so matching on it means parsing a log stream. An early exit is a startup failure whatever the cause, and a deterministic one still fails the run after 3 attempts, with the uptime and attempt count in the message.
• A crash later in the test stays immediately fatal. The 30 second window separates "failed to start" from "died during the test", so fail-fast for real regressions is unchanged.
• Each restart logs a warning naming the uptime, window and attempt, so a retried run explains itself.

Verification

cargo build -p apollo_node --all-features and cargo clippy -p apollo_node --all-targets --all-features -- -D warnings are clean.

The retry path has no automated test: reproducing it means racing a real port bind against a child process, which is the same non-determinism this change absorbs.

[reviewable-StarkWare]

This change is