[Feature] Improve (dis-)connect handling between peers

[kaimast]

Fixes #3893. The PR is quite large, below is a high-level overview of the changes.

Better Connection Handling

The main purpose of this PR is to propagate errors during handshakes "up", so the error is logged at the appropriate location. It also enables filtering out benign errors such as "Already connecting".

This is achieved by using a concrete ConnectionError type instead of anyhow::Error. Previously, the code also used Option or booleans to indicate a duplicate connection, which did not make the code very readable. To replace those, the new disconnect reasons, mentioned below, are very useful.

New Disconnect Reasons

The PR adds a four new disconnect reasons "already connecting", "already connected", "connecting to myself", "no untrusted external peers allowed". This avoids those confusing peer disconnected before sending Message::ChallengeResponse` messages.

We previously added support for unknown disconnect reasons (see #4018), so this can be added without causing problems.

Significant Reduction of Connection-Related Warnings

Nodes still generated a significant number of warnings even with these changes. One culprit was that the heartbeat logic sometimes reattempted failed connections too frequently. The PR now adds a mandatory 10-second cooldown period between connection attempts.
In the future, we should consider increasing the cooldowns with every failed attempt.

Additionally, I also expanded the notion of a grace period at startup, where nodes do not log errors, such as "no connected validators". Now, nodes do not generate warnings in the first minute about not being fully connected.
We already did this for the stake-related checks, but not for other router/gateway errors.

To test these changes, the PR adds a new check to devnet_ci.sh that limits the number of warnings a node is allowed to generate. For now, I limited them to 10 per node.
There are some connections related errors at startup that we cannot remove easily; nodes all start concurrently and then fail to connect to each other at the first attempt.

[ljedrz]

Some of the goals of this PR are achieved with #3900 (which is also a bugfix). In addition, I can see that some of the changes here are conflicting with the aforementioned PR, which deduplicates some of the related code.

That being said, the idea to introduce a new, concrete error type is solid, and so is the Result change in the Handshake protocol. Since I've been tinkering with peering quite a lot lately, my recommendation would be as follows:

1. close this PR for now, as the code it is applied to can still be compacted, making the change surface smaller
2. simplify/deduplicate the network-related code further (we still have plenty of duplicate functionalities between the Gateway and the Router)
3. update and extend the connectivity/peering tests to make sure they cover everything we need (it's been a long time since their introduction)
4. reconsider these changes and apply them in small chunks, without any unrelated adjustments

[vicsn]

Agreed with all of the above except:

simplify/deduplicate the network-related code further (we still have plenty of duplicate functionalities between the Gateway and the Router)

Please no refactors unless agreed upon and resolving an existing issue.

[ljedrz]

Please no refactors unless agreed upon and resolving an existing issue.

Fair enough; I was mostly thinking about the prospect of a new node type (BootstrapClient), and some of the existing issues (like #3888), which are likely to naturally lead to such a result.

[kaimast]

Left some comments and suggestions.

Thanks! I think I addressed everything.

Note: Sometimes the devnet test fails due to an error in syncing. This is something I am fixing in another PR and, ideally, should not hold up this one.

[ljedrz]

LGTM with some final nits 👌.

[vicsn]

Pending Lukasz' approval!

[ljedrz]

LGTM

[kaimast]

I resolved a conflict with staging and it dismissed the reviews again. Please re-approve.