feat(mse): Message Stream Encryption (MSE) support

[lingdiansr]

Closes #617

Summary

Adds full MSE (Message Stream Encryption) support to rqbit: DH-768 key exchange + RC4 stream cipher + SHA-1 key derivation, the incoming (responder) and outgoing (initiator) handshakes, plaintext sniffing/fallback, and a three-state MseMode config — all wired into peer connections (peer_connection.rs for outgoing, session.rs for incoming). No dead code: every piece is referenced by a live call site or protocol test.

Commit history is organized top-down (wiring → incoming → outgoing) so each step is reviewable on its own.

Size: 13 files, +1706 / −24 (net ~+1682).

What's in it

• MseMode config (mse/mod.rs, default Disabled) — three states: Disabled (plaintext only), Enabled (prefer MSE, plaintext redial on failure), Forced (require MSE, drop peers that don't complete the handshake). Threaded through SessionOptions, PeerConnectionOptions, and ManagedTorrentOptions.
• Crypto primitives
  • mse/rc4.rs (140 LoC) — RC4 stream cipher (new / apply_keystream / discard, incl. the MSE drop-1024 step). RC4 is cryptographically broken but MSE requires it for wire-level interoperability.
  • mse/dh768.rs (249 LoC) — DH-768 key exchange: fixed Azureus-spec 768-bit group (incl. the 769-bit modular-reduction borrow case), degenerate-key rejection.
  • mse/stream.rs (187 LoC) — Rc4Reader / Rc4Writer wrappers that encrypt/decrypt post-handshake traffic.
• Handshakes (mse/mod.rs)
  • Outgoing (initiator): send Ya + PadA, probe the first 20 bytes for a plaintext BT handshake (2s sniff window), then complete the exchange; OutgoingOutcome (Encrypted / PlaintextPeer).
  • Incoming (responder): sniff the BT prefix, otherwise run the acceptor (YA → YB+PadB → req1/SKEY → VC → provide → PadC → IA → PadD); resolves the obfuscated info hash via a SKEY-hash snapshot of the torrent db; IncomingOutcome (Encrypted / Plaintext).
• Wiring
  • Outgoing: connect_with_mse_fallback in peer_connection.rs — MSE attempt, plaintext redial on a fresh connection on failure/plaintext detection; mse_applied controls whether the handshake was already consumed as IA.
  • Incoming: check_incoming_connection in session.rs — RC4-wrapped streams continue encrypted; plaintext peers accepted unless Forced.
• Tests (18 MSE tests): rc4 vectors, dh768 external vectors, stream wrappers, duplex handshake, plaintext sniff fallback, zero-length-IA, fragmented-prefix replay, fresh-redial, Disabled single-connection, Forced failure, plus a #[cfg(test)] connection injector in StreamConnector.

Default value

MseMode defaults to Disabled, so this merge is a zero-behavior change: every existing connection takes the plaintext path exactly as before. Users opt in via SessionOptions::mse_mode. This follows the precedent of the uTP support PR (#322, "disabled by default"). A unit test locks the default to Disabled.

Provenance

This MSE implementation was developed on my rqbit-9 fork (lingdiansr/rqbit, branch mse-dev, based on v9.0.0) to fix a real-world problem: librqbit lacked MSE, so peers that require encryption (Xunlei, BitComet, etc.) dropped plaintext handshakes — swarms ran 85–97% dead peers and downloads stalled below 1 MB/s (FluxDown issue #341). The implementation follows the Azureus/libtorrent MSE spec (DH-768 + RC4 + SHA-1 key derivation); the fork is consumed by FluxDown PR 475 via a git dependency, where it measured 29 MB/s (Ubuntu) and 20–24.5 MB/s (Win11 pure-DHT magnet) vs <1 MB/s before, with encrypted peer connections now succeeding.

Verification

• cargo check -p librqbit --all-targets — clean
• cargo test -p librqbit --lib — 50 passed, 0 failed, 5 ignored (incl. 18 MSE tests + the default-Disabled lock)
• cargo clippy -p librqbit --all-targets — no dead_code / unused warnings
• Functionality confirmed reachable while default-off: with SessionOptions::mse_mode temporarily set to Enabled, all MSE handshake/fallback tests still pass

Notes on authorship

The implementation was developed with AI assistance (code generation and review), then verified by hand: I traced the 9.0 peer-I/O adaptation points (into_vectored_compat, single-read ReadBuf limitation), ran the full test suite, and validated the crypto primitives against external test vectors.

[ikatson]

Thanks for the PR

I will not be merging dead code, we need a change of strategy here - got burnt by this when someone tried doing btv2 recently and never finished- this left useless dead code around.

Either send a stack of PRs that implement the feature fully,
Or implement top down, not bottom up, with anyhow::bail stubs for not yet implemented pieces.
Alternatively, if the core needs to be refactored to be flexible enough to support encryption (eg readbuf), those can be separate PRs in the stack. If those refactors are reasonable I might merge as is even if encryption is never committed.

[lingdiansr]

Thanks for the PR

I will not be merging dead code, we need a change of strategy here - got burnt by this when someone tried doing btv2 recently and never finished- this left useless dead code around.

Either send a stack of PRs that implement the feature fully, Or implement top down, not bottom up, with anyhow::bail stubs for not yet implemented pieces. Alternatively, if the core needs to be refactored to be flexible enough to support encryption (eg readbuf), those can be separate PRs in the stack. If those refactors are reasonable I might merge as is even if encryption is never committed.

Thanks for the detailed feedback — this is exactly the guidance I needed, and I've restructured the PR accordingly.

Top-down, implemented fully, no dead code. Instead of the bottom-up primitives-first approach, I've reorganized the work into a top-down series following the "wiring → handshake" order you described:

1. feat(mse): add MseMode config and wire MSE into peer connections — the config surface (MseMode three-state, default Disabled) threaded through SessionOptions / PeerConnectionOptions / ManagedTorrentOptions, plus the two call sites (connect_with_mse_fallback in peer_connection.rs, check_incoming_connection in session.rs) — at this commit they call bail! placeholders, exactly as you suggested.
2. feat(mse): implement incoming MSE handshake (responder side) — fills in the incoming hook with the real handshake.
3. feat(mse): implement outgoing MSE handshake with plaintext fallback — fills in the outgoing hook (including the plaintext redial fallback).

Every commit is a complete, compilable slice, and the final PR has no dead code — every function is referenced by a live call site or a protocol test. I also added a unit test locking MseMode::default() == Disabled, so the feature merges as a zero-behavior change (following the uTP "disabled by default" precedent).

To be explicit about the lessons from btv2: nothing here is a placeholder left for a later commit — all three commits together form the full MSE implementation (DH-768 + RC4 + SHA-1 key derivation, incoming/outgoing handshakes, plaintext sniffing and fallback), all wired and tested. The top-down ordering is only about commit structure so each step is reviewable independently.

[ikatson]

It's not too big, so let's just shrink into one commit, don't bother with splitting into top-down

I haven't finished looking in detail, but this is a first high-level pass.

Main concerns are:

• lots of custom code for crypto. Need to try to use crates for it, esp with hardware acceleration if any (openssl etc)
• spaghetti code where mse becomes intrusive into the rest of the logic

[lingdiansr]

It's not too big, so let's just shrink into one commit, don't bother with splitting into top-down
Main concerns are:

• lots of custom code for crypto. Need to try to use crates for it, esp with hardware acceleration if any (openssl etc)
• spaghetti code where mse becomes intrusive into the rest of the logic

Thanks again for the review — I've reworked the PR accordingly. It's now a single commit, and I've addressed both concerns plus the inline comments. Response below.

Crypto crates. You were right about the wheel-invention. To be upfront: I originally started from the libtorrent / transmission implementations (which inline both RC4 and the DH-768 group), and I wasn't familiar enough with the Rust crypto ecosystem to question that assumption — so I mirrored the reference engines rather than looking for a better fit. I've now replaced both:

• DH-768 → crypto-bigint. The 768-bit group is MSE-specified (not an OpenSSL standard group), so the prime is hardcoded and the modular exponentiation is done by the crate; the hand-rolled bigint mod_reduce/powm is gone, verified against external vectors.
• RC4 → rc4 crate. Since RustCrypto's rc4 has no Clone/seek, I rewrote the writer following how libtorrent/transmission handle it — the PadB/VC scan computes the encrypted-VC pattern once and searches the raw bytes, and Rc4Writer buffers ciphertext and only reports full writes or Pending, keeping the keystream position consistent. SHA-1 already goes through sha1w.

One caveat on hardware acceleration: RC4 and this fixed DH-768 group have no meaningful HWA available (OpenSSL 3 removed RC4; the group isn't standard), so those two are pure-Rust — only SHA-1 gets a hardware backend today. For a per-connection handshake I don't think that's a real loss.

Leftover code. Apologies — the SocketAddr on MseForced and the ConnectionKind::Tcp guard were leftovers from a debugging pass I failed to clean up before opening. Both are gone now: the error carries no address, and MSE applies to any connection kind (uTP/SOCKS included) with mse_mode as the only switch.

Abstraction. Implemented per your sketch: outgoing via StreamConnector::connect_with_handshake -> OutgoingHandshake, incoming via accept_with_handshake -> IncomingHandshake (struct, not tuple). session.rs/peer_connection.rs carry no MSE details — the only thing that stays in the session is building the SKEY→info_hash lookup (it needs the torrent db), passed in as a closure.

Happy to adjust further if anything still looks off.