Skip to content
Open
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions serverconn/e2e_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -205,9 +205,14 @@ func (s *testServer) handleRequest(ctx context.Context,
ProtocolVersion: 1,
ArkProtocolVersion: 1,
Sender: s.serverMailboxID,
Recipient: env.Rpc.ReplyTo,
Headers: headers,
Body: body,

// Mirror the production responder (waved's handleInboundRPC),
// which answers the envelope sender rather than the request's
// ReplyTo. The two are equal here, so this changes nothing
// today -- it keeps the double honest if they ever diverge.
Recipient: env.Sender,
Headers: headers,
Body: body,
Rpc: &mailboxpb.RpcMeta{
Kind: mailboxpb.RpcMeta_KIND_RESPONSE,
CorrelationId: env.Rpc.CorrelationId,
Expand Down
23 changes: 21 additions & 2 deletions waved/server.go
Original file line number Diff line number Diff line change
Expand Up @@ -3605,8 +3605,27 @@ func (s *Server) handleInboundRPC(ctx context.Context,
}

responseEnv := &mailboxpb.Envelope{
Sender: s.localMailboxID,
Recipient: env.Rpc.ReplyTo,
Sender: s.localMailboxID,

// A response belongs to whoever sent the request, so address
// it to the sender rather than to the request's ReplyTo.
//
// The two agree for every producer in this repo: each sets
// Sender and Rpc.ReplyTo to the same LocalMailboxID (see
// serverconn/unary_facade.go, serverconn/actor.go,
// serverconn/heartbeat.go and the operator's clientconn
// equivalents). So this changes nothing for a well-formed
// peer, and it stops a request choosing a destination that
// disagrees with where it came from.
//
// It also fixes an absent ReplyTo. That used to produce an
// empty recipient, which the mailbox store rejects outright,
// so the caller lost its answer entirely.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 major — This comment claims more than the change delivers. Removing the empty-ReplyTo cause is right, but the silence is what made that bug expensive, and the silence is still here.

SendResponse.Status is this codebase's canonical application-level failure channel — mailboxconn.StatusError, StatusMailboxVersionUnsupported, StatusArkVersionMismatch, StatusUpgradeRequired all travel that way, never as a transport error. Three of the four Edge.Send call sites in the repo fold it correctly through edgeResponseError (serverconn/unary_facade.go:143, serverconn/actor.go:1109, serverconn/heartbeat.go:117). This one, at line 3644, is the only one that discards it:

_, err = edge.Send(ctx, &mailboxpb.SendRequest{
        Envelope: responseEnv,
})
if err != nil {
        return fmt.Errorf("send RPC response: %w", err)
}

So any rejection that arrives as Status.Ok=false — a version mismatch, an unknown recipient, whatever the operator adds next — returns nil here. handleInboundRPC reports success, the ingress loop acks the request as dispatched, and the operator's waiter blocks until it times out. Nothing is logged on either side. That is the same failure mode the empty-ReplyTo bug had; this PR removed one cause of it and left the detector missing, so the next cause will be just as invisible.

This also makes the claim on these lines unfalsifiable from inside waved: whether the store really "rejects outright" is exactly what this send path cannot see. (The new test's recordingReplyEdge accepts everything, so it does not establish the rejection either.)

Suggest folding this send into the same helper as its three siblings:

resp, err := edge.Send(ctx, &mailboxpb.SendRequest{
        Envelope: responseEnv,
})
if sErr := edgeResponseError("send rpc response", resp, err); sErr != nil {
        return sErr
}

(edgeResponseError is unexported in serverconn, so this needs a small exported wrapper — worth it to stop waved being the one send path that cannot see a rejection.)

To be clear: this is pre-existing and not introduced here, and I am not blocking on it. But the comment at these lines asserts the lost-answer problem is fixed, and with the status still dropped it is only narrowed.

//
// Rpc.ReplyTo is therefore advisory from the responder's
// point of view: producers still set it, and nothing here
// reads it.
Recipient: env.Sender,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 minorenv.Sender is now the sole input that decides where the answer goes, and it is the one field this function does not guard.

The preamble checks its two siblings:

if env.Rpc == nil { ... }
if env.Body == nil { ... }

and serverconn's ingress validation (validateInboundEnvelope, serverconn/inbound_version.go:18) only compares the two version fields — it never looks at Sender. I grepped the inbound path: nothing anywhere asserts env.Sender != "".

So an inbound KIND_REQUEST with an empty Sender produces exactly the empty-Recipient response this PR set out to eliminate. The hazard moved from Rpc.ReplyTo to Sender rather than going away — and combined with the dropped SendResponse.Status above, the loss is silent. The new test covers absent ReplyTo but never absent Sender, so the one case that is still broken is the one case not exercised.

A third guard alongside the existing two closes it, and turns a silent drop into a dispatch error the ingress loop logs:

if env.Sender == "" {
        return fmt.Errorf("missing envelope sender")
}

Worth a fourth table entry in TestHandleInboundRPCAnswersTheSender asserting the handler errors and sends nothing.

Headers: headers,
Body: body,
Rpc: &mailboxpb.RpcMeta{
Expand Down
127 changes: 127 additions & 0 deletions waved/server_reply_to_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
package waved

import (
"context"
"testing"

mailboxpb "github.com/lightninglabs/wavelength/mailbox/pb"
mailboxrpc "github.com/lightninglabs/wavelength/mailbox/rpc"
"github.com/stretchr/testify/require"
"google.golang.org/grpc"
"google.golang.org/protobuf/types/known/anypb"
)

// recordingReplyEdge captures the envelopes handleInboundRPC sends so a test
// can assert where a response was addressed.
type recordingReplyEdge struct {
sent []*mailboxpb.Envelope
}

// Send records the outbound envelope and reports success.
func (r *recordingReplyEdge) Send(_ context.Context, in *mailboxpb.SendRequest,
_ ...grpc.CallOption) (*mailboxpb.SendResponse, error) {

r.sent = append(r.sent, in.Envelope)

return &mailboxpb.SendResponse{
Status: &mailboxpb.Status{
Ok: true,
},
}, nil
}

// Pull is unused by these tests.
func (r *recordingReplyEdge) Pull(_ context.Context, _ *mailboxpb.PullRequest,
_ ...grpc.CallOption) (*mailboxpb.PullResponse, error) {

return &mailboxpb.PullResponse{
Status: &mailboxpb.Status{
Ok: true,
},
}, nil
}

// AckUpTo is unused by these tests.
func (r *recordingReplyEdge) AckUpTo(_ context.Context,
_ *mailboxpb.AckUpToRequest, _ ...grpc.CallOption) (
*mailboxpb.AckUpToResponse, error) {

return &mailboxpb.AckUpToResponse{
Status: &mailboxpb.Status{
Ok: true,
},
}, nil
}

// TestHandleInboundRPCAnswersTheSender asserts an inbound request is answered
// to the mailbox it came from, whatever its Rpc.ReplyTo says.
//
// The method dispatched here is deliberately unregistered, so ServeRPC fails
// and the handler takes its error path. That is the interesting path to pin:
// it still sends a response envelope, so it still has to address one.
func TestHandleInboundRPCAnswersTheSender(t *testing.T) {
t.Parallel()

const (
operatorMailboxID = "operator-1"
otherMailboxID = "somebody-else"
daemonMailboxID = "this-daemon"
)

tests := []struct {
name string
replyTo string
}{{
name: "matching reply-to",
replyTo: operatorMailboxID,
}, {
// Previously produced an empty Recipient, which the mailbox
// store rejects, so the caller never got its answer.
name: "absent reply-to",
replyTo: "",
}, {
name: "reply-to naming another mailbox",
replyTo: otherMailboxID,
}}

for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()

edge := &recordingReplyEdge{}
s := newCompatTestServer(t, edge)
s.mailboxMux = mailboxrpc.NewServeMux()
s.localMailboxID = daemonMailboxID

env := &mailboxpb.Envelope{
Sender: operatorMailboxID,
Body: &anypb.Any{},
Rpc: &mailboxpb.RpcMeta{
Kind: mailboxpb.
RpcMeta_KIND_REQUEST,
Service: "svc.Unregistered",
Method: "Method",
CorrelationId: "corr-1",
ReplyTo: tc.replyTo,
},
}

err := s.handleInboundRPC(t.Context(), edge, env)
require.NoError(t, err)

require.Len(t, edge.sent, 1)
require.Equal(
t, operatorMailboxID, edge.sent[0].Recipient,
)
require.NotEqual(
t, otherMailboxID, edge.sent[0].Recipient,
)

// The other half of the envelope: the response is from
// us, whoever it is addressed to.
require.Equal(
t, daemonMailboxID, edge.sent[0].Sender,
)
})
}
}
Loading