Proposal: align the TCP stack's connection close and segment responses with RFC 9293 #155

🤖 AI (Claude Code) was used for the investigation and for writing this text

Overview

We run long-lived agents inside Gondolin VMs and reach host-side services through tcp.hosts. Problems we hit there led us to look into the userspace TCP stack.

We checked host/src/qemu/network-stack.ts and host/src/qemu/net.ts (as of 0705083) against RFC 9293, and based on that we propose the fixes below. We would appreciate your view on how to proceed and on the items in group B.

Group A
Fixes for clear bugs. We will send one small PR per bug.

Group B
Deviations where we cannot tell whether the current behavior is intended. We would like to discuss first whether to fix them and, if so, in what form. For the ones you decide on, we will prepare PRs in the same way as group A.

Group C
RFC requirements that we judged not applicable to Gondolin's position, or practically unnecessary. We do not propose any change.

Method and environment

We compared the code against the per-state segment processing in RFC 9293 §3.10.7 (and the state diagram in §3.3.2), and against every row of the requirements table in Appendix B.

• Host: Linux (WSL2)
• Guest: Ubuntu 24.04 image (kernel 6.18.54-0-virt)
• Gondolin: 0.12.0 (the relevant code is the same in 0.12.0 and main)

Gondolin's position we assumed

• It only accepts guest SYNs and never opens connections itself (ingress goes through virtio tcp_open)
• It handles IPv4 only
• Above its TCP there is no application other than the relay in net.ts

Group A - bugs we propose to fix

A1. Zero-length unacceptable segments are not ACKed, so keepalive probes get no reply

What happens
A Linux keepalive probe is a zero-length segment with seq = RCV.NXT - 1. §3.10.7.4 says an unacceptable segment "should" be answered with <SEQ=SND.NXT><ACK=RCV.NXT><CTL=ACK>. handleTCP only re-ACKs inside if (payload.length > 0), so the probe gets no reply. After TCP_KEEPCNT probes, the guest kernel drops the connection with ETIMEDOUT.

How we checked
We connected through tcp.hosts to a host-side server that accepts a request and never responds.

In real use as well, long calls from Claude Code's (Bun) MCP client were failing after 66–74s.

History
Not ACKing zero-length segments dates back to the first version. #37 added discarding duplicates of segments with payload and re-ACKing them, to prevent duplicated SSH bytes.

Fix
#156

A2. A connection the guest closes first is never removed (no LAST-ACK)

What happens

1. The guest sends its FIN first → state = "FIN_SENT" (CLOSE-WAIT)
2. The upstream closes and Gondolin sends its FIN → state = "FIN_WAIT"
3. The guest ACKs Gondolin's FIN, but nothing removes the session

A session is only removed when a guest FIN arrives in FIN_WAIT or CLOSED_BY_REMOTE, on RST, on error, on a policy rejection (rejectTcpFlow), or when payload is dropped in the TX queue (teardownDroppedTcpSession), and the guest's FIN has already been received.

TCP sessions have no idle timeout, so natTable grows without bound. When the guest later reuses the same 4-tuple, the new SYN hits the leftover session and is ignored (A3).

How we checked
We connected with curl --local-port 40001 to HTTP through tcp.hosts, with the client closing first.

History
These state transitions date back to the first version (754a78a). We found nothing indicating that not handling the LAST-ACK case was intended.

Fix
#157

A3. A SYN for an existing session gets no reply

What happens
Only SYN && !session is handled. A SYN that hits an existing session gets neither a challenge ACK nor an RST. §3.10.7.4, following RFC 5961, says that a SYN received in a synchronized state is answered with a challenge ACK (<SEQ=SND.NXT><ACK=RCV.NXT><CTL=ACK>) regardless of its sequence number. Even with A2 fixed, a session left behind for any other reason (A4, or something we missed) causes the same ~130s connect() hang, so we fix this separately from A2. With a challenge ACK, a guest in SYN-SENT answers that unacceptable ACK with an RST (half-open recovery, §3.5), and we expect Gondolin to then remove the old session so the next SYN retransmit opens a new connection (we will confirm this in the PR).

How we checked
In the same measurement as A2, we confirmed via /proc/net/tcp that the guest socket stays in SYN_SENT with growing retransmits. Nothing comes back from Gondolin.

History
The SYN branch dates back to the first version (754a78a). We found nothing indicating that not handling a SYN on an existing session was intended.

Fix
#158

A4. A guest FIN before any payload leaves the session behind

What happens
Gondolin returns the SYN-ACK before contacting the upstream (handleTcpConnect in net.ts), and on the raw path it creates the upstream socket when the first payload arrives (ensureTcpSocket). When the guest sends a FIN before any payload (nc -z, or connecting and closing right away), handleTcpClose has no socket and only deletes from tcpSessions. After that neither handleTcpEnd nor handleTcpClosed is called, so no FIN goes back to the guest, and the natTable entry stays in FIN_SENT. The guest stays in FIN-WAIT-2, and the leftover entry leads to the A3 hang. The HTTP/TLS paths also have no upstream socket at this point (see B3).

How we checked
We connected from the guest to a tcp.hosts target and closed without sending anything. No connection reached the host-side server, and the guest socket stayed in FIN_WAIT2 with no FIN from Gondolin even after 150s. That the leftover entry leads to the A3 hang comes from reading the code.

History
We found nothing in the commit history or comments indicating intent.

Fix
#159

A5. Data that arrives before the upstream connects is discarded on a guest FIN (raw/mapped TCP)

What happens
When data and then a FIN arrive before the upstream connect() finishes, handleTcpClose sets session.pendingWrites = [] and then calls socket.end(). By the time the connection completes there is nothing left to write. The guest has already been sent an ACK for that data. This happens with clients that send a request and half-close right away (printf … | nc -N, etc.).

How we checked
When the guest sent hello (5 bytes) and closed immediately, the host-side server received 0 bytes. When it waited 0.5s after sending before closing, all 5 bytes arrived.

History
We found nothing in the commit history or comments indicating intent.

Fix
#160

Group B - items we would like to discuss before fixing

B1. The RST for a segment with no session is not in the RFC form

• Behavior: The RST sent for a segment with no session has SEQ=0 and no ACK bit (network-stack.ts 813–825)
• RFC: §3.10.7.1: <SEQ=SEG.ACK><CTL=RST> if the incoming segment has ACK, otherwise <SEQ=0><ACK=SEG.SEQ+SEG.LEN><CTL=RST,ACK>
• Possible consequence: A guest in a synchronized state discards it as out of window (RFC 5961 checks; we did not measure the guest's reaction). After Gondolin removes a session (teardownDroppedTcpSession, etc.), the guest keeps retransmitting until its own timeout
• Question: OK to change it to the RFC form? (a one-place change)

B2. CLOSED_BY_REMOTE loses whether our FIN was sent

• Behavior: CLOSED_BY_REMOTE overwrites state and does not record whether our FIN was sent. A guest FIN in this state removes the session immediately (1008–1013, 1551–1556)
• RFC: §3.10.4 CLOSE-WAIT: queued data and the FIN are still sent
• Possible consequence: If data for the guest is still queued when the upstream closes (small guest window) and the guest half-closes then, the remaining data and our FIN are discarded
• Question: Likely fixed by the same state split as A2. Fold it into A2 or keep it separate?

B3. A guest half-close is treated as an abort on the HTTP/TLS/WebSocket paths

• Behavior: On the HTTP/TLS/WebSocket paths, a guest FIN (half-close) is treated as an abort: the response is stopped and no FIN is returned (net.ts 1143–1167, http.ts 965/1072)
• RFC: §3.6.1: data in the other direction continues after a half-close
• Possible consequence: A client that half-closes after sending its request does not get the response
• Question: Propagating guest close to the flow-resume waiters in TCP fixes #71 was deliberate. Is treating a half-close (destroy: false) the same as RST (destroy: true) also intended?

B4. Mapped TCP does not relay an upstream half-close

• Behavior: The upstream net.Socket keeps the default allowHalfOpen: false (ensureTcpSocket)
• RFC: §3.6 / SHLD-3
• Possible consequence: After the upstream half-closes, data the guest sends is ACKed and then silently discarded (not measured; depends on Node's automatic end)
• Question: Should mapped TCP relay half-close in both directions? At least, should undeliverable data get an RST?

B5. No receive-window backpressure; overflow aborts the connection

• Behavior: The receive window is always 65535, and writes to the upstream get no backpressure. Beyond 4 MiB the connection is aborted (pinned by caps guest->upstream pendingWrites and aborts on overflow)
• RFC: §3.8.6
• Possible consequence: A large upload to a slow mapped-TCP upstream is cut with an RST instead of being slowed down
• Question: We read this as intended. Do you want window-based backpressure, or keep aborting at the cap?

B6. A dropped SYN-ACK or FIN is not retransmitted

• Behavior: No retransmission. Dropping payload tears down the session (intended, per the comment and test). A dropped SYN-ACK or FIN is not recovered. The guest's SYN retransmit is ignored (A3)
• RFC: MUST-19
• Possible consequence: Only when the TX queue (8 MiB) overflows. The guest hangs in connect() or never sees EOF
• Question: Would you accept a minimal fix (resend the SYN-ACK on a duplicate SYN, treat a dropped FIN like dropped payload)?

B7. No MSS option in the SYN-ACK

• Behavior: The SYN-ACK has no MSS option, so the guest assumes 536. The sender uses a fixed 1460 and ignores the guest's MSS (1036, 1386)
• RFC: MUST-14/15/16, SHLD-5
• Possible consequence: Guest→host traffic likely uses 536-byte segments (~2.7× the frames; not measured). It only becomes a problem if the guest lowers its MTU
• Question: Are you interested in adding an MSS option to the SYN-ACK and honoring the guest's MSS?

B8. ACKs are not aggregated

• Behavior: One bare ACK per data segment, no aggregation (949–960)
• RFC: MUST-58/59, SHLD-18
• Possible consequence: Performance only. Aggregating per writeToNetwork call would change when ACKs appear in tests that call handleTCP directly
• Question: Worth doing?

B9. TCP sessions have no idle timeout or count limit

• Behavior: TCP sessions have neither an idle timeout nor a count limit (UDP has both)
• RFC: §3.10.8 user timeout
• Possible consequence: Session leaks (A2, A4) accumulate for the VM's lifetime
• Question: Do you want a cap or an idle timeout as a safety net?

Group C - judged not applicable or practically unnecessary