erts: support endian-aware packet framing by humdrum00001010 · Pull Request #11441 · erlang/otp

humdrum00001010 · 2026-08-09T17:12:55Z

open_port/2, gen_tcp, and erlang:decode_packet/3 only support big-endian length prefixes, forcing little-endian protocols to implement framing themselves.

Add three-byte packet headers and explicit big- or little-endian framing for two-, three-, and four-byte prefixes. Port and TCP options use {packet, {N, Endian}}, while decode_packet/3 accepts {N, Endian}. Existing scalar packet forms remain big-endian. The implementation covers spawned and file-descriptor ports, both gen_tcp backends, documentation, and regression tests.

Closes #11411

CLAassistant · 2026-08-09T17:13:01Z

All committers have signed the CLA.

github-actions · 2026-08-09T17:13:45Z

CT Test Results

No tests were run for this PR. This is either because the build failed, or the PR is based on a branch without GH actions tests configured.

Results for commit 7dea5d4

To speed up review, make sure that you have read Contributing to Erlang/OTP and that all checks pass.

See the TESTING and DEVELOPMENT HowTo guides for details about how to run test locally.

Artifacts

No CT logs found
No HTML docs found
No Windows Installer found

// Erlang/OTP Github Action Bot

jhogberg · 2026-08-13T06:41:21Z

+        if (tp[2] == am_little) {
+            prefix->byte_order = ERTS_PACKET_BYTE_ORDER_LITTLE;
+        } else if (tp[2] != am_big) {
+            return 0;
+        }


Thanks for the updates to cover all the different variants, I could've sworn the previous implementation had native endian as well, but I can't find it here anymore. Can you add that as well everywhere? It's fairly easy to implement along the lines of:

Suggested change

if (tp[2] == am_little) {

prefix->byte_order = ERTS_PACKET_BYTE_ORDER_LITTLE;

} else if (tp[2] != am_big) {

return 0;

}

if (tp[2] == am_little) {

prefix->byte_order = ERTS_PACKET_BYTE_ORDER_LITTLE;

} else if (tp[2] == am_native) {

#if !defined(WORDS_BIGENDIAN)

prefix->byte_order = ERTS_PACKET_BYTE_ORDER_LITTLE;

#endif

} else if (tp[2] != am_big) {

return 0;

}

Of course! And I'm simplifying my code changes! when ready, I will change the PR status

jhogberg · 2026-08-13T11:18:06Z

+        switch (packet[2]) {
+        case am_big:
+            switch (packet_bytes) {
+            case 2: type = TCP_PB_2_BIG; break;
+            case 3: type = TCP_PB_3_BIG; break;
+            case 4: type = TCP_PB_4_BIG; break;
+            }
+            break;
+        case am_little:
+            switch (packet_bytes) {
+            case 2: type = TCP_PB_2_LITTLE; break;
+            case 3: type = TCP_PB_3_LITTLE; break;
+            case 4: type = TCP_PB_4_LITTLE; break;
+            }
+            break;
+        case am_native:
+            switch (packet_bytes) {
+            case 2: type = TCP_PB_2_NATIVE; break;
+            case 3: type = TCP_PB_3_NATIVE; break;
+            case 4: type = TCP_PB_4_NATIVE; break;
+            }
+            break;


The implementation can be simplified a bit by removing TCP_PB_2_NATIVE et al and changing the switch as follows:

Suggested change

switch (packet[2]) {

case am_big:

switch (packet_bytes) {

case 2: type = TCP_PB_2_BIG; break;

case 3: type = TCP_PB_3_BIG; break;

case 4: type = TCP_PB_4_BIG; break;

}

break;

case am_little:

switch (packet_bytes) {

case 2: type = TCP_PB_2_LITTLE; break;

case 3: type = TCP_PB_3_LITTLE; break;

case 4: type = TCP_PB_4_LITTLE; break;

}

break;

case am_native:

switch (packet_bytes) {

case 2: type = TCP_PB_2_NATIVE; break;

case 3: type = TCP_PB_3_NATIVE; break;

case 4: type = TCP_PB_4_NATIVE; break;

}

break;

switch (packet[2]) {

#if defined(WORDS_BIGENDIAN)

case am_native:

#endif

case am_big:

switch (packet_bytes) {

case 2: type = TCP_PB_2_BIG; break;

case 3: type = TCP_PB_3_BIG; break;

case 4: type = TCP_PB_4_BIG; break;

}

break;

#if !defined(WORDS_BIGENDIAN)

case am_native:

#endif

case am_little:

switch (packet_bytes) {

case 2: type = TCP_PB_2_LITTLE; break;

case 3: type = TCP_PB_3_LITTLE; break;

case 4: type = TCP_PB_4_LITTLE; break;

}

break;

jhogberg · 2026-08-13T11:23:30Z


-#endif
+ERTS_GLB_INLINE void
+erts_sys_driver_reverse_packet_header(byte *header, int packet_bytes);


Changing endian with this function before/after the get/put operations makes the code rather hard to follow, I think it's better to be explicit with put_xx_little / get_xx_little / put_xx / get_xx instead.

I think I'm ready now. Thank you for your review.

Parse packet byte order in open_port and use explicit endian-specific get and put operations in both sys-driver backends. Verify both wire directions and cached Windows fd reuse with behavioral port tests.

Accept endian-qualified packet widths in decode_packet and both TCP backends. Resolve native byte order at the option boundary and retain only concrete big- and little-endian parser types.

Apply the packet contract to SSL connection options, runtime setopts, and framing. A synchronized raw TLS peer verifies exact wire bytes without allowing matching encoder and decoder mistakes to cancel out.

Document three-byte headers and explicit big, little, and native tuple forms for open_port and decode_packet. Keep scalar packet widths as the backward-compatible big-endian shorthand.

Declare the term-spec bounds only when SCTP support is built, since their consumers are compiled out otherwise.

jhogberg · 2026-08-14T06:52:49Z

-	ASSERT(desc->inet.htype == TCP_PB_HTTP_BIN ||
-	       desc->inet.htype == TCP_PB_HTTPH_BIN);
+    if (desc->inet.htype == TCP_PB_HTTP_BIN ||
+        desc->inet.htype == TCP_PB_HTTPH_BIN) {


This is a behavioral change, the assertion was obviously wrong (without ever firing) but we cannot blindly correct the code to fit it. The PS team needs to take a look. @IngelaAndin

Yes, I tried to fix it because it didn't fire. Let me revert

jhogberg · 2026-08-14T06:55:50Z

+        int htype_changed;
+
+        htype_changed =
+            (desc->stype == SOCK_STREAM) && (desc->htype != old_htype);
+        if (htype_changed) {
+            tcp_descriptor *tdesc = (tcp_descriptor *) desc;
+            /* Reparse buffered input using the new header type. */
+            tdesc->i_remain = 0;
+        }


What's the purpose of this change (and the related ones further down)?

It was for lifecycle bug fix: inet:setopts(Socket, [{packet, NewMode}]) wasn't able to change the recv's mode. But it's out-of-scope of this PR. Sorry. It came out while fixing tests. I should've been more careful.

Nobody's run into that bug so far, so I'd say there's no problem keeping it in this PR, just break it out into a separate commit and write a message for it (and maybe an improved testcase just for targeting this). We can always backport it if there's a need.

jhogberg · 2026-08-14T06:58:11Z

+        if (dd->packet_header_endianness ==
+            ERTS_SYS_DRIVER_PACKET_HEADER_ENDIAN_LITTLE)
+            put_little_int16(len, lbp);
+        else
+            put_int16(len, lbp);


Nitpick (applies elsewhere too):

Suggested change

if (dd->packet_header_endianness ==

ERTS_SYS_DRIVER_PACKET_HEADER_ENDIAN_LITTLE)

put_little_int16(len, lbp);

else

put_int16(len, lbp);

if (dd->packet_header_endianness ==

ERTS_SYS_DRIVER_PACKET_HEADER_ENDIAN_LITTLE) {

put_little_int16(len, lbp);

} else {

put_int16(len, lbp);

}

jhogberg · 2026-08-14T06:59:14Z

+        ASSERT(0);
+        return;


Suggested change

ASSERT(0);

return;

ERTS_UNREACHABLE;

jhogberg · 2026-08-14T07:00:17Z

+        ASSERT(0);
+        driver_failure_posix(dp->port_num, EINVAL);
+        driver_free_binary(bin);
+        return;


Suggested change

ASSERT(0);

driver_failure_posix(dp->port_num, EINVAL);

driver_free_binary(bin);

return;

ERTS_UNREACHABLE;

humdrum00001010 · 2026-08-14T07:09:40Z

    int aid;
    int req;
    int i = 0;
 #ifdef HAVE_SCTP


This one came from ./otp_build check warning, unused variable, discovered while passing CI in local. should I revert?

This is fine, just break it into a separate commit.

rickard-green added the team:VM Assigned to OTP team VM label Aug 10, 2026

humdrum00001010 force-pushed the erts/open-port-packet-endian branch from 38d5264 to 122cfae Compare August 12, 2026 16:10

humdrum00001010 changed the title ~~erts: support endian-aware port framing~~ erts: support endian-aware packet framing Aug 12, 2026

humdrum00001010 marked this pull request as draft August 12, 2026 16:51

humdrum00001010 force-pushed the erts/open-port-packet-endian branch from 122cfae to 3d73e92 Compare August 12, 2026 17:50

humdrum00001010 marked this pull request as ready for review August 12, 2026 18:04

humdrum00001010 marked this pull request as draft August 12, 2026 20:09

humdrum00001010 force-pushed the erts/open-port-packet-endian branch from 3d73e92 to b1c8539 Compare August 12, 2026 21:04

jhogberg reviewed Aug 13, 2026

View reviewed changes

humdrum00001010 force-pushed the erts/open-port-packet-endian branch from b1c8539 to 6a9c7c4 Compare August 13, 2026 09:23

jhogberg reviewed Aug 13, 2026

View reviewed changes

humdrum00001010 force-pushed the erts/open-port-packet-endian branch from 6a9c7c4 to e1d9c9c Compare August 13, 2026 13:24

humdrum00001010 marked this pull request as ready for review August 13, 2026 13:31

erts: Add endian-aware packet framing to ports

37df654

Parse packet byte order in open_port and use explicit endian-specific get and put operations in both sys-driver backends. Verify both wire directions and cached Windows fd reuse with behavioral port tests.

humdrum00001010 force-pushed the erts/open-port-packet-endian branch from e1d9c9c to 65bc1b9 Compare August 13, 2026 13:40

humdrum00001010 added 4 commits August 13, 2026 23:42

erts, kernel: Add endian-aware packet options

e7f75ba

Accept endian-qualified packet widths in decode_packet and both TCP backends. Resolve native byte order at the option boundary and retain only concrete big- and little-endian parser types.

ssl: Add endian-aware packet framing

f37f317

Apply the packet contract to SSL connection options, runtime setopts, and framing. A synchronized raw TLS peer verifies exact wire bytes without allowing matching encoder and decoder mistakes to cancel out.

erts: Document packet byte order

dd230b6

Document three-byte headers and explicit big, little, and native tuple forms for open_port and decode_packet. Keep scalar packet widths as the backward-compatible big-endian shorthand.

erts: Guard SCTP-only variables

7dea5d4

Declare the term-spec bounds only when SCTP support is built, since their consumers are compiled out otherwise.

humdrum00001010 force-pushed the erts/open-port-packet-endian branch from 65bc1b9 to 7dea5d4 Compare August 13, 2026 15:11

jhogberg reviewed Aug 14, 2026

View reviewed changes

humdrum00001010 commented Aug 14, 2026

View reviewed changes

Uh oh!

Conversation

humdrum00001010 commented Aug 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CLAassistant commented Aug 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CT Test Results

Artifacts

Uh oh!

Choose a reason for hiding this comment

Uh oh!

humdrum00001010 Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

humdrum00001010 Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

humdrum00001010 Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

humdrum00001010 Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

humdrum00001010 commented Aug 9, 2026 •

edited

Loading

CLAassistant commented Aug 9, 2026 •

edited

Loading

github-actions Bot commented Aug 9, 2026 •

edited

Loading

humdrum00001010 Aug 13, 2026 •

edited

Loading

humdrum00001010 Aug 13, 2026 •

edited

Loading

humdrum00001010 Aug 14, 2026 •

edited

Loading

humdrum00001010 Aug 14, 2026 •

edited

Loading