Add basic rateless codec support

[joe-calderon-ditto]

Hello,

This PR adds basic support for a rateless codec, working toward the RaptorQ / non-systematic codec support discussed in #7.

What's included

• A new rateless codec path through the encoder/segment layers, with the necessary message and session plumbing to carry it. A new underspecified scheme was added because I was unsure if rateless feel under the umbrella of small block codes, and I didn't know if using 6 would conflate general rateless codes to RaptorQ.
• API additions in normApi.h / normApi.cpp to enable and configure it.
• An examples/normRatelessTest.cpp example demonstrating the feature.

Notes

• This is an initial, "basic" implementation intended as a starting point rather than a complete solution — feedback on the approach would be appreciated.
• I'd be happy to adjust naming, API shape, or integration points to better match how you'd like rateless/non-systematic codecs to be handled in NORM.
• I added NormTelemetry so rateless encoders could implement their own rate control for both proactive and reactive parity using metrics provided by norm.

If the interface looks ok, I can follow this pr with an actual implementation of a rateless codec.

[bebopagogo]

Wow - this has been on my "to do" list for some time. I will take a look and happy to dig in to help make it happen. I understand this has the plumbing but not not a specific associated FEC codec yet.

…

On Thu, Jun 18, 2026 at 5:24 PM joe-calderon-ditto ***@***.***> wrote: Hi, and thanks for NORM! This PR adds basic support for a rateless codec, working toward the RaptorQ / non-systematic codec support discussed in #7 <#7>. What's included - A new rateless codec path through the encoder/segment layers, with the necessary message and session plumbing to carry it. - API additions in normApi.h / normApi.cpp to enable and configure it. - An examples/normRatelessTest.cpp example demonstrating the feature. Notes - This is an initial, "basic" implementation intended as a starting point rather than a complete solution — feedback on the approach is very welcome. - I'd be happy to adjust naming, API shape, or integration points to better match how you'd like rateless/non-systematic codecs to be handled in NORM. Happy to iterate on anything here. Thanks for taking a look! ------------------------------ You can view, comment on, or merge this pull request online at: #102 Commit Summary - ae61d13 <ae61d13> basic rateless codec support File Changes (13 files <https://github.com/USNavalResearchLaboratory/norm/pull/102/files>) - *A* examples/normRatelessTest.cpp <https://github.com/USNavalResearchLaboratory/norm/pull/102/files#diff-6563b1e42eb2fce1b1a3b50c9cedbd02478628f0bbfefa1cdd2d8f5862c22697> (88) - *M* include/normApi.h <https://github.com/USNavalResearchLaboratory/norm/pull/102/files#diff-8eb79be75a5573f3dc4fcbc7c10b54eed0375fd8872cba1478388f22a21da5cd> (25) - *M* include/normEncoder.h <https://github.com/USNavalResearchLaboratory/norm/pull/102/files#diff-fb7ad839a6113c0b09f5c88d745ff10a37ef49a9ebdf4365027c56b698336595> (15) - *M* include/normMessage.h <https://github.com/USNavalResearchLaboratory/norm/pull/102/files#diff-9911dcfd8dfc3e1a02823efbd9ad1ff3c816d9f551885812e471b69c3b2d355f> (119) - *M* include/normSegment.h <https://github.com/USNavalResearchLaboratory/norm/pull/102/files#diff-056aed8ebabbb7114279285abe3f96e3e0778c1cacf2703dbb74350939b0e633> (6) - *M* include/normSession.h <https://github.com/USNavalResearchLaboratory/norm/pull/102/files#diff-8222cbf93cf5599c2d07e8dd0e2c325d24d62343ecd4cce512ba235f2ab946c7> (37) - *M* makefiles/Makefile.common <https://github.com/USNavalResearchLaboratory/norm/pull/102/files#diff-3d2ceb33e0b368b59c5c4810416b8306279e50eb91f899a733c669fc44847e4b> (10) - *M* src/common/normApi.cpp <https://github.com/USNavalResearchLaboratory/norm/pull/102/files#diff-680b29152839941d69b4be26dc22458a2e59c852acfd2ac99b49e0ed98dc982e> (30) - *M* src/common/normNode.cpp <https://github.com/USNavalResearchLaboratory/norm/pull/102/files#diff-96cb940e8c3c74bfc5464c5c9a8f33fb124c7110231158f434910331d40648d2> (133) - *M* src/common/normObject.cpp <https://github.com/USNavalResearchLaboratory/norm/pull/102/files#diff-3cd98c2cd78f3a9aea980bea194ac67364806223fc43c0db3485d34fea5a7806> (57) - *M* src/common/normSegment.cpp <https://github.com/USNavalResearchLaboratory/norm/pull/102/files#diff-aaacccbc9456e5ded80895c1d433e0acfcfe68fc19419f1f66a0893a5012fdd9> (30) - *M* src/common/normSession.cpp <https://github.com/USNavalResearchLaboratory/norm/pull/102/files#diff-a6282efb4f8010a67e0701fd87d5dc6a7756469158575df45866dd48570b482a> (61) - *M* wscript <https://github.com/USNavalResearchLaboratory/norm/pull/102/files#diff-b79856260b3cc9b3e958c83a55bae1e46e31d232125b925c581022786a7bc042> (2) Patch Links: - https://github.com/USNavalResearchLaboratory/norm/pull/102.patch - https://github.com/USNavalResearchLaboratory/norm/pull/102.diff — Reply to this email directly, view it on GitHub <#102?email_source=notifications&email_token=ABU46GP6WPR2U5KSBZSZBZD5ARM2HA5CNFSNUABEM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UF4ZTQOJUGUZTKMZQGWTHEZLBONXW5KTTOVRHGY3SNFRGKZFFMV3GK3TUVRTG633UMVZF6Y3MNFRWW>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABU46GOWLORZLIRFYFWTEQL5ARM2HAVCNFSNUABFKJSXA33TNF2G64TZHMZDANZVGUZDMNBRHNEXG43VMU5TINRZGYYDMNRYGEYKC5QC> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/ABU46GKUN6YZM23G2DHGGRL5ARM2HA5CNFSNUABEM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UF4ZTQOJUGUZTKMZQGWTHEZLBONXW5KTTOVRHGY3SNFRGKZFFMV3GK3TUVJTG633UMVZF62LPOM> and Android <https://github.com/notifications/mobile/android/ABU46GKMBUHFDVDUZFFYMND5ARM2HA5CNFSNUABEM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UF4ZTQOJUGUZTKMZQGWTHEZLBONXW5KTTOVRHGY3SNFRGKZFFMV3GK3TUVZTG633UMVZF6YLOMRZG62LE>. Download it today! You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- Brian Adamson ***@***.***>

[bebopagogo]

I've been looking through the changes you made and overall it looks like it's pretty much on the right track. So - great job on figuring that stuff out as the logic of my code is definitely not obvious with a lot of cruft from it's development history as research code to get the protocol operation sorted out. I do have a couple of initial questions/comments:

1. I notice you have the Encoder/Decoder factory list and instances attached to the NormSession object. It could make sense to have those be part of the NormSessionMgr object? This isn't really critical, but in the case of a NormSessionMgr (corresponds to the NORM API instance) managing multiple sessions, there would only need to be a single 'factory' for each registered FEC codec.

2. In the NormSession::SenderHandleNackMessage() method where you added code to process the NormRepairRequest::ERASURES repair request form, I think your added code ends up causing NormBlock::TxUpdate() to set "pending_mask" so that "numErasures" of parity packets are marked for transmission and assuming that for rateless codes they are always generating "fresh" (innovative) parity packets that should be OK. This makes use of the current NormBlock implementation that was designed for block codes so that not a lot of code change is needed. It seems it could be possible to update NormBlock to allow for representation of rateless code in without needing to use the bitmasks (i.e. no need to manage the parity bitmask on the sender side if they are a "don't care" for rateless codes), but probably the approach you took here makes best sense for initial implementation and further refinement of the NormBlock class definition could be a later refinement. No real question here other than to confirm that I understand what you implemented here?

3. Also not really a question, but I do think it would be useful to expand your MockRatelessEncoder to something that could support testing of the basic rateless support code you have added. For example, it could just select random source symbols from the block to transmit as virtual "parity" packets. This would of course be a horribly inefficient 'rateless' code as it would take multiple NACK rounds or a lot of "auto parity" to randomly transmit the correct source symbols to make the receiver(s) happy (i.e. to be able to decode the block/object being transmitted). But if small-ish objects were used for testing with modest packet loss, it could validate the logic of the code you have added. I may try to do this if you don't have time or interest in doing that in lieu of having an actual code to test for the moment since I don't think it would difficult (just a little hash function or something to select unique random segments in the range of [0, ndata-1] and use those a surrogate for actual rateless code parity packets. I just can't guarantee how quickly I can get around to that, but wanted to run that idea by you?

4. Finally, did you have a specific RaptorQ implementation in mind? It looks like "nanorq" might be a good candidate for the NORM code base as a native C/C++ implementation?

[joe-calderon-ditto]

Thanks for taking the time to look through this. I will address (1) and (3) in a follow up commit. You are 100% spot on (2) we can leave as a future refinement. For (4), I'm obviously biased towards nanorq, I recently added a lightweight core interface that would work well with norm, but I'm concerned about complicating the build system too much. I planned to take a closer look at integration approaches after we hashed this pr out.

[joe-calderon-ditto]

(1) and (3) are addressed. I realized decode() handling needed more care because rateless (unlike mds) codes can fail even with k symbols.

[bebopagogo]

Still looking over the stuff you've added. Need a little time to understand what the FecLayout stuff is all about and the memory barrier added to the normMessage.cpp file for that, etc. I wasn't familiar with the sync_syncrhonize() call and don't yet understand why that's needed versus a mutex or something? If you want to share any "cliff notes" on that, that would be helpful.

[joe-calderon-ditto]

The synchronization might not be needed at all, I was just trying ensure safe updates to FecLayout during codec registrations. I tried to use mutex from protolib but they didn't seem wired. I honestly would appreciate feedback on how best to handle, or rework custom codec registration entirely.

[bebopagogo]

I will take a look at it. The ProtoMutex class in ProtoThread should work. It's a very thin wrapper around pthread mutexes on Unix and Critical Sections on Windows that are both pretty lightweight. The protoThread.cpp file hasn't been included in the build since the ProtoDispatcher has all the same mechanisms built into it since it predated the creation of the ProtoThread class. The plan is to eventually have ProtoDispatcher just use some of the ProtoThread stuff but that refactoring hasn't been done yet.

At a glance the synchronization call may be a little heavier weight than needed and when I did a search on that there were some recommendations to use some newer APIs than that. I think protecting the FEC Codec Registry held by the instance could be protected by a mutex to avoid a writer changing things while a reader is accessing it.

FYI - from your rename of one of your commits I see you may some other improvements in the works. I am also working on some new congestion control enhancements to include a BBR-like delay-based mechanism that can hopefully approach the performance of the ECN-based mode without the dependency on ECN enabled routers in the path