Support byte ranges in binary patterns - #11473
Open
josevalim wants to merge 2 commits into
Open
Conversation
Contributor
CT Test Results 5 files 545 suites 1h 54m 15s ⏱️ For more details on these failures, see this check. Results for commit 8555f4a. ♻️ This comment has been updated with latest results. To speed up review, make sure that you have read Contributing to Erlang/OTP and that all checks pass. See the TESTING and DEVELOPMENT HowTo guides for details about how to run test locally. Artifacts
// Erlang/OTP Github Action Bot |
Allow non-empty lists of inclusive byte-range tuples
in binary:compile_pattern/1 and the binary match, matches,
split, and replace operations. Normalize overlapping
ranges and compile them to the new br pattern type.
Use SIMD for up to 16 normalized entries, with equality
checks for singletons and one-sided comparisons for ranges
touching 0 or 255. Use the 256-byte membership table for
tails, unsupported SIMD targets, and larger range sets.
ASCII validation throughput on arm64 macOS (million calls/s):
SIMD Unicode SWAR Guard
Valid ASCII 14.40 2.74 2.74 0.41
Invalid last 13.78 3.00 2.74 0.41
Invalid first 42.30 148.81 91.83 104.38
SIMD is the one added by this pull request, Unicode stands
for the current unicode:bin_is_7bit/1. Guard is the equivalent
guard imlpementation and the SWAR one is an optimized version
of that. The native unicode:bin_is_7bit/1 can be removed in
a future commit.
As shown in the benchmarks above, the new byte ranges are
clear winners for validating data, as most times you are
expecting to fully traverse binaries without matches.
For parsing, byte ranges can be faster than guards, but may
be beaten by SWAR on certain payloads.
josevalim
force-pushed
the
jv-binary-matches-byte-ranges
branch
from
August 12, 2026 11:44
4bc3603 to
8555f4a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Allow non-empty lists of inclusive byte-range tuples
in binary:compile_pattern/1 and the binary match, matches,
split, and replace operations. Those ranges are compiled
into a new pattern which uses SIMD for up to 16 normalized
entries.
ASCII validation throughput on arm64 macOS (million calls/s):
SIMD is the one added by this pull request, Unicode is
unicode:bin_is_7bit/1. Guard is the equivalent guardimplementation and a SWAR optimized guard. The native
unicode:bin_is_7bit/1 can be removed in a future commit.
As shown in the benchmarks above, the new byte ranges are
clear winners for validating data, as most times you are
expecting to fully traverse binaries without matches.
For parsing, byte ranges can be faster than guards, but may
be beaten by SWAR on certain payloads.
Implementation notes
The first commit is actually #11468. Without that commit,
the SWAR usage in json is much faster than this implementation
due to the cost of compiling the pattern. With that commit,
they are in the same order of magnitude, and one edges the
other depending on how frequent the matches are (with SIMD
being up to 3x faster when there are no matches). This pull
request will be rebased if and when that one is merged.
Also note this functionality could be added by adding new
functions, such as find_byte and find_bytes, but I thought
adding new compile patterns is the best approach as we can
reuse all infrastructure for match/matches/replace/split.