feat: SIMD-accelerated glz::validate_utf8 (NEON/AVX2/SSSE3) with scalar fallback

[stephenberry]

Summary

Makes glz::validate_utf8 substantially faster, especially for large and multibyte-heavy text, by adding a SIMD UTF-8 range validator with a scalar fallback. Builds on the earlier consolidation in this PR (validate_utf8 backed by a single utf8_stream_validator engine).

Two layers:

1. Scalar path (all sizes < 512 B, and any build without SIMD): glz::validate_utf8 runs one pass of utf8_stream_validator (consume + complete). Single decoding engine, no rule duplication.
2. SIMD path (>= 512 B): a range/lookup validator (Lemire / simdjson algorithm) implemented for NEON, AVX2, and SSSE3 in glaze/simd/utf8.hpp, dispatched from validate_utf8.

The 512 B threshold is chosen so no input size regresses — below it, the fixed per-call SIMD block/remainder overhead doesn't pay off, so scalar wins; above it, SIMD wins decisively.

Performance (Apple M1, NEON, vs the scalar path)

payload	speedup
512 B ascii	+64%
1 KiB ascii	+82%
16 KiB ascii	+100%
64 KiB ascii	+107%
4 KiB multibyte-heavy	+320%
256 KiB multibyte-heavy	+318%

The multibyte case is the headline: the scalar path validates codepoint-by-codepoint (~2.4 GB/s); the SIMD path validates 16/32 bytes per iteration regardless of content (~10 GB/s).

Correctness

• Each ISA implementation (NEON, AVX2, SSSE3) was fuzzed against an independent scalar oracle: 19M cases per ISA covering random bytes, boundary-clustered bytes, every codepoint class, surrogates, overlong/too-large encodings, and truncations. The x86 paths were executed via Rosetta on the dev machine. 0 mismatches.
• The integrated glz::validate_utf8 (with dispatch) was differentially verified on all four configs — NEON, AVX2, SSSE3, x86_64 scalar baseline — across sizes spanning the 512 B threshold. 0 mismatches.
• A new differential test in utf8_test.cpp cross-checks validate_utf8 against a scalar reference (~300k cases/run, sizes weighted around 512 B), so every CI architecture verifies its own SIMD path going forward.
• Existing utf8_test (now 18 tests / 227 asserts), websocket_close_test, and websocket_client_test pass.

Notes / portability

• Dispatch is compile-time, matching glaze's existing GLZ_USE_* SIMD convention: AVX2 if __AVX2__, else the 128-bit path if GLZ_USE_SSE2 && __SSSE3__, else NEON on AArch64, else scalar. A default-baseline x86_64 build (no -mssse3/AVX2) uses the scalar path — no runtime CPU dispatch.
• parse.hpp now includes glaze/simd/utf8.hpp (which pulls the intrinsics header on SIMD targets). Disable with GLZ_DISABLE_SIMD.
• glz::validate_utf8 remains a public, stateless API with unchanged semantics.

[annihilatorq]

Hi Stephen, I'm slowly getting started with SIMD and wanted to ask you a question that's been on my mind for a while regarding Glaze and other projects that use SIMD without runtime dispatch. If it's not too much trouble, could you please explain why Glaze uses compile-time dispatch to determine the executable instruction set?

As far as I understand, for server applications this is definitely more useful than runtime dispatch, since you can build the project with -march=native, in which case there will be absolutely no overhead on the if statements or indirect calls (assuming that the cpuid was parsed only once and then cached somewhere), and consequently, there would be no potential branch misses and/or no indirect branch overhead. Still, wouldn't it be useful to have an option to enable runtime dispatch for the SIMD part in Glaze (mainly for projects that run on different CPUs, e.g. desktop applications)?

I understand that this would likely have a slight impact on the instruction cache, degrade (or probably even remove) possibility for inlining, etc., which would result in a slight performance drop. But what about client applications, for example, where it’s unknown which instruction sets are supported? Or am I underestimating the cost of runtime dispatch, and is it actually much more expensive than I think?

[stephenberry]

@annihilatorq, I don't think the cost of runtime dispatch would be too large for bulk SIMD operations like UTF8 validation. In my experience, the benefits of -march=native go beyond SIMD, and include other instructions like BMI2, that can have additional performance improvements. Glaze allows you to compile without SIMD, and instead uses SWAR (SIMD Within A Register) so you still get high performance if you want to be more flexible. But, if you're targeting peak performance with SIMD then you probably want the other native optimizations as well. I just don't think there is a very big gap between SWAR and native SIMD that justifies the additional complexities of a runtime dispatched SIMD. But, if there's a real use case for it and benchmarks to show for it, I'm always eager to learn.