I store billions of compressed payloads and am stripping the first 4 bytes during storage, as they add up. My reader code looks roughly like this, with the problematic part highlighted in FastDecodeValue()
const (
zstdMagic = "\x28\xB5\x2F\xFD"
zstdMagicLen = len(zstdMagic)
)
var valBuf, scratchBuf []byte
zctx := zstd.NewCtx()
for ... {
valBuf, scratchBuf, err = FastDecodeValue(zctx, valLen, valBuf, valRaw, scratchBuf)
...
}
func FastDecodeValue(zctx zstd.Ctx, outLen int, out, in, scratch []byte) ([]byte, []byte, error) {
if cap(out) < int(outLen) {
out = make([]byte, outLen)
} else {
out = out[:outLen]
}
if cap(scratch) < zstdMagicLen+len(in) {
scratch = make([]byte, zstdMagicLen+len(in))
}
_, err = zctx.DecompressInto(
out,
append(append(scratch[:0], zstdMagic...), in...), // <--- it would be great to remove this copying entirely
)
if err != nil {
return nil, nil, xerrors.Errorf("decompression failed: %w", err)
}
return out, scratch, nil
}
As the title says - it would be great for the wrapper to somehow expose ZSTD_f_zstd1_magicless.
I store billions of compressed payloads and am stripping the first 4 bytes during storage, as they add up. My reader code looks roughly like this, with the problematic part highlighted in
FastDecodeValue()