Skip to content

fix int64 truncation in oomm for values past the largest prefix - #172

Open
Nashit-h wants to merge 1 commit into
dustin:masterfrom
Nashit-h:oomm-int64-truncation
Open

Nashit-h wants to merge 1 commit into
dustin:masterfrom
Nashit-h:oomm-int64-truncation

Conversation

@Nashit-h

@Nashit-h Nashit-h commented Oct 5, 2026

Copy link
Copy Markdown

oomm stops dividing once it reaches the largest prefix, so for inputs at or above 2^63 QB the leftover quotient no longer fits an int64 and n.Int64() hands back the truncated low bits: BigBytes(2^63 QB) prints "-9223372036854775808.0 QB", BigBytes(2^64 QB) prints "0.0 QB", and BigIBytes(2^164) prints "0.0 QiB". Convert the quotient through big.Float instead, which rounds the same way as float64(int64) for values that fit, so nothing below the boundary changes (inputs past the float64 range now come out as "+Inf QB" rather than an arbitrary 19-digit number). Added TestBigBytesBeyondInt64 with SI and IEC cases, which fails on master and passes with the change; go test -race ./..., go vet and gofmt are clean.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant