shuf: fix memory and CPU usage with -i LOW-HIGH and small -n COUNT

[xelan]

shuf -i L-H creates an in-memory array with a "virtual line" for every number in the range, even if -n COUNT asks for only a few of them: "shuf -i 1-2222222222 -n 1" dies trying to allocate ~17 gigabytes, and "shuf -i 1-99999999 -n 1" needs ~800 megabytes and takes seconds where GNU shuf needs a millisecond.

If COUNT is small enough (outlines^2 / 2 < numlines), pick COUNT distinct random numbers from the range instead of creating and shuffling the array: the expected cost of the duplicate checking is then lower than the cost of creating the array. Otherwise keep the old array method, so full-range permutations behave as before.

This PR was made with the help of Claude Fable, as C is not my main programming language. Added tests and refined the code to keep it compatible while fixing the issue, but please double-check.

Fixes #109

Changeset

File	Changes	Added	Removed
coreutils/shuf.c	70	+53	−17
testsuite/shuf.tests (new)	55	+55	−0
Total	125	+108	−17

Code size

Function	Old (bytes)	New (bytes)	Delta
shuf_main	557	698	+141

[robang74]

While running time issue is solved in 2023, these tests fails by allocating too much memory, then SKIP= is added.

# Must not try to allocate an array for the whole range
# (~17 gigabytes here): www.github.com/mirror/busybox/issues/109

SKIP=
testing "shuf -i with huge range prints distinct numbers" \
	"shuf -i 1-2222222222 -n 4 | sort -un | wc -l" \
	"4\n" "" ""

SKIP=
testing "shuf -i with huge range prints numbers within it" \
	"shuf -i 2000000000-2222222222 -n 3 | awk '\$1 < 2000000000 || \$1 > 2222222222'" \
	"" "" ""

SKIP=
testing "shuf -i with huge range and -n 0 prints nothing" \
	"shuf -i 1-2222222222 -n 0" \
	"" "" ""

[robang74]

rand() uniformity issue in shuf.c as noted by the TODO

This can be moved in libbb.a and serves others applets like zcip.c, awk.c, ntpd.c, telnetd.c and tls.c

Please, consider that rand() isn't strong in terms of PractRand testing and potentially should be replaced with umkaos.c internal function.

Moreover, the integration of umkaos internals (from uchaosys/kdev) in busybox support the idea to replace ./miscutils/seedrng.c also

• https://github.com/robang74/busybox

• https://github.com/robang74/uchaosys/blob/v074/kdev/umkaos.c

commit 451122a (HEAD -> uchaosys)
Author: Roberto A. Foglietta roberto.foglietta@gmail.com
Date:   Tue Jul 21 15:09:50 2026 +0200

    shuf: random non-uniformity fix (todo->done), v4
   
    TODO:
   
    the method was seriously non-uniform when numlines is very large.
    For example, with numlines of 0xf0000000,
        values of (r % numlines) in [0, 0x0fffffff] range
        are more likely: e.g. r=1 and r=0xf0000001 both map to 1,
        whereas only one value, r=0xefffffff, maps to 0xefffffff.
   
    DONE:
   
    A djb2 similar approach has been used to provide a more flat
    random distribution, and adding a min to max range.
   
       text    data     bss     dec     hex filename
        661       0       0     661     295 coreutils/shuf.o
        719       0       0     719     2cf coreutils/shuf.o v2
        758       0       0     758     2f6 coreutils/shuf.o v3
                                +97
   
    roberto@x280[5]:~/robang74/busybox$ ./rtest 0 0xf0000000 1000000 | ent
    Entropy = 7.999939 bits per byte.
   
    Optimum compression would reduce the size
    of this 3000000 byte file by 0 percent.
   
    Chi square distribution for 3000000 samples is 253.71, and randomly
    would exceed this value 51.10 percent of the times.
   
    Arithmetic mean value of data bytes is 127.5258 (127.5 = random).
    Monte Carlo value for Pi is 3.140096000 (error 0.05 percent).
    Serial correlation coefficient is -0.000707 (totally uncorrelated = 0.0).
   
    v1 --> v2:
    - random generates in size_t that folds many times into every
      number within the unsigned range which is usually enough to
      have a decent uniformity, while 64 bit constants mixing the
      rand() values into a murmur3-like space.
   
    v2 --> v3:
    - it doesn't block if rand() always return 0 because totally
      broken and increases the Montecarlo precision dramattically
   
    v3 --> v4:
    - It includes testsuite/randtest.c for further tests with ent
   
    Signed-off-by: Roberto A. Foglietta roberto.foglietta@gmail.com

[robang74]

Requires:

• shuf: random non-uniformity fix (todo->done), v5

This patch can be applied once having reasonably fixed the isse
of non-uniformity rand() issue otherwise the goto again can create
an (almost) infinite loop. Since the Montercarlo precision error is
assesed by now, it meas that the new random() function is uniform
enough to be trusted in exit from the again-loop.

Please, provide a patch with your e-mail address as authorship signing-off:

• Signed-off-by: Andreas Erhard username@domain, I will add co-authored-by (my signature, email)

The integrated patch is on the bugfixes branch:

• robang74@b64ef44

[robang74]

updated