Tool to better understand lance dataset storage and compression

[dentiny]

Hi community, I'm using lance in my day job and found there're a few confusion for MLE (and mostly, agents) common usage.

• why lance dataset is larger or smaller than parquet/iceberg/etc counterparts in terms of storage space
• which column(s) take most of the storage space, and how could we optimize it

During my investigation, there're a few common pitfalls

• Agents are using stale versions for lance SDK, with old storage version support; for example, structural encoding and blob type support is not available in storage version v2.0
• General-purpose compression (i.e., LZ4/zstd) is not on by default, which I found super useful for text and json blobs
• Semi-structured types like json should be avoided for query performance and storage cost

So I vibe coded this storage inspector to help

• It prints out top columns which take most of storage
• Heuristically, it samples and tries to do a comparison between different storage versions to see if we save any storage
  • note, generally speaking we should upgrade to the latest stable version, but we don't use monorepo, so upgrade version means changing multiple repos (i.e., training and data processing) because lance doesn't guarantee forward compatibililty, it's helpful to evaluate the ROI
• Heuristically, it samples and tries to try a few encoding and compression techniques and see the storage saving
• Decompression and decoding penalty is considered as an option; i.e. lz4 provides better (de)compression throughput but worse ratio

Example CLI output
Top storage columns

Rank	Column	Type	On Disk	Share	Files	Current Encoding
1	embedding	FixedSizeList(64 x Float32)	24.41 MiB	43%	1/1	fullzip / fixed-size-list+flat / none
2	event_json	Utf8	18.32 MiB	32%	1/1	miniblock / variable+flat / none
3	payload	Binary	12.70 MiB	22%	1/1	miniblock / variable+flat / none
4	row_id	UInt64	783.16 KiB	1%	1/1	miniblock / flat / none
5	status	Utf8	391.63 KiB	<1%	1/1	miniblock / flat+dictionary+variable / none

Format changes

Target	Status	Projected File	Gain / Cost	Capabilities
2.2	INFO	56.61 MiB	cost 25.00 KiB	constant-layout, larger-miniblocks, variable-packed-struct
2.3 (next/unstable)	INFO	56.61 MiB	cost 25.00 KiB	sparse-structural-layout

Encoding changes

Column	Current Encoding	Suggested Encoding	Format	Projected File	Savings
row_id	miniblock / flat / none	auto(miniblock) / auto(bss+flat) / zstd:6	2.1 → 2.2	55.95 MiB	646.25 KiB (1%)
status	miniblock / flat+dictionary+variable / none	auto(miniblock) / auto(bss+flat+dictionary+variable) / zstd:12+lz4	2.1 → 2.2	56.25 MiB	336.26 KiB (<1%)
event_json	miniblock / variable+flat / none	auto(miniblock) / auto(variable+flat) / zstd:6	2.1 → 2.2	41.32 MiB	15.26 MiB (26%)
payload	miniblock / variable+flat / none	auto(miniblock) / auto(variable+flat) / zstd:12	2.1 → 2.2	46.78 MiB	9.80 MiB (17%)

Example UI (should be easy to adapt)

Related tool to check lance data file internals: https://github.com/dentiny/lance-file-internal-viewer