Hello 👋
What I'm trying to do
I maintain VirtualiZarr (https://github.com/zarr-developers/VirtualiZarr), a tool that builds "virtual" Zarr datasets over archival files. Instead of copying or decoding data, we scan a file once and record, for every chunk, its byte offset and length within the file, plus the codec used to compress it. That index (a "chunk manifest") lets Zarr/Xarray read the original HDF4/HDF5/etc. bytes directly from disk or object storage, lazily, without duplicating the data.
We already do this for HDF5 (via h5py + the library's H5Dget_chunk_info / H5Dchunk_iter), and I'd like to support HDF4 the same way, ideally by wrapping pyhdf rather than reimplementing HDF4's on-disk format parsing ourselves.
Why I currently can't with pyhdf
As far as I can tell, pyhdf wraps the SD / VS / V APIs, which are all data-reading interfaces — they return decoded values, but not the physical location of that data inside the file. For my use case the decoded values are exactly what I want to avoid reading; I only need the on-disk layout.
The HDF4 C library does expose this via the "data info" family of functions, e.g.:
SDgetdatainfo — offset(s) and length(s) of an SD array's data blocks (including per-chunk info for chunked datasets, and multiple blocks for linked/non-atomic chunks)VSgetdatainfo — offset/length for VdataHgetdatainfo / ANgetdatainfo / GRgetdatainfo — the equivalent for other object types
These are the same functions The HDF Group's own h4mapwriter uses here every byte lives. They appear not to be wrapped by pyhdf today.
Ask
Would you be open to exposing the *getdatainfo functions (at minimum SDgetdatainfo, and ideally Hgetdatainfo / VSgetdatainfo) through pyhdf? Even a thin, low-level binding returning the raw (offset, length) arrays would allow building chunk manifests on top.
Hello 👋
What I'm trying to do
I maintain VirtualiZarr (https://github.com/zarr-developers/VirtualiZarr), a tool that builds "virtual" Zarr datasets over archival files. Instead of copying or decoding data, we scan a file once and record, for every chunk, its byte offset and length within the file, plus the codec used to compress it. That index (a "chunk manifest") lets Zarr/Xarray read the original HDF4/HDF5/etc. bytes directly from disk or object storage, lazily, without duplicating the data.
We already do this for HDF5 (via h5py + the library's
H5Dget_chunk_info/H5Dchunk_iter), and I'd like to support HDF4 the same way, ideally by wrapping pyhdf rather than reimplementing HDF4's on-disk format parsing ourselves.Why I currently can't with pyhdf
As far as I can tell, pyhdf wraps the SD / VS / V APIs, which are all data-reading interfaces — they return decoded values, but not the physical location of that data inside the file. For my use case the decoded values are exactly what I want to avoid reading; I only need the on-disk layout.
The HDF4 C library does expose this via the "data info" family of functions, e.g.:
SDgetdatainfo— offset(s) and length(s) of an SD array's data blocks (including per-chunk info for chunked datasets, and multiple blocks for linked/non-atomic chunks)VSgetdatainfo— offset/length for VdataHgetdatainfo/ANgetdatainfo/GRgetdatainfo— the equivalent for other object typesThese are the same functions The HDF Group's own
h4mapwriteruses here every byte lives. They appear not to be wrapped by pyhdf today.Ask
Would you be open to exposing the
*getdatainfofunctions (at minimumSDgetdatainfo, and ideallyHgetdatainfo/VSgetdatainfo) through pyhdf? Even a thin, low-level binding returning the raw (offset, length) arrays would allow building chunk manifests on top.