Don't alter OME-XML BigEndian values for int8/uint8 data#151
Don't alter OME-XML BigEndian values for int8/uint8 data#151melissalinkert wants to merge 3 commits intoglencoesoftware:masterfrom
Conversation
The endianness coming from the Zarr library may be different in this case, since it doesn't technically matter what the endianness is for int8/uint8 data. However, this prevents the endianness from mismatching if a uint8 Image is followed by a uint16 (or greater) Image.
Yajing826
left a comment
Yajing826
left a comment
There was a problem hiding this comment.
I was able to successfully run the conversion and load the data into OMERO Plus. Thanks!
sbesson
left a comment
sbesson
left a comment
There was a problem hiding this comment.
Tested with a Zarr v2 dataset containing a mixture of arrays with "|u1" and ">u2 dtypes.
Without this PR, the conversion fails with the Endian mismatch error (including unreplace {} values) and it succeeds with this PR included.
A few questions:
- should we wait for #152 to be included to retest this PR? should we also test it against v3 data?
- has this always been an issue or is this a recent regression?
- even though endianness is not relevant for single byte data types, should it also be the responsibility of
bioformats2rawto try and maintain consistency between the underlying Zarr array metadata andPixels.BigEndian?
In terms of testing, I assume it should be possible to create a sample OME-XML with a mixture of data types?
If an RC with this change is needed very urgently (defer to @Yajing826), then I would say test this first, and then double-check this case with v2 and v3 as part of the review of #152. If an RC with this change is not needed urgently, then it might be better to get #152 in first and then we can re-evaluate this.
I believe this has always been an issue. It will only appear in the case where there is a int8/uint8 series before a higher byte depth image, which isn't super common with the data that we typically convert.
It really depends on what happens in the underlying library. For jzarr: https://github.com/zarr-developers/jzarr/blob/main/src/main/java/com/bc/zarr/ZarrHeader.java#L66-L72 So that forces writing int8/uint8 with undefined endianness in all cases, but will return the system-native endianness when reading the same data back. I don't see how we can work around that, but definitely worth re-evaluating for zarr-java. |
|
And for zarr-java, note that the only int8/uint8 data types allowed for v2 are unspecified endianness: |
It's not urgent, I only needed it for internal testing, and so far no customer has had this issue directly. |
sbesson
left a comment
sbesson
left a comment
There was a problem hiding this comment.
Using the private source file, I generated various OME-Zarr datasets both with bioformats2raw 0.11.0 (v2) and bioformats2raw 0.12.0rc2 (v2 and v3).
After conversion, confirmed that the versions result in mixed endianness and that the switch to zarr-java uses little endian - see glencoesoftware/bioformats2raw#317
omero@ngff:/mnt/data/seb/raw2ometiff_151$ grep dtype failing_dataset_0.11.0.zarr/*/0/.zarray
failing_dataset_0.11.0.zarr/0/0/.zarray: "dtype" : "|u1",
failing_dataset_0.11.0.zarr/1/0/.zarray: "dtype" : "|u1",
failing_dataset_0.11.0.zarr/2/0/.zarray: "dtype" : ">u2",
omero@ngff:/mnt/data/seb/raw2ometiff_151$ grep dtype failing_dataset_0.12.0rc3_v2.zarr/*/0/.zarray
failing_dataset_0.12.0rc3_v2.zarr/0/0/.zarray: "dtype" : "|u1",
failing_dataset_0.12.0rc3_v2.zarr/1/0/.zarray: "dtype" : "|u1",
failing_dataset_0.12.0rc3_v2.zarr/2/0/.zarray: "dtype" : "<u2",
omero@ngff:/mnt/data/seb/raw2ometiff_151$ grep endian failing_dataset_0.12.0rc3_v3.zarr/*/0/zarr.json
failing_dataset_0.12.0rc3_v3.zarr/0/0/zarr.json: "endian" : "little"
failing_dataset_0.12.0rc3_v3.zarr/0/0/zarr.json: "endian" : "little"
failing_dataset_0.12.0rc3_v3.zarr/1/0/zarr.json: "endian" : "little"
failing_dataset_0.12.0rc3_v3.zarr/1/0/zarr.json: "endian" : "little"
failing_dataset_0.12.0rc3_v3.zarr/2/0/zarr.json: "endian" : "little"
failing_dataset_0.12.0rc3_v3.zarr/2/0/zarr.json: "endian" : "little"
Trying to convert these Zarr datasets both with raw2ometiff 0.9.0 and this PR yields the following result
| raw2ometiff 0.9.0 | raw2ometiff PR 151 | |
|---|---|---|
| failing_dataset_0.11.0.zarr | fail (Endian mismatch in series {} (expected {}) | pass |
| failing_dataset_0.12.0-rc3_v2.zarr | pass | fail (Endian mismatch in series 2 (expected false) |
| failing_dataset_0.12.0-rc3_v3.zarr | fail ('.zgroup' expected but is not readable or missing in store) | fail (Endian mismatch in series 2 (expected false) |
While this fixes the behavior for the Zarr datasets converted with bioformats2raw 0.11.0, there seems to be another issue when converting with bioformats2raw 0.12.0rc3.
As discussed earlier, this appears to be the result of the mismatch between the OME-XML and Zarr array metadata that is fixed in glencoesoftware/bioformats2raw#317. If I convert the test dataset with glencoesoftware/bioformats2raw#317 using both @Yajing826, if you could please verify that the two PRs together resolve this issue with the following steps:
|
3 participants
The endianness coming from the Zarr library may be different in this case, since it doesn't technically matter what the endianness is for int8/uint8 data. However, this prevents the endianness from mismatching if a uint8 Image is followed by a uint16 (or greater) Image.
Without this change, relevant test data would have thrown a
FormatExceptionwith the (incorrectly formatted)Endian mismatch...message. With this change, conversion should succeed. This may need a little reworking when we switch to zarr-java for v2 reading.I wanted to add a test for this case, but the fake file structure doesn't currently allow creating multiple series with different pixel types, so will need to give that some more thought.