Report Modulo dimensions as separate axes (RFC-3) - #330
Conversation
|
@sbesson mentioned a discussion with @jni around providing test data for RFC-3. We're likely to start seriously testing this functionality in the next few weeks (once corresponding raw2ometiff work is done). If there are datasets we should include in testing, it would be helpful to collect them under https://zenodo.org/communities/bio-formats. |
|
For reference, representative samples were discussed in the context of an imagesc.zulip thread. Seconding Melissa's request and as discussed during the OME-Zarr 1.0 meeting in Heidelberg a few weeks ago, public and permissively licensed datasets are essential to support the upcoming OME-Zarr 1.0 work and allow the generation of derived OME-Zarr samples under similar use, modification and distribution terms. In the thread above, the FLIM ICS datasets would qualify as they have been placed in the public domain under CC0. |
|
Hello team! Please see clbarnes/ome-zarr-rfc3-data#1 for some real datasets. One of those (EBSD) is just "I got permission from the authors to share it", but unlicensed. I'll double check with them now whether I can make it more public and how they would like it cited if -BY. The other (FLIM, BioImage Archive S-BIAD1967) is CC-BY 4.0. |
|
I've confirmed with the creators of the EBSD dataset that they are happy with CC-BY 4.0. Here is the attribution string:
|
|
Thanks @jni! |
sbesson
left a comment
sbesson
left a comment
There was a problem hiding this comment.
The RFC-3 proposes to update the OME-Zarr specification to relax the requirement to have between 2 and 5 dimensions as well as the constraints on the axis naming and ordering.
In the context of bioformats2raw, the most amenable use case is the extension of the OME model to support 6, 7 or 8 dimensions via the modulo annotation - see https://ome-model.readthedocs.io/en/stable/developers/6d-7d-and-8d-storage.html. This extension is implemented across several reader of Bio-Formats e.g. to deal with FLIM file formats which are one of the motivations behind the RFC - see https://ngff.openmicroscopy.org/rfc/3/index.html#fluorescence-lifetime-imaging-microscopy-flim.
To test this change, I collected all the public samples of the curated QA repository that are read using this extension. Each sample has been:
- converted to OME-Zarr 0.5 using a build of bioformats2raw with this PR
- converted to OME-Zarr 1.0-DEV using a build of bioformats2raw with this PR
- converted from OME-Zarr 1.0-DEV to OME-TIFF using a build of raw2ometiff with glencoesoftware/raw2ometiff#156 included
- converted to OME-TIFF using Bio-Formats 8.5.0 and bfconvert
The following generation script has been used
#! /bin/sh
# Generates OME-Zarr 0.5 and 1.0-DEV (with RFC-3) from a series of
# public sample files implementing the Modulo extension of the OME model
# The following application are expected alongside this script and the source files
# - the command-line Bio-Formats 8.5.0 utilities
# - a build of bioformats2raw with https://github.com/glencoesoftware/bioformats2raw/pull/330
# - a build of raw2ometiff with https://github.com/glencoesoftware/raw2ometiff/pull/156
set -e
set -x
BASE_PATH=~/Desktop/RFC-3
BF_MAX_MEM=8g
# Clean the generated folders
rm -rfv $BASE_PATH/ome-zarr
rm -rfv $BASE_PATH/ome-tiff
for format in ics oir ome-tiff sdt spc; do
mkdir -p $BASE_PATH/ome-zarr/0.5/$format
mkdir -p $BASE_PATH/ome-zarr/1.0-DEV/$format
mkdir -p $BASE_PATH/ome-tiff/raw2ometiff/$format
mkdir -p $BASE_PATH/ome-tiff/bfconvert/$format
done
convert () {
echo "Converting $1"
filename="${1%.*}"
./bioformats2raw-0.13.0-SNAPSHOT/bin/bioformats2raw $BASE_PATH/sources/$1 $BASE_PATH/ome-zarr/0.5/$filename.zarr --ngff-version 0.5 -p
./bioformats2raw-0.13.0-SNAPSHOT/bin/bioformats2raw $BASE_PATH/sources/$1 $BASE_PATH/ome-zarr/1.0-DEV/$filename.zarr --ngff-version 1.0-DEV -p
./raw2ometiff-0.11.0-SNAPSHOT/bin/raw2ometiff $BASE_PATH/ome-zarr/1.0-DEV/$filename.zarr $BASE_PATH/ome-tiff/raw2ometiff/$filename.ome.tiff -p
./bftools/bfconvert $BASE_PATH/sources/$1 $BASE_PATH/ome-tiff/bfconvert/$filename.ome.tiff -compression LZW
}
# https://downloads.openmicroscopy.org/images/ICS/gh-4197/
convert ics/20240612_training_test-tiles_01_stitched.ics
# https://downloads.openmicroscopy.org/images/SPC-FIFO/biofisika/
convert spc/conv-256x256.spc
# https://downloads.openmicroscopy.org/images/Olympus-OIR/gh-4205/zenodo-12773657/
for filename in DAPI_mCherry_4T_5Lambda-420-630-w10nm-s50nm DAPI_mCherry_4Z_5Lambda-420-630-w10nm-s50nm DAPI-mCherry_3T_4Z_5Lambda-420-630-w10nm-s50nm; do
convert oir/$filename.oir
done
# https://downloads.openmicroscopy.org/images/OME-TIFF/2016-06/modulo/
for filename in FLIM-ModuloAlongC FLIM-ModuloAlongT-TSCPC LAMBDA-ModuloAlongZ-ModuloAlongT SPIM-ModuloAlongZ; do
convert ome-tiff/$filename.ome.tiff
done
# https://downloads.openmicroscopy.org/images/SDT/gh-4198/
convert sdt/FocalCheck_A1_20x_8xzoom_800nm.sdtand all sources and secondary data (OME-TIFF & OME-Zarr) has been uploaded to the public gs-public-zarr-dev AWS S3 bucket under the RFC-3 prefix - see https://gs-public-zarr-dev.s3.amazonaws.com/index.html#RFC-3. Note this is a development bucket and all data stored there is not expected to be persistent and will be cleaned up eventually.
This raises the question of the validation of this output. The OME NGFF validator can be used to point at some of these samples e.g. https://ome.github.io/ome-ngff-validator/?source=https://gs-public-zarr-dev.s3.amazonaws.com/RFC-3/ome-zarr/1.0-DEV/oir/DAPI-mCherry_3T_4Z_5Lambda-420-630-w10nm-s50nm.zarr/0/ or https://ome.github.io/ome-ngff-validator/?source=https://gs-public-zarr-dev.s3.amazonaws.com/RFC-3/ome-zarr/1.0-DEV/ome-tiff/LAMBDA-ModuloAlongZ-ModuloAlongT.ome.zarr/0. The absence of schema makes it hard to validate the metadata. At minimum, the data is chunked alongside more than 5 dimensions and the individual chunks can be loaded.
Next steps will be to:
- generate additional variants of these datasets using compact dimensions (no singleton) and sharding
- validate the OME-TIFF filesets and compare them to the ground-truth generated by
bftools
| @@ -1247,7 +1248,8 @@ public SupportedVersions getNGFFVersion() { | |||
| * @return true if Zarr v3 data should be written | |||
| */ | |||
| public boolean getV3() { | |||
There was a problem hiding this comment.
Given we will need to keep maintaining this everytime we had a new versions, I wonder if this should be moved to SupportedVersions so that we have an API to retrieve the underlying Zarr versions associated with each version of the OME-Zarr specification.
| int[] offset = new int[axes.size()]; | ||
| Arrays.fill(offset, 0); | ||
|
|
||
| boolean useModulo = getNGFFVersion() == SupportedVersions.NGFF_DEV; |
There was a problem hiding this comment.
Same as above, as we add new versions, this will need to be kept up-to-date. Almost feels like we need an API of the like of supportsRFC3(value) for each version.
There was a problem hiding this comment.
131e394 has one approach to this, but open to other ideas.
sbesson
left a comment
sbesson
left a comment
There was a problem hiding this comment.
Retesting the last commit, I came across an error while using the new development version and the --compact option to reduce the number of singleton dimensions
sbesson@Sebastien-GS-MacBook-Pro-2025 RFC-3 % ./bioformats2raw-0.13.0-SNAPSHOT/bin/bioformats2raw /Users/sbesson/Desktop/RFC-3/sources/ics/20240612_training_test-tiles_01_stitched.ics /Users/sbesson/Desktop/RFC-3/ome-zarr/1.0-DEV-compact/ics/20240612_training_test-tiles_01_stitched.zarr --ngff-version 1.0-DEV --compact -p
2026-08-04 11:15:57,482 [main] WARN o.x.m.e.h.ExperimentTypeEnumHandler - Unknown ExperimentType value 'null' will be stored as "Other"
2026-08-04 11:15:57,512 [main] WARN o.x.m.e.h.ExperimentTypeEnumHandler - Unknown ExperimentType value 'null' will be stored as "Other"
2026-08-04 11:15:57,545 [main] WARN o.x.m.e.h.ExperimentTypeEnumHandler - Unknown ExperimentType value 'null' will be stored as "Other"
2026-08-04 11:15:57,552 [main] WARN o.x.m.e.h.ExperimentTypeEnumHandler - Unknown ExperimentType value 'null' will be stored as "Other"
2026-08-04 11:15:57,560 [main] WARN o.x.m.e.h.ExperimentTypeEnumHandler - Unknown ExperimentType value 'null' will be stored as "Other"
[0/0] 62% [===================================================================================== ] 541/864 (0:00:03 / 0:00:01)
2026-08-04 11:16:01,184 [pool-1-thread-2] ERROR c.g.bioformats2raw.Converter - Failure processing chunk; resolution=0 plane=28 xx=2048 yy=2048 zz=4 width=1024 height=277 depth=1
java.lang.RuntimeException: dev.zarr.zarrjava.ZarrException: Error in decoding blosc.
at dev.zarr.zarrjava.v3.codec.core.ShardingIndexedCodec.lambda$decodeInternal$3(ShardingIndexedCodec.java:238)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
at java.base/java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:408)
at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:736)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:661)
at dev.zarr.zarrjava.v3.codec.core.ShardingIndexedCodec.decodeInternal(ShardingIndexedCodec.java:212)
at dev.zarr.zarrjava.v3.codec.core.ShardingIndexedCodec.decode(ShardingIndexedCodec.java:101)
at dev.zarr.zarrjava.core.codec.CodecPipeline.decode(CodecPipeline.java:122)
at dev.zarr.zarrjava.core.Array.readChunk(Array.java:181)
at dev.zarr.zarrjava.core.Array.lambda$write$0(Array.java:116)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
at java.base/java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:408)
at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:736)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:661)
at dev.zarr.zarrjava.core.Array.write(Array.java:99)
at dev.zarr.zarrjava.core.Array.write(Array.java:309)
at com.glencoesoftware.bioformats2raw.Converter.writeBytes(Converter.java:2015)
at com.glencoesoftware.bioformats2raw.Converter.processChunk(Converter.java:2495)
at com.glencoesoftware.bioformats2raw.Converter.lambda$saveResolutions$6(Converter.java:2727)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: dev.zarr.zarrjava.ZarrException: Error in decoding blosc.
at dev.zarr.zarrjava.core.codec.core.BloscCodec.decode(BloscCodec.java:26)
at dev.zarr.zarrjava.core.codec.CodecPipeline.decode(CodecPipeline.java:114)
at dev.zarr.zarrjava.v3.codec.core.ShardingIndexedCodec.lambda$decodeInternal$3(ShardingIndexedCodec.java:231)
... 39 common frames omitted
Caused by: java.lang.RuntimeException: Buffer does not contain valid blosc-encoded contents
at com.scalableminds.bloscjava.Blosc._decompress(Native Method)
at com.scalableminds.bloscjava.Blosc.decompress(Blosc.java:141)
at com.scalableminds.bloscjava.Blosc.decompress(Blosc.java:151)
at dev.zarr.zarrjava.core.codec.core.BloscCodec.decode(BloscCodec.java:24)
... 41 common frames omitted
2026-08-04 11:16:01,184 [pool-1-thread-1] ERROR c.g.bioformats2raw.Converter - Failure processing chunk; resolution=0 plane=20 xx=2048 yy=2048 zz=4 width=1024 height=277 depth=1
java.lang.RuntimeException: dev.zarr.zarrjava.ZarrException: Error in decoding blosc.
at dev.zarr.zarrjava.v3.codec.core.ShardingIndexedCodec.lambda$decodeInternal$3(ShardingIndexedCodec.java:238)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
at java.base/java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:408)
at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:736)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:661)
at dev.zarr.zarrjava.v3.codec.core.ShardingIndexedCodec.decodeInternal(ShardingIndexedCodec.java:212)
at dev.zarr.zarrjava.v3.codec.core.ShardingIndexedCodec.decode(ShardingIndexedCodec.java:101)
at dev.zarr.zarrjava.core.codec.CodecPipeline.decode(CodecPipeline.java:122)
at dev.zarr.zarrjava.core.Array.readChunk(Array.java:181)
at dev.zarr.zarrjava.core.Array.lambda$write$0(Array.java:116)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
at java.base/java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:408)
at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:736)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:661)
at dev.zarr.zarrjava.core.Array.write(Array.java:99)
at dev.zarr.zarrjava.core.Array.write(Array.java:309)
at com.glencoesoftware.bioformats2raw.Converter.writeBytes(Converter.java:2015)
at com.glencoesoftware.bioformats2raw.Converter.processChunk(Converter.java:2495)
at com.glencoesoftware.bioformats2raw.Converter.lambda$saveResolutions$6(Converter.java:2727)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: dev.zarr.zarrjava.ZarrException: Error in decoding blosc.
at dev.zarr.zarrjava.core.codec.core.BloscCodec.decode(BloscCodec.java:26)
at dev.zarr.zarrjava.core.codec.CodecPipeline.decode(CodecPipeline.java:114)
at dev.zarr.zarrjava.v3.codec.core.ShardingIndexedCodec.lambda$decodeInternal$3(ShardingIndexedCodec.java:231)
... 39 common frames omitted
Caused by: java.lang.RuntimeException: Buffer does not contain valid blosc-encoded contents
at com.scalableminds.bloscjava.Blosc._decompress(Native Method)
at com.scalableminds.bloscjava.Blosc.decompress(Blosc.java:141)
at com.scalableminds.bloscjava.Blosc.decompress(Blosc.java:151)
at dev.zarr.zarrjava.core.codec.core.BloscCodec.decode(BloscCodec.java:24)
| NGFF_05("0.5"); | ||
| NGFF_01("0.1", 2, null), | ||
| NGFF_04("0.4", 2, null), | ||
| NGFF_05("0.5", 3, null), |
There was a problem hiding this comment.
For completeness, proposing
| NGFF_05("0.5", 3, null), | |
| NGFF_05("0.5", 3, new Integer[] {2}), |
| NGFF_01("0.1", 2, null), | ||
| NGFF_04("0.4", 2, null), | ||
| NGFF_05("0.5", 3, null), | ||
| NGFF_DEV("1.0-DEV", 3, new Integer[] {3}); |
There was a problem hiding this comment.
| NGFF_DEV("1.0-DEV", 3, new Integer[] {3}); | |
| NGFF_DEV("1.0-DEV", 3, new Integer[] {2, 3}); |
as per above
|
I haven't so far been able to reproduce the error with |
|
I compared the data generated by @sbesson in The following sources were compared to their raw2ometiff converted counterpart OME-TIFF (via NGFF 1.0-DEV): The following were checked for consistency:
Because No issues were found in the consistency checks above. |
|
I was eventually able to reproduce an error with In this test data, I was a little tempted to instead forbid removing only one of the two axes when a modulo dimension is present; either both have length 1 and are removed, or at least one has length > 1 so they both stay. That's maybe a little easier to think about, but isn't really in the spirit of what |
I agree this is a edge case. The last commits fix the issue but then running through the I assume if we decide to allow one of the modulo dimensions to be dropped if length is 1, we will need to handle this special case in |
Yeah, that error is coming from the logic that is supposed to check that the OME-XML dimensions and the Zarr array dimensions all match. |
|
Thanks, I went through another round of generation of I will validate these samples more closely tomorrow but my initial vote would be to keep the |
|
With the current state of this PR and glencoesoftware/raw2ometiff#156, The When writing the Zarr metadata, the actual raw2ometiff uses the |
|
With 67f9da3, the compact .ics test above results in axes: As discussed separately today, that means the original |
sbesson
left a comment
sbesson
left a comment
There was a problem hiding this comment.
With the latest state of this PR and glencoesoftware/raw2ometiff#156, I regenerated a complete set of datasets from the same sources as #330 (review) and going through the following conversion workflows:
- original file format -> OME-Zarr 0.5 (default) -> OME-TIFF
- original file format -> OME-Zarr 0.5 (--compact) -> OME-TIFF
- original file format -> OME-Zarr 0.9.dev1 -> OME-TIFF
- original file format -> OME-Zarr 0.9.dev1 -> OME-TIFF
- original file format -> OME-TIFF (via bftools)
All datasets have been uploaded to s3://gs-public-zarr-dev/RFC-3 (see https://gs-public-zarr-dev.s3.amazonaws.com/index.html#RFC-3/) and an initial look at these datasets through the validator looks as expected.
I will perform a round of more in-depth validation overthe next few days. Note also progress is happening on the specification and schemas in ome/ngff-spec#190. There is an outstanding question about the dev version number that might affect this PR and the samples.
|
To simplify the dissemination and validation process, the table below contains the list of all generated OME-Zarr datasets (versions 0.5 and 0.9.dev.1, with and without --compact) that can be accessed via the OME validator
|
|
Looking more carefully at the changes in ome/ngff-spec#190, I see that includes RFC-5, which introduces a required See also https://ngff.openmicroscopy.org/rfc/5/index.html#multiscales-metadata |
|
Another question which came up during the review of the samples is how the order of the axes is chosen. For instance https://ome.github.io/ome-ngff-validator/?source=https://gs-public-zarr-dev.s3.amazonaws.com/RFC-3/ome-zarr/0.9.dev1/compact/spc/conv-256x256.zarr/0/ has |
|
See |
4 participants
See https://ngff.openmicroscopy.org/rfc/3/#proposal, fixes #328.
Minimally tested with https://downloads.openmicroscopy.org/images/OME-TIFF/2016-06/modulo/, but needs unit tests which I need to think about a bit more as that doesn't appear to be possible with just .fake files. I think I also want to get the corresponding raw2ometiff changes in place before assigning for review, since it will be much easier to test when the input data can be directly compared with an output OME-TIFF.