Transformations between coordinate systems with different number of dimensions (e.g. mutichannel -> single channel)

[clbarnes]

Use case

A multichannel image is labelled with a single channel. How do you convert coordinates between the single-channel and multi-channel data?

I refer to channels a few times below but they should be read as "indices in a dropped or added dimension".

Problems

Invertibility

What does it mean to invert this transform? You can't recover the lost data, although you might be able to infer it by translating a coordinate into each channel.

Coordinate count

If you were using samples from a YX label image to index into an RGB image, you would want to triplicate your input coordinates so that you request one pixel in each colour channel. Changing the number of coordinates is not done by any existing transforms and would break some assumptions required by high-performance implementations (which might prefer to allocate or re-use fixed-size output buffers).

Possible solutions

1. Special-case channel dimensions

You rarely want to transform things through channels so you could just make transformations ignore them and transparently transform between two spaces which differ only in the channel dimension - SpatialData does this in some cases.

This sidesteps the invertibility and coordinate count issues, for this specific case, but introduces a half-equality between axes which differ only in channels.

I would prefer a more generic solution as there may be other use cases, and it would be nice for coordinate transformations not to have to reach out to the axes metadata at all. Plus you might want to do colour space transformations etc..

2. Relax constraints of byDimension

Currently, byDimension transformations need all axis dimensions to be represented in the input and output. If we allowed it to omit output dimensions, they could be removed; by using some marker (null or -1) axes, they could be added.

This is somewhat intuitive and relaxing constraints is generally nice, but overloading the functionality of an existing transform adds cognitive overhead. We would still need a new transformation to create axes unless we special-cased "if the input_axes has a null, use 0".

{
  "type": "byDimension",
  "input": "in",
  "output": "out",
  "transformations": [
    {
      "input_axes": [0, 1, 2],
      "output_axes": [0, 1],
      "transformation": { "type": "identity" }
    }
  ]
}

{
  "type": "byDimension",
  "input": "in",
  "output": "out",
  "transformations": [
    {
      "input_axes": [0, 1, null],
      "output_axes": [0, 1, 2],
      // implicitly use 0 for all coordinates in last axis?
      "transformation": { "type": "identity" }
    }
  ]
}

Does not resolve the issues of coordinate count or how to spread your coordinate request into other channels (or locations in the dropped dimension).

3. New transformations

We could create new transformations, dropDimension and newDimension, to handle this case.

They would almost always be used in a bijection as they would not be invertible. The newDimension transformation would often be used with a translation to push the coordinates into some other channel.

Does not resolve coordinate count issue.

[d-v-b]

seems like the real problem is that the "channel dimension" is treated the same as a spatial dimension in the first place. there should probably be a way to declare up front which axes of an array map to coordinates that can be transformed.

[d-v-b]

could you get rid of byDimension and instead have a selectDimension transform that selects a subset of the dimensions of the input? bonus points for identifying axes by name in transform metadata, because iirc the spec requires that the dimension_names field be complete on all the arrays.

[clbarnes]

there should probably be a way to declare up front which axes of an array map to coordinates that can be transformed.

There's already the discrete flag on axes - are there discrete axes which should be transformable?

bonus points for identifying axes by name in transform metadata

That's how we started with the byDimension, but changed it to the axis indices: #358

[clbarnes]

Another use case: having an initial array as e.g. a time series and then cutting out individual frames/ time points for labelling.