Skip to content

feat(transforms): add mask_fn to DetConSTransform for callable segmentation#1981

Open
saud5150 wants to merge 1 commit into
lightly-ai:masterfrom
saud5150:feat/detcon-transform-mask-fn
Open

feat(transforms): add mask_fn to DetConSTransform for callable segmentation#1981
saud5150 wants to merge 1 commit into
lightly-ai:masterfrom
saud5150:feat/detcon-transform-mask-fn

Conversation

@saud5150

Copy link
Copy Markdown
Contributor

Closes #1972.

Description

  • My change is breaking

Using unsupervised masks with DetConS previously required a helper function that ran the segmenter, wrapped the output into a Mask, and threaded it through the transform by hand (see #1969, #1971). This adds an optional mask_fn that maps an image to segmentation labels, so scikit-image / cv2 segmenters work via functools.partial:

DetConSTransform(mask_fn=partial(felzenszwalb, scale=100, sigma=0.5, min_size=20))

The mask is generated once per image inside the transform — so it runs in the dataloader workers and both views share the same segmentation — and is wrapped into a torchvision.tv_tensors.Mask internally, removing all user-side Mask(...) boilerplate. Typed Callable[[Image | Tensor], Image | Tensor] per the torchvision v2 conventions. Purely additive: the existing grid_size and pre-segmented-Mask paths are untouched; grid_size and mask_fn are mutually exclusive (both → ValueError). Scope is DetConSTransform, the shared DetCon augmentation transform — there is no separate DetConBTransform.

On the "override the forward method" note: DetConSTransform extends MultiViewTransformV2, which dispatches via __call__ (there is no forward in the hierarchy), so I overrode __call__. I generate the mask there — once per image, before the view split — so both views share one segmentation and the segmenter is not run twice. Happy to move it into a v2 Transform if you'd prefer, but that path runs mask_fn per-view.

Tests

  • My change is covered by existing tests.
  • My change needs new tests.
  • I have added/adapted the tests accordingly.
  • I have manually tested the change.

Added test_mask_fn and test_grid_size_and_mask_fn_are_mutually_exclusive in tests/transforms/test_detcon_transform.py (patterns from test_generate_grid_mask / test_given_masks). Manually verified end-to-end with real felzenszwalb on a PIL image: transform(pil_img) returns two (3, 96, 96) image views and matching (1, 96, 96) int64 masks, with labels preserved as integers across randomized crop/rotate/resize.

make format        # All checks passed
make lint          # All checks passed
make type-check    # Success: no issues found in 541 source files
python -m pytest tests/transforms/test_detcon_transform.py   # 5 passed
make generate-example-notebooks

Before → after in examples/pytorch/detcon.py:

# before: hand-rolled helper threads the mask through the transform
def transform(pil_img):
    tv_img = _to_image(pil_img)
    segments = felzenszwalb(np.array(pil_img), scale=100, sigma=0.5, min_size=20).astype(np.int64)
    segments = np.clip(segments, 0, num_cls - 1)
    mask = Mask(torch.from_numpy(segments).unsqueeze(0))
    return _detcons_transform(tv_img, mask)

# after: the transform builds the mask itself
transform = DetConSTransform(mask_fn=felzenszwalb_mask, input_size=96)

Documentation

  • I have added docstrings to all public functions/methods.
  • My change requires a change to the documentation ( .rst files).
  • I have updated the documentation accordingly.
  • The autodocs update the documentation accordingly.

mask_fn is documented in the DetConSTransform docstring (three mask-sourcing modes, plus a note that it must be picklable for DataLoader workers). The examples/pytorch/detcon.py change flows into the rendered example page via its literalinclude; the notebook is regenerated.

Implications / comments / further issues

None. The public API, the grid_size path, and the pre-segmented-mask path are all unchanged; mask_fn is additive.

…tation

Using unsupervised masks with DetConS previously required a helper
function that ran the segmenter, wrapped the result into a Mask, and
threaded it through the transform manually (see lightly-ai#1969, lightly-ai#1971).

Add an optional mask_fn that maps an image to segmentation labels. The
mask is generated once per image inside the transform (so it runs in the
dataloader workers and both views share the same segmentation) and is
wrapped into a torchvision Mask internally, so scikit-image / cv2
segmenters such as felzenszwalb work via functools.partial without any
Mask boilerplate. mask_fn and grid_size are mutually exclusive.

Simplify examples/pytorch/detcon.py to use mask_fn, dropping the
hand-rolled mask helper.

Closes lightly-ai#1972.

@liopeer liopeer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Comment thread lightly/transforms/detcon_transform.py
@liopeer liopeer enabled auto-merge (squash) July 15, 2026 07:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DetCon Transform: Allow passing Callable for unsupervised segmentation

2 participants