feat(transforms): add mask_fn to DetConSTransform for callable segmentation#1981
Open
saud5150 wants to merge 1 commit into
Open
feat(transforms): add mask_fn to DetConSTransform for callable segmentation#1981saud5150 wants to merge 1 commit into
saud5150 wants to merge 1 commit into
Conversation
…tation Using unsupervised masks with DetConS previously required a helper function that ran the segmenter, wrapped the result into a Mask, and threaded it through the transform manually (see lightly-ai#1969, lightly-ai#1971). Add an optional mask_fn that maps an image to segmentation labels. The mask is generated once per image inside the transform (so it runs in the dataloader workers and both views share the same segmentation) and is wrapped into a torchvision Mask internally, so scikit-image / cv2 segmenters such as felzenszwalb work via functools.partial without any Mask boilerplate. mask_fn and grid_size are mutually exclusive. Simplify examples/pytorch/detcon.py to use mask_fn, dropping the hand-rolled mask helper. Closes lightly-ai#1972.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1972.
Description
Using unsupervised masks with DetConS previously required a helper function that ran the segmenter, wrapped the output into a
Mask, and threaded it through the transform by hand (see #1969, #1971). This adds an optionalmask_fnthat maps an image to segmentation labels, so scikit-image / cv2 segmenters work viafunctools.partial:The mask is generated once per image inside the transform — so it runs in the dataloader workers and both views share the same segmentation — and is wrapped into a
torchvision.tv_tensors.Maskinternally, removing all user-sideMask(...)boilerplate. TypedCallable[[Image | Tensor], Image | Tensor]per the torchvision v2 conventions. Purely additive: the existinggrid_sizeand pre-segmented-Maskpaths are untouched;grid_sizeandmask_fnare mutually exclusive (both →ValueError). Scope isDetConSTransform, the shared DetCon augmentation transform — there is no separateDetConBTransform.On the "override the forward method" note:
DetConSTransformextendsMultiViewTransformV2, which dispatches via__call__(there is noforwardin the hierarchy), so I overrode__call__. I generate the mask there — once per image, before the view split — so both views share one segmentation and the segmenter is not run twice. Happy to move it into a v2Transformif you'd prefer, but that path runsmask_fnper-view.Tests
Added
test_mask_fnandtest_grid_size_and_mask_fn_are_mutually_exclusiveintests/transforms/test_detcon_transform.py(patterns fromtest_generate_grid_mask/test_given_masks). Manually verified end-to-end with realfelzenszwalbon a PIL image:transform(pil_img)returns two(3, 96, 96)image views and matching(1, 96, 96)int64 masks, with labels preserved as integers across randomized crop/rotate/resize.Before → after in
examples/pytorch/detcon.py:Documentation
.rstfiles).mask_fnis documented in theDetConSTransformdocstring (three mask-sourcing modes, plus a note that it must be picklable forDataLoaderworkers). Theexamples/pytorch/detcon.pychange flows into the rendered example page via itsliteralinclude; the notebook is regenerated.Implications / comments / further issues
None. The public API, the
grid_sizepath, and the pre-segmented-mask path are all unchanged;mask_fnis additive.