[JEWEL-1285] Update JDialogRenderer to Support Custom Window Shape

[DanielSouzaBertoldi]

Context

During the development of the Got It component, we ran into a limitation of our current JDialogRenderer. Currently, it can only render windows with rounded corners. While this works perfectly for menus and other types of popups, it fails to provide the best UX for the Got It component since it has a custom shape.

So, this PR paves the way so that our Got It component can be rendered in all its glory!

Changes

• Created new variations of Popup that take windowShape: ((IntSize) -> java.awt.Shape)? = null,. Deprecated variations without it.
• Added windowShape: ((IntSize) -> java.awt.Shape)? = null, lambda parameter to PopupRenderer and passed it down to both JDialogRenderer and JBPopupRenderer
• Updated JDialogRenderer code to apply the custom window shape defined by the factory lambda if:
  1. compose.interop.blending is disabled
  2. windowShape is not null

Demo

Scenario	Screenshot
Before
After
Custom Popup Enabled, Window Shape Provided but Compose Blending Disabled	Screen.Recording.2026-03-05.at.16.08.49.mov

Closeup of Jagged Arrow Shape

Note

Keep in mind this only ever happens if users provide a custom shape via the new factory parameter AND don't enable compose.interop.blending!

Scenario	Screenshot
Arrow pointing up
Arrow pointing down
Arrow pointing left

Curiously, this minor artifact is not present when the arrow is to the right. No idea why, though.

Release notes

New features

• New variations of Popup is available which take a windowShape factory. With this, you can create a custom shape that'll be applied to the window which holds the popup. Bear in mind that this factory is only useful in the following scenario:
  • You have the system property jewel.customPopupRender flag enabled
  • Your project is running as a Desktop app. IJP plugins are unaffected by this change
  • Your project does not have the system property compose.interop.blending enabled
• Keep in mind that forcing to clip the window to a shape has some limitations: Swing's WindowTranslucency can't render concave corners properly since it forces pixels to be either completely off or on. This means that depending on your custom shape, you can have jagged edges. For this reason, it's highly encouraged to also enable compose.interop.blending in your system properties.

Deprecated API

• Popup components without the windowShape parameter are now deprecated.

[daaria-s]

Thank you for the updates! LGTM

[AlexVanGogen]

It's not entirely true: blendingEnabled can be true even without compose.interop.blending. Having a transparent Panel.background is enough.

(And I'm not sure that dialog panels cannot have a custom background, different from Panel.background, but it's out of scope for this PR anyway)

[AlexVanGogen]

WDYT about mentioning compose.interop.blending and transparent panel here? They also affect whether windowShape gets invoked