fix(windowing/input): harden window targeting and pointer reporting

[nisavid]

 

• 
  • src/abs_pointer.rs
  • src/cli.rs
  • src/command_runner.rs
  • src/diagnostics.rs
  • src/gnome_extension.rs
  • src/server.rs
  • src/windowing/backends/kwin.rs
• 
  • src/abs_pointer.rs
  • src/cli.rs
  • src/command_runner.rs
  • src/diagnostics.rs
  • src/gnome_extension.rs
  • src/server.rs
  • src/windowing/backends/kwin.rs
  • src/windowing/mod.rs
• 
  • CHANGELOG.md
• 
  • src/windowing/backends/kwin.rs

Summary

• Correct absolute-uinput axis bounds, preserve unsupported-button fallback, and carry requested and emitted coordinates together so clamped clicks report where they land.
• Bound all GNOME extension setup commands to the shared two-second process supervisor while retaining conservative extension-reload reporting.
• Authenticate KWin callbacks to the current bus owner and one script response, bound the full transaction, preserve cleanup ownership under collisions, and classify Plasma 6 native Wayland and Xwayland windows.
• Advertise AT-SPI capability only when the accessibility bus and toolkit state can build a usable tree.

Provenance and scope

This PR began as the standalone export of generic Computer Use changes in two bounded downstream ranges:

• 26db7074..781201d7
• 98f2338d..19239c06

The audited downstream Computer Use tree is 0f122b7cdeb26bf13faacaf8b16445ee18a7d62e, retained by downstream audit commit 4762b207. The current PR comparison is 663930fef8d1bee64b7b8f833b68cff3196059e3..39efa804b32c6f443a0c1d2230579c482ceb599f.

Review and manual validation added narrow corrections on the same affected surfaces: bounded GNOME setup commands, emitted pointer reporting, and Plasma 6 client classification. Downstream app authorization, identity, browser-host, cursor side-channel, package, and numeric-version contracts remain out of scope.

Review path

1. Start with src/command_runner.rs and src/gnome_extension.rs for the bounded setup-command path and extension reload contract.
2. Review src/abs_pointer.rs, src/server.rs, and src/cli.rs for the pointer landing contract and fallback behavior.
3. Continue with src/windowing/backends/kwin.rs: owner resolution, callback envelope, one-shot delivery, transaction timeout, collision-safe cleanup, Plasma 6 client classification, then the private-bus adversarial tests.
4. Finish with the small capability-map correction in src/diagnostics.rs.

Author-run operational evidence

These completed scenarios summarize the evidence already gathered; they are not a request to repeat the full matrix.

Status	Surface	Expected contract	Observed result
✅ Passed	Plasma 6/KWin targeting	Focus exact native Wayland and Xwayland targets across normal, minimized, fullscreen, second-monitor, and off-desktop states; classify both client types	Every activation reached the intended window; native Kate reported client_type: wayland and Xmessage reported client_type: x11.
✅ Passed	KWin failure isolation	Reject a stale ID promptly and isolate concurrent script transactions	The stale ID failed in 46.4 ms; 50 activation/list cycles across five MCP processes completed without crossed responses or script residue.
✅ Passed	Absolute pointer mapping	Map the mixed-scale seam and exclusive desktop edge correctly; report every clamp	(3440,720) landed at local x=0 on the borrowed monitor. (5744,1440) emitted as (5743,1439) through MCP and abs-test; negative and oversized inputs also clamped with no generic warning.
✅ Passed	GNOME Shell targeting	Focus exact compositor IDs across normal, minimized, second-monitor, and fullscreen states	Focus completed in 31.1–37.8 ms. B stayed within the secondary head, A matched the primary fullscreen rectangle before restoring to 640×440, and receipts were A, B, B, A.
✅ Passed	GNOME stale target	Reject a closed-window ID without disturbing focus	No window matched window_id 3718609245. returned in 14.1 ms with no focus result, while A stayed focused.
✅ Passed	Portal fallback	Fail closed without cross-backend replay; deliver one in-bounds click and reject off-screen input	The incomplete layout failed closed. After the guest-only xrandr addition, (500,400) produced one B receipt at (483,395); (2048,100) returned ok: false with the generic warning, no receipt, and no ydotool replay.
⚠️ Passed with limitation	Ydotool fallback	Deliver in-bounds input and retain the off-screen warning	(200,200) returned ok: true and produced one fullscreen receipt. (2048,100) kept the warning but acceleration distorted the landing onto the visible target's left edge, which is why absolute uinput remains preferred.
➖ Not run	Optional GNOME X11	Exercise a genuine GNOME-on-Xorg session	Fedora 44 provided neither that session nor xdotool; Xwayland was not counted. A focused regression still covers xdotool spawn failure routing to ydotool.

How it was run

• Core — Plasma 6/KWin Wayland targeting and transaction isolation (run; host-mutating): On Plasma Wayland with native and Xwayland scratch windows, 1.2×/1.0× monitors, and a temporary desktop, I ran cargo build --locked, target/debug/computer-use-linux doctor, and target/debug/computer-use-linux windows, then drove list_windows, activate_window, and focused_window through a direct MCP harness without persistent configuration. doctor found KWin, AT-SPI, focus, and development input ready. Cleanup: closed both window sets, removed the desktop, and found no script or journal residue.
• Optional — mixed-scale absolute-pointer boundaries (run; host-input injection): With writable /dev/uinput and harmless fullscreen targets on both displays, I clicked around the seam and ran target/debug/computer-use-linux abs-test 5744 1440. Cleanup: closed both targets and found no remaining window or KWin script.
• Required — GNOME Shell Wayland targeting (run; disposable guest): On Fedora 44 with GNOME Shell/Mutter 50.4 and two scale-1 1024×768 heads, nonce-labelled GTK windows supplied compositor IDs and click receipts while list, focus, move, and activation went through MCP. Cleanup: closed both targets, confirmed a fresh list contained neither target, and terminated every MCP server; the guest and its volumes remain available for review follow-up.
• Required — extension-disabled pointer fallbacks (run; disposable guest): Before installing the extension in the same guest, I disabled the preferred absolute-pointer backend, forced portal and ydotool independently, and checked delivery with GTK receipts plus five passing caller-contract tests. Cleanup: carried the fixture into the GNOME targeting scenario above, where it was closed; the guest-only xrandr addition and portal consent remain in the retained guest.

Revision binding: KWin, pointer, and stress coverage exercised 6c5a7c9; Plasma 6 classification in 2d7ab1f was retested on native Wayland and Xwayland; 39efa80 only records the result. The GNOME checkout had no product-path difference from 39efa804, and the installed extension byte-matched it.

Automated verification

• Static analysis: cargo fmt --all -- --check, cargo check --locked --all-targets, and cargo clippy --locked --all-targets -- -D warnings passed.
• Tests: cargo test --locked windowing:: -- --nocapture passed all 56 windowing tests. cargo test --locked --no-fail-fast -- --test-threads=1 passed 255 of 257 library tests; both failures are unchanged ydotool harness tests. COSMIC's 7 tests and the library-export test passed. The focused fakeroot cargo test --locked ydotool::tests::accepts_supported_raw_cli_without_xdg_runtime_dir -- --exact --nocapture passed; the continuous-output timing test still exceeds its one-second sandbox threshold.
• Installer: bash -n install.sh scripts/install_sh_test.sh and scripts/install_sh_test.sh passed all 12 cases.
• Build and release paths: cargo build --locked, RUSTDOCFLAGS="-D warnings" cargo doc --locked --no-deps --document-private-items, and cargo publish --dry-run --locked passed.
• MCP safety: scripts/mcp_safety_check.py --binary target/debug/computer-use-linux passed for all 18 tools at version 0.4.9.
• Hosted CI: All required checks passed, including agnix, the clean-host suite, and Revuto; release and publication jobs were skipped as expected.

Remaining

The published branch is at its final shape: upstream/main remains 663930fef8d1bee64b7b8f833b68cff3196059e3, the remote PR head remains 39efa804b32c6f443a0c1d2230579c482ceb599f, and the base is an ancestor with no commits to reconcile. Live validation produced no product changes, and nothing else is queued. This revision is ready for the maintainer's full review.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[revuto-review]

This is an auto review done by revuto.

---

Revuto completed the review and found no evidence-backed concerns.

[avifenesh]

Two fixes needed before merge. Pointer axis maxima/clamp look correct and tested.

1. P1 — New Registry Peer.Ping / gdbus introspect calls use unbounded cmd.output(). GetAddress can succeed on the session bus while the a11y socket is stale, so doctor / setup_accessibility can hang instead of returning a blocker. Wrap both probes in the existing 2s command_runner::output_blocking_with_timeout.
2. P2 — When that Registry check fails, readiness still says AT-SPI is disabled and recommends setup_accessibility / IsEnabled. Surface accessibility.at_spi_bus.detail and do not recommend toolkit enable when the bus was discovered but Registry is unreachable.

Optional (not blocking): exclusive-edge clicks now land on the last pixel while off_screen_note_for_point still warns that the input landed on no visible pixel. Align that warning with the new clamp.

[revuto-review]

This is an auto review done by revuto.

---

Revuto completed the review and found no evidence-backed concerns.

[nisavid]

@avifenesh I removed both Registry-probe findings by dropping that out-of-range experiment entirely. The current branch retains only the bounded source-range exports, and the two old review threads are resolved with exact-head evidence.

The current immutable head is bebbdcc41df3dac1cacb79ad3bca3c48072c06eb. All hosted checks pass, and Revuto's fresh review on that exact head found no evidence-backed concerns. Could you refresh your review or otherwise give the current head an explicit disposition? GitHub does not let me re-request your review through the API with my repository permission.

[avifenesh]

Thanks for the fast turnaround, and for tracing the two findings back to source range instead of patching around them. That's the right instinct and it made this easy to verify.

I re-ran the full CI set verbatim against bebbdcc4 in a clean worktree — fmt, check, clippy, 262 + 7 + 1 tests, installer regressions, rustdoc, publish --dry-run, audit, MCP safety (18 tools / 0.4.9), agnix, npm wrapper, zod schema. All green locally, matching hosted. doctor from the PR-head binary on GNOME 50.1 / Wayland still advertises at_spi, reports no new blockers, degrades cleanly.

Both blocking points are resolved. Confirmed by reading the diff, not just the reply: src/diagnostics.rs at head has no Registry reachability probe, and the only surviving accessibility change is src/diagnostics.rs:353 routing capability advertisement through the existing can_build_accessibility_tree. P2 has no new failure state left to reconcile. Threads closed.

Three things before I can approve.

1 — Drop the Niri backend. Policy call on my side, not a code-quality one. niri is GPL-3.0-or-later (workspace Cargo.toml, inherited by niri-ipc on every published version) and this project is MIT. That's the incompatible direction, and it's a hard line for me regardless of how the integration is written.

I want to be clear that I see what you did here. You deliberately avoided the niri-ipc crate, kept it to a runtime niri msg exec, and left Cargo.toml and Cargo.lock genuinely untouched — I checked, and that's accurate. I'll also grant that exec across a process boundary is not linking, so you can make a decent argument that nothing attaches. The line I'm still not crossing is that the local NiriWindow / NiriWindowLayout projection is derived from reading niri-ipc/src/lib.rs. No dependency, no verbatim copy — but the shape came from GPL source, and I'd rather not carry that provenance at all than have to argue about where the boundary sits later, in an MIT tree other people build on.

For context on why I'm rigid about this exact case: wanting IPC-level desktop control without depending on niri is precisely why I built agent-workspace-linux. It gives an agent a hidden, agent-owned desktop and browser over MCP — full window/input/browser control, real compositor semantics, IPC the whole way down — and none of it routes through a copyleft compositor. MIT, so it composes cleanly downstream. It also solves something computer-use-linux structurally can't: the agent gets absolute control of a machine without seizing control of your machine. No focus fights, no stray clicks landing in your editor, no screenshots of your real desktop.

If the Niri work came from wanting a better compositor target to drive, that's where I'd point you — and I mean that as an actual invitation, not a deflection. Good contributors are rare and I'd rather keep working with you than win an argument. There's real work over there I'd hand you tomorrow.

Reverting 45fbafb and bebbdcc should be clean; nothing else here depends on them, so the pointer, KWin, GNOME-extension, and AT-SPI corrections stand on their own. That also removes the README compositor row and the CHANGELOG Added block, which resolves the validation question I'd otherwise have raised — the backend has no real-niri coverage, only JSON fixtures.

2 — Same class as P1, new instance. gnome_extension_enabled() (src/gnome_extension.rs:271-283) calls gsettings get org.gnome.shell enabled-extensions through raw command.output(), reached from setup_window_targeting_report() at line 41. A stalled dconf or session bus hangs setup_window_targeting with no deadline. I know that file already has three unbounded .output() calls, so this matches local style — but it's the exact pattern we just bounded, in a PR titled "bounded corrections." Route it through command_runner::output_blocking_with_timeout with the 2s COMMAND_TIMEOUT. Bounding the three pre-existing ones too would be welcome; I won't hold the PR on those.

3 — The optional item, now nearly free. clamp_coordinates (src/abs_pointer.rs:42) clamps to width-1, so x == width lands on the last visible pixel, while off_screen_note_for_point (src/server.rs:2830) still tests x >= mx+mw and warns "the input landed on no visible pixel." This PR introduced that disagreement; one-line fix while you're in there.

One process note, not a complaint: my review was against da3b3d4 — 3 files, +233/−19. Head is +1112/−261 across 11 files on top of that, including the KWin callback-authentication rework I haven't looked at yet. So this isn't a refresh, it's a first review of new work. Naming it so neither of us mistakes a re-request for a rubber stamp. What I've read of the KWin side looks right: the org.kde.KWin owner check plus method-kind, envelope, and one-shot CAS gating in accept() is the correct shape, and the collision path properly leaves the incumbent registration alone. Caveat that it's all validated against the fake-scripting-on-private-bus harness — nothing here has touched real KWin, and I have no Plasma session to check on either.

Sort out 1–3 and I'll do a full pass on the remaining scope. Appreciate the work.

[nisavid]

@avifenesh Thanks for the careful pass and for the pointer to agent-workspace-linux.

The Niri backend wasn’t motivated by my own setup—I don’t use Niri. It entered the downstream tree through ilysenko/codex-desktop-linux#917, and I carried it over while auditing the downstream Computer Use changes. I treated it as a generic interoperability change and missed your stricter license-provenance policy. I’m happy to remove it rather than leave that boundary for this repository to own.

agent-workspace-linux is an interesting model, particularly the separation between an agent-owned environment and the live desktop. I’m committed to Wayland on my own systems, so I don’t want to promise work on an X11-oriented stack. I could imagine exploring a native-Wayland isolated backend—or a shared interface with clearly separate host and isolated implementations—later, but I’ll keep that separate from this PR.

I’ll address item 2 systematically: route the new gsettings query and all three pre-existing unbounded command invocations in gnome_extension.rs through the common two-second timeout path, with timeout and cleanup coverage. I’ll also fix item 3 by making the diagnostic reflect where the selected input backend actually lands.

You’re right that the expanded patch deserves a complete first review, not a quick recheck. I have a real Plasma/KWin Wayland session available, so before I ask you to look again I’ll test the revised commit there: native-Wayland and Xwayland window listing and activation, minimized and off-desktop windows, multiple monitors, stale-target recovery, repeated and concurrent transactions, cleanup, and coordinate boundaries. I’ll include the environment and results with the revision.

I’ll follow up once the changes, tests, and live KWin pass are complete.

[revuto-review]

This is an auto review done by revuto.

---

Revuto completed the review and found no evidence-backed concerns.

[nisavid]

@avifenesh I’ve pushed the implementation revision (6c5a7c9).

Niri and its registry, diagnostics, environment, test, and documentation wiring are gone. All four GNOME extension subprocess paths now use the shared two-second blocking runner; timeout coverage verifies process-group cleanup, including descendants holding output pipes open.

The pointer result is now a PointerLanding value carrying requested and emitted coordinates together. The MCP click response and hidden abs-test path both report the emitted point after edge clamping, while portal, xdotool, and ydotool fallbacks retain the general off-screen diagnostic. This also closes a second-caller bug found while tightening the change.

The relevant local gates passed: formatting, compilation and linting, targeted behavioral tests, installer regressions, and the 18-tool MCP contract. This sandbox still trips two library tests that also fail on bebbdcc4; the /tmp ownership case passes under fakeroot. Hosted CI should give us the clean-host result.

I still owe the Plasma/KWin run I described above. I’ll follow up once I’ve had a chance to complete it, with the environment, results, and any resulting fixes, before asking for a full review.

[revuto-review]

This is an auto review done by revuto.

---

Revuto completed the review and found no evidence-backed concerns.

[nisavid]

@avifenesh I completed the Plasma/KWin pass and updated the PR description with the environment, commands, and detailed results.

On Plasma/KWin 6.7.4 Wayland, native Wayland and Xwayland targets behaved correctly across normal, minimized, fullscreen, second-monitor, and separate-desktop cases. A stale target was rejected in 46.4 ms, and 50 cycles across five concurrent MCP processes completed without crossed responses or leftover scripts.

The run caught one additional Plasma 6 compatibility issue: KWin no longer exposes the legacy waylandClient and x11Client flags. I fixed the classifier and verified that native Kate reports wayland while Xmessage reports x11.

I also replaced the original subjective pointer check with two fullscreen Wayland observers. They confirmed both sides of the mixed-scale monitor seam, including the seam coordinate at the second monitor’s first pixel. A request at the combined desktop’s exclusive bottom-right boundary was clamped to the last visible pixel and returned the clamp-specific result instead of the misleading off-screen warning.

All test windows, the temporary desktop, and KWin scripts were cleaned up. All hosted checks pass on commit 39efa80.

The revised scope is ready for the full pass you mentioned. When you have a chance, I’d appreciate another look.

[avifenesh]

Thanks @nisavid — went through the revision and the Plasma/KWin results.

On your updates:

• The PointerLanding requested+emitted split is the right contract. Returning the clamp-specific result at the desktop's exclusive boundary instead of the misleading off-screen warning is exactly the behavior I want, and the two fullscreen Wayland observers on the mixed-scale seam are solid evidence. Good catch on the second-caller bug while tightening it.
• Plasma 6 classifier fix (legacy waylandClient/x11Client flags gone) verified with native Kate → wayland / Xmessage → x11 — convincing.
• The two library-test failures reproducing on base bebbdcc4 in your sandbox are pre-existing, and hosted CI is green across all checks here (cargo test, MCP safety contract, installer regressions, revuto), so that's settled.
• Dropping Niri wholesale and unifying the four GNOME extension subprocess paths on the shared 2s runner with process-group cleanup (including pipe-holding descendants) — good, that was the risky surface.

What I still need before the full line-by-line review:

1. A GNOME Shell pass equivalent to your KWin matrix. The PR reworks gnome_extension.rs (reload contract + bounded setup commands), but the live validation you posted covers Plasma only. Same shape of evidence please: GNOME version, Wayland session, normal/minimized/fullscreen/second-monitor targets, one stale-target rejection timing. X11 GNOME too if it's cheap for you.
2. One run on a session without the extension — portal/xdotool/ydotool fallback path only. Just confirm the generic off-screen diagnostic still fires and fallback callers see no behavior change from the PointerLanding refactor.
3. Explicit ready signal: confirm the branch is at its final shape (rebased on current main, nothing else queued), and I'll do the full review pass.

1 and 2 can be short — environment line + observed results, same style as your KWin writeup.

[nisavid]

@avifenesh Thanks for laying out the remaining matrix so clearly. I've run both passes now and put the full environment, timings, receipts, and cleanup notes in the PR description rather than make you dig through another long comment here.

The short version: GNOME 50.4 on Wayland focused the intended window in the normal, minimized, second-monitor, and fullscreen cases, then rejected a stale ID in 14.1 ms without disturbing focus. With the extension absent, the portal still failed closed off-screen; ydotool kept the warning, along with its existing acceleration-sensitive landing caveat. The Fedora 44 guest has no genuine GNOME X11 session, so I left that optional variant out.

I refreshed against current main as well. The branch is unchanged and nothing else is queued—this is the final shape, ready for the full pass you mentioned.

[avifenesh]

Approving. Ran the gate set verbatim against 39efa80 in a clean worktree first — I wasn't going to sign off on the KWin rework from the diff alone.

fmt clean. clippy surfaces two nonminimal_bool hits at src/diagnostics.rs:466 and src/server.rs:3615, both in code you never touched and both reproducing on main under a newer local toolchain than CI pins — mine to clean up, not yours. Tests: 255 pass serialized, and the two process-group reaping failures are the sandbox; main fails four of that same family here, so this branch is strictly better than what it lands on. Hosted CI green across all 12 on the head.

On the three items:

1 — Niri. Gone cleanly, and nothing else leaned on it. Thanks for taking the license point at face value rather than arguing where the boundary sits. The agent-workspace-linux offer stands whenever you want it, and I hear you on Wayland — that's a fair reason to pass.

2 — Bounded commands. You did the systematic version instead of the minimum I asked for: all four subprocess paths on the shared 2s runner, with timeout coverage proving process-group cleanup including descendants holding output pipes open. That's the version I wanted and hadn't asked for.

3 — Clamp vs off-screen warning. Checked this rather than took it. capture_space_rect() returns origin (0,0) with the same dimensions AbsPointer::create is handed, so the clamp note and the off-screen warning are exactly complementary on the absolute-pointer path — nothing was lost by moving off_screen_note_for_point down to the fallback branch. The PointerLanding requested/emitted pair is the right contract, and moving the button check ahead of ensure_abs_pointer means an unsupported button falls through without paying for device creation.

On the KWin work, the part I hadn't reviewed before: owner check, method kind, envelope, and the one-shot CAS in accept() close the callback off properly, and both generated scripts stamp backend/pluginName, so a well-formed foreign payload can't starve the channel. Gating cleanup on owns_callback is what makes the collision path safe — the duplicate-path test asserting the incumbent's registration survives with zero load/start/unload calls is the one I'd have written. Plasma 6 classification via serialize(window) object-class parsing is stringly, but it's guarded and it's what KWin leaves you now the legacy flags are gone. Good catch finding that on your own run rather than shipping it broken.

The GNOME and extension-absent passes close what I asked for. GNOME-on-X11 being unavailable in your guest is fine — flagging it not-run instead of counting Xwayland is the honest call, and the xdotool spawn-failure regression covers the routing anyway.

Squashing on merge; the branch carries the Niri add-and-remove round trip and there's no reason for main to.

Appreciate the work on this one — tracing findings back to the source range instead of patching around them, standing up a real Plasma session to validate, and re-running a matrix rather than asserting it would pass. This is the shape of contribution I want more of here. Thank you.

---

Generated by Claude Code