diff --git a/.gitattributes b/.gitattributes index c1cb2a89bcfb..1cce978fe491 100644 --- a/.gitattributes +++ b/.gitattributes @@ -36,3 +36,8 @@ *.d.ts -whitespace src/BloomBrowserUI/compilerconfig.json -whitespace changelog -whitespace + +# Vendored third-party artifacts must stay byte-identical to their upstream +# source (see src/BloomBrowserUI/keymanweb/README.md) - no EOL conversion, +# no whitespace enforcement. +src/BloomBrowserUI/keymanweb/** -text -whitespace diff --git a/Design/Keyboards/implementation-handoff.md b/Design/Keyboards/implementation-handoff.md new file mode 100644 index 000000000000..7527d55e9707 --- /dev/null +++ b/Design/Keyboards/implementation-handoff.md @@ -0,0 +1,57 @@ +# Handoff: Implement the per-language keyboard setting (branch `Keyman`) + +**For:** a fresh agent session (Opus) implementing this feature. +**The plan (authoritative):** `Design/Keyboards/keyboard-setting-plan.md` — read it first; this document only adds context the plan doesn't carry. + +## Where things stand + +- Branch has two commits beyond master relevant here: `af6f479a2` (the KeymanWeb POC — hard-coded Thai `thai_kedmanee`, CDN-loaded engine, all browser-side in `src/BloomBrowserUI/bookEdit/js/keymanWebIntegration.ts` + hooks in `bloomEditing.ts`) and `f8376794a` (unrelated skill helper). +- **No implementation of the plan has started.** The planning session ended right after the plan was written. +- The design was worked out interactively with John and he made explicit decisions. **Do not relitigate them** (see "Decisions and their rationale" below). He reviewed the plan and said it is good. + +## Non-negotiable first step: the spike (plan item 0) + +Do the WebView2 OS-keyboard-switching spike **before** building anything else. libpalaso's `WindowsKeyboardSwitchingAdapter` activates TSF profiles with `TF_IPPMF_FORPROCESS` (process-scoped), but WebView2 keyboard input is consumed in separate `msedgewebview2.exe` processes — so activation may not affect where the user actually types. This is a go/no-go gate for the "active OS switching in v1" decision. Fallbacks to try, in order, are in the plan. If all fail, go back to John with findings before descoping; the stored setting shape does not change either way. + +## Decisions and their rationale (so you can defend them, not re-ask them) + +1. **The setting is a resolution policy, not a keyboard picker.** Default "Automatic" = per-machine cascade: OS input method for the language if this machine has one (this *includes* installed Keyman-for-Windows keyboards, which register as ordinary TSF TIPs — no Keyman-specific detection code needed), else the collection's cached KeymanWeb fallback (top suggestion from the Keyman search API). Rationale: Keyman-for-Windows already works underneath WebView2 with zero cooperation from Bloom, so "do nothing" IS the installed-Keyman path; KMW exists for machines with nothing installed. +2. **Conflicts are handled at settings time, not edit time.** The edit view honors whatever the setting resolves to. John was explicit: "The Edit view should do whatever the selector in the settings has chosen." +3. **Active switching in v1** (FieldWorks-style: focusing a Thai field activates the installed Thai keyboard; focusing English switches back). John chose this over passive knowingly, subject to the spike. +4. **Setting lives per-language on `WritingSystem` in `.bloomCollection`** (syncs via Team Collections). The per-machine resolution of Automatic is what makes syncing safe (Keyman-equipped teammate → OS wins; bare machine → KMW kicks in). John himself proposed "OS keyboard first, default KeymanWeb keyboard second." +5. **Offline-first is a hard requirement.** John: "we need to have a strong offline story here where everything keeps working if you're offline... we have already cached everything that we need." Engine vendored into Bloom; keyboard `.js` + OSK font downloaded into `/Keyboards/` at selection/suggestion time. +6. **Smart default on new language:** if the OS lacks a keyboard for it and we're online, default to the Keyman API's top suggestion; if offline, silently retry later. If an OS keyboard exists, Automatic surfaces/pre-selects it. +7. **Long-press is disabled per-field when KMW attaches** — John probed this twice; the accepted rationale: (a) long-press alternates are variants of the physical key, which is meaningless under remapping; (b) both libraries intercept the same key events; (c) it extends the existing BL-1071 policy (window-level disable when a system input processor is active) to field granularity. Caveat given to John: the event conflict is reasoned, not empirically demonstrated with KMW — if they demonstrably coexist, this is a one-line decision to revisit; the nonsense-alternates argument holds regardless. + +## Verified facts — do NOT re-research these + +- **Keyman search API** (chooser list): `https://api.keyman.com/search/2.0?q=l:id:` → `keyboards[]` with `id`, `name`, `match.finalWeight`, download counts. There is **no** official "default keyboard for language" flag; order by `finalWeight`, top = de-facto suggestion. Verified live 2026-07-09. +- **Keyman download metadata**: `https://api.keyman.com/cloud/4.0/keyboards/{id}/{lang}?languageidtype=bcp47` → `options.keyboardBaseUri` (`https://s.keyman.com/keyboard/`), `options.fontBaseUri`, `keyboard.filename`, per-language `font`/`oskFont`. Verified live. +- **No npm package for the KeymanWeb engine** (`@keymanapp/keymanweb` is a 404; registry search found nothing). Vendor the 18.0.x release artifacts (MIT license). The POC pins `18.0.249` and its comments document why `root`/`resources`/`fonts` must all be set in `keyman.init` — keep those comments. +- **KMW API facts**: `addKeyboards({id, filename, languages...})` object form registers a local stub (no cloud); `attachToControl` + `setKeyboardForControl(el, id, lang)` give per-field keyboards; `setActiveKeyboard` is global. The POC's `HasLoaded` polling exists because `setActiveKeyboard` rejects before the lazily-fetched keyboard JS arrives ("Cannot read properties of null (reading 'metadata')") — keep that workaround. +- **libpalaso** (reflected from the actual `SIL.Windows.Forms.Keyboarding` 18.0.0-beta0014 DLL in `output/Debug/AnyCPU`): `KeyboardController.Initialize()` (Bloom never calls it today; the existing `IsFormUsingInputProcessor` in `KeyboardingConfigApi.cs` works without it), `Instance.AvailableKeyboards` (`IKeyboardDefinition {Id, Name, Locale, Layout, IsAvailable, Activate()}`), `TryGetKeyboard(id)`, `ActivateDefaultKeyboard()`, `ActiveKeyboard`. Activation path ends in `ITfInputProcessorProfileMgr.ActivateProfile(..., TF_IPPMF_FORPROCESS)` — the spike risk. libpalaso master (2025 commits) has fresh trace logging in exactly this switching path, so escalating to that team is viable. +- **Team Collections** sync only `RootLevelCollectionFilesIn` (`.bloomCollection`, `customCollectionStyles.css`, `configuration.txt`, `ReaderTools*.json`) plus hard-coded folders `"Allowed Words"` and `"Sample Texts"` — `TeamCollection.cs:~1100-1147`, `FolderTeamCollection.cs:~421-422, ~586-587`. Adding `"Keyboards"` requires touching all three spots. +- **BloomServer** serves arbitrary disk files via `/bloom/` (`LocalHostPathToFilePath`, ~line 1141); use `path.ToLocalhost()`. Vite's `viteStaticCopy` (vite.config.mts:626-648) copies non-ts/pug/less files from `src/BloomBrowserUI` to `output/browser`, so a vendored `src/BloomBrowserUI/keymanweb/` folder ships on build — but static copy is **build-only**; check `scripts/dev.mjs` watcher behavior and ask John to run one build to seed `output/browser/keymanweb/` (the agent must never run `yarn build`). +- **Settings plumbing pattern to mirror**: `CollectionSettingsApi.UpdatePendingFontName` → `CollectionSettingsDialog.PendingFontSelections` (line ~57) → committed in `UpdateLanguageSettings` (~line 591) → `ChangeThatRequiresRestart()`. Keyboard changes should trigger restart too, which makes per-session resolution caching safe. +- **Data-div scrub hooks** (`BookData.cs`): `_attributesNotToCopy` ~1685, `_attributesToRemoveIfAbsent` ~1715, `_classesNotToCopy` ~1722, `_classesToRemoveIfAbsent` ~1738, `GetAttributesToSave` ~1770. `dir` needs a value-sensitive skip (only strip `dir="ltr"`; Bloom itself only writes `dir="rtl"` — same reasoning as the JS scrub comment at `bloomEditing.ts:~182`). + +## Open questions to raise with John during implementation + +- **Which XLF file** the new settings strings belong in (likely `Bloom.xlf` next to `CollectionSettingsDialog.BookMakingTab.DefaultFontFor`, but the xlf-strings skill says ask about priority file). +- Whether the long-press/KMW conflict reproduces empirically (cheap to test once KMW attach is working; see decision 7). +- L4+ languages get Automatic-only behavior with no settings row in v1 — confirm that's acceptable when the UI lands. + +## Working-style notes for this user (John) + +- He explicitly wants to be **talked with, not deferred to** — give genuine opinions, push back, and engage conversationally with his answers rather than just recording them. +- Anything posted under his account (PR comments etc.) must start with `[Claude ]`. +- Finished work lands in **"Personal Review"** on his board — never auto-promote to Peer Review (see the `personal-board` skill). +- Never `yarn build` (a --watch build may be running); never npm; `dotnet test` never with `--no-build`; only edit localization under `DistFiles/localization/en/`. + +## Suggested skills + +- `.github/skills/xlf-strings/SKILL.md` (repo skill) — **mandatory** when adding the settings-UI strings (plan item 6). +- `run-bloom` — for the spike (item 0) and all manual verification (plan's Verification section). +- `verify` — before committing each substantive slice. +- `preflight` — once implementation is complete, to push, open the draft PR, and run the bot gauntlet. +- `personal-board` — move the card to "Personal Review" when done (never Peer Review). diff --git a/Design/Keyboards/keyboard-setting-plan.md b/Design/Keyboards/keyboard-setting-plan.md new file mode 100644 index 000000000000..16c568847e69 --- /dev/null +++ b/Design/Keyboards/keyboard-setting-plan.md @@ -0,0 +1,96 @@ +# Per-language keyboard setting: generalize the KeymanWeb POC + holistic OS keyboard support + +## Context + +The `Keyman` branch has a POC (commit af6f479a2) that hard-codes the Thai `thai_kedmanee` KeymanWeb keyboard into the edit view, loaded from the Keyman CDN. We now generalize this into a real per-language **keyboard setting**, designed holistically around one key fact: **Keyman for Windows (and any Windows input method) is a TSF input processor that already works underneath WebView2** — it's the ambient condition, not a feature to build. So the setting answers: *"Should Bloom itself supply a keyboard for this language, and if so, which one?"* + +Decisions made with John: +- **Default = Automatic cascade, resolved per machine at edit time**: if this machine's OS has an input method for the field's language (includes installed Keyman keyboards) → activate it, never attach KeymanWeb. Otherwise → attach the collection's cached KMW fallback keyboard (default = top suggestion from the Keyman search API, fetched/cached first time online; silent retry when offline). +- **Active OS switching in v1** (FieldWorks-style): focusing a field switches the Windows input method to match its language; focusing English switches back. Browser → C# focus notification, libpalaso `KeyboardController`. +- **Chooser = Automatic + full list**: per-language dropdown offers Automatic (showing what it resolves to on this machine), then installed input methods, then Keyman-cloud keyboards (ordered by popularity). Picking an explicit item pins it. Conflicts are resolved at settings time; edit view just honors the setting. +- **Setting lives per-language in collection settings** (on `WritingSystem`, next to `FontName`, in `.bloomCollection`), syncing via Team Collections. Per-machine resolution of Automatic makes that safe (Keyman-equipped teammate → OS wins; bare machine → KMW kicks in). +- **Offline-first in v1**: vendor the KMW 18 engine with Bloom; download the chosen/suggested keyboard's `.js` + OSK font into the collection folder so it syncs and typing never needs the network. + +## Data model & serialization + +- `WritingSystem.Keyboard` (string, new field; skip for sign language): + - `""` / absent → Automatic (default) + - `"system:"` → pinned installed input method + - `"kmw:@"` → pinned KeymanWeb keyboard +- `WritingSystem.CachedKmwFallbackKeyboard` (string): top search-API result, used only by Automatic; empty until fetched. +- Serialize in both existing forms following the `FontName` pattern in `SaveToXElementInternal`/`ReadFromXmlInternal`: legacy `Language{n}Keyboard` / `Language{n}CachedKmwKeyboard` and unnumbered form inside ``. Copy in `Clone()`. +- New value type `src/BloomExe/Keyboarding/KeyboardSetting.cs`: `Parse`/`ToString`, `Kind {Automatic, System, KeymanWeb}`. + +## Resolution cascade (C#, per machine, cached per lang per session) + +For focused field lang X (match language subtag, ignoring region/script): +1. `system:` → `KeyboardController.Instance.TryGetKeyboard(id)` → `Activate()`; no KMW. Not installed here → fall through to Automatic. +2. `kmw:@` → reply "attach KMW"; ensure files cached (background retry when online). +3. Automatic → best matching `AvailableKeyboards` entry (exact locale > language-only) → activate, no KMW. Else `CachedKmwFallbackKeyboard` (background-fetch if empty & online) → attach KMW. Else `ActivateDefaultKeyboard()`, no KMW. +4. Langs with no `WritingSystem` (`z`, `*`, source-bubble langs) and all non-KMW fields → `ActivateDefaultKeyboard()` (switch back for English fields). + +Settings changes mark `ChangeThatRequiresRestart()` (mirroring fonts), so session caching is safe. + +## Keyman API facts (verified live) + +- Chooser list: `https://api.keyman.com/search/2.0?q=l:id:` → `keyboards[]` with `id`, `name`, `match.finalWeight`, downloads. No official "default" flag — order by `finalWeight`; top = de-facto suggestion. +- Download metadata: `https://api.keyman.com/cloud/4.0/keyboards/{id}/{lang}?languageidtype=bcp47` → `options.keyboardBaseUri`, `keyboard.filename`, per-language `font`/`oskFont`. +- No npm package for the KMW engine — vendor MIT-licensed release artifacts (18.0.x). + +## Ordered work items + +### 0. De-risking spike FIRST (throwaway): OS switching under WebView2 +libpalaso's `WindowsKeyboardSwitchingAdapter` uses `ITfInputProcessorProfileMgr.ActivateProfile(..., TF_IPPMF_FORPROCESS)` — process-scoped, while WebView2 input lives in separate `msedgewebview2.exe` processes. **Must prove activation affects typing in a bloom-editable before building on it.** +- Temp endpoint in `src/BloomExe/web/controllers/KeyboardingConfigApi.cs` (`handleOnUiThread: true`): `KeyboardController.Initialize()` once, then `TryGetKeyboard(id).Activate()`. Test via run-bloom with a Windows Thai layout + a Keyman keyboard installed. +- If it fails, evaluate in order: `WM_INPUTLANGCHANGEREQUEST` to the focused WebView2 HWND; calling `ActivateProfile` ourselves with `TF_IPPMF_FORSESSION`; escalate to libpalaso team (their master has fresh tracing in this exact path). Worst case v1 ships KMW + passive OS behavior; setting shape unchanged. +- Also verify `KeyboardController.Initialize()` doesn't disturb the existing `useLongpress` check, and measure init cost. + +### 1. Model + serialization +`src/BloomExe/Collection/WritingSystem.cs`, new `src/BloomExe/Keyboarding/KeyboardSetting.cs`; tests in `src/BloomTests/Collection/CollectionSettingsTests.cs` (round-trip both XML forms, defaults, Parse cases). + +### 2. Startup init + enumeration +`src/BloomExe/Program.cs`: `KeyboardController.Initialize()` as a low-priority startup action on the UI thread (try/catch + NonFatalProblem), `Shutdown()` beside `Sldr.Cleanup()`. New `src/BloomExe/Keyboarding/OsKeyboards.cs`: `GetInstalledKeyboardsForLanguage(tag)`, `FindBestForLanguage(tag)`, `TryActivate(id)`, `ActivateDefault()` — UI-thread-only. + +### 3. Keyman cloud client + collection cache + TC sync +New `src/BloomExe/Keyboarding/KeymanCloudClient.cs` (5s-timeout HttpClient; JSON parsing split into pure methods tested with canned fixtures) and `CollectionKeyboardCache.cs` (`/Keyboards/` with `.js`, `.json` manifest, `fonts/`; atomic temp+rename writes; `GetJsUrl` via `ToLocalhost()`). +Team Collections: add `"Keyboards"` alongside `"Allowed Words"`/`"Sample Texts"` in `TeamCollection.cs` (~1100–1147) and `FolderTeamCollection.cs` (~421, ~586). Extend existing TC sync tests. + +### 4. Edit-time endpoint + OS switching +New `src/BloomExe/Keyboarding/KeyboardResolver.cs` (cascade + `ConcurrentDictionary` session cache + background fallback fetch that saves `CachedKmwFallbackKeyboard` — marshal the save; don't clobber an open settings dialog). +`KeyboardingConfigApi.cs`: `POST keyboarding/fieldFocused {lang}` with `handleOnUiThread: true` → activation side-effect (only when `ActiveKeyboard` differs) + reply `{useKmw, keyboardId, languageTag, keyboardFileUrl, fontFamily?, fontUrls?, oskFontFamily?, oskFontUrls?}`. Inject `CollectionSettings` via the existing autofac registration (`ProjectContext.cs` 175/420). One HTTP call per focus change, never per keystroke. + +### 5. Vendor engine + generalize JS integration +New `src/BloomBrowserUI/keymanweb/` (engine js + OSK resources + osk font + LICENSE + README with version/update procedure) — vite's `viteStaticCopy` ships it to `output/browser` automatically; verify `scripts/dev.mjs` watcher behavior and note that the developer must run one build to seed it (do NOT run `yarn build` as agent). +Rewrite `src/BloomBrowserUI/bookEdit/js/keymanWebIntegration.ts`: engine from `/bloom/keymanweb/keymanweb.js`, `keyman.init({attachType:"manual", root/resources/fonts: "/bloom/keymanweb/"})` (keep the POC's hard-won comments); on focusin post `keyboarding/fieldFocused`; if `useKmw` → `addKeyboards({id, filename: keyboardFileUrl, ...})` (local stub, no cloud), keep the POC's `HasLoaded` poll, `attachToControl` + `setKeyboardForControl` (WeakSet), `setActiveKeyboard` + `osk.show`; else `osk.hide()` and do nothing (C# already switched the OS keyboard). Client cache `Map` to skip re-registration; still post every focus so C# can switch OS keyboards. Skip `z`/`*`/empty langs. Keep the existing focusin hook in `bloomEditing.ts` (~975–986) and its `Cleanup()` scrub. + +### 6. Settings UI (Book Making tab) +`CollectionSettingsApi.cs`: `GET settings/keyboardsForLanguage?languageNumber=` → `{current, automaticResolvesTo: {kind, displayName}, installed: [{id,name}], cloud: [{id,name,downloads}]}` (cloud proxied through C#, empty offline); `POST settings/setKeyboardForLanguage` using the pending pattern (`PendingKeyboardSelections` array in `CollectionSettingsDialog.cs` init ~96–101, commit in `UpdateLanguageSettings` — extend signature + its tests; `ChangeThatRequiresRestart()` on change). On OK-commit of a `kmw:` pin, kick `EnsureDownloaded` in background. +New `src/BloomBrowserUI/react_components/keyboardSection.tsx` (MUI Select: Automatic w/ dynamic "resolves to" secondary text, installed group, cloud group), rendered per language via `fontScriptSettingsControl.tsx`/`singleFontSection.tsx`; extend `currentFontData` to carry `languageTag` + `keyboard`. +**Localizable strings → follow `.github/skills/xlf-strings/SKILL.md`** (only `DistFiles/localization/en/`; IDs like `CollectionSettingsDialog.BookMakingTab.KeyboardFor` beside `...DefaultFontFor` in `Bloom.xlf`; ask John which xlf file if priority is unclear). + +### 7. C#-side scrub (data-div/xmatter) — closes the POC's flagged gap +`src/BloomExe/Book/BookData.cs`: add `"keymanweb-font"` to `_classesNotToCopy` (~1722) AND `_classesToRemoveIfAbsent` (~1738, heals POC-era pollution); `"inputmode"` to `_attributesNotToCopy` (~1685) AND `_attributesToRemoveIfAbsent` (~1715); value-sensitive skip of `dir="ltr"` in `GetAttributesToSave` (~1770; Bloom only ever writes `dir="rtl"`). Tests in `BookDataTests` (polluted xmatter → clean data-div + clean re-emitted page; healing case). Keep the JS scrub unchanged. + +### 8. Longpress interplay + polish +When KMW attaches to an editable, disable longpress on that element (`activateLongPressFor`, `bloomEditing.ts` ~1812–1863); OS-switched fields keep longpress (the global `IsFormUsingInputProcessor` check already covers desktop Keyman). L4+ languages: resolver handles them (Automatic-only; no settings row in v1). Sign language: nothing. + +## Verification + +- C#: `dotnet test` (never `--no-build`) — `CollectionSettingsTests`, new Keyboarding tests, `BookDataTests`, TC sync tests. +- JS: `yarn lint`, `yarn typecheck`, vitest for keymanWebIntegration (mock bloomApi). **No `yarn build`.** +- Manual via run-bloom skill: + 1. Thai collection, no Thai OS keyboard → Automatic fetches+caches fallback, OSK appears, typing remaps; English fields unaffected, OSK hides. + 2. Install Windows Thai Kedmanee → Automatic now activates OS keyboard on focus, no KMW; English field switches back. + 3. Pin a cloud keyboard while an OS keyboard exists → KMW wins (setting honored). + 4. Offline with a new language → no errors; suggestion fetched silently next time online. + 5. Team collection: `Keyboards/` folder syncs both directions. + 6. Save a book with xmatter fields → saved HTML free of `keymanweb-font`/`inputmode`/`dir="ltr"`. + +## Risks (ranked) + +1. **OS activation under WebView2** — spiked first (item 0) with named fallbacks; scope decision point if it fails. +2. TSF/Keyman TIP quirks (TIP vs HKL profiles, IME conversion state for CJK) — cover in spike. +3. First-focus latency (engine + keyboard load) — mitigate by prefetching `fieldFocused` for L1–L3 at page load. +4. ckeditor/undo interaction with KMW in contenteditables — watch source bubbles & canvas text during manual verification. +5. Dev-mode serving of vendored engine (static copy is build-only) — verify `dev.mjs`, document. +6. Background save of `CachedKmwFallbackKeyboard` racing an open settings dialog — only write that one field, marshal to UI thread. diff --git a/Design/Keyboards/spike-os-switching-findings.md b/Design/Keyboards/spike-os-switching-findings.md new file mode 100644 index 000000000000..86bef4e8947e --- /dev/null +++ b/Design/Keyboards/spike-os-switching-findings.md @@ -0,0 +1,150 @@ +# Spike findings: OS keyboard switching under WebView2 (plan item 0) + +**Date:** 2026-07-09 +**Author:** Claude Opus 4.8 (spike teammate) +**Instance under test:** Bloom launched from source via `./go.sh` (collection "FAL Thai Test"), +WebView2 content hosted in a separate `msedgewebview2.exe` process. + +## Verdict + +**GO-VIA-FALLBACK.** + +- The plan's assumed mechanism — libpalaso `KeyboardController` → `IKeyboardDefinition.Activate()`, + which ends in `ITfInputProcessorProfileMgr.ActivateProfile(..., TF_IPPMF_FORPROCESS)` — **does NOT + change what is typed into a bloom-editable hosted in WebView2.** This is the plan's ranked-#1 risk, + and it is realized. **NO-GO for that mechanism as written.** +- **Active OS switching is still achievable in v1**, but item 2/4 must switch the input language of + **Bloom's own foreground top-level window** (which WebView2 follows), not call libpalaso's + process-scoped `Activate()`. Posting `WM_INPUTLANGCHANGEREQUEST` to Bloom's main-form HWND + demonstrably switched the WebView2 process's input thread to the target layout, scoped to Bloom + (a co-running Chrome/Meet was unaffected). +- One honest caveat (see "What I could not do"): the gold-standard test — real OS-level keystrokes + into a focused field producing Thai glyphs — could not be run because the machine's user was in a + live video meeting and reliable key injection requires stealing foreground. The conclusion rests on + two strong, converging instruments instead (per-thread `GetKeyboardLayout` and the webview's own + `navigator.keyboard.getLayoutMap()`). A 30-second manual confirmation is recommended before building. + +## The core architectural fact + +Bloom is `pid A` (`Bloom.exe`). Its WebView2 keyboard input is processed in a **separate process** +`pid B` (`msedgewebview2.exe`) — the window that receives keystrokes is a +`Chrome_RenderWidgetHostHWND` / `Chrome_WidgetWin_1` owned by `pid B`, reparented under Bloom's window. +`TF_IPPMF_FORPROCESS` activates a TSF profile **for the calling process only** (`pid A`). It therefore +cannot, by definition, affect `pid B`'s input threads. This was then confirmed empirically. + +## Evidence + +Two instruments, both non-intrusive: +1. **`GetKeyboardLayout(threadId)`** for the UI thread of `Bloom.exe`, the render-widget thread of + `msedgewebview2.exe`, and (as a control) a co-running Chrome/Meet window. `0x04090409` = US English, + `0x041E041E` = Thai Kedmanee. +2. **`navigator.keyboard.getLayoutMap()`** read over CDP inside the bloom-editable's document — reports + the character each physical key would produce under the webview's *current* layout. Supported in + this WebView2 build. + +| Action | Bloom UI thread HKL | **WebView thread HKL** | getLayoutMap (KeyA / Semicolon) | +|---|---|---|---| +| Baseline (default) | `0x04090409` US | **`0x04090409` US** | `a` / `;` | +| libpalaso `Activate()` Thai (**FORPROCESS**) | `0x041E041E` Thai | **`0x04090409` US — UNCHANGED** | `a` / `;` — UNCHANGED | +| `WM_INPUTLANGCHANGEREQUEST` → child webview HWND | (n/a) | **`0x04090409` US — UNCHANGED** | unchanged | +| our TSF `ActivateProfile` **FORSESSION** (Bloom in background) | Thai | **US — UNCHANGED** | unchanged | +| our TSF `ActivateProfile` **FORSESSION** (Bloom **foreground**) | Thai | **US — UNCHANGED** | unchanged | +| **`WM_INPUTLANGCHANGEREQUEST` → Bloom's foreground TOP window** | Thai | **`0x041E041E` Thai — CHANGED** | changed (see caveat) | +| restore: `WM_INPUTLANGCHANGEREQUEST`(US) → foreground TOP | Thai | **`0x04090409` US — restored** | `a` / `;` restored | + +Control throughout: the co-running Chrome/Meet process stayed `0x04090409` US — the working mechanism +is scoped to Bloom + its webview, it does **not** hijack the whole session's other apps. + +### Fallbacks, evaluated in the plan's order +- **(a) `WM_INPUTLANGCHANGEREQUEST` to the focused WebView2 HWND.** Posting to the *child* webview + windows (`Chrome_RenderWidgetHostHWND`, `Chrome_WidgetWin_1`): **no effect.** Posting to Bloom's + **foreground top-level window** (the same message a language-bar / Win+Space switch generates): + **WORKS** — the webview process's input thread switched to the posted HKL. This is the recommended + fallback. +- **(b) our own `ActivateProfile` with `TF_IPPMF_FORSESSION`.** Implemented via minimal TSF COM interop + (`ITfInputProcessorProfileMgr.ActivateProfile`, profile type = keyboard layout). The call returned + success and changed `Bloom.exe`'s own thread, but produced **no observed change in the webview + thread's HKL or getLayoutMap**, even with Bloom foreground. Either the session-wide activation does + not propagate to an already-running child process's input thread the way `WM_INPUTLANGCHANGEREQUEST` + does, or my interop is incomplete (TSF activation for HKL-backed layouts is finicky). Not pursued + further because fallback (a) already works and is far simpler. + +### The `getLayoutMap` caveat (why not a slam-dunk) +`getLayoutMap()` is live (it tracked the working switch: `;` → `ö` → `;`), but it is **focus-dependent +and lags** — it refreshes when the webview processes a layout-change while focused, not on a bare +`GetKeyboardLayout` change. In the one run that flipped the webview thread to Thai, `getLayoutMap` still +reported a **Swedish-ish** mapping (`KeyA`→`a`, `Semicolon`→`ö`), *not* Thai (`KeyA`→`ฟ`). So the two +instruments briefly disagreed. Interpretation: the **thread HKL is authoritative** for what Windows +feeds Chromium at keystroke time (→ Thai), and `getLayoutMap` was a stale cache because the webview +wasn't OS-focused. But because I could not fire a real keystroke to settle it 100%, I am flagging it. + +## `KeyboardController.Initialize()` cost +Measured by timing the one-time `KeyboardController.Initialize()` inside the spike endpoint: +**~160–500 ms on the first call** (495 ms and 457 ms on two cold instances; 160–177 ms on a warmer one), +**one-time** — subsequent `AvailableKeyboards` enumeration is instant. It enumerated **22 installed +input methods**, including the target `th-TH_Thai Kedmanee_Thai Kedmanee`. Confirms the plan's item-2 +approach of doing `Initialize()` once as a low-priority startup action on the UI thread is fine; budget +up to ~½ second on first use. + +## Effect of `Initialize()` on the existing `IsFormUsingInputProcessor` check +**None.** The existing `keyboarding/useLongpress` endpoint returned `"true"` **both before and after** +`KeyboardController.Initialize()` on the same running instance. `Initialize()` does not disturb the +`IsFormUsingInputProcessor(form)` code path that item 8 / BL-1071 relies on. + +## What item 2/4 should do differently from the plan +1. **Do NOT use `IKeyboardDefinition.Activate()` / libpalaso's `KeyboardController` to *activate* the + keyboard for the edit view.** Its `TF_IPPMF_FORPROCESS` activation stays in `Bloom.exe` and never + reaches the `msedgewebview2.exe` process, so typing is unaffected. (libpalaso is still fine for + *enumeration* — `AvailableKeyboards`, `TryGetKeyboard`, resolving the HKL — just not for the + activation that must affect typing.) +2. **Switch the input language on Bloom's foreground main window instead**, and let WebView2 follow. + The proven channel is `PostMessage(mainFormHwnd, WM_INPUTLANGCHANGEREQUEST, INPUTLANGCHANGE_SYSCHARSET, + hkl)`. Equivalent candidates worth trying in the real implementation: `ActivateKeyboardLayout(hkl, …)` + executed **on Bloom's UI thread while it is foreground**. The activation must happen when Bloom is + foreground and the field focused — which is exactly the real-world condition (a user is typing), so + this is not a limitation in production, only in headless testing. +3. **Resolve the target to an HKL.** libpalaso's `AvailableKeyboards[].Id` is a composite string + (e.g. `th-TH_Thai Kedmanee_Thai Kedmanee`), not an HKL. Item 4 will need the actual `HKL` + (`LoadKeyboardLayout`/the layout list) to post in `WM_INPUTLANGCHANGEREQUEST`. Plan the resolver to + carry the HKL, not just libpalaso's Id. +4. **Verify with real typing, not `getLayoutMap`.** `getLayoutMap` lags when unfocused and briefly + mis-reported the active layout here. The manual verification (focus a Thai field, type home-row + `asdfjkl;`, expect Thai glyphs; focus English, expect Latin) remains the source of truth. +5. **The passive/ambient premise still holds** and is unaffected by this finding: a user *manually* + switching to an installed Keyman/OS keyboard (Win+Space, session-wide) works under WebView2 as the + plan assumes. This spike only concerns Bloom *programmatically driving* the switch, which is where + FORPROCESS fails and the foreground-window approach succeeds. + +## Machine-state changes and how to undo them +1. **Added Thai Kedmanee (`041E:0000041E`) to the Windows `th` language** (it previously had an empty + input-method list). **Left installed** per the task brief so feature work can proceed. To undo: + ```powershell + $list = Get-WinUserLanguageList + foreach ($e in $list) { if ($e.LanguageTag -eq 'th') { $e.InputMethodTips.Clear() } } + Set-WinUserLanguageList $list -Force + ``` + (Use the `foreach` *statement*, not `Get-WinUserLanguageList | ForEach-Object` — the cmdlet returns a + generic `List` the pipeline does not unroll; enumerate by `foreach`/index.) +2. **Foreground-lock timeout:** I attempted to set `SPI_SETFOREGROUNDLOCKTIMEOUT` to 0 to make + `SetForegroundWindow` reliable; the set **did not take** (it stayed pinned at `0x7FFFFFFF`). **No net + change** — nothing to undo. +3. **Bloom + its WebView2 layout:** left the WebView2 process back on **US** (restored and verified). + The throwaway Bloom instance's WinForms *shell* thread was left showing Thai — inconsequential (users + type in the webview, not the shell) and gone when the instance is killed. **No other app was + affected** (Chrome/Meet stayed US throughout). +4. **No software was installed.** No Keyman-for-Windows install. No npm, no `yarn build`, no commits. + +## Spike code (uncommitted, to be discarded) +Temporary endpoints in `src/BloomExe/web/controllers/KeyboardingConfigApi.cs`, all marked +`// SPIKE - temporary`: +- `keyboarding/spikeList` — one-time `KeyboardController.Initialize()` (timed) + `AvailableKeyboards` + JSON + `IsFormUsingInputProcessor` readout. +- `keyboarding/spikeActivate?id=` — `TryGetKeyboard(id).Activate()` (FORPROCESS) / + `ActivateDefaultKeyboard()`. +- `keyboarding/spikeForSession?hkl=&lang=` — our TSF `ActivateProfile(..., TF_IPPMF_FORSESSION)` interop. +- `keyboarding/fieldFocused` — a `useKmw:false` **stub** added only so the orchestrator's item-5 browser + code (which POSTs `fieldFocused` on every focus) stops raising "Cannot Find API Endpoint" dialogs that + wrecked the test. Not part of the answer; discard with the rest. + +Test tooling used (in the session scratchpad, outside the repo): a CDP driver +(`editDriver.mjs` — list/focus/clear/read/`layoutmap`) and PowerShell HKL/thread-layout probes. diff --git a/DistFiles/localization/en/Bloom.xlf b/DistFiles/localization/en/Bloom.xlf index dd64c53c3e5c..c43ace415442 100644 --- a/DistFiles/localization/en/Bloom.xlf +++ b/DistFiles/localization/en/Bloom.xlf @@ -408,6 +408,51 @@ ID: CollectionSettingsDialog.BookMakingTab.DefaultFontFor {0} is a language name. + + Keyboard for {0} + ID: CollectionSettingsDialog.BookMakingTab.KeyboardFor + {0} is a language name. + + + Automatic + ID: CollectionSettingsDialog.BookMakingTab.Keyboard.Automatic + First item of the per-language Keyboard dropdown in the Book Making tab of Settings. Means "let Bloom figure out the best keyboard for this language on this machine". + + + Off + ID: CollectionSettingsDialog.BookMakingTab.Keyboard.Off + An item in the per-language Keyboard dropdown in the Book Making tab of Settings. Means Bloom should not change or supply a keyboard for this language; it leaves whatever keyboard the user has active. + + + Currently: {0} + ID: CollectionSettingsDialog.BookMakingTab.Keyboard.AutomaticResolvesTo + Secondary/subtext line under "Automatic" in the per-language Keyboard dropdown. {0} is the name of the keyboard or input method that Automatic currently resolves to on this machine (may itself be untranslated, e.g. an OS input method name). + + + Installed input methods + ID: CollectionSettingsDialog.BookMakingTab.Keyboard.InstalledGroup + Group header in the per-language Keyboard dropdown, above the list of keyboards/input methods already installed on this machine. + + + Keyman (online) keyboards + ID: CollectionSettingsDialog.BookMakingTab.Keyboard.CloudGroup + Group header in the per-language Keyboard dropdown, above the list of keyboards available from the Keyman online keyboard service. "Keyman" is a product name and must not be translated. + + + Not available offline + ID: CollectionSettingsDialog.BookMakingTab.Keyboard.CloudUnavailable + Shown in place of the online Keyman keyboard list, in the per-language Keyboard dropdown, when Bloom cannot reach the Keyman server (e.g. no internet connection). + + + {0} downloads + ID: CollectionSettingsDialog.BookMakingTab.Keyboard.DownloadCount + Small subtext shown under a Keyman online keyboard's name in the per-language Keyboard dropdown. {0} is a number of times that keyboard has been downloaded. + + + Default + ID: CollectionSettingsDialog.BookMakingTab.Keyboard.NoneResolvesTo + Used as the "Currently: {0}" value under "Automatic" in the per-language Keyboard dropdown, when this machine has no matching OS input method and no Keyman keyboard has been suggested yet for this language. + Front/Back Matter Pack ID: CollectionSettingsDialog.BookMakingTab.Front/BackMatterPack @@ -1563,6 +1608,11 @@ This sentence is too long for this level. ID: EditTab.EditTab.Toolbox.LeveledReaderTool.SentenceTooLong + + Show on-screen keyboard + ID: EditTab.EditableControls.ShowOnScreenKeyboard + Tooltip on a small keyboard icon that appears at the corner of a text box when that box is focused and uses a Keyman on-screen keyboard. Clicking the icon re-opens the on-screen keyboard. + Your subscription includes this feature. ID: EditTab.SubscriptionEnabled diff --git a/src/BloomBrowserUI/.prettierignore b/src/BloomBrowserUI/.prettierignore index 32ddda6fc80b..60be50fc6ad3 100644 --- a/src/BloomBrowserUI/.prettierignore +++ b/src/BloomBrowserUI/.prettierignore @@ -1,4 +1,5 @@ lib +keymanweb typings Readium modified_libraries diff --git a/src/BloomBrowserUI/bookEdit/StyleEditor/fontSelectComponent.tsx b/src/BloomBrowserUI/bookEdit/StyleEditor/fontSelectComponent.tsx index 2f2a75b4bc8a..d3cad81f4cb3 100644 --- a/src/BloomBrowserUI/bookEdit/StyleEditor/fontSelectComponent.tsx +++ b/src/BloomBrowserUI/bookEdit/StyleEditor/fontSelectComponent.tsx @@ -41,6 +41,9 @@ interface FontSelectProps { // Use this if you need to modify the style of popup menus by increasing z-index // (e.g., to make the popup be in front of the bloom font dialog) popoverZindex?: string; + // Applied to the underlying select's FormControl. Emotion routes a `css` prop + // on this component here, so callers can size the control. + className?: string; } const FontSelectComponent: React.FunctionComponent = ( @@ -131,6 +134,7 @@ const FontSelectComponent: React.FunctionComponent = ( currentValue={textValue} onChangeHandler={handleFontChange} popoverZindex={props.popoverZindex} + className={props.className} > {getMenuItemsFromFontMetaData()} diff --git a/src/BloomBrowserUI/bookEdit/css/editMode.less b/src/BloomBrowserUI/bookEdit/css/editMode.less index be291292c0f3..5681da0e01ac 100644 --- a/src/BloomBrowserUI/bookEdit/css/editMode.less +++ b/src/BloomBrowserUI/bookEdit/css/editMode.less @@ -165,6 +165,37 @@ body:has(.qtip.topic-chooser-hint-bubble:hover) pointer-events: auto; } +// The per-editable React controls strip (language tag, keyboard indicator; see +// editableControls/). Its POSITIONING is unconditional (not gated on :hover): +// the strip is a real DOM sibling inserted right after each editable, so if it +// were ever position:static it would fall into the translation group's flex +// column and add a phantom gap between multilingual editables. Absolute +// positioning removes it from flow entirely. The manager sets top/left/width/ +// height to overlay the editable; the strip inside pins to the bottom-right. +// padding matches the old tip's 2px inset (Bl-10017). +// +// Its VISIBILITY, however, rides on the same page-hover gate as the format cog +// and the structure borders (see the hover block below): when the mouse leaves +// the page we hide all editing chrome so the user sees the page as it will +// print. So the whole strip is hidden by default and revealed only on hover. +// (Per-control conditions compose on top of this: e.g. the keyboard indicator is +// only rendered by React when the field is focused and uses a KMW keyboard, so +// it appears only when focused AND the page is hovered.) +.bloom-editableControls { + position: absolute; + pointer-events: none; // clicks fall through to the text; the keyboard button + // turns them back on for itself + z-index: @formatButtonZIndex; + display: flex; + flex-direction: column; + justify-content: flex-end; + align-items: flex-end; + padding: 0 2px 2px 0; + box-sizing: border-box; + overflow: visible; + visibility: hidden; +} + // Keep together here all the effects we want when hovering (previously: also when focus-within) // the bloom-page. All of them need to also apply when the canvas element context controls are hovered, // even though that is no longer a child of the bloom-page. (That's what the second rule is for.) @@ -259,24 +290,14 @@ body:has(#canvas-element-context-controls:hover) .bloom-page, } } - /*Put in little grey language tooltips in the bottom-right of the editable divs*/ - .languageTip, - .bloom-editable[contentEditable="true"][data-languageTipContent]:not( - [data-languageTipContent=""] - ):after { - position: absolute; - right: 0; - /*Same grey color as pageLabel*/ - color: var(--language-tag-color); //rgba(0, 0, 0, 0.2); - font-family: @UIFontStack; - font-size: small; - font-style: normal; - font-weight: normal; - line-height: 1; //else it will draw up in the box somewhere if the font is large - text-shadow: none; - content: attr(data-languageTipContent); - bottom: 2px; // See Bl-10017 - margin-right: 2px; + // Reveal the per-editable controls strip (language tag, keyboard indicator, + // any future controls) only while the page is hovered, exactly like the + // format cog below and the structure borders above. Its positioning is set + // unconditionally (see the top-level .bloom-editableControls rule); only its + // visibility is gated here, so moving the mouse off the page hides it along + // with the rest of the editing chrome. + .bloom-editableControls { + visibility: visible; } // show some structure when the page is hovered // Show the padding, multilingual gap, etc as colored areas. @@ -1316,16 +1337,10 @@ body { .bloom-dragHandle { right: calc(100% + 1.4px); } - - [data-languageTipContent]:after { - content: "" !important; // suppress these, now in canvas element context controls - } -} - -[data-bubble*="caption"] { - [data-languageTipContent]:after { - right: calc(100% + 2.8px) !important; - } + // (The old data-languageTipContent :after suppression that lived here is + // gone: the corner tag is now a React control, which the editableControls + // manager already suppresses inside canvas elements. The language is shown + // in the canvas element context controls.) } // hideAllCKEditors is toggled by bloomEditing.ts to prevent bl-12448 diff --git a/src/BloomBrowserUI/bookEdit/editableControls/EditableControls.spec.tsx b/src/BloomBrowserUI/bookEdit/editableControls/EditableControls.spec.tsx new file mode 100644 index 000000000000..cceba3c761ce --- /dev/null +++ b/src/BloomBrowserUI/bookEdit/editableControls/EditableControls.spec.tsx @@ -0,0 +1,154 @@ +import { afterEach, describe, expect, it, vi, beforeEach } from "vitest"; +import { + render, + screen, + fireEvent, + cleanup, + act, +} from "@testing-library/react"; + +// Control the keyman integration surface the component depends on. The event +// name must match the real module's constant so dispatching below reaches the +// component's listener. +vi.mock("../js/keymanWebIntegration", () => ({ + kFieldKeyboardInfoEvent: "bloom-fieldKeyboardInfo", + getKeyboardInfoFor: vi.fn(), + showOsk: vi.fn(), +})); + +import { + getKeyboardInfoFor, + showOsk, + kFieldKeyboardInfoEvent, + FieldKeyboardInfo, +} from "../js/keymanWebIntegration"; +import { EditableControls } from "./EditableControls"; + +const getKeyboardInfoForMock = vi.mocked(getKeyboardInfoFor); +const showOskMock = vi.mocked(showOsk); + +// A bloom-editable to adorn. lang "en" resolves to "English" via the shared +// localizationManager mock in vitest.setup.ts. +function makeEditable(lang: string): HTMLElement { + const editable = document.createElement("div"); + editable.className = "bloom-editable"; + editable.setAttribute("lang", lang); + document.body.appendChild(editable); + return editable; +} + +const kmwInfo: FieldKeyboardInfo = { + useKmw: true, + keyboardId: "thai_kedmanee", + languageTag: "th", + keyboardFileUrl: "some.js", +}; + +describe("EditableControls", () => { + beforeEach(() => { + vi.clearAllMocks(); + getKeyboardInfoForMock.mockReturnValue(undefined); + }); + + afterEach(() => { + cleanup(); + document.body.innerHTML = ""; + }); + + it("shows the localized language tag when showTag is true", () => { + const editable = makeEditable("en"); + render( + , + ); + expect(screen.getByText("English")).toBeInTheDocument(); + }); + + it("does not show the language tag when showTag is false", () => { + const editable = makeEditable("en"); + render( + , + ); + expect(screen.queryByText("English")).not.toBeInTheDocument(); + }); + + it("does not show the keyboard indicator when the field is not focused, even with a KMW keyboard", () => { + getKeyboardInfoForMock.mockReturnValue(kmwInfo); + const editable = makeEditable("th"); + render( + , + ); + expect(screen.queryByRole("button")).not.toBeInTheDocument(); + }); + + it("does not show the keyboard indicator when focused but the field does not use KMW", () => { + getKeyboardInfoForMock.mockReturnValue({ + ...kmwInfo, + useKmw: false, + }); + const editable = makeEditable("en"); + render( + , + ); + expect(screen.queryByRole("button")).not.toBeInTheDocument(); + }); + + it("shows the keyboard indicator when focused with a KMW keyboard, and re-shows the OSK on click", () => { + getKeyboardInfoForMock.mockReturnValue(kmwInfo); + const editable = makeEditable("th"); + render( + , + ); + + const button = screen.getByRole("button"); + // Sanity: showOsk not called until the user clicks. + expect(showOskMock).not.toHaveBeenCalled(); + + fireEvent.click(button); + expect(showOskMock).toHaveBeenCalledTimes(1); + }); + + it("appears when keyboard info arrives asynchronously via the notification event", () => { + // No info at mount time: the field was focused before the fieldFocused + // POST resolved. + getKeyboardInfoForMock.mockReturnValue(undefined); + const editable = makeEditable("th"); + render( + , + ); + expect(screen.queryByRole("button")).not.toBeInTheDocument(); + + act(() => { + editable.dispatchEvent( + new CustomEvent(kFieldKeyboardInfoEvent, { + detail: kmwInfo, + }), + ); + }); + + expect(screen.getByRole("button")).toBeInTheDocument(); + }); +}); diff --git a/src/BloomBrowserUI/bookEdit/editableControls/EditableControls.tsx b/src/BloomBrowserUI/bookEdit/editableControls/EditableControls.tsx new file mode 100644 index 000000000000..0ba628dfcd3f --- /dev/null +++ b/src/BloomBrowserUI/bookEdit/editableControls/EditableControls.tsx @@ -0,0 +1,147 @@ +import { css } from "@emotion/react"; +import * as React from "react"; +import { useState, useEffect } from "react"; +import { ThemeProvider } from "@mui/material/styles"; +import { default as KeyboardIcon } from "@mui/icons-material/Keyboard"; +import { lightTheme } from "../../bloomMaterialUITheme"; +import { useL10n } from "../../react_components/l10nHooks"; +import theOneLocalizationManager from "../../lib/localizationManager/localizationManager"; +import { + FieldKeyboardInfo, + getKeyboardInfoFor, + kFieldKeyboardInfoEvent, + showOsk, +} from "../js/keymanWebIntegration"; + +// Look up the display name for a language code, matching the old jQuery +// AddLanguageTags logic: use the localized name, falling back to the raw code +// if we have no localization for it. Exported so the canvas-element context +// controls can compute the same name without depending on the now-removed +// data-languageTipContent attribute. +export function getLanguageDisplayName(langCode: string | null): string { + if (!langCode) return ""; + return theOneLocalizationManager.getLanguageName(langCode) || langCode; +} + +// A small strip of React-hosted controls that is attached (as an out-of-flow +// sibling, never a child) to a single .bloom-editable. It currently hosts the +// language tag (formerly a CSS ::after pseudo-element) and a keyboard indicator +// that re-shows the KeymanWeb on-screen keyboard. See editableControlsManager.ts +// for the non-React lifecycle/positioning owner. +// +// props.editable the bloom-editable this strip adorns +// props.focused whether that editable currently has focus (drives the +// keyboard indicator, which is only relevant for the field the +// user is actually typing in) +// props.showTag whether the language tag should be shown (the manager decides +// this from the same rules the old AddLanguageTags used) +export const EditableControls: React.FunctionComponent<{ + editable: HTMLElement; + focused: boolean; + showTag: boolean; +}> = (props) => { + const tagText = getLanguageDisplayName(props.editable.getAttribute("lang")); + + // The keyboard info arrives asynchronously: the field's focusin handler + // POSTs keyboarding/fieldFocused and only then knows whether this field uses + // a KeymanWeb keyboard. This strip is mounted before that (at page setup), + // so we seed from whatever is already cached and then listen for the + // notification dispatched on the editable when a (possibly later) response + // comes back. + const [keyboardInfo, setKeyboardInfo] = useState< + FieldKeyboardInfo | undefined + >(() => getKeyboardInfoFor(props.editable)); + + useEffect(() => { + const handler = (e: Event) => { + setKeyboardInfo((e as CustomEvent).detail); + }; + props.editable.addEventListener(kFieldKeyboardInfoEvent, handler); + // Pick up anything that was resolved between our initial render and this + // effect running. + setKeyboardInfo(getKeyboardInfoFor(props.editable)); + return () => { + props.editable.removeEventListener( + kFieldKeyboardInfoEvent, + handler, + ); + }; + }, [props.editable]); + + const showKeyboardIndicator = props.focused && !!keyboardInfo?.useKmw; + + const showOskTooltip = useL10n( + "Show on-screen keyboard", + "EditTab.EditableControls.ShowOnScreenKeyboard", + ); + + return ( + +
+ {showKeyboardIndicator && ( + + )} + {props.showTag && tagText && ( + + {tagText} + + )} +
+ + ); +}; diff --git a/src/BloomBrowserUI/bookEdit/editableControls/editableControlsManager.ts b/src/BloomBrowserUI/bookEdit/editableControls/editableControlsManager.ts new file mode 100644 index 000000000000..fbd02bc74fe7 --- /dev/null +++ b/src/BloomBrowserUI/bookEdit/editableControls/editableControlsManager.ts @@ -0,0 +1,224 @@ +// Non-React lifecycle owner for the per-editable React controls (see +// EditableControls.tsx). This module inserts, positions, updates, and removes a +// small container div for each qualifying .bloom-editable, and drives the +// React root inside it. It replaces the old jQuery AddLanguageTags mechanism +// (which set a data-languageTipContent attribute rendered by a CSS ::after). +// +// The container is inserted as a sibling immediately after the editable (inside +// its .bloom-translationGroup), NOT as a child. This inherits the BL-12118 +// safety of the #formatButton precedent: ckeditor only ever touches the +// editables themselves, and putting interactive UI inside a contentEditable +// broke ctrl-A in WebView2. The container carries the bloom-ui class, so it is +// stripped from the saved book HTML by both the C# save path and Cleanup() in +// bloomEditing.ts, guaranteeing the book is never polluted. + +import * as React from "react"; +import { renderRoot, unmountRoot } from "../../utils/reactRender"; +import { kCanvasElementSelector } from "../toolbox/canvas/canvasElementConstants"; +import { EditableControls } from "./EditableControls"; + +export { getLanguageDisplayName } from "./EditableControls"; + +const kControlsContainerClass = "bloom-editableControls"; + +// Every editable we currently manage, mapped to its controls container. A plain +// Map (not WeakMap) so we can iterate it to reposition/cleanup; entries for +// editables removed from the DOM are pruned lazily (isConnected checks) and on +// cleanup. +const containersByEditable = new Map(); + +// One ResizeObserver per translation group; disconnected and rebuilt whenever +// the set of managed editables changes. +let observers: ResizeObserver[] = []; + +// The editable that currently has focus (drives the keyboard indicator). Null +// when focus is not in any editable. +let focusedEditable: HTMLElement | null = null; + +// Guards one-time wiring of the document-level focusout listener. +let focusoutListenerAttached = false; + +// Decide whether the language tag should be visible for this editable. Ported +// from the old AddLanguageTags qualification rules (bloomEditing.ts): the +// keyboard indicator may still be shown even when the tag is not, so this only +// governs the tag. +function shouldShowTag(editable: HTMLElement): boolean { + const lang = editable.getAttribute("lang"); + // "*" and empty are "any language" placeholders; "z" is the never-visible + // prototype-block marker (and looking it up would produce a + // missing-localization toast). + if (!lang || lang === "*" || lang === "z") return false; + + // bloom-hideLanguageNameDisplay on the editable or any ancestor turns tags + // off (e.g. for a whole page). closest() covers the self case too. + if (editable.closest(".bloom-hideLanguageNameDisplay")) return false; + + // Inside a canvas element the language is shown in the context controls box, + // not as a corner tag. + if (editable.closest("[data-bubble]")) return false; + + // With a really small box the tag fights for space with hint bubbles, so we + // suppress it — but not inside canvas elements, where boxes are small by + // design and the tag lives elsewhere anyway. + const inCanvasElement = !!editable.closest(kCanvasElementSelector); + if (editable.offsetWidth < 100 && !inCanvasElement) return false; + + return true; +} + +// Render (or re-render) the React controls for one editable with current props. +function renderControls(editable: HTMLElement): void { + const container = containersByEditable.get(editable); + if (!container) return; + renderRoot( + React.createElement(EditableControls, { + editable, + focused: editable === focusedEditable, + showTag: shouldShowTag(editable), + }), + container, + ); +} + +// Position one container to overlay its editable exactly. The container is +// absolutely positioned and shares an offset parent with the editable (they are +// siblings), so the editable's own offset* box — already in the page's +// unscaled coordinate space — positions it correctly at any zoom without any +// getBoundingClientRect/scale arithmetic. The right-aligned strip inside then +// sits in the editable's lower-right corner. Hidden editables (offsetHeight 0) +// hide their strip. +function positionContainer( + editable: HTMLElement, + container: HTMLElement, +): void { + if (editable.offsetHeight === 0) { + container.style.display = "none"; + return; + } + container.style.display = ""; + container.style.top = editable.offsetTop + "px"; + container.style.left = editable.offsetLeft + "px"; + container.style.width = editable.offsetWidth + "px"; + container.style.height = editable.offsetHeight + "px"; +} + +// Disconnect and rebuild the ResizeObservers over all currently-managed, +// still-connected editables. We observe each translation group AND each of its +// editables: typing that grows one box changes the group's size (moving later +// boxes) and also the box's own size, and both need the strips realigned. +function rebuildPositioning(): void { + observers.forEach((o) => o.disconnect()); + observers = []; + + // Prune editables that have left the DOM, then group the rest by their + // translation group (falling back to the parent element for the rare + // page types that have no translation group, e.g. arithmetic templates). + const groups = new Map(); + for (const [editable, container] of containersByEditable) { + if (!editable.isConnected) { + unmountRoot(container); + container.remove(); + containersByEditable.delete(editable); + continue; + } + const group = (editable.closest(".bloom-translationGroup") ?? + editable.parentElement) as HTMLElement | null; + if (!group) continue; + const list = groups.get(group) ?? []; + list.push(editable); + groups.set(group, list); + } + + for (const [group, editables] of groups) { + const observer = new ResizeObserver(() => { + editables.forEach((editable) => { + const container = containersByEditable.get(editable); + if (container) positionContainer(editable, container); + }); + }); + observer.observe(group); + editables.forEach((editable) => { + observer.observe(editable); + const container = containersByEditable.get(editable); + if (container) positionContainer(editable, container); + }); + observers.push(observer); + } +} + +// When focus leaves all editables (e.g. to the toolbox), drop the keyboard +// indicator on the previously-focused field. +function attachFocusoutListenerOnce(): void { + if (focusoutListenerAttached) return; + focusoutListenerAttached = true; + document.body.addEventListener("focusout", () => { + // The related focus target isn't reliably available synchronously; check + // on the next tick where document.activeElement, is settled. + window.setTimeout(() => { + const active = document.activeElement; + if (!active || !active.closest(".bloom-editable")) { + const previous = focusedEditable; + focusedEditable = null; + if (previous) renderControls(previous); + } + }, 0); + }); +} + +// Ensure a controls container exists for every qualifying editable in the +// container, render the React controls into each, and (re)establish +// positioning. Idempotent: an editable that already has a container is simply +// re-rendered, not duplicated. Called from bloomEditing SetupElements, which +// covers page bootstrap as well as newly-added canvas elements and image +// descriptions. +export function setupEditableControls(container: HTMLElement): void { + attachFocusoutListenerOnce(); + + const editables = Array.from( + container.querySelectorAll( + ".bloom-editable[contenteditable=true]", + ), + ); + for (const editable of editables) { + // Prototype blocks (lang "z") are never visible; don't waste a React + // root on them. + if (editable.getAttribute("lang") === "z") continue; + + let controlsContainer = containersByEditable.get(editable); + if (!controlsContainer) { + controlsContainer = document.createElement("div"); + controlsContainer.className = `bloom-ui ${kControlsContainerClass}`; + controlsContainer.setAttribute("contenteditable", "false"); + editable.insertAdjacentElement("afterend", controlsContainer); + containersByEditable.set(editable, controlsContainer); + } + renderControls(editable); + } + + rebuildPositioning(); +} + +// Called from bloomEditing's focusin handler after keyboard attachment, so the +// focused field's keyboard indicator can appear (and the previously-focused +// field's disappear). +export function notifyEditableFocused(editable: HTMLElement): void { + const previous = focusedEditable; + focusedEditable = editable; + if (previous && previous !== editable) renderControls(previous); + renderControls(editable); +} + +// Unmount every React root and remove every container. Called from +// removeEditingDebris before the page is captured for saving; the bloom-ui +// stripping is the backstop, but unmounting first keeps React from touching a +// detached DOM. +export function cleanupEditableControls(): void { + observers.forEach((o) => o.disconnect()); + observers = []; + for (const [, container] of containersByEditable) { + unmountRoot(container); + container.remove(); + } + containersByEditable.clear(); + focusedEditable = null; +} diff --git a/src/BloomBrowserUI/bookEdit/js/bloomEditing.ts b/src/BloomBrowserUI/bookEdit/js/bloomEditing.ts index 7e9f489870d1..1db42afd5a0a 100644 --- a/src/BloomBrowserUI/bookEdit/js/bloomEditing.ts +++ b/src/BloomBrowserUI/bookEdit/js/bloomEditing.ts @@ -54,6 +54,12 @@ import { getToolboxBundleExports, } from "./workspaceFrames"; import { showInvisibles, hideInvisibles } from "./showInvisibles"; +import { attachKeymanWebIfNeeded } from "./keymanWebIntegration"; +import { + cleanupEditableControls, + notifyEditableFocused, + setupEditableControls, +} from "../editableControls/editableControlsManager"; //promise may be needed to run tests with phantomjs //import promise = require('es6-promise'); @@ -168,6 +174,27 @@ function Cleanup() { TrimTrailingLineBreaksInDivs(this); }); + // Scrub KeymanWeb attach artifacts (see keymanWebIntegration.ts). The engine + // decorates every control it attaches to: it adds the "keymanweb-font" class, + // sets inputmode="none", forces dir="ltr", and can leave an empty style="". + // None of these belong in the saved book, so remove them here. This runs + // unconditionally (not only when Keyman loaded this session) so books already + // polluted by an earlier session get cleaned on their next save. + $("div.bloom-editable").each(function () { + $(this).removeClass("keymanweb-font"); + $(this).removeAttr("inputmode"); + // Bloom itself only ever sets dir="rtl" (via C# TranslationGroupManager), + // never "ltr", so a "ltr" value can only be KeymanWeb's; leave "rtl" alone. + if ($(this).attr("dir") === "ltr") { + $(this).removeAttr("dir"); + } + // Remove the empty style attribute Keyman can leave behind, but keep any + // real inline styles. + if ($(this).attr("style") === "") { + $(this).removeAttr("style"); + } + }); + cleanupImages(); cleanupOrigami(); cleanupNiceScroll(); @@ -319,63 +346,6 @@ function AddEditKeyHandlers(container) { // so that it can be caught even when the focus isn't on the browser } -// Add little language tags. (At one point we limited this to visible .bloom-editable divs, -// probably as an optimization since there can be other-language divs present but hidden. -// But there may be yet others that are not visible when we run this but which soon will be, -// such as image descriptions. We don't seem to need the optimization, so let's just do -// them all.) -function AddLanguageTags(container) { - $(container) - .find(".bloom-editable[contentEditable=true]") - .each(function () { - const $this = $(this); - - // If this DIV already had a language tag, remove the content in case we decide the situation has changed. - if ($this.hasAttr("data-languageTipContent")) { - $this.removeAttr("data-languageTipContent"); - } - - // With a really small box that also had a hint qtip, there wasn't enough room and the two fought - // with each other, leading to flashing back and forth - // Of course that was from when Language Tags were qtips too, but I think I'll leave the restriction for now. - // August 2024: for canvas elements, the language is now displayed in the context controls box, and isn't - // a problem for small text boxes. - if ($this.width() < 100 && !this.closest(kCanvasElementSelector)) { - return; - } - - const key = $this.attr("lang"); - if (key !== undefined && (key === "*" || key.length < 1)) { - return; //seeing a "*" was confusing even to me - } - // z is not a real language, it is used for prototype blocks, which are NEVER visible. - // Searching for it causes missing-localization toasts if attempted. - if (key === "z") { - return; - } - - // if this or any parent element has the class bloom-hideLanguageNameDisplay, we don't want to show any of these tags - // first usage (for instance) was turning off language tags for a whole page - if ( - $this.hasClass("bloom-hideLanguageNameDisplay") || - $this.parents(".bloom-hideLanguageNameDisplay").length !== 0 - ) { - return; - } - - let whatToSay = ""; - if (key !== undefined) { - whatToSay = theOneLocalizationManager.getText(key); - if (!whatToSay) { - whatToSay = key; //just show the code - } - } - - // Put whatToSay into data attribute for pickup by the css - $this.attr("data-languageTipContent", whatToSay); - }); -} - function SetBookCopyrightAndLicenseButtonVisibility(container) { const shouldShowButton = !$(container).find("DIV.copyright").text(); $(container) @@ -768,7 +738,7 @@ export function SetupElements( } }); - AddLanguageTags(container); + setupEditableControls(container); //only make things deletable if they have the deletable class *and* page customization is enabled $(container) @@ -957,6 +927,11 @@ export function SetupElements( editBox.closest(".bloom-userCannotModifyStyles").length === 0 ) { editor.AttachToBox(editBox.get(0)); + const focusedEditable = editBox.get(0)!; + attachKeymanWebIfNeeded(focusedEditable).catch((err) => + console.error("attachKeymanWebIfNeeded failed", err), + ); + notifyEditableFocused(focusedEditable); } }); @@ -1282,6 +1257,7 @@ export function localizeCkeditorTooltips(bar: JQuery) { // This is invoked when we are about to change pages. function removeEditingDebris() { resetAbovePageControls(); + cleanupEditableControls(); // We are mirroring the Change Layout mode toggle behavior here, in case the user changes // pages while the Change Layout mode toggle is on. // The DOM here is for just one page, so there's only ever one marginBox. diff --git a/src/BloomBrowserUI/bookEdit/js/canvasElementManager/CanvasElementContextControls.tsx b/src/BloomBrowserUI/bookEdit/js/canvasElementManager/CanvasElementContextControls.tsx index ec60a37290df..8d8a7ec418ce 100644 --- a/src/BloomBrowserUI/bookEdit/js/canvasElementManager/CanvasElementContextControls.tsx +++ b/src/BloomBrowserUI/bookEdit/js/canvasElementManager/CanvasElementContextControls.tsx @@ -35,6 +35,7 @@ import { getMenuSections, getToolbarItems, } from "../../toolbox/canvas/canvasControlResolution"; +import { getLanguageDisplayName } from "../../editableControls/editableControlsManager"; interface IMenuItemWithSubmenu extends ILocalizableMenuItemProps { subMenu?: ILocalizableMenuItemProps[]; @@ -63,7 +64,12 @@ const CanvasElementContextControls: React.FunctionComponent<{ const editable = props.canvasElement.getElementsByClassName( "bloom-editable bloom-visibility-code-on", )[0] as HTMLElement | undefined; - const langName = editable?.getAttribute("data-languagetipcontent"); + // Formerly read from a data-languageTipContent attribute set by the old + // AddLanguageTags. That mechanism is gone (the corner tag is now a React + // control); compute the same display name from the editable's lang here. + const langName = getLanguageDisplayName( + editable?.getAttribute("lang") ?? null, + ); const setMenuOpen = (open: boolean, launchingDialog?: boolean) => { // Even though we've done our best to tell the MUI menu NOT to steal focus, it seems it still does... // or some other code somewhere is doing it when we choose a menu item. So we tell the CanvasElementManager diff --git a/src/BloomBrowserUI/bookEdit/js/keymanWebIntegration.spec.ts b/src/BloomBrowserUI/bookEdit/js/keymanWebIntegration.spec.ts new file mode 100644 index 000000000000..15edf1072f5b --- /dev/null +++ b/src/BloomBrowserUI/bookEdit/js/keymanWebIntegration.spec.ts @@ -0,0 +1,323 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; + +vi.mock("../../utils/bloomApi", () => ({ + postJsonAsync: vi.fn(), + postBoolean: vi.fn(), + // The keyboarding/oskVisible endpoint doesn't exist yet; simulate that by + // invoking the error callback, which leaves the desired-visible default + // (true) in place. + get: vi.fn((_url, _success, error?: () => void) => error?.()), +})); + +vi.mock("../longPressShared", () => ({ + setKmwAttached: vi.fn(), +})); + +import { postJsonAsync } from "../../utils/bloomApi"; +import { setKmwAttached } from "../longPressShared"; +import type { attachKeymanWebIfNeeded as AttachFn } from "./keymanWebIntegration"; + +const postJsonAsyncMock = vi.mocked(postJsonAsync); +const setKmwAttachedMock = vi.mocked(setKmwAttached); + +// Fake KeymanWeb engine object. The real keymanweb.js (vendored, not loaded in +// tests) is what defines window.keyman when it finishes loading; we simulate +// that by installing this fake on window before letting the mocked