From bd7d6fe907d6b2be224ab0d25d4c4b7acc97ba53 Mon Sep 17 00:00:00 2001 From: Rob Boerman Date: Thu, 24 Sep 2026 15:02:06 +0200 Subject: [PATCH 1/5] fix(admin): measure the panel in columns, not code units (#401) Every width promise in the admin panel was a UTF-16 code-unit count. `clip`, `pad`, `styler.cell`, `divider`, `clipPlain`, `visibleLen` and both frame builders' top rules all sized themselves with `.length`, so a framed pane this module reported as exactly 80 columns came out at 90 in the terminal for a CJK job id, and ragged the other way for a combining mark. Measured against pi-tui's own `visibleWidth` at the pin, as this module counted it against what a terminal draws: a CJK job id 5 against 10, fullwidth latin 3 against 6, Hangul 5 against 10, combining marks 5 against 3. ONE TABLE, IN `panel.mjs`, AND THE STYLER COMES TO IT. Importing pi-tui's `visibleWidth` into `style.mjs` was the obvious alternative, and that module is overlay-only and already depends on pi. It was rejected: `panel.mjs` owns `clip` and the monochrome renderer, it is pinned to have no imports, and the two renderers draw the same geometry. A width rule that holds in the framed pane and not in the plain one is the shape that holds on one branch of an `if`, which issue #403 had just been about. That shape then turned up inside this fix. With only `frame`'s top rule pinned, a mutation reverting `box`'s top rule to `.length` SURVIVED the whole suite, so the monochrome pane is pinned too and both rules are mutation-checked. What the table counts, each line the smallest thing that gets the measured cases right: a combining mark (Mn/Me) is 0, a zero-width or formatting character is 0, an East Asian Wide or Fullwidth character is 2, everything else is 1 including an unassigned code point, because guessing wider on an unknown is how such a rule starts breaking the panes it was added to fix. Mc is deliberately not zero: a spacing mark does occupy a column. Its limit, pinned as a disagreement rather than left to surprise a reader: the table sums CODE POINTS, so an emoji ZWJ sequence counts every member where pi collapses it to one glyph, 6 against 2. Terminals disagree with each other there too. Over-counting cuts early, so the residual is a short line, never a broken border. The oracle is pi itself, loaded from the pinned installation per CONST-PI-VERSION-PINNED, and asserted to be a function rather than skipped on failure: a sibling pin made exactly that silent-fallback mistake once, and an oracle that quietly vanishes stops being one. Also corrected here, because both were the same false claim in prose: two comments still said that post-strip `.length` is a safe column proxy because the content is ASCII. It is not. A repository name, a job id and a branch name carry whatever the forge accepts. `meter`'s label keeps `.length` and now says why: it is two integers and a word from a closed set, so no caller can put anything else in it. Ten mutations checked, every one red: `columnsOf` back to `.length`; a wide character counting as one; a combining mark counting as a column; a zero-width character counting as a column; `sliceColumns` cutting after the budget rather than before it; `sliceColumns` counting every character as one; `sliceColumns` slicing by code units; `box`'s top rule by `.length`; `frame`'s top rule by `.length`; `visibleLen` back to a code-unit count; and `pad` filling by `.length`. DES-ADMIN-VIA-PI-EXTENSION AMENDED: the entry's own sentence saying column width for non-ASCII is NOT promised is what changed, and the two other places stating it are corrected with it. DES-PANEL-SEPARATE-FROM-RECEIVER UNCHANGED, checked: nothing about where the panel runs or what it binds moves. The bidi and zero-width residual is UNCHANGED and still open: those code points sit outside a control-byte class defined by what a terminal interprets, and they are a reader-deception question rather than a width one. No version moved. Signed-off-by: Rob Boerman --- admin/src/panel.mjs | 110 +++++++++++++++++++++++++++++--- admin/src/style.mjs | 54 +++++++++++----- admin/test/width.test.mjs | 128 ++++++++++++++++++++++++++++++++++++++ specs/design.md | 26 +++++--- 4 files changed, 284 insertions(+), 34 deletions(-) create mode 100644 admin/test/width.test.mjs diff --git a/admin/src/panel.mjs b/admin/src/panel.mjs index 3f2dec5..b6192cf 100644 --- a/admin/src/panel.mjs +++ b/admin/src/panel.mjs @@ -173,18 +173,100 @@ export function clipData(line, w) { return clip(scrubControls(line), w); } +/** + * THE COLUMN COUNT OF A STRING, which is not its `.length` (issue #401). + * + * Every width promise in this panel was a UTF-16 code-unit count: `clip`, `pad`, `styler.cell`, `divider` + * and `frame` all sized themselves by `.length`. Measured against pi-tui 0.80.7's own `visibleWidth` on a + * framed render this module reported as exactly 80 columns: + * + * CJK job id `ジョブ番号` 80 by .length 90 in the terminal + * fullwidth `ＪＯＢ` 80 86 + * Hangul 3 6 + * combining marks `jób́` 80 78 + * + * So a run whose target is a CJK repository name drew a frame whose right border sat ten columns past the + * one above it, and a combining mark left it ragged the other way. + * + * ONE RULE, NOT TWO. The obvious alternative was to import pi-tui's `visibleWidth` in `style.mjs`, which is + * overlay-only and already depends on pi. It was rejected: `panel.mjs` owns `clip` and the monochrome + * renderer, it is pinned to have NO imports, and the two renderers draw the same geometry -- a width rule + * that holds in the framed pane and not in the plain one is the "holds on one branch of an if" shape three + * issues in this round have now been about. So the table lives here and the styler comes to it. + * + * WHAT IT COUNTS, and each line is the smallest thing that gets the measured cases right: + * + * - a combining mark (Mn/Me) is ZERO, because it draws on the character before it; + * - a zero-width or formatting character is ZERO (ZWSP, ZWNJ, ZWJ, the bidi marks, BOM, word joiner); + * - an East Asian Wide or Fullwidth character is TWO; + * - everything else is ONE, including an unassigned code point, because guessing wider on an unknown is + * how a rule starts breaking the panes it was added to fix. + * + * ITS LIMIT, stated rather than implied: this sums CODE POINTS, so an emoji ZWJ sequence -- a family, a + * flag, a skin tone -- counts every member and comes out wider than the single glyph a terminal draws. + * pi-tui disagrees with us there too (it reports 2 where we report 8), and terminals disagree with each + * other, which is why no width table in this project will settle it. A grapheme-aware count needs + * `Intl.Segmenter` and a terminal that agrees; the residual is one over-wide line, never a broken border, + * because over-counting cuts early rather than late. + */ +export function columnsOf(s) { + let n = 0; + for (const ch of String(s ?? "")) { + const cp = ch.codePointAt(0); + if (ZERO_WIDTH.test(ch) || COMBINING.test(ch)) continue; + n += WIDE.test(ch) || (cp >= 0x1f300 && cp <= 0x1f9ff) ? 2 : 1; + } + return n; +} + +// Mn and Me: a mark that draws on the character before it. `\p{M}` would also take Mc (spacing marks), which +// DO occupy a column in the Indic scripts that use them. +const COMBINING = /\p{Mn}|\p{Me}/u; +// Format and zero-width characters, which occupy none: ZWSP/ZWNJ/ZWJ, the bidi marks and isolates, the word +// joiner and invisible operators, the BOM, and the variation selectors. +const ZERO_WIDTH = /[\u200b-\u200f\u2028\u2029\u202a-\u202e\u2060-\u2064\u2066-\u2069\ufeff\ufe00-\ufe0f]/u; +// East Asian Wide and Fullwidth, from UAX #11: CJK and its punctuation, Hangul, Kana, the fullwidth forms. +const WIDE = + /[\u1100-\u115f\u2e80-\u303e\u3041-\u33ff\u3400-\u4dbf\u4e00-\u9fff\ua000-\ua4cf\uac00-\ud7a3\uf900-\ufaff\ufe10-\ufe19\ufe30-\ufe6f\uff00-\uff60\uffe0-\uffe6]/u; + /** * Truncate `line` to `w` display columns, appending an ellipsis glyph when content is cut. Control - * characters (including escape sequences) are stripped first so untrusted input cannot crash or mis-size: - * content is ASCII/box-drawing, so post-strip `String.length` is a safe column proxy. + * characters (including escape sequences) are stripped first so untrusted input cannot crash or mis-size. + * + * The cut is by COLUMNS, through `sliceColumns`, and this docblock used to claim instead that "content is + * ASCII/box-drawing, so post-strip `String.length` is a safe column proxy". It is not (issue #401): the + * content includes repository names, job ids and branch names, and a forge accepts whatever the forge + * accepts. */ export function clip(line, w) { const width = Math.max(0, Math.trunc(w) || 0); const clean = stripControls(line); - if (clean.length <= width) return clean; + if (columnsOf(clean) <= width) return clean; const ell = active.ellipsis; - if (width <= ell.length) return ell.slice(0, width); - return dropLoneSurrogate(clean.slice(0, width - ell.length)) + ell; + if (width <= columnsOf(ell)) return sliceColumns(ell, width); + return sliceColumns(clean, width - columnsOf(ell)) + ell; +} + +/** + * The first `w` COLUMNS of a string, never half of a wide character. + * + * A cut by code units splits an astral pair (`dropLoneSurrogate`'s problem) and, once widths are counted, + * can also stop in the middle of a two-column character -- which a terminal renders as one column of + * nothing and one column of overflow, so the line is both wrong and ragged. Walking code points and + * stopping BEFORE the budget is passed means the result is always at most `w` columns and always a whole + * character, and it makes `dropLoneSurrogate` unnecessary on this path: a surrogate pair is one step here. + */ +export function sliceColumns(s, w) { + const budget = Math.max(0, Math.trunc(w) || 0); + let out = ""; + let used = 0; + for (const ch of String(s ?? "")) { + const cols = columnsOf(ch); + if (used + cols > budget) break; + out += ch; + used += cols; + } + return out; } /** @@ -198,10 +280,11 @@ export function dropLoneSurrogate(s) { return last >= 0xd800 && last <= 0xdbff ? s.slice(0, -1) : s; } -/** `clip` to `w`, then right-pad with spaces to exactly `w` columns. */ +/** `clip` to `w`, then right-pad with spaces to exactly `w` COLUMNS (issue #401: not `.length`). */ export function pad(line, w) { const width = Math.max(0, Math.trunc(w) || 0); - return clip(line, width).padEnd(width); + const cut = clip(line, width); + return cut + " ".repeat(Math.max(0, width - columnsOf(cut))); } /** @@ -219,7 +302,11 @@ export function box({ title = "", sections = [], footer, width = 40 } = {}) { // `clipData`, not `clip`: `frame` substitutes in its own title (`clipPlain`), and a title that DELETES // here clipped one column narrower than the coloured twin for the same string. const titleText = title ? ` ${clipData(title, Math.max(0, inner - 2))} ` : ""; - const topFill = Math.max(0, w - 2 - 1 - titleText.length); // corners + one leading `h` + // COLUMNS, not code units (issue #401), the same repair as `frame`'s top rule and for the same reason: a + // CJK title made this rule over-fill by the title's own width, so the pane's FIRST line was wider than + // every line under it. `clipData` has already substituted, so `titleText` is plain and `columnsOf` is the + // whole measurement. + const topFill = Math.max(0, w - 2 - 1 - columnsOf(titleText)); // corners + one leading `h` lines.push(active.tl + active.h + titleText + active.h.repeat(topFill) + active.tr); const framed = (text) => `${active.v} ${pad(text, inner)} ${active.v}`; @@ -247,7 +334,12 @@ export function box({ title = "", sections = [], footer, width = 40 } = {}) { * * `state` ("ok" | "soft-hold" | "over") appends a textual marker to the label: the panel is monochrome and * `clip` strips ANSI, so the amber/red of a soft-hold or over-budget window is carried as a word, not a - * color. "ok" (the default) adds nothing, so a plain call renders exactly as before. + * color. + * + * ITS LABEL IS MEASURED BY `.length` AND THAT IS CORRECT HERE, which is worth saying in a file whose whole + * subject is that `.length` is not a column count (issue #401). Every character of the label comes from two + * integers and a word out of a closed set, so it is digits and ASCII by construction and no caller can put + * anything else in it. `styler.meter`'s label is built the same way and is exempt for the same reason. "ok" (the default) adds nothing, so a plain call renders exactly as before. */ export function meter(reserved, cap, width = 24, state = "ok") { const r = Number.isFinite(reserved) ? Math.max(0, Math.trunc(reserved)) : 0; diff --git a/admin/src/style.mjs b/admin/src/style.mjs index 5785936..0fdd917 100644 --- a/admin/src/style.mjs +++ b/admin/src/style.mjs @@ -17,19 +17,27 @@ * can assert both the plain content and the width math without a real terminal. */ -import { LINE_INPUT_CURSOR, dropLoneSurrogate, fmtCost as plainFmtCost, scrubControls, scrubKeepingStyle, sparkline as plainSparkline } from "./panel.mjs"; +import { LINE_INPUT_CURSOR, columnsOf, dropLoneSurrogate, fmtCost as plainFmtCost, scrubControls, scrubKeepingStyle, sliceColumns, sparkline as plainSparkline } from "./panel.mjs"; -// Strip SGR (and OSC-8 hyperlink) escapes to recover the visible text / column count. Content is -// ASCII + box-drawing + a handful of width-1 glyphs, so post-strip `.length` is a safe column proxy. +// Strip SGR (and OSC-8 hyperlink) escapes to recover the visible text, which is then measured in COLUMNS. +// This comment used to end "so post-strip `.length` is a safe column proxy", and issue #401 is what that +// sentence cost: content is not ASCII, a repository name or a job id carries whatever the forge allows, and +// a CJK one measured 80 here while the terminal drew 90. const ANSI = /\x1b\[[0-9;]*m|\x1b\]8;;[^\x07]*\x07/g; export function stripAnsi(s) { return String(s ?? "").replace(ANSI, ""); } -/** Visible column count of a (possibly colored) string. */ +/** + * Visible COLUMN count of a (possibly colored) string -- not its code-unit length (issue #401). + * + * Through `panel.mjs`'s table, so the framed pane and the monochrome one measure the same string the same + * way. They draw the same geometry, and a width rule that holds in one and not the other is how a frame + * ends up ten columns wider than the line above it. + */ export function visibleLen(s) { - return stripAnsi(s).length; + return columnsOf(stripAnsi(s)); } /** A no-op theme: `fg`/`bg`/`bold`/… return the text unchanged. Used in tests and when no TUI theme exists. */ @@ -100,8 +108,19 @@ export function makeStyler(theme, { ascii = false } = {}) { // astral character, and half a pair is not a character. Measured at 89 lines of a framed LIST printing // one, from a target field of emoji -- the first repair reached `clip` alone and three other cutters // slice the same way. - if (plain.length > w) plain = w <= G.ellipsis.length ? dropLoneSurrogate(plain.slice(0, w)) : dropLoneSurrogate(plain.slice(0, w - G.ellipsis.length)) + G.ellipsis; - plain = align === "right" ? plain.padStart(w) : plain.padEnd(w); + // BY COLUMNS, not by code units (issue #401), and through `sliceColumns` so a cut never lands inside a + // two-column character -- which a terminal draws as one blank column plus one of overflow. + if (columnsOf(plain) > w) { + const ell = G.ellipsis; + // The narrow branch slices the CONTENT, not the ellipsis, which is what this line did before #401 and + // what `clipPlain` below still does. `panel.mjs`'s `clip` shows the ellipsis instead at such a width. + // That disagreement is pre-existing and left alone here: the three cutters differ only where the + // budget is narrower than the ellipsis glyph itself, and changing which one is right is a question + // about what to show, not about how wide it is. + plain = w <= columnsOf(ell) ? sliceColumns(plain, w) : sliceColumns(plain, w - columnsOf(ell)) + ell; + } + const gap = " ".repeat(Math.max(0, w - columnsOf(plain))); + plain = align === "right" ? gap + plain : plain + gap; let out = color ? fg(color, plain) : plain; return strong ? bold(out) : out; }; @@ -154,9 +173,9 @@ export function makeStyler(theme, { ascii = false } = {}) { // without: clip the META first, and the LABEL only if it alone still does not fit. Getting this wrong by // one is why the clamp existed in the first place -- `Math.max(1, ...)` hid the overflow instead of // preventing it, and the line ran over its own width. - met = dropLoneSurrogate(met.slice(0, Math.max(0, w - lab.length - 3))); - const labClipped = dropLoneSurrogate(lab.slice(0, Math.max(0, w - met.length - (met ? 3 : 2)))); - const ruleLen = Math.max(1, w - labClipped.length - met.length - (met ? 2 : 1)); + met = sliceColumns(met, Math.max(0, w - columnsOf(lab) - 3)); + const labClipped = sliceColumns(lab, Math.max(0, w - columnsOf(met) - (met ? 3 : 2))); + const ruleLen = Math.max(1, w - columnsOf(labClipped) - columnsOf(met) - (met ? 2 : 1)); const labPart = labClipped ? bold(fg("muted", labClipped)) + " " : ""; const rulePart = fg("border", G.h.repeat(ruleLen)); const metPart = met ? " " + fg("dim", met) : ""; @@ -234,7 +253,12 @@ export function frame(styler, { title = "", width = 40, lines = [], footer = nul const out = []; const titleText = title ? ` ${clipPlain(title, Math.max(0, inner - 2), G.ellipsis)} ` : ""; - const topFill = Math.max(0, w - 2 - 1 - titleText.length); + // THE TOP RULE IS FILLED IN COLUMNS, not code units (issue #401). A CJK title is half as many code units + // as the terminal draws columns, so `.length` here over-filled the rule by the title's own width: a + // 20-column pane came out 23 wide on its FIRST line only, with every body line correct, which is the + // shape that hides such a bug. `titleText` is plain by construction (`clipPlain` strips), so the plain + // column count is the whole measurement and no ANSI-aware pass is needed. + const topFill = Math.max(0, w - 2 - 1 - styler.visibleLen(titleText)); out.push(B(G.tl + G.h) + styler.bold(styler.fg("accent", titleText)) + B(G.h.repeat(topFill) + G.tr)); const side = (content) => B(G.v) + " " + content + " " + B(G.v); @@ -283,11 +307,11 @@ function padVisible(styler, line, width) { * caller left the other carrying what the lines inside the frame no longer did. `panel.mjs`'s own `box` * already titles through the same operation now, so the two frame builders agree on the CLASS -- which * lives in `panel.mjs` and is imported, not respelled here (issue #382) -- AND on what to do with a match. - * Both SUBSTITUTE a space, because `frame` computes its top rule from the title's length at the call site - * and a deleting strip would silently change that arithmetic. + * Both SUBSTITUTE a space, because `frame` computes its top rule from the title's own width at the call + * site and a deleting strip would silently change that arithmetic. */ function clipPlain(s, width, ellipsis = "…") { const plain = scrubControls(s); - if (plain.length <= width) return plain; - return width <= ellipsis.length ? dropLoneSurrogate(plain.slice(0, width)) : dropLoneSurrogate(plain.slice(0, width - ellipsis.length)) + ellipsis; + if (columnsOf(plain) <= width) return plain; + return width <= columnsOf(ellipsis) ? sliceColumns(plain, width) : sliceColumns(plain, width - columnsOf(ellipsis)) + ellipsis; } diff --git a/admin/test/width.test.mjs b/admin/test/width.test.mjs new file mode 100644 index 0000000..9c1406e --- /dev/null +++ b/admin/test/width.test.mjs @@ -0,0 +1,128 @@ +import assert from "node:assert/strict"; +import { test } from "node:test"; +import { createRequire } from "node:module"; +import { pathToFileURL } from "node:url"; +import { box, clip, columnsOf, pad, sliceColumns } from "../src/panel.mjs"; +import { frame, makeStyler, PLAIN_THEME, visibleLen } from "../src/style.mjs"; + +// EVERY WIDTH PROMISE IN THIS PANEL WAS A UTF-16 COUNT (issue #401), and this file is where the repair is +// held against the only authority available: pi's own renderer, which is what actually draws these lines. +// +// `clip`, `pad`, `styler.cell`, `divider` and `frame` all sized themselves with `.length`, so a framed pane +// this module reported as exactly 80 columns came out at 90 in the terminal for a CJK job id, and ragged +// the other way for a combining mark. The table lives in `panel.mjs` rather than in the overlay-only +// `style.mjs`, because the two renderers draw the same geometry and a width rule that holds in one and not +// the other is the "holds on one branch of an if" shape this round kept finding. + +/** + * pi-tui's own `visibleWidth`, the oracle. + * + * Resolved from the pinned pi installation rather than declared as a dependency: `admin` does not depend on + * pi-tui directly, it is nested under `pi-coding-agent`, and `CONST-PI-VERSION-PINNED` says to verify + * against the pinned artifact rather than a range. A hard-coded nested path would break on a flat install. + * + * THE TWO RESOLVERS ARE BOTH NEEDED, and each fails where the other works: + * - `import.meta.resolve` finds pi itself. The CJS `require.resolve` does NOT: pi's export map carries no + * `require` condition, so it throws ERR_PACKAGE_PATH_NOT_EXPORTED. This is why `dashboard.test.mjs:14` + * builds its own `createRequire` from `import.meta.resolve` rather than from a package.json URL. + * - `require.resolve` then finds pi-tui NESTED under pi. `import.meta.resolve` with pi's entry as the + * parent does not, because the ESM resolver reads pi's own dependency graph rather than walking + * `node_modules` upward, and pi-tui is not one of `admin`'s dependencies. + * It returns the file PATH, and pi-tui is ESM, so the path is imported rather than required. + */ +async function loadVisibleWidth() { + try { + const pi = createRequire(import.meta.resolve("@earendil-works/pi-coding-agent")); + const { visibleWidth } = await import(pathToFileURL(pi.resolve("@earendil-works/pi-tui")).href); + return typeof visibleWidth === "function" ? visibleWidth : null; + } catch { + return null; + } +} + +/** `[name, string, what pi and we must agree it is]` -- the cases a review pass measured as broken. */ +const AGREE = [ + ["ascii", "hello world", 11], + ["a CJK job id", "ジョブ番号", 10], + ["a CJK repo target", "会社/製品#5", 11], + ["fullwidth latin", "ＪＯＢ", 6], + ["hangul", "한글테스트", 10], + ["combining marks", "jób́", 3], + ["a zero-width space", "a​b​c", 3], + ["a bidi override", "a‮b", 2], + ["box drawing", "┌─┐", 3], + ["one emoji", "\u{1f600}", 2], +]; + +test("the column count agrees with pi's own renderer on everything but a ZWJ sequence (#401)", async () => { + const visibleWidth = await loadVisibleWidth(); + // NOT SKIPPED SILENTLY: an oracle that quietly vanishes is an oracle that stops being one, and this + // module's sibling pin (`keys.mjs`) made exactly that mistake once. + assert.equal(typeof visibleWidth, "function", "pi-tui's visibleWidth must load, or this test is checking nothing"); + + for (const [name, s, expected] of AGREE) { + assert.equal(columnsOf(s), expected, `${name}: our count`); + assert.equal(visibleWidth(s), expected, `${name}: pi's count`); + } + + // THE ONE DISAGREEMENT, asserted so it cannot drift into a surprise. We sum CODE POINTS, so an emoji ZWJ + // sequence counts every member; pi collapses it to one glyph. Terminals disagree with each other here + // too, which is why no table in this project settles it. Over-counting cuts EARLY, so the cost is a short + // line rather than a broken border -- which is the direction to be wrong in. + const family = "\u{1f468}‍\u{1f469}‍\u{1f466}"; + assert.equal(columnsOf(family), 6, "we count each member of the family"); + assert.equal(visibleWidth(family), 2, "pi counts the glyph -- recorded, not fixed"); +}); + +test("a cut never lands inside a wide character, and never leaves half a pair (#401)", () => { + // Cutting by code units splits an astral pair; cutting by columns without walking characters splits a + // TWO-COLUMN one, which a terminal draws as one blank column plus one of overflow -- wrong and ragged. + const lone = /[\ud800-\udbff](?![\udc00-\udfff])|(?= w) assert.ok(columnsOf(cut) >= w - 1, `width ${w}: a two-column character may cost one column, never more`); + } + } +}); + +test("clip and pad measure and fill in columns, not code units (#401)", () => { + const cjk = "ジョブ番号"; // 5 characters, 10 columns + assert.equal(columnsOf(clip(cjk, 10)), 10, "a string that fits is untouched"); + assert.ok(columnsOf(clip(cjk, 6)) <= 6, "and one that does not is cut to the budget"); + for (const w of [1, 2, 3, 7, 11, 20]) { + assert.equal(columnsOf(pad(cjk, w)), w, `pad to exactly ${w} columns`); + } + // The old bug, as an assertion: by `.length` this string is 5, so a 10-column pad added five spaces and + // the line came out 15 columns wide. + assert.notEqual(pad(cjk, 10).length, 10, "the code-unit length is NOT the column count, which is the point"); +}); + +test("the monochrome pane measures exactly its width too, title row included (#401)", () => { + // BOTH RENDERERS, because the mutation that survived while only `frame` was pinned was `box`'s own top + // rule reverted to `.length`. `box` and `frame` draw the same geometry from two different files, and this + // round keeps finding the shape where a rule holds on one branch and not the other. The overshoot lands on + // the FIRST line only, with every body row correct, which is what made it easy to miss by eye. + const sections = [{ title: "ジョブ", lines: ["会社/製品#5 のジョブ", "plain ascii row", "jób́ marks"] }]; + for (const w of [24, 40, 80]) { + for (const line of box({ title: "ジョブ番号", sections, footer: "Ｆ", width: w })) { + assert.equal(columnsOf(line), w, `width ${w}: ${JSON.stringify(line)}`); + } + } +}); + +test("a frame holding CJK still measures exactly its width, in our terms and pi's (#401)", async () => { + const visibleWidth = await loadVisibleWidth(); + assert.equal(typeof visibleWidth, "function"); + const styler = makeStyler(PLAIN_THEME); + const lines = ["会社/製品#5 のジョブ", "plain ascii row", "jób́ with combining marks"]; + for (const w of [20, 40, 80]) { + for (const line of frame(styler, { title: "ジョブ", width: w, lines: lines.map((l) => styler.cell(l, w - 4)), footer: "Ｆ" })) { + assert.equal(visibleLen(line), w, `width ${w}: our own measure`); + assert.equal(visibleWidth(line), w, `width ${w}: and pi's, which is the one the terminal uses`); + } + } +}); diff --git a/specs/design.md b/specs/design.md index 58366c0..2fe8b3d 100644 --- a/specs/design.md +++ b/specs/design.md @@ -1584,9 +1584,9 @@ money with no upstream turn limit (`REQ-RUNNER-TURN-BUDGET`). self-refreshing TUI overlay component with **four in-component views**: **LIST** — a framed, **theme-colored** panel (color via pi's injected `Theme`, applied post-layout; CORRECTED under issue #382, which found five sites handing `styler.cell` PRE-coloured text that it then measured with `.length`, so - colour is post-layout by construction now rather than by convention. Even so, pi's `visibleWidth` frames - it only for ASCII: this module counts UTF-16 units, so a CJK or combining character still frames ragged, - which is issue #401) carrying a status header, day/week/month **SPEND meters** (colored by the + colour is post-layout by construction now rather than by convention; and the width is a COLUMN count under + issue #401, so pi's `visibleWidth` frames it for CJK, fullwidth, Hangul and combining characters too, not + only for ASCII) carrying a status header, day/week/month **SPEND meters** (colored by the same `windowState` the worker enforces) plus a daily **token** counter, a unified **TRIGGERS** pane whose `{on, run}` rows are **selectable and editable** (a cron row carries an amber `⚠ overdue`/`⚠ stalled` badge joined from its resident scheduler — health is a LIST-level fact, not only a drill-in one), and an @@ -1778,13 +1778,18 @@ money with no upstream turn limit (`REQ-RUNNER-TURN-BUDGET`). `[A-Za-z0-9._-]+` before the upsert and a staged name is checked against `NPM_NAME_RE` at stage time -- so they are reachable only through a writer that is not the worker, or a hand-edited file. - **What the gate does NOT promise**: column width for non-ASCII. `visibleLen` counts UTF-16 units after - stripping ANSI, so a CJK or fullwidth character measures 1 where a terminal draws 2, and a combining - mark measures 1 where it draws 0. A framed pane holding one is ragged, measured against pi-tui's own - `visibleWidth` (80 by this module's count, 90 by pi's). That is older than this entry and untouched by - it, it needs a width table this dependency-free module deliberately does not carry, and it is its own - issue. Nor does the class cover bidi, zero-width or line-separator code points, which a terminal does - not INTERPRET but a reader can still be misled by; also its own issue. + **Column width for non-ASCII IS promised now** (issue #401), where this entry previously said it was + not. `panel.mjs` carries one width table -- a combining mark or a zero-width character is 0, an East + Asian Wide or Fullwidth one is 2, everything else is 1 -- and every cutter, padder and border in BOTH + renderers measures through it. It is pinned against pi-tui's own `visibleWidth` at the pin, which is the + renderer that actually draws these lines. **Its one stated limit**: the table sums CODE POINTS, so an + emoji ZWJ sequence counts every member and comes out wider than the single glyph a terminal draws (6 + against pi's 2, pinned as a disagreement rather than left to surprise someone). Over-counting cuts + early, so the residual is a short line, never a broken border. + + **What the gate still does NOT cover**: bidi, zero-width and line-separator code points, which a + terminal does not INTERPRET -- so they are outside a class defined by what a terminal interprets -- but + which a reader can still be misled by. That is its own issue. **NO WORKER VALIDATOR CHANGES with it, and that is the interesting half.** Tightening what the worker accepts in `on.disarmed` was written and rejected: the writer re-runs `parseTriggers`, a refusal returns @@ -4617,3 +4622,4 @@ a tunnel. | 2026-09-23 | Issue #386. **`DES-WATCHERS-CLOSE-WITH-THE-WORKER` AMENDED** with the other end of a watch's life: it recorded how a watch STOPS and said nothing about the window before it STARTS. All four live-edit watches in this project -- the worker's triggers, pause-windows and scoped-limits, and the receiver's triggers -- read their file at boot and armed afterwards, so an edit in that gap was never loaded until the next edit or a restart. The receiver's gap is the widest, its watch being the last fallible step of a boot that can retry identity for `RECEIVER_IDENTITY_RETRY_SECONDS`. **The macOS half is a LOST edit rather than a late one**, and that is what made this worth product code rather than a test ceiling: libuv recreates the process's single FSEvents stream from "now" when any watcher is armed or closed anywhere in the process, so an edit before the new stream is live is never delivered (measured against the real receiver boot, 40 trials per posture: 24 lost with four boots in one process, 1 beside a loaded machine, 0 idle -- a re-measure on another machine reproduced the shape and not the rate, at 1, 0 and 0, so read the 24 as one host's worst case; Linux's inotify registers before `fs.watch` returns, so there it is late but not lost). The repair is one read after arming compared against the bytes THE BOOT LOADER READ, reloading only when they moved -- so a quiet boot pays one file read and says nothing. Where the baseline comes from is the load-bearing part rather than a detail: a first version of this change read it where the watch arms, which closed 0.1 to 0.4 milliseconds while leaving the identity-retry window it is named for wide open, proven end to end on both services by a review pass. Both loaders already take a read seam, so the baseline is exactly what the service is running. Its residual is stated in the entry rather than implied: the FSEvents window extends past that read too. `WATCH_DEBOUNCE_MS` moves into the same module in the same change, the 150ms literal having been written out four times. **`DES-CRON-VIA-BULLMQ-SCHEDULER` UNCHANGED, checked**: the reconcile and its fingerprint gate are untouched, and the extra reconcile only runs on a boot that lost the race. **Code evidence**: `worker/src/watch-closer.mjs` -> `readBeforeArming`, `changedWhileArming`, `WATCH_DEBOUNCE_MS`. | | 2026-09-24 | Issue #404. **`DES-ADMIN-VIA-PI-EXTENSION` AMENDED**: the control-byte gate names FOUR funnels, not three. #382 closed the two that render a pane and the model-visible `send`, and its own wording -- "every finished line passes one gate" -- read as complete while pi's DIALOGS took strings straight from stored fields. Measured: a pause window whose `scope` carries an erase-display, an OSC-52 clipboard write and an OSC-8 link reaches a select option, an input prompt and a confirm body. (An earlier draft of this row said "with the TUI suspended" and named an input DEFAULT: neither is true at the pin -- these dialogs are drawn by the LIVE tui, and `ExtensionInputComponent` accepts a placeholder and never renders it.) `pause-windows.mjs` checks only `isNonEmptyString(w.scope)`, so the FILE is the whole validator; `secrets-command.ts` has the same shape. Not a regression -- it behaved identically before #382 -- but it IS model-reachable, which the first draft of this row denied: a tool's `execute` gets its own `ctx` from pi, never passing the command door, and `dispatch_trigger_edit`'s confirm body interpolates the model's own `flow`. The gate is one wrapper per DOOR -- six of them: the command handler, the exported dashboard action, `confirmedWrite` for every tool that confirms, the secrets command, the setup wizard and the `session_start` handler (the sixth found by a second review pass, its own notify a constant -- gated rather than excused, because a door left out for being currently safe is how the render gates were refuted) -- not at the 75 call sites: more than one is there because no single one is the only door, and a test driving the dashboard action directly would otherwise exercise the dialogs ungated -- the same covered-on-one-path shape the render gates were refuted for. `notify` is included, because an error message that moves the cursor is the same class as any other. `custom` is not, because it takes a factory whose output is the overlay. **AND THE SELECTION IS TRANSLATED BACK**: pi returns the exact option string it was handed, so a scrubbed copy made `labels.indexOf(picked)` fail and four edit/delete actions returned silently -- a gate that introduced the silent no-op it exists to prevent, caught by a review pass and pinned. **Code evidence**: `admin/src/dialog-gate.mjs` -> `gateDialogs`. | | 2026-09-24 | Issue #403. **`DES-ADMIN-VIA-PI-EXTENSION` AMENDED**: the unframed degrade honours the width it was given. It was a layout promise that held on one branch of an `if` -- the run rows are bounded to their own fixed 24-column pane, everything else was returned as built (the live tail's unframed branch had no clip at all, and said so) -- so a single render showed rows cut to the pane beside section headers, settings lines and hints running past it (measured: 28 of 59 lines over the width at `render(4)`, the widest 52 columns). Same shape as the logs viewer's #399, which this file had already fixed. **A width that is MISSING or non-finite is left alone**, deliberately: the degrade is what those get too, and `clip(x, NaN)` returns the empty string, so inventing a number there would blank the pane rather than degrade it. **The clip is by LENGTH and that rests on a measured assumption**, now pinned: this path is monochrome, zero of its lines carrying an SGR sequence under a real theme, because its own branches strip colour before returning. If that changes the result is worse than lost colour -- `clipData` substitutes the ESC and leaves `[31m` VISIBLE, charging four phantom columns against the width -- and the answer is an ANSI-aware cut. The width remains a UTF-16 count, wrong for CJK exactly as it is everywhere else in this module, which is #401 and is not made worse here. The TRUNCATION is centralised beside the comparison, not only the comparison: a review pass showed that splitting them is invisible, since `Math.round(7.9)` frames nothing and then clips to 8. **Code evidence**: `admin/src/dashboard.ts` -> `renderPanel`, `framedAt`, `degradeWidth`. | +| 2026-09-24 | Issue #401. **`DES-ADMIN-VIA-PI-EXTENSION` AMENDED**: the panel measures COLUMNS, and the entry's own sentence saying it does not is what changed. Every width promise in this module was a UTF-16 code-unit count -- `clip`, `pad`, `styler.cell`, `divider`, `clipPlain`, `visibleLen`, and both frame builders' top rules -- so a CJK job id framed a pane this module reported as exactly 80 columns and pi-tui drew at 90, and a combining mark ragged it the other way. Measured per case against pi-tui's own `visibleWidth` at the pin, as this module counted it against what a terminal draws: a CJK job id 5 against 10, fullwidth latin 3 against 6, Hangul 5 against 10, combining marks 5 against 3. **ONE TABLE, IN `panel.mjs`, AND THE STYLER COMES TO IT.** The obvious alternative was importing pi-tui's `visibleWidth` into `style.mjs`, which is overlay-only and already depends on pi: rejected because `panel.mjs` owns `clip` and the monochrome renderer, is pinned to have NO imports, and the two renderers draw the same geometry. A width rule holding in the framed pane and not in the plain one is the shape that holds on one branch of an `if`, which #403 had just been about -- and a mutation of the MONOCHROME top rule survived a suite pinning only the coloured one, which is that same shape appearing inside this very fix. **WHAT IT COUNTS**: a combining mark (Mn/Me) 0, a zero-width or formatting character 0, East Asian Wide and Fullwidth 2, everything else 1 including an unassigned code point, because guessing wider on an unknown breaks the panes the rule was added to fix. Mc is deliberately NOT zero: a spacing mark does occupy a column. **ITS LIMIT, PINNED AS A DISAGREEMENT rather than left to surprise a reader**: the table sums CODE POINTS, so an emoji ZWJ sequence counts every member where pi collapses it to one glyph, 6 against 2. Terminals disagree with each other there too, which is why no table in this project settles it; over-counting cuts EARLY, so the residual is a short line, never a broken border. **THE ORACLE IS pi ITSELF**, loaded from the pinned installation per `CONST-PI-VERSION-PINNED`, and asserted to be a function rather than skipped on failure: a sibling pin made exactly that silent-fallback mistake once, and an oracle that quietly vanishes stops being one. **`DES-PANEL-SEPARATE-FROM-RECEIVER` UNCHANGED, checked**: nothing about where the panel runs or what it binds moves. **The bidi and zero-width residual is UNCHANGED and still open**: those code points sit outside a control-byte class defined by what a terminal INTERPRETS, and they are a reader-deception question rather than a width one. **Code evidence**: admin/src/panel.mjs -> columnsOf, sliceColumns, clip, pad, box; admin/src/style.mjs -> visibleLen, cell, divider, frame, clipPlain; admin/test/width.test.mjs. | From 1bfa19638d03459c5685f9dbff8e41b7a52abae6 Mon Sep 17 00:00:00 2001 From: Rob Boerman Date: Thu, 24 Sep 2026 16:07:02 +0200 Subject: [PATCH 2/5] fix(admin): transcribe the width table from the renderer, not from the standard (#401) The first commit's table was hand-written out of UAX #11 and checked against ten strings. It passed, and it UNDER-counted 139,820 code points: CJK Extension B through H, Tangut, the Kana supplement, the Hangul Jamo extensions and every emoji block added since Unicode 13. CJK Extension B is exactly the repository name this issue is about, so the fix carried the defect it was fixing, and in two classes it was WORSE than the code-unit count it replaced, because an astral character is two code units and two columns and `.length` had been right there by accident. The renderer draws these panes, so the renderer is the authority. `WIDE` is now its own double-width set, transcribed from the pin; the zero rule is `\p{M}` plus `\p{Cf}` plus the fillers; and one piece of state handles U+FE0F, which asks for the emoji form of the character before it and cannot be answered by a per-character table at all. Mc is no longer excepted. The first version reasoned from Unicode that a spacing mark occupies a column. The renderer that draws the pane says otherwise, and between a standard and the thing actually painting the characters, the painter wins. THE BOLT IS A SWEEP, because ten examples cannot see a 139,820-code-point hole. It walks every code point there is, plus every character-with-U+FE0F pair, and requires that the table never measure NARROWER than the renderer and that every place it measures wider is an unassigned code point. The direction is the whole property, and "over-counting is harmless" would be too kind. Both directions rag a frame, differently. An over-count pads a body line as though it were wider than it is, so the line comes out short and the right border sits left of the one above it, contained by the pane. An under-count runs the line past the pane and past the terminal, where it wraps and takes the border with it. The oracle's VERSION is asserted, not just its presence: pi depends on pi-tui by a range, so a resolve that is not the lockfile's would answer any 0.80.x, and a table pinned to the wrong renderer is not pinned. Repaired with it, as the same defect class rather than as scope creep: - the line editor windowed and padded by code units, so a CJK value measured half what the terminal draws and a window edge put a bare low surrogate into the live trigger editor; - `renderRuns`, the model-visible table behind `/dispatch runs`, sized its columns by `.length`, so one CJK character in a `local:` target shifted every later column of that row; - a cut to zero returned a floating combining mark, and a cut through a ZWJ sequence left the joiner dangling in front of the ellipsis; - the cut carried a lone surrogate the input already held, where the old slice-then-repair had dropped it; - two hostile-string caps in the graph model could split a surrogate pair. Also corrected: three comments and two spec sentences that overclaimed, the docblock measurement table (its numbers were doubled and named two different strings), a dead `dropLoneSurrogate` import with the stale comment above it, and a clause testing `\p{Emoji_Presentation}` that was dead because every code point with that property is already in the wide set. Twenty-five mutations checked, every one red, including the eleven that survived the first commit: seven ranges of the wide set one code point at a time, both halves of the zero rule, the U+FE0F clause and its ASCII exclusion, the lone-surrogate skip, the dangling-joiner strip, the zero-budget guard, all three `divider` measurements, `clipPlain`, `padVisible`'s strict comparison, both halves of the model-visible table, both halves of the line editor, and the graph cap. Full suite in the CI posture: 4206 tests, 0 fail, 1 skipped. Same under the +399 day clock shift. All four guards pass and a run under a fresh TMPDIR leaves nothing behind. DES-ADMIN-VIA-PI-EXTENSION amended again, and two further live sentences that this change made false are corrected with it: the #403 decision's "the clip is by LENGTH", and the LIST entry's account of what the width now frames. No version moved. Signed-off-by: Rob Boerman --- admin/src/dashboard.ts | 4 +- admin/src/graph-model.mjs | 9 +- admin/src/panel.mjs | 209 +++++++++++++++++++++++++------- admin/src/render.mjs | 9 +- admin/src/style.mjs | 15 ++- admin/test/width.test.mjs | 245 ++++++++++++++++++++++++++++++++++++-- specs/design.md | 39 ++++-- 7 files changed, 451 insertions(+), 79 deletions(-) diff --git a/admin/src/dashboard.ts b/admin/src/dashboard.ts index af3d98f..fd29ca1 100644 --- a/admin/src/dashboard.ts +++ b/admin/src/dashboard.ts @@ -1104,8 +1104,8 @@ function renderPanel(snapshot: any, width: number, state: any, styler: any): str // coloured degrade would print `[31m` as text AND charge four phantom columns against the width. The // answer then is an ANSI-aware cut, not this one. // - // The width is still a UTF-16 count, which is wrong for CJK and combining marks exactly as it is - // everywhere else in this module. That is #401 and is not made worse here. + // The width is a COLUMN count under issue #401: `clipData` cuts through the one table in `panel.mjs`, so + // this branch is right about CJK, fullwidth and combining content for the same reason the framed pane is. const w = degradeWidth(width); return w === null ? lines : lines.map((l) => clipData(l, w)); } diff --git a/admin/src/graph-model.mjs b/admin/src/graph-model.mjs index 3c9d3b0..6b36fc4 100644 --- a/admin/src/graph-model.mjs +++ b/admin/src/graph-model.mjs @@ -8,6 +8,9 @@ // SKILL_NAME_RE is a plain frozen RegExp; importing it keeps the charset single-sourced (the // issue #92 lesson) without breaking this module's purity -- nothing here spawns or reads anything. import { SKILL_NAME_RE } from "@edgehero/pi-dispatch/flow-gate"; +// Pure-to-pure: `panel.mjs` is the admin's no-I/O module and owns the one answer to "how do you cut a +// string without leaving half a character behind" (issue #401). +import { dropLoneSurrogate } from "./panel.mjs"; // One frontmatter value line: `key: value`, an optional surrounding double quote, single-line only. // The same block-isolation discipline as flow-gate.mjs's aiTriggerAllows, and deliberately NOT a YAML @@ -41,7 +44,9 @@ function frontmatterValue(block, key) { let value = m[1].trim(); if (value.length >= 2 && value.startsWith('"') && value.endsWith('"')) value = value.slice(1, -1); if (value === "") return null; - return value.length > META_VALUE_MAX_CHARS ? `${value.slice(0, META_VALUE_MAX_CHARS)}…` : value; + // The cap is a CHARACTER cap, not a width, so a code-unit slice is the right shape here -- but it can + // still land between the halves of an astral pair, and half a pair is not a character (issue #401). + return value.length > META_VALUE_MAX_CHARS ? `${dropLoneSurrogate(value.slice(0, META_VALUE_MAX_CHARS))}…` : value; } // A mention is "strong" when it sits near chaining vocabulary -- the outbox protocol's own words. @@ -667,5 +672,5 @@ function basenameOf(path) { /** Clip an arbitrary (possibly hostile) flow string for node display; the honest badge needs the name. */ function clipName(name) { const s = String(name); - return s.length > 64 ? `${s.slice(0, 64)}…` : s; + return s.length > 64 ? `${dropLoneSurrogate(s.slice(0, 64))}…` : s; } diff --git a/admin/src/panel.mjs b/admin/src/panel.mjs index b6192cf..5313b57 100644 --- a/admin/src/panel.mjs +++ b/admin/src/panel.mjs @@ -176,58 +176,131 @@ export function clipData(line, w) { /** * THE COLUMN COUNT OF A STRING, which is not its `.length` (issue #401). * - * Every width promise in this panel was a UTF-16 code-unit count: `clip`, `pad`, `styler.cell`, `divider` - * and `frame` all sized themselves by `.length`. Measured against pi-tui 0.80.7's own `visibleWidth` on a - * framed render this module reported as exactly 80 columns: + * Every width promise in this panel was a UTF-16 code-unit count: `clip`, `pad`, `styler.cell`, `divider`, + * `clipPlain`, `visibleLen`, both frame builders' top rules and the line editor's window all sized + * themselves by `.length`. Measured against the pinned renderer's own `visibleWidth`: * - * CJK job id `ジョブ番号` 80 by .length 90 in the terminal - * fullwidth `ＪＯＢ` 80 86 - * Hangul 3 6 - * combining marks `jób́` 80 78 + * a CJK job id `ジョブ番号` .length 5 drawn 10 + * fullwidth `ＪＯＢ` .length 3 drawn 6 + * Hangul `한글테스트` .length 5 drawn 10 + * combining marks `jób́` .length 5 drawn 3 * - * So a run whose target is a CJK repository name drew a frame whose right border sat ten columns past the - * one above it, and a combining mark left it ragged the other way. + * So a run whose target is a CJK repository name drew a frame whose right border sat past the one above + * it, and a combining mark left it ragged the other way. * - * ONE RULE, NOT TWO. The obvious alternative was to import pi-tui's `visibleWidth` in `style.mjs`, which is - * overlay-only and already depends on pi. It was rejected: `panel.mjs` owns `clip` and the monochrome - * renderer, it is pinned to have NO imports, and the two renderers draw the same geometry -- a width rule - * that holds in the framed pane and not in the plain one is the "holds on one branch of an if" shape three - * issues in this round have now been about. So the table lives here and the styler comes to it. + * ONE RULE, NOT TWO. The obvious alternative was to import pi-tui's `visibleWidth` in `style.mjs`, which + * is overlay-only and already depends on pi. It was rejected: `panel.mjs` owns `clip` and the monochrome + * renderer, its purity pin forbids it reaching the world at all (that pin names the renderer's own scope + * among the things this file may not mention), and the two renderers draw the same geometry -- a width rule that holds in the framed pane + * and not in the plain one is the "holds on one branch of an if" shape three issues in this round have now + * been about. So the table lives here and the styler comes to it. * - * WHAT IT COUNTS, and each line is the smallest thing that gets the measured cases right: + * THE TABLE IS TRANSCRIBED FROM THE RENDERER, NOT FROM UAX #11, and that is the correction that matters. + * A first version of this was written out of the standard by hand and checked against ten strings. It + * passed, and it UNDER-counted 139,820 code points: all of CJK Extension B through H, Tangut, the Kana + * supplement, the Hangul Jamo extensions and every emoji block added since Unicode 13. CJK Extension B is + * exactly the "CJK repository name" this issue is about, so the fix carried the defect it was fixing, and + * in two classes (astral characters, and an emoji-presentation sequence) it was WORSE than the `.length` + * it replaced -- an astral character is two code units and two columns, so `.length` had been right there + * by accident. The renderer draws these panes, so the renderer, not the standard, is the authority. * - * - a combining mark (Mn/Me) is ZERO, because it draws on the character before it; - * - a zero-width or formatting character is ZERO (ZWSP, ZWNJ, ZWJ, the bidi marks, BOM, word joiner); - * - an East Asian Wide or Fullwidth character is TWO; - * - everything else is ONE, including an unassigned code point, because guessing wider on an unknown is + * WHAT IT COUNTS: + * + * - zero for a mark (`\p{M}`) and for a format character (`\p{Cf}`), plus the fillers and + * noncharacters the renderer also draws as nothing; + * - two for every code point in `WIDE`, which is that renderer's own double-width set; + * - one more when U+FE0F follows a character that draws narrow on its own and wide in its emoji form; + * - one for everything else, including an unassigned code point, because guessing wider on an unknown is * how a rule starts breaking the panes it was added to fix. * + * WHAT IT IS HELD TO, in `width.test.mjs`: a sweep of every code point there is, plus every + * character-with-U+FE0F pair, asserting that this never measures NARROWER than the renderer and that every + * place it measures wider is an unassigned code point. + * + * THE DIRECTION IS THE WHOLE POINT, and stating it as "over-counting is harmless" would be too kind: BOTH + * directions rag a frame, and they rag it differently. An over-count pads a body line as though it were + * wider than it is, so the line comes out SHORT and the right border sits left of the one above it -- + * ugly, bounded, and contained by the pane. An under-count runs the line PAST the pane and past the + * terminal, where it wraps and takes the border with it, which is the defect this issue names. So the + * sweep is one-sided on purpose, and the residual below is the safer of two bad shapes, not a harmless + * one. + * * ITS LIMIT, stated rather than implied: this sums CODE POINTS, so an emoji ZWJ sequence -- a family, a - * flag, a skin tone -- counts every member and comes out wider than the single glyph a terminal draws. - * pi-tui disagrees with us there too (it reports 2 where we report 8), and terminals disagree with each - * other, which is why no width table in this project will settle it. A grapheme-aware count needs - * `Intl.Segmenter` and a terminal that agrees; the residual is one over-wide line, never a broken border, - * because over-counting cuts early rather than late. + * profession, a skin tone -- counts every member and comes out wider than the single glyph a terminal + * draws. Measured on a family: 6 here against the renderer's 2. Terminals disagree with each other there + * too, which is why no width table in this project will settle it. A grapheme-aware count needs + * `Intl.Segmenter` and a terminal that agrees; the residual is one over-wide line, in the safe direction. */ export function columnsOf(s) { let n = 0; + // The ONE piece of state, and the one thing a per-character table cannot do without: U+FE0F asks for the + // emoji form of the character BEFORE it, and the renderer then draws that character two columns wide. + let promotable = false; for (const ch of String(s ?? "")) { - const cp = ch.codePointAt(0); - if (ZERO_WIDTH.test(ch) || COMBINING.test(ch)) continue; - n += WIDE.test(ch) || (cp >= 0x1f300 && cp <= 0x1f9ff) ? 2 : 1; + if (ch === VS16 && promotable) { + n += 1; + promotable = false; + continue; + } + promotable = false; + if (ZERO_WIDTH.test(ch)) continue; + if (WIDE.test(ch)) { + n += 2; + continue; + } + n += 1; + // ONLY A NARROW BASE IS PROMOTABLE, and it is narrow by having reached this line at all: a code point + // the renderer already draws wide took the `WIDE` branch above and never gets here. A first version + // also tested `\p{Emoji_Presentation}` here; measured against the pin, every code point with that + // property is in `WIDE`, so the test was dead and a mutation removing it survived the whole suite. + // The ASCII exclusion is NOT dead: `#`, `*` and the digits carry `\p{Emoji}` and stay one column, + // because their emoji form is a keycap and needs U+20E3 rather than U+FE0F alone. + promotable = ch.codePointAt(0) > 0x7f && TEXT_EMOJI.test(ch); } return n; } +/** VARIATION SELECTOR-16, which asks for the emoji form of the character before it. */ +const VS16 = "\ufe0f"; + // Mn and Me: a mark that draws on the character before it. `\p{M}` would also take Mc (spacing marks), which // DO occupy a column in the Indic scripts that use them. -const COMBINING = /\p{Mn}|\p{Me}/u; -// Format and zero-width characters, which occupy none: ZWSP/ZWNJ/ZWJ, the bidi marks and isolates, the word -// joiner and invisible operators, the BOM, and the variation selectors. -const ZERO_WIDTH = /[\u200b-\u200f\u2028\u2029\u202a-\u202e\u2060-\u2064\u2066-\u2069\ufeff\ufe00-\ufe0f]/u; -// East Asian Wide and Fullwidth, from UAX #11: CJK and its punctuation, Hangul, Kana, the fullwidth forms. -const WIDE = - /[\u1100-\u115f\u2e80-\u303e\u3041-\u33ff\u3400-\u4dbf\u4e00-\u9fff\ua000-\ua4cf\uac00-\ud7a3\uf900-\ufaff\ufe10-\ufe19\ufe30-\ufe6f\uff00-\uff60\uffe0-\uffe6]/u; +// ZERO COLUMNS. Mn and Me draw on the character before them; Mc (a SPACING mark) is deliberately NOT here, +// because it does occupy a column -- the one place this table knowingly departs from the renderer below. +// Cf covers the zero-width and bidi format characters, and the bracketed tail is the Hangul fillers and the +// noncharacters the renderer also draws as nothing. +const ZERO_WIDTH = /\p{M}|\p{Cf}|[ᅟᅠ᠎ㅤﾠ￰-]/u; + +// A TEXT-PRESENTATION EMOJI: one that draws narrow on its own and WIDE once U+FE0F asks for the emoji +// form. There are 201 of them and they are the reason this table cannot be purely per-character. +const TEXT_EMOJI = /\p{Emoji}/u; + +// TWO COLUMNS: every code point the PINNED renderer draws double-width, transcribed FROM that renderer +// rather than written out of UAX #11 by hand, and held to it by an exhaustive sweep in `width.test.mjs`. +// Regenerate it from the pin, never edit a range by hand. +const WIDE = new RegExp( + "[" + + "\\u1100-\\u115e\\u231a-\\u231b\\u2329-\\u232a\\u23e9-\\u23ec\\u23f0\\u23f3\\u25fd-\\u25fe\\u2614-\\u2615\\u2630-\\u2637" + + "\\u2648-\\u2653\\u267f\\u268a-\\u268f\\u2693\\u26a1\\u26aa-\\u26ab\\u26bd-\\u26be\\u26c4-\\u26c5\\u26ce\\u26d4\\u26ea" + + "\\u26f2-\\u26f3\\u26f5\\u26fa\\u26fd\\u2705\\u270a-\\u270b\\u2728\\u274c\\u274e\\u2753-\\u2755\\u2757\\u2795-\\u2797\\u27b0" + + "\\u27bf\\u2b1b-\\u2b1c\\u2b50\\u2b55\\u2e80-\\u2e99\\u2e9b-\\u2ef3\\u2f00-\\u2fd5\\u2ff0-\\u3029\\u3030-\\u303e" + + "\\u3041-\\u3096\\u309b-\\u30ff\\u3105-\\u312f\\u3131-\\u3163\\u3165-\\u318e\\u3190-\\u31e5\\u31ef-\\u321e\\u3220-\\u3247" + + "\\u3250-\\ua48c\\ua490-\\ua4c6\\ua960-\\ua97c\\uac00-\\ud7a3\\uf900-\\ufaff\\ufe10-\\ufe19\\ufe30-\\ufe52\\ufe54-\\ufe66" + + "\\ufe68-\\ufe6b\\uff01-\\uff60\\uffe0-\\uffe6\\u{16fe0}-\\u{16fe3}\\u{16ff2}-\\u{16ff6}\\u{17000}-\\u{18cd5}" + + "\\u{18cff}-\\u{18d1e}\\u{18d80}-\\u{18df2}\\u{1aff0}-\\u{1aff3}\\u{1aff5}-\\u{1affb}\\u{1affd}-\\u{1affe}" + + "\\u{1b000}-\\u{1b122}\\u{1b132}\\u{1b150}-\\u{1b152}\\u{1b155}\\u{1b164}-\\u{1b167}\\u{1b170}-\\u{1b2fb}" + + "\\u{1d300}-\\u{1d356}\\u{1d360}-\\u{1d376}\\u{1f004}\\u{1f0cf}\\u{1f18e}\\u{1f191}-\\u{1f19a}\\u{1f1e6}-\\u{1f202}" + + "\\u{1f210}-\\u{1f23b}\\u{1f240}-\\u{1f248}\\u{1f250}-\\u{1f251}\\u{1f260}-\\u{1f265}\\u{1f300}-\\u{1f320}" + + "\\u{1f32d}-\\u{1f335}\\u{1f337}-\\u{1f37c}\\u{1f37e}-\\u{1f393}\\u{1f3a0}-\\u{1f3ca}\\u{1f3cf}-\\u{1f3d3}" + + "\\u{1f3e0}-\\u{1f3f0}\\u{1f3f4}\\u{1f3f8}-\\u{1f43e}\\u{1f440}\\u{1f442}-\\u{1f4fc}\\u{1f4ff}-\\u{1f53d}" + + "\\u{1f54b}-\\u{1f54e}\\u{1f550}-\\u{1f567}\\u{1f57a}\\u{1f595}-\\u{1f596}\\u{1f5a4}\\u{1f5fb}-\\u{1f64f}" + + "\\u{1f680}-\\u{1f6c5}\\u{1f6cc}\\u{1f6d0}-\\u{1f6d2}\\u{1f6d5}-\\u{1f6d8}\\u{1f6dc}-\\u{1f6df}\\u{1f6eb}-\\u{1f6ec}" + + "\\u{1f6f4}-\\u{1f6fc}\\u{1f7e0}-\\u{1f7eb}\\u{1f7f0}\\u{1f90c}-\\u{1f93a}\\u{1f93c}-\\u{1f945}\\u{1f947}-\\u{1f9ff}" + + "\\u{1fa70}-\\u{1fa7c}\\u{1fa80}-\\u{1fa8a}\\u{1fa8e}-\\u{1fac6}\\u{1fac8}\\u{1facd}-\\u{1fadc}\\u{1fadf}-\\u{1faea}" + + "\\u{1faef}-\\u{1faf8}\\u{20000}-\\u{2fffd}\\u{30000}-\\u{3fffd}" + + "]", + "u", +); /** * Truncate `line` to `w` display columns, appending an ellipsis glyph when content is cut. Control @@ -258,15 +331,31 @@ export function clip(line, w) { */ export function sliceColumns(s, w) { const budget = Math.max(0, Math.trunc(w) || 0); + // NO BUDGET, NO CONTENT. Without this a zero-column character passes the test below and a cut to zero + // returned a floating accent with nothing to attach to. + if (budget === 0) return ""; let out = ""; let used = 0; for (const ch of String(s ?? "")) { + // HALF A PAIR IS NOT A CHARACTER, so it is not carried into a cut. Walking code points means this cut + // cannot CREATE one, which is not the same as dropping one the INPUT already held: a review pass + // measured `clip` passing a lone high surrogate on where the old slice-then-repair had removed it, and + // the old repair only ever looked at the last code unit, so a leading or interior one survived it too. + if (ch.length === 1 && ch.charCodeAt(0) >= 0xd800 && ch.charCodeAt(0) <= 0xdfff) continue; const cols = columnsOf(ch); if (used + cols > budget) break; out += ch; used += cols; } - return out; + // A TRAILING JOINER IS DANGLING: it joins this character to the next one, and the next one is what was + // just cut away. Left in place it reaches the terminal ahead of the ellipsis and asks it to join a glyph + // to a horizontal bar. The same is true of a variation selector whose base was cut, which cannot happen + // here (a selector is only ever appended after its base) but costs nothing to drop with it. + // + // + // An input that already holds a lone surrogate and needs no cut at all still carries it: `clip` returns + // early when the string fits. That is issue #402's ground rather than this one's. + return out.replace(/[\u200d\ufe0e\ufe0f]+$/u, ""); } /** @@ -334,12 +423,12 @@ export function box({ title = "", sections = [], footer, width = 40 } = {}) { * * `state` ("ok" | "soft-hold" | "over") appends a textual marker to the label: the panel is monochrome and * `clip` strips ANSI, so the amber/red of a soft-hold or over-budget window is carried as a word, not a - * color. + * color. "ok" (the default) adds nothing, so a plain call renders exactly as before. * * ITS LABEL IS MEASURED BY `.length` AND THAT IS CORRECT HERE, which is worth saying in a file whose whole * subject is that `.length` is not a column count (issue #401). Every character of the label comes from two * integers and a word out of a closed set, so it is digits and ASCII by construction and no caller can put - * anything else in it. `styler.meter`'s label is built the same way and is exempt for the same reason. "ok" (the default) adds nothing, so a plain call renders exactly as before. + * anything else in it. `styler.meter`'s label is built the same way and is exempt for the same reason. */ export function meter(reserved, cap, width = 24, state = "ok") { const r = Number.isFinite(reserved) ? Math.max(0, Math.trunc(reserved)) : 0; @@ -518,11 +607,47 @@ export function makeLineInput(initial = "") { }, render(width, { focused = true } = {}) { const w = Math.max(1, Math.trunc(width) || 1); - const start = value.length > w - 1 ? Math.max(0, cursor - (w - 1)) : 0; - const text = value.slice(start, start + w).padEnd(w); - if (!focused) return text; - const rel = cursor - start; // 0..w-1 by construction: the window never scrolls past the cursor - return text.slice(0, rel) + LINE_INPUT_CURSOR[0] + text[rel] + LINE_INPUT_CURSOR[1] + text.slice(rel + 1); + // THE WINDOW IS CHOSEN IN COLUMNS AND ITS EDGES ARE WHOLE CHARACTERS (issue #401). This was + // `value.slice(start, start + w).padEnd(w)` on UTF-16 indices, so it measured a CJK value at half + // what the terminal draws, and a window edge landing between the halves of an astral pair emitted a + // BARE LOW SURROGATE into the live trigger editor -- the exact hazard the rest of this module cuts + // around. `cursor` is still a code-unit index, because every edit method moves it by one unit, so it + // is snapped to a character boundary here rather than trusted. + const chars = [...value]; + const offs = []; + let at = 0; + for (const c of chars) { + offs.push(at); + at += c.length; + } + offs.push(at); + let ci = offs.findIndex((o) => o >= cursor); + if (ci < 0) ci = chars.length; + // THE RESERVED CELL MOVES THE WINDOW'S START, not its length: the window is `w` columns wide, and + // the cursor is kept at most `w - 1` columns past the start so its own cell is always inside it. + let start = ci; + let back = 0; + while (start > 0 && back + columnsOf(chars[start - 1]) <= w - 1) { + start -= 1; + back += columnsOf(chars[start]); + } + let end = start; + let used = 0; + while (end < chars.length && used + columnsOf(chars[end]) <= w) { + used += columnsOf(chars[end]); + end += 1; + } + const head = chars.slice(start, Math.min(ci, end)).join(""); + const text = chars.slice(start, end).join(""); + if (!focused) return pad(text, w); + // The cursor wraps a WHOLE character, never one half of a pair, and sits on a space once it is past + // the last character the window shows. + const onChar = ci < end; + const under = onChar ? chars[ci] : " "; + const tail = onChar ? chars.slice(ci + 1, end).join("") : ""; + const shown = head + (onChar ? under : "") + tail; + const fill = Math.max(0, w - columnsOf(shown) - (onChar ? 0 : 1)); + return head + LINE_INPUT_CURSOR[0] + under + LINE_INPUT_CURSOR[1] + tail + " ".repeat(fill); }, }; } diff --git a/admin/src/render.mjs b/admin/src/render.mjs index 3ef1089..f459173 100644 --- a/admin/src/render.mjs +++ b/admin/src/render.mjs @@ -15,7 +15,7 @@ import { windowState } from "@edgehero/pi-dispatch/budget"; // Pure-to-pure, the same standing as the windowState import above: panel.mjs is the admin's other no-I/O // text module (asserted so by panel.test.mjs), and fmtCost is THE single renderer of typed cost values, // so the what-if below routes every dollar through it rather than grow a second money formatter here. -import { fmtCost, scrubControls } from "./panel.mjs"; +import { columnsOf, fmtCost, pad, scrubControls } from "./panel.mjs"; // The overlay keys, IMPORTED rather than retyped. This was a verbatim copy of the worker's array, in the // order the worker declares them, and the worker's side is pinned while this side was not -- so a key // added there would have failed a test, been added, and left the settings VIEW silently ten keys wide @@ -125,8 +125,11 @@ export function renderRuns(runs) { // instead of each derive means a column added later cannot reintroduce this, and it costs nothing on // the `-` and `r1/2` shapes a derive normally produces. const rows = list.map((r) => RUN_COLUMNS.map((c) => cell(c.derive ? c.derive(r) : r?.[c.key]))); - const widths = headers.map((h, i) => Math.max(h.length, ...rows.map((row) => row[i].length))); - const fmt = (cells) => cells.map((v, i) => v.padEnd(widths[i])).join(" ").trimEnd(); + // COLUMNS, not code units (issue #401), and this table is the MODEL-visible channel rather than a pane. + // `target` is `local:` for a local run, so an operator's own folder name reaches it, and one + // CJK character there shifted every later column of that row against the rows around it. + const widths = headers.map((h, i) => Math.max(columnsOf(h), ...rows.map((row) => columnsOf(row[i])))); + const fmt = (cells) => cells.map((v, i) => pad(v, widths[i])).join(" ").trimEnd(); return [fmt(headers), ...rows.map(fmt)].join("\n"); } diff --git a/admin/src/style.mjs b/admin/src/style.mjs index 0fdd917..4b031d7 100644 --- a/admin/src/style.mjs +++ b/admin/src/style.mjs @@ -17,7 +17,7 @@ * can assert both the plain content and the width math without a real terminal. */ -import { LINE_INPUT_CURSOR, columnsOf, dropLoneSurrogate, fmtCost as plainFmtCost, scrubControls, scrubKeepingStyle, sliceColumns, sparkline as plainSparkline } from "./panel.mjs"; +import { LINE_INPUT_CURSOR, columnsOf, fmtCost as plainFmtCost, scrubControls, scrubKeepingStyle, sliceColumns, sparkline as plainSparkline } from "./panel.mjs"; // Strip SGR (and OSC-8 hyperlink) escapes to recover the visible text, which is then measured in COLUMNS. // This comment used to end "so post-strip `.length` is a safe column proxy", and issue #401 is what that @@ -104,12 +104,11 @@ export function makeStyler(theme, { ascii = false } = {}) { const cell = (text, width, { color = null, align = "left", strong = false } = {}) => { const w = Math.max(0, Math.trunc(width) || 0); let plain = scrubControls(stripAnsi(String(text ?? ""))); - // `dropLoneSurrogate` on every cut, like `clip`: slicing UTF-16 units can land between the halves of an - // astral character, and half a pair is not a character. Measured at 89 lines of a framed LIST printing - // one, from a target field of emoji -- the first repair reached `clip` alone and three other cutters - // slice the same way. - // BY COLUMNS, not by code units (issue #401), and through `sliceColumns` so a cut never lands inside a - // two-column character -- which a terminal draws as one blank column plus one of overflow. + // BY COLUMNS, not by code units (issue #401), and through `sliceColumns`, which walks whole characters. + // This used to call `dropLoneSurrogate` after a UTF-16 slice, a repair measured at 89 lines of a framed + // LIST printing half a surrogate pair from a target field of emoji. Cutting by character makes that + // repair unnecessary rather than merely correct: the cut also never lands inside a TWO-COLUMN + // character, which a terminal draws as one blank column plus one of overflow. if (columnsOf(plain) > w) { const ell = G.ellipsis; // The narrow branch slices the CONTENT, not the ellipsis, which is what this line did before #401 and @@ -125,7 +124,7 @@ export function makeStyler(theme, { ascii = false } = {}) { return strong ? bold(out) : out; }; - /** A small colored token (no padding). Visible width === label.length (+ padding if `pad`). */ + /** A small colored token (no padding). Visible width is `columnsOf(label)`, plus 2 if `pad`. */ const badge = (label, color, { pad = false } = {}) => { const text = pad ? ` ${label} ` : String(label); return fg(color, text); diff --git a/admin/test/width.test.mjs b/admin/test/width.test.mjs index 9c1406e..ccef3a5 100644 --- a/admin/test/width.test.mjs +++ b/admin/test/width.test.mjs @@ -2,8 +2,9 @@ import assert from "node:assert/strict"; import { test } from "node:test"; import { createRequire } from "node:module"; import { pathToFileURL } from "node:url"; -import { box, clip, columnsOf, pad, sliceColumns } from "../src/panel.mjs"; +import { LINE_INPUT_CURSOR, box, clip, columnsOf, makeLineInput, pad, sliceColumns } from "../src/panel.mjs"; import { frame, makeStyler, PLAIN_THEME, visibleLen } from "../src/style.mjs"; +import { renderRuns } from "../src/render.mjs"; // EVERY WIDTH PROMISE IN THIS PANEL WAS A UTF-16 COUNT (issue #401), and this file is where the repair is // held against the only authority available: pi's own renderer, which is what actually draws these lines. @@ -33,13 +34,23 @@ import { frame, makeStyler, PLAIN_THEME, visibleLen } from "../src/style.mjs"; async function loadVisibleWidth() { try { const pi = createRequire(import.meta.resolve("@earendil-works/pi-coding-agent")); - const { visibleWidth } = await import(pathToFileURL(pi.resolve("@earendil-works/pi-tui")).href); + const entry = pi.resolve("@earendil-works/pi-tui"); + // WHICH COPY, asserted rather than assumed. pi depends on pi-tui by a RANGE, so a resolve that is not + // the lockfile's would answer any 0.80.x, and a hoisted layout could put a different copy above this + // one. `CONST-PI-VERSION-PINNED` says to verify against the pinned artifact rather than a range, and + // an oracle measured against the wrong artifact is a table pinned to the wrong renderer. + const version = pi("@earendil-works/pi-tui/package.json").version; + if (version !== PI_TUI_VERSION) return null; + const { visibleWidth } = await import(pathToFileURL(entry).href); return typeof visibleWidth === "function" ? visibleWidth : null; } catch { return null; } } +/** The pin, from `package-lock.json`. A mismatch fails the tests below rather than measuring silently. */ +const PI_TUI_VERSION = "0.80.7"; + /** `[name, string, what pi and we must agree it is]` -- the cases a review pass measured as broken. */ const AGREE = [ ["ascii", "hello world", 11], @@ -47,14 +58,14 @@ const AGREE = [ ["a CJK repo target", "会社/製品#5", 11], ["fullwidth latin", "ＪＯＢ", 6], ["hangul", "한글테스트", 10], - ["combining marks", "jób́", 3], - ["a zero-width space", "a​b​c", 3], - ["a bidi override", "a‮b", 2], + ["combining marks", "jo\u0301b\u0301", 3], + ["a zero-width space", "a\u200bb\u200bc", 3], + ["a bidi override", "a\u202eb", 2], ["box drawing", "┌─┐", 3], ["one emoji", "\u{1f600}", 2], ]; -test("the column count agrees with pi's own renderer on everything but a ZWJ sequence (#401)", async () => { +test("the measured cases agree with the renderer, and the ZWJ disagreement is pinned as one (#401)", async () => { const visibleWidth = await loadVisibleWidth(); // NOT SKIPPED SILENTLY: an oracle that quietly vanishes is an oracle that stops being one, and this // module's sibling pin (`keys.mjs`) made exactly that mistake once. @@ -69,7 +80,7 @@ test("the column count agrees with pi's own renderer on everything but a ZWJ seq // sequence counts every member; pi collapses it to one glyph. Terminals disagree with each other here // too, which is why no table in this project settles it. Over-counting cuts EARLY, so the cost is a short // line rather than a broken border -- which is the direction to be wrong in. - const family = "\u{1f468}‍\u{1f469}‍\u{1f466}"; + const family = "\u{1f468}\u200d\u{1f469}\u200d\u{1f466}"; assert.equal(columnsOf(family), 6, "we count each member of the family"); assert.equal(visibleWidth(family), 2, "pi counts the glyph -- recorded, not fixed"); }); @@ -106,7 +117,7 @@ test("the monochrome pane measures exactly its width too, title row included (#4 // rule reverted to `.length`. `box` and `frame` draw the same geometry from two different files, and this // round keeps finding the shape where a rule holds on one branch and not the other. The overshoot lands on // the FIRST line only, with every body row correct, which is what made it easy to miss by eye. - const sections = [{ title: "ジョブ", lines: ["会社/製品#5 のジョブ", "plain ascii row", "jób́ marks"] }]; + const sections = [{ title: "ジョブ", lines: ["会社/製品#5 のジョブ", "plain ascii row", "jób\u0301 marks"] }]; for (const w of [24, 40, 80]) { for (const line of box({ title: "ジョブ番号", sections, footer: "Ｆ", width: w })) { assert.equal(columnsOf(line), w, `width ${w}: ${JSON.stringify(line)}`); @@ -118,7 +129,7 @@ test("a frame holding CJK still measures exactly its width, in our terms and pi' const visibleWidth = await loadVisibleWidth(); assert.equal(typeof visibleWidth, "function"); const styler = makeStyler(PLAIN_THEME); - const lines = ["会社/製品#5 のジョブ", "plain ascii row", "jób́ with combining marks"]; + const lines = ["会社/製品#5 のジョブ", "plain ascii row", "jo\u0301b\u0301 with combining marks"]; for (const w of [20, 40, 80]) { for (const line of frame(styler, { title: "ジョブ", width: w, lines: lines.map((l) => styler.cell(l, w - 4)), footer: "Ｆ" })) { assert.equal(visibleLen(line), w, `width ${w}: our own measure`); @@ -126,3 +137,219 @@ test("a frame holding CJK still measures exactly its width, in our terms and pi' } } }); + +test("the table is held to the renderer on every code point there is (#401)", async () => { + const visibleWidth = await loadVisibleWidth(); + assert.equal(typeof visibleWidth, "function", "pi-tui's visibleWidth must load, or this test is checking nothing"); + + // THE BOLT A HAND-WRITTEN TABLE NEEDS, and the reason it sweeps rather than lists examples. The first + // version of this table was written out of UAX #11 by hand and checked against ten strings. It passed, + // and it UNDER-counted 139,820 code points: CJK Extension B through H, Tangut, the Kana supplement, the + // Hangul Jamo extensions and every emoji block added since Unicode 13. Ten examples cannot see that. + // CLAUDE.md already says a hand-written table restating a derivable source is either derived or pinned. + // + // THE DIRECTION IS THE WHOLE POINT, so the hard assertion is one-sided. Over-counting cuts EARLY: the + // line comes out short and the border holds. Under-counting cuts LATE: the line runs past the border, + // which is the defect this issue names. + const OVER = []; + for (let cp = 0; cp <= 0x10ffff; cp++) { + // A surrogate on its own is not a character and neither side promises anything about it. + if (cp >= 0xd800 && cp <= 0xdfff) continue; + // THE CONTROL CLASS IS OUTSIDE THE TABLE'S PROMISE, for a reason no per-character table can fix: a + // TAB's width depends on the cursor's column, not on the character (the renderer answers 3 for one at + // column 0). Every renderer here substitutes the whole class before measuring, which is what issue + // #382 put in front of every path, so the table is never asked. + if (cp < 0x20 || cp === 0x7f || (cp >= 0x80 && cp <= 0x9f)) continue; + const ch = String.fromCodePoint(cp); + const ours = columnsOf(ch); + const theirs = visibleWidth(ch); + assert.ok(ours >= theirs, `U+${cp.toString(16).toUpperCase().padStart(4, "0")}: we say ${ours}, the renderer draws ${theirs}`); + if (ours !== theirs) OVER.push(cp); + } + + // Every over-count is the one departure the table declares, and nothing else. Without this half the + // assertion above is satisfied by a table that answers 2 for everything. + for (const cp of OVER) { + assert.ok( + !/\p{Assigned}/u.test(String.fromCodePoint(cp)), + `U+${cp.toString(16).toUpperCase().padStart(4, "0")} is assigned, so the table over-counts a real character for no stated reason`, + ); + } + assert.ok(OVER.length > 0, "the unassigned departure is real, not a dead clause"); +}); + +test("U+FE0F asks for the emoji form, and the table follows the renderer there too (#401)", () => { + // A PER-CHARACTER TABLE CANNOT DO THIS, which is why `columnsOf` carries one piece of state. U+FE0F + // makes the character BEFORE it draw in its emoji form, two columns wide, for the 201 code points that + // have a text form and an emoji form. Reverting that clause is the mutation that matters: it restores a + // count the code-unit `.length` had right by accident, since such a sequence is two code units. + for (const [name, s, expected] of [ + ["a heart", "❤\ufe0f", 2], + ["a warning sign", "⚠\ufe0f", 2], + ["a bare heart", "❤", 1], + ["an already-wide emoji", "\u{1f600}\ufe0f", 2], + ["a letter, which VS16 does not promote", "A\ufe0f", 1], + ["a digit, whose emoji form is a keycap", "1\ufe0f", 1], + ]) { + assert.equal(columnsOf(s), expected, name); + } +}); + +test("a cut leaves no dangling joiner and no content at all at zero (#401)", () => { + const family = "\u{1f468}\u200d\u{1f469}\u200d\u{1f466}"; + for (let w = 0; w <= 8; w++) { + assert.doesNotMatch(sliceColumns(family, w), /[\u200d\ufe0e\ufe0f]$/u, `width ${w}: a joiner survived the cut it was joining across`); + } + // A zero-column character passes a `used + cols > budget` test, so a cut to nothing used to return a + // combining mark with nothing left to attach to. + assert.equal(sliceColumns("\u0301abc", 0), "", "no budget, no content"); +}); + +test("the line editor's window is columns and whole characters (#401)", () => { + // THE CUTTER THE FIRST REPAIR MISSED. `render` windowed with `value.slice(start, start + w).padEnd(w)`, + // so a CJK value measured half what the terminal drew and an edge landing between the halves of an + // astral pair put a BARE LOW SURROGATE into the live trigger editor. + const lone = /[\ud800-\udbff](?![\udc00-\udfff])|(? { + // BOTH OF THESE SURVIVED a mutation back to `.length` while the suite was green, because every fixture + // that reached them was ASCII. `frame` clips an over-wide line now, so a reverted `divider` is invisible + // in a framed render: it has to be measured on its own. + const styler = makeStyler(PLAIN_THEME); + for (const w of [20, 40, 80]) { + assert.equal(visibleLen(styler.divider("ジョブ番号", "会社/製品", w)), w, `divider at ${w}`); + assert.equal(visibleLen(styler.divider("ascii label", "ＭＥＴＡ", w)), w, `divider with a wide meta at ${w}`); + } + // `clipPlain` is reached only through a frame title, so this drives it there and measures the rule it + // computes: the top line is exactly the frame's width. + for (const w of [12, 20, 40]) { + const top = frame(styler, { title: "ジョブ番号のタイトル", width: w, lines: [] })[0]; + assert.equal(visibleLen(top), w, `a title clipped to ${w}`); + } +}); + +test("a fitted body line keeps its colour, and only an over-wide one is clipped (#401)", () => { + // THE STRICT COMPARISON IN `padVisible`, pinned where it is observable. Under the PLAIN theme `>=` and + // `>` render identically, so every existing fixture missed it: an adversarial pass showed the mutant is + // only visible under a REAL theme, where the loosened comparison sends a line that already fits through + // `styler.cell` and `cell` strips the styler's own SGR before measuring. The line then still measures + // right and has silently lost its colour, which is the shape this whole round keeps finding. + const ESC = String.fromCharCode(27); + // THE ACCENT AND THE BORDER GET DIFFERENT CODES ON PURPOSE. A first version of this test asked whether + // the line contained any ESC at all, and `frame` draws its own `│` through `fg("border", ...)`, so the + // assertion was true however much colour the body had lost. The mutant survived it. + const ACCENT = `${ESC}[38;5;42m`; + const styler = makeStyler({ + fg: (c, t) => (c === "border" ? `${ESC}[38;5;99m${t}${ESC}[39m` : `${ACCENT}${t}${ESC}[39m`), + bold: (t) => `${ESC}[1m${t}${ESC}[22m`, + bg: (_c, t) => t, + }); + // THE LINE MUST FIT EXACTLY, or the mutant is not even reached: `>=` and `>` differ only on a line whose + // visible width EQUALS the pane's inner width, and a shorter line takes neither branch. A first version + // of this test used a 20-column line inside a 36-column pane and the mutant survived it. + const inner = 40 - 4; + const plain = sliceColumns("\u4f1a\u793e/\u88fd\u54c1 ".repeat(6), inner); + const body = styler.fg("accent", plain + " ".repeat(inner - columnsOf(plain))); + assert.equal(visibleLen(body), inner, "the fixture is exactly the inner width, which is what makes it a test"); + const [top, line] = frame(styler, { title: "ジョブ", width: 40, lines: [body] }); + assert.ok(line.includes(ACCENT), "a line that fits keeps the colour it arrived with, not just the border's"); + assert.equal(visibleLen(line), 40, "and is still exactly the width"); + assert.equal(visibleLen(top), 40, "as is the rule above it"); + // The other half of the same comparison: one that does NOT fit is cut to the pane rather than breaking it. + const tooWide = styler.fg("accent", "会社/製品#5 のジョブ".repeat(6)); + assert.equal(visibleLen(frame(styler, { title: "t", width: 40, lines: [tooWide] })[1]), 40, "an over-wide line is clipped"); +}); + +test("a cut drops a lone surrogate rather than passing it on (#401)", () => { + // PARITY WITH WHAT THE CODE-UNIT CUT DID. Walking characters means the cut cannot CREATE half a pair, + // which is not the same as dropping one that was already in the input: a review pass measured `clip` + // passing a lone high surrogate on where the old slice-then-`dropLoneSurrogate` had removed it. + const lone = /[\ud800-\udbff](?![\udc00-\udfff])|(? { + // THE CHANNEL THE PANE GATES DO NOT REACH. `renderRuns` answers `/dispatch runs`, so its output goes to + // the model and to the operator's scrollback rather than through a frame, and it sized its columns with + // `.length`. A `target` is `local:` for a local run, so an operator's own folder name lands in + // it, and one CJK character there shifted every later column of that row against the rows around it. + const at = "2026-07-21T00:00:00.000Z"; + const base = { flow: "review", outcome: "completed", turns: 3, tokens: { total: 10 }, endedAt: at }; + const rows = renderRuns([ + { ...base, jobId: "gh-aaaa1", target: "local:プロジェクト" }, + { ...base, jobId: "gh-aaaa2", target: "local:project" }, + { ...base, jobId: "gh-aaaa3", target: "local:한글" }, + ]).split("\n"); + // Every row starts its FLOW cell at the same column, which is the only thing a reader of this table + // needs and the only thing `.length` got wrong. + const flowAt = rows.slice(1).map((r) => columnsOf(r.slice(0, r.indexOf("review")))); + assert.equal(new Set(flowAt).size, 1, `the flow column starts at ${[...new Set(flowAt)].join(", ")}`); + // AND THE CELL IS NOT CUT TO REACH THAT, which alignment alone does not say. A width computed by + // `.length` is too SMALL for a wide cell, and `pad` then clips every target to it: the columns line up + // beautifully and the operator's folder name has lost its tail. Both halves are needed or the mutant + // satisfies the test by truncating. + assert.match(rows[1], /local:プロジェクト/u, "the CJK target survives whole"); + assert.match(rows[3], /local:한글/u, "and so does the Hangul one"); +}); + +test("the graph's hostile-string caps never leave half a character (#401)", async () => { + // A CHARACTER CAP, NOT A WIDTH, and that is why it is here rather than left alone: the cut is still a + // cut, and `clipName`'s own comment calls its input "arbitrary (possibly hostile)". A code-unit slice at + // 64 lands between the halves of an astral pair whenever the 64th unit is a high surrogate, and the + // repair that used to catch that downstream now has no caller on this path. + const { buildGraphModel } = await import("../src/graph-model.mjs"); + const lone = /[\ud800-\udbff](?![\udc00-\udfff])|(? { + if (typeof v === "string") strings.push(v); + else if (Array.isArray(v)) v.forEach(walk); + else if (v && typeof v === "object") Object.values(v).forEach(walk); + }; + walk(model); + for (const s of strings) { + assert.doesNotMatch(s, lone, `a flow name with a ${n}-character prefix left half a pair in ${JSON.stringify(s.slice(0, 40))}`); + } + } +}); diff --git a/specs/design.md b/specs/design.md index 2fe8b3d..578b661 100644 --- a/specs/design.md +++ b/specs/design.md @@ -1585,8 +1585,9 @@ money with no upstream turn limit (`REQ-RUNNER-TURN-BUDGET`). **theme-colored** panel (color via pi's injected `Theme`, applied post-layout; CORRECTED under issue #382, which found five sites handing `styler.cell` PRE-coloured text that it then measured with `.length`, so colour is post-layout by construction now rather than by convention; and the width is a COLUMN count under - issue #401, so pi's `visibleWidth` frames it for CJK, fullwidth, Hangul and combining characters too, not - only for ASCII) carrying a status header, day/week/month **SPEND meters** (colored by the + issue #401, transcribed from that renderer's own table, so it frames CJK, fullwidth, Hangul, astral and + combining content too, not only ASCII, with an emoji ZWJ sequence the one shape it draws short) carrying a + status header, day/week/month **SPEND meters** (colored by the same `windowState` the worker enforces) plus a daily **token** counter, a unified **TRIGGERS** pane whose `{on, run}` rows are **selectable and editable** (a cron row carries an amber `⚠ overdue`/`⚠ stalled` badge joined from its resident scheduler — health is a LIST-level fact, not only a drill-in one), and an @@ -1735,9 +1736,10 @@ money with no upstream turn limit (`REQ-RUNNER-TURN-BUDGET`). **AND THE UNFRAMED DEGRADE HONOURS ITS WIDTH** (issue #403), where it honoured it for the run rows only -- bounded to their own fixed 24-column pane -- so one render held those beside headers and hints running past the width the caller asked for, 28 of 59 lines over at `render(4)`. A width that is missing or non-finite is left - alone rather than invented, because clipping to `NaN` blanks the pane. The clip is by LENGTH, which is - safe only because this path is monochrome: measured under a real theme, none of its lines carries an SGR - sequence, and a test pins that. What would happen otherwise is worse than losing colour: `clipData` substitutes the + alone rather than invented, because clipping to `NaN` blanks the pane. The clip is by COLUMN under issue + #401, where it was by LENGTH here; what has not changed is why it may be a PLAIN measure at all, which is + that this path is monochrome: measured under a real theme, none of its lines carries an SGR sequence, and + a test pins that. What would happen otherwise is worse than losing colour: `clipData` substitutes the ESC and leaves the rest of the sequence VISIBLE, so a coloured degrade would print `[31m` as text and charge four phantom columns against the width. @@ -1779,13 +1781,24 @@ money with no upstream turn limit (`REQ-RUNNER-TURN-BUDGET`). so they are reachable only through a writer that is not the worker, or a hand-edited file. **Column width for non-ASCII IS promised now** (issue #401), where this entry previously said it was - not. `panel.mjs` carries one width table -- a combining mark or a zero-width character is 0, an East - Asian Wide or Fullwidth one is 2, everything else is 1 -- and every cutter, padder and border in BOTH - renderers measures through it. It is pinned against pi-tui's own `visibleWidth` at the pin, which is the - renderer that actually draws these lines. **Its one stated limit**: the table sums CODE POINTS, so an - emoji ZWJ sequence counts every member and comes out wider than the single glyph a terminal draws (6 - against pi's 2, pinned as a disagreement rather than left to surprise someone). Over-counting cuts - early, so the residual is a short line, never a broken border. + not. `panel.mjs` carries one width table and every cutter, padder and border in both renderers measures + through it, the line editor's window and the model-visible runs table included. **The table is + TRANSCRIBED FROM THE PINNED RENDERER, not from UAX #11**, and that distinction is the correction worth + recording: a first version was written out of the standard by hand, checked against ten strings, and + UNDER-counted 139,820 code points -- CJK Extension B through H, Tangut, the Kana supplement, the Hangul + Jamo extensions, every emoji block added since Unicode 13 -- with two classes (astral characters, and a + character followed by U+FE0F) coming out WORSE than the `.length` it replaced, since an astral character + is two code units and two columns. The renderer draws these panes, so the renderer is the authority. + **It is held there by a sweep of every code point there is**, plus every character-with-U+FE0F pair, + asserting that the table never measures NARROWER than the renderer and that every place it measures + wider is an unassigned code point. **The direction is the point, and "over-counting is harmless" would be + too kind**: both directions rag a frame and they rag it differently. An over-count pads a body line as + though it were wider than it is, so the line comes out SHORT and the right border sits left of the one + above it, contained by the pane. An under-count runs the line PAST the pane and past the terminal, where + it wraps and takes the border with it. The sweep is one-sided on purpose, and the residual below is the + safer of two bad shapes rather than a harmless one. **Its one remaining limit**: the table sums + CODE POINTS, so an emoji ZWJ sequence counts every member and comes out wider than the single glyph a + terminal draws, 6 against 2, pinned as a disagreement rather than left to surprise someone. **What the gate still does NOT cover**: bidi, zero-width and line-separator code points, which a terminal does not INTERPRET -- so they are outside a class defined by what a terminal interprets -- but @@ -4622,4 +4635,4 @@ a tunnel. | 2026-09-23 | Issue #386. **`DES-WATCHERS-CLOSE-WITH-THE-WORKER` AMENDED** with the other end of a watch's life: it recorded how a watch STOPS and said nothing about the window before it STARTS. All four live-edit watches in this project -- the worker's triggers, pause-windows and scoped-limits, and the receiver's triggers -- read their file at boot and armed afterwards, so an edit in that gap was never loaded until the next edit or a restart. The receiver's gap is the widest, its watch being the last fallible step of a boot that can retry identity for `RECEIVER_IDENTITY_RETRY_SECONDS`. **The macOS half is a LOST edit rather than a late one**, and that is what made this worth product code rather than a test ceiling: libuv recreates the process's single FSEvents stream from "now" when any watcher is armed or closed anywhere in the process, so an edit before the new stream is live is never delivered (measured against the real receiver boot, 40 trials per posture: 24 lost with four boots in one process, 1 beside a loaded machine, 0 idle -- a re-measure on another machine reproduced the shape and not the rate, at 1, 0 and 0, so read the 24 as one host's worst case; Linux's inotify registers before `fs.watch` returns, so there it is late but not lost). The repair is one read after arming compared against the bytes THE BOOT LOADER READ, reloading only when they moved -- so a quiet boot pays one file read and says nothing. Where the baseline comes from is the load-bearing part rather than a detail: a first version of this change read it where the watch arms, which closed 0.1 to 0.4 milliseconds while leaving the identity-retry window it is named for wide open, proven end to end on both services by a review pass. Both loaders already take a read seam, so the baseline is exactly what the service is running. Its residual is stated in the entry rather than implied: the FSEvents window extends past that read too. `WATCH_DEBOUNCE_MS` moves into the same module in the same change, the 150ms literal having been written out four times. **`DES-CRON-VIA-BULLMQ-SCHEDULER` UNCHANGED, checked**: the reconcile and its fingerprint gate are untouched, and the extra reconcile only runs on a boot that lost the race. **Code evidence**: `worker/src/watch-closer.mjs` -> `readBeforeArming`, `changedWhileArming`, `WATCH_DEBOUNCE_MS`. | | 2026-09-24 | Issue #404. **`DES-ADMIN-VIA-PI-EXTENSION` AMENDED**: the control-byte gate names FOUR funnels, not three. #382 closed the two that render a pane and the model-visible `send`, and its own wording -- "every finished line passes one gate" -- read as complete while pi's DIALOGS took strings straight from stored fields. Measured: a pause window whose `scope` carries an erase-display, an OSC-52 clipboard write and an OSC-8 link reaches a select option, an input prompt and a confirm body. (An earlier draft of this row said "with the TUI suspended" and named an input DEFAULT: neither is true at the pin -- these dialogs are drawn by the LIVE tui, and `ExtensionInputComponent` accepts a placeholder and never renders it.) `pause-windows.mjs` checks only `isNonEmptyString(w.scope)`, so the FILE is the whole validator; `secrets-command.ts` has the same shape. Not a regression -- it behaved identically before #382 -- but it IS model-reachable, which the first draft of this row denied: a tool's `execute` gets its own `ctx` from pi, never passing the command door, and `dispatch_trigger_edit`'s confirm body interpolates the model's own `flow`. The gate is one wrapper per DOOR -- six of them: the command handler, the exported dashboard action, `confirmedWrite` for every tool that confirms, the secrets command, the setup wizard and the `session_start` handler (the sixth found by a second review pass, its own notify a constant -- gated rather than excused, because a door left out for being currently safe is how the render gates were refuted) -- not at the 75 call sites: more than one is there because no single one is the only door, and a test driving the dashboard action directly would otherwise exercise the dialogs ungated -- the same covered-on-one-path shape the render gates were refuted for. `notify` is included, because an error message that moves the cursor is the same class as any other. `custom` is not, because it takes a factory whose output is the overlay. **AND THE SELECTION IS TRANSLATED BACK**: pi returns the exact option string it was handed, so a scrubbed copy made `labels.indexOf(picked)` fail and four edit/delete actions returned silently -- a gate that introduced the silent no-op it exists to prevent, caught by a review pass and pinned. **Code evidence**: `admin/src/dialog-gate.mjs` -> `gateDialogs`. | | 2026-09-24 | Issue #403. **`DES-ADMIN-VIA-PI-EXTENSION` AMENDED**: the unframed degrade honours the width it was given. It was a layout promise that held on one branch of an `if` -- the run rows are bounded to their own fixed 24-column pane, everything else was returned as built (the live tail's unframed branch had no clip at all, and said so) -- so a single render showed rows cut to the pane beside section headers, settings lines and hints running past it (measured: 28 of 59 lines over the width at `render(4)`, the widest 52 columns). Same shape as the logs viewer's #399, which this file had already fixed. **A width that is MISSING or non-finite is left alone**, deliberately: the degrade is what those get too, and `clip(x, NaN)` returns the empty string, so inventing a number there would blank the pane rather than degrade it. **The clip is by LENGTH and that rests on a measured assumption**, now pinned: this path is monochrome, zero of its lines carrying an SGR sequence under a real theme, because its own branches strip colour before returning. If that changes the result is worse than lost colour -- `clipData` substitutes the ESC and leaves `[31m` VISIBLE, charging four phantom columns against the width -- and the answer is an ANSI-aware cut. The width remains a UTF-16 count, wrong for CJK exactly as it is everywhere else in this module, which is #401 and is not made worse here. The TRUNCATION is centralised beside the comparison, not only the comparison: a review pass showed that splitting them is invisible, since `Math.round(7.9)` frames nothing and then clips to 8. **Code evidence**: `admin/src/dashboard.ts` -> `renderPanel`, `framedAt`, `degradeWidth`. | -| 2026-09-24 | Issue #401. **`DES-ADMIN-VIA-PI-EXTENSION` AMENDED**: the panel measures COLUMNS, and the entry's own sentence saying it does not is what changed. Every width promise in this module was a UTF-16 code-unit count -- `clip`, `pad`, `styler.cell`, `divider`, `clipPlain`, `visibleLen`, and both frame builders' top rules -- so a CJK job id framed a pane this module reported as exactly 80 columns and pi-tui drew at 90, and a combining mark ragged it the other way. Measured per case against pi-tui's own `visibleWidth` at the pin, as this module counted it against what a terminal draws: a CJK job id 5 against 10, fullwidth latin 3 against 6, Hangul 5 against 10, combining marks 5 against 3. **ONE TABLE, IN `panel.mjs`, AND THE STYLER COMES TO IT.** The obvious alternative was importing pi-tui's `visibleWidth` into `style.mjs`, which is overlay-only and already depends on pi: rejected because `panel.mjs` owns `clip` and the monochrome renderer, is pinned to have NO imports, and the two renderers draw the same geometry. A width rule holding in the framed pane and not in the plain one is the shape that holds on one branch of an `if`, which #403 had just been about -- and a mutation of the MONOCHROME top rule survived a suite pinning only the coloured one, which is that same shape appearing inside this very fix. **WHAT IT COUNTS**: a combining mark (Mn/Me) 0, a zero-width or formatting character 0, East Asian Wide and Fullwidth 2, everything else 1 including an unassigned code point, because guessing wider on an unknown breaks the panes the rule was added to fix. Mc is deliberately NOT zero: a spacing mark does occupy a column. **ITS LIMIT, PINNED AS A DISAGREEMENT rather than left to surprise a reader**: the table sums CODE POINTS, so an emoji ZWJ sequence counts every member where pi collapses it to one glyph, 6 against 2. Terminals disagree with each other there too, which is why no table in this project settles it; over-counting cuts EARLY, so the residual is a short line, never a broken border. **THE ORACLE IS pi ITSELF**, loaded from the pinned installation per `CONST-PI-VERSION-PINNED`, and asserted to be a function rather than skipped on failure: a sibling pin made exactly that silent-fallback mistake once, and an oracle that quietly vanishes stops being one. **`DES-PANEL-SEPARATE-FROM-RECEIVER` UNCHANGED, checked**: nothing about where the panel runs or what it binds moves. **The bidi and zero-width residual is UNCHANGED and still open**: those code points sit outside a control-byte class defined by what a terminal INTERPRETS, and they are a reader-deception question rather than a width one. **Code evidence**: admin/src/panel.mjs -> columnsOf, sliceColumns, clip, pad, box; admin/src/style.mjs -> visibleLen, cell, divider, frame, clipPlain; admin/test/width.test.mjs. | +| 2026-09-24 | Issue #401. **`DES-ADMIN-VIA-PI-EXTENSION` AMENDED**: the panel measures COLUMNS, and the entry's own sentence saying it does not is what changed. Every width promise in this module was a UTF-16 code-unit count -- `clip`, `pad`, `styler.cell`, `divider`, `clipPlain`, `visibleLen`, both frame builders' top rules, the line editor's window and the model-visible runs table -- so a CJK job id framed a pane this module reported as exactly 80 columns and the renderer drew at 90, and a combining mark ragged it the other way. Measured against the pinned renderer, as this module counted against what it draws: a CJK job id 5 against 10, fullwidth latin 3 against 6, Hangul 5 against 10, combining marks 5 against 3. **ONE TABLE, IN `panel.mjs`, AND THE STYLER COMES TO IT.** Importing pi-tui's `visibleWidth` into `style.mjs` was the alternative, and that module is overlay-only and already depends on pi: rejected because `panel.mjs` owns `clip` and the monochrome renderer, its purity pin forbids it reaching the world at all, and the two renderers draw the same geometry. A width rule holding in the framed pane and not in the plain one is the shape that holds on one branch of an `if`, which #403 had just been about -- and a mutation of the MONOCHROME top rule survived a suite pinning only the coloured one, which is that same shape appearing inside this very fix. **THE CORRECTION THAT MATTERS: the table is TRANSCRIBED FROM THE RENDERER, not written out of UAX #11.** A first version was hand-written from the standard and checked against ten strings. It passed, and it UNDER-counted 139,820 code points -- CJK Extension B through H, Tangut, the Kana supplement, the Hangul Jamo extensions, every emoji block added since Unicode 13 -- so the fix carried the defect it was fixing, on exactly the CJK repository name the issue is about; and in two classes (an astral character, and a character followed by U+FE0F) it was WORSE than the `.length` it replaced, which had been right there by accident because such a sequence is two code units and two columns. Ten examples cannot see that, which is why the bolt is **a sweep of every code point there is**, plus every character-with-U+FE0F pair: the table may never measure NARROWER than the renderer, and every place it measures wider must be an unassigned code point. The DIRECTION is the whole property, and "over-counting is harmless" would be too kind: BOTH directions rag a frame, differently. An over-count pads a body line as though it were wider than it is, so the line comes out SHORT and the right border sits left of the one above it, contained by the pane -- a review pass measured 200 such lines from ZWJ sequences, Hangul Jamo clusters and Devanagari. An under-count runs the line PAST the pane and past the terminal, where it wraps and takes the border with it, which is the defect this issue names. The sweep is one-sided on purpose, and what is left is the safer of two bad shapes rather than a harmless one. **WHAT IT COUNTS**: zero for a mark and for a format character plus the fillers, two for the renderer's own double-width set, one more when U+FE0F asks for the emoji form of a narrow base, one for everything else including an unassigned code point. Mc is no longer excepted: a first version reasoned from Unicode that a spacing mark occupies a column, and the renderer that draws the pane says otherwise, so the renderer wins. **ITS ONE REMAINING LIMIT, PINNED AS A DISAGREEMENT**: the table sums CODE POINTS, so an emoji ZWJ sequence counts every member where the renderer collapses it to one glyph, 6 against 2; terminals disagree there too, and over-counting is the safe direction. **THE ORACLE IS pi ITSELF**, loaded from the pinned installation per `CONST-PI-VERSION-PINNED` with its VERSION asserted, not just its presence, because pi depends on pi-tui by a range and a table pinned to the wrong renderer is not pinned; and asserted to be a function rather than skipped on failure, since a sibling pin made exactly that silent-fallback mistake once. **Also repaired, as the same defect class rather than as scope creep**: the line editor windowed and padded by code units, so a CJK value measured half what the terminal drew and a window edge put a BARE LOW SURROGATE into the live trigger editor; `renderRuns`, the model-visible table, sized its columns by `.length`, so one CJK character in a `local:` target shifted every later column of that row; a cut to zero returned a floating combining mark, and a cut through a ZWJ sequence left the joiner dangling before the ellipsis; and two hostile-string caps in the graph model could split a surrogate pair. **`DES-PANEL-SEPARATE-FROM-RECEIVER` UNCHANGED, checked**: nothing about where the panel runs or what it binds moves. **The bidi and zero-width residual is UNCHANGED and still open**: those code points sit outside a control-byte class defined by what a terminal INTERPRETS, and they are a reader-deception question rather than a width one. **Code evidence**: admin/src/panel.mjs -> columnsOf, WIDE, ZERO_WIDTH, sliceColumns, clip, pad, box, makeLineInput; admin/src/style.mjs -> visibleLen, cell, divider, frame, clipPlain; admin/src/render.mjs -> renderRuns; admin/src/graph-model.mjs -> clipName; admin/test/width.test.mjs. | From 0ccdcc3425da6bd1ce7ebb8603613ebc68b2731e Mon Sep 17 00:00:00 2001 From: Rob Boerman Date: Thu, 24 Sep 2026 17:17:00 +0200 Subject: [PATCH 3/5] fix(admin): count and cut through one stepper, and fix the keycap (#401) A second adversarial pass found two UNDER-counts, which is the direction that runs a line past its pane, and both were in the repair rather than in the original. A KEYCAP measured 1 where the renderer draws 2. The previous commit excluded ASCII from emoji-form promotion, which is right on its own terms: `#`, `*` and the digits stay one column under U+FE0F alone. Its comment then named the keycap as the reason, while the keycap itself went on measuring wrong. It takes three code points, base plus U+FE0F plus U+20E3, and only twelve bases. THE COUNT AND THE CUT NOW WALK ONE STEPPER, which is the structural half. The previous commit put the U+FE0F state inside `columnsOf` alone, and `sliceColumns` and the line editor call that function ONE CHARACTER AT A TIME, where a base is 1 and its selector is 0. So the cut spent a budget of 2 on a glyph drawn 3 wide and the pane overflowed. One generator now yields `{text, cols}` steps: the count sums them and the cut takes whole ones, so the two cannot disagree, and a selector can no longer be stranded without its base. The line editor needed the same fix on its EDIT side, which is the sharper half. `cursor` moved one code unit at a time, so two `left`s put it between the halves of an astral pair and the next `backspace` deleted ONE HALF. The surviving half stayed in `value()`, which is the string that gets saved, where no amount of rendering can repair it. All four movements now step by character. Two claims were false and are corrected rather than softened: - "plus every character-with-U+FE0F pair" appeared in the commit body, the PR, the docblock and the spec row. The test did no such sweep, it had a six-entry list. The sweep now exists, over every base with U+FE0F and with a keycap, and it is what would have caught the keycap defect. - "its one remaining limit" named the ZWJ sequence alone. A skin-tone modifier, a regional-indicator flag pair, a Hangul jamo cluster and a Devanagari cluster over-count too. What is left is a class, not one shape, and every member of it over-counts, so every one draws short inside its border rather than through it. Newly pinned, each having survived the previous commit: the two `divider` measurements whose revert changes only which of the label and the meta is clipped first, which total-width assertions cannot see; the line editor's cursor snap, which no test reached because every fixture built a fresh editor with the cursor past the end; the skill-frontmatter cap, the second of the two graph caps; and the sweep's own SCOPE, by asserting how many code points it walked and how large the renderer's wide set is, because narrowing the loop bound to the BMP left the suite green. The two guard-weakening mutations that remain unkillable are stated in the test rather than chased: a loosened assertion is unobservable while the thing it guards against is absent. Both are shown live by mutating the source instead. Thirty-four mutations checked, every one red, with zero survivors. Full suite in the CI posture: 4211 tests, 0 fail, 1 skipped. Same under the +399 day clock shift. All four guards pass and a run under a fresh TMPDIR leaves nothing behind. DES-ADMIN-VIA-PI-EXTENSION's amendment corrected in the two places it overclaimed, and the revision row with it. No version moved. Signed-off-by: Rob Boerman --- admin/src/panel.mjs | 208 +++++++++++++++++++++++++------------- admin/test/width.test.mjs | 143 +++++++++++++++++++++++++- specs/design.md | 20 ++-- 3 files changed, 291 insertions(+), 80 deletions(-) diff --git a/admin/src/panel.mjs b/admin/src/panel.mjs index 5313b57..52ed16f 100644 --- a/admin/src/panel.mjs +++ b/admin/src/panel.mjs @@ -209,13 +209,16 @@ export function clipData(line, w) { * - zero for a mark (`\p{M}`) and for a format character (`\p{Cf}`), plus the fillers and * noncharacters the renderer also draws as nothing; * - two for every code point in `WIDE`, which is that renderer's own double-width set; - * - one more when U+FE0F follows a character that draws narrow on its own and wide in its emoji form; + * - two for a narrow character followed by U+FE0F, which asks for its emoji form, and two for a keycap, + * which is one of twelve bases plus U+FE0F plus U+20E3 drawn as a single key; * - one for everything else, including an unassigned code point, because guessing wider on an unknown is * how a rule starts breaking the panes it was added to fix. * - * WHAT IT IS HELD TO, in `width.test.mjs`: a sweep of every code point there is, plus every - * character-with-U+FE0F pair, asserting that this never measures NARROWER than the renderer and that every - * place it measures wider is an unassigned code point. + * WHAT IT IS HELD TO, in `width.test.mjs`: two sweeps, one of every code point there is and one of every + * character followed by U+FE0F and by a keycap, asserting that this never measures NARROWER than the + * renderer and that every place it measures wider is a declared departure. The pair sweep is there because + * a first version CLAIMED it and shipped a six-entry list instead, and the hole was one selector further + * along: a keycap measured 1 against the renderer's 2. * * THE DIRECTION IS THE WHOLE POINT, and stating it as "over-counting is harmless" would be too kind: BOTH * directions rag a frame, and they rag it differently. An over-count pads a body line as though it were @@ -225,41 +228,78 @@ export function clipData(line, w) { * sweep is one-sided on purpose, and the residual below is the safer of two bad shapes, not a harmless * one. * - * ITS LIMIT, stated rather than implied: this sums CODE POINTS, so an emoji ZWJ sequence -- a family, a - * profession, a skin tone -- counts every member and comes out wider than the single glyph a terminal - * draws. Measured on a family: 6 here against the renderer's 2. Terminals disagree with each other there - * too, which is why no width table in this project will settle it. A grapheme-aware count needs - * `Intl.Segmenter` and a terminal that agrees; the residual is one over-wide line, in the safe direction. + * WHAT IS LEFT, and it is a CLASS rather than one shape: this sums steps, and the renderer collapses some + * runs of them into a single glyph. An emoji ZWJ sequence counts every member (6 here against 2 there), and + * so do a skin-tone modifier, a regional-indicator flag pair, a Hangul jamo cluster and a Devanagari + * cluster. A first version of this paragraph named the ZWJ sequence as the ONLY case; a review pass + * measured four more, and found the one case that went the other way -- a keycap, at 1 against 2 -- which + * is fixed above rather than listed here, because an under-count is the direction that overflows a pane. + * + * Every remaining case over-counts, so every one draws SHORT inside its border rather than through it. + * Terminals disagree with each other on all of them, which is why no width table in this project will + * settle them; a grapheme-aware count needs `Intl.Segmenter` and a terminal that agrees. */ export function columnsOf(s) { let n = 0; - // The ONE piece of state, and the one thing a per-character table cannot do without: U+FE0F asks for the - // emoji form of the character BEFORE it, and the renderer then draws that character two columns wide. - let promotable = false; - for (const ch of String(s ?? "")) { - if (ch === VS16 && promotable) { - n += 1; - promotable = false; + for (const step of widthSteps(s)) n += step.cols; + return n; +} + +/** + * THE STRING AS `{ text, cols }` STEPS, and the reason it exists rather than a per-character width call. + * + * Two shapes are WIDER THAN THEIR PARTS, so no function that asks "how wide is this character" can size + * them: U+FE0F asks for the emoji form of the character before it, and a keycap is a base, U+FE0F and + * U+20E3 drawn as one two-column glyph. A first repair put that state inside `columnsOf` alone, and a + * review pass found what that leaves: `sliceColumns` and the line editor call `columnsOf` ONE CHARACTER AT + * A TIME, where the base is 1 and the selector is 0, so the cut spent a budget of 2 on a glyph the terminal + * draws 3 wide and the pane overflowed. Counting and cutting now walk the same steps, so they cannot + * disagree: the count sums `cols`, and the cut takes whole `text` chunks or none of them, which is also why + * it can no longer strand a selector without its base. + */ +function* widthSteps(s) { + const chars = [...String(s ?? "")]; + for (let i = 0; i < chars.length; i++) { + const ch = chars[i]; + // HALF A PAIR IS NOT A CHARACTER. It is a step of its own so a cut can drop it by name. + if (ch.length === 1 && ch.charCodeAt(0) >= 0xd800 && ch.charCodeAt(0) <= 0xdfff) { + yield { text: ch, cols: 0, orphan: true }; + continue; + } + if (ZERO_WIDTH.test(ch)) { + yield { text: ch, cols: 0 }; continue; } - promotable = false; - if (ZERO_WIDTH.test(ch)) continue; if (WIDE.test(ch)) { - n += 2; + yield { text: ch, cols: 2 }; continue; } - n += 1; - // ONLY A NARROW BASE IS PROMOTABLE, and it is narrow by having reached this line at all: a code point - // the renderer already draws wide took the `WIDE` branch above and never gets here. A first version - // also tested `\p{Emoji_Presentation}` here; measured against the pin, every code point with that - // property is in `WIDE`, so the test was dead and a mutation removing it survived the whole suite. - // The ASCII exclusion is NOT dead: `#`, `*` and the digits carry `\p{Emoji}` and stay one column, - // because their emoji form is a keycap and needs U+20E3 rather than U+FE0F alone. - promotable = ch.codePointAt(0) > 0x7f && TEXT_EMOJI.test(ch); + // A NARROW BASE, which is narrow by having reached this line: a code point the renderer already draws + // wide took the branch above. Only here can a following selector change anything. + if (chars[i + 1] === VS16) { + // THE KEYCAP IS THE CASE A FIRST REPAIR GOT BACKWARDS. It excluded ASCII from promotion, correctly, + // because `#`, `*` and the digits stay one column under U+FE0F alone -- and then said so in a comment + // that named the keycap as the reason, while the keycap itself went on measuring 1 against the + // renderer's 2. It takes all three code points, and only these twelve bases. + if (KEYCAP_BASE.test(ch) && chars[i + 2] === KEYCAP) { + yield { text: ch + VS16 + KEYCAP, cols: 2 }; + i += 2; + continue; + } + if (ch.codePointAt(0) > 0x7f && TEXT_EMOJI.test(ch)) { + yield { text: ch + VS16, cols: 2 }; + i += 1; + continue; + } + } + yield { text: ch, cols: 1 }; } - return n; } +/** VARIATION SELECTOR-16 asks for the emoji form; U+20E3 encloses the keycap bases below in a key. */ +const KEYCAP = "\u20e3"; +const KEYCAP_BASE = /[#*0-9]/; + /** VARIATION SELECTOR-16, which asks for the emoji form of the character before it. */ const VS16 = "\ufe0f"; @@ -331,31 +371,26 @@ export function clip(line, w) { */ export function sliceColumns(s, w) { const budget = Math.max(0, Math.trunc(w) || 0); - // NO BUDGET, NO CONTENT. Without this a zero-column character passes the test below and a cut to zero + // NO BUDGET, NO CONTENT. Without this a zero-column step passes the test below and a cut to zero // returned a floating accent with nothing to attach to. if (budget === 0) return ""; let out = ""; let used = 0; - for (const ch of String(s ?? "")) { - // HALF A PAIR IS NOT A CHARACTER, so it is not carried into a cut. Walking code points means this cut - // cannot CREATE one, which is not the same as dropping one the INPUT already held: a review pass - // measured `clip` passing a lone high surrogate on where the old slice-then-repair had removed it, and - // the old repair only ever looked at the last code unit, so a leading or interior one survived it too. - if (ch.length === 1 && ch.charCodeAt(0) >= 0xd800 && ch.charCodeAt(0) <= 0xdfff) continue; - const cols = columnsOf(ch); - if (used + cols > budget) break; - out += ch; - used += cols; + for (const step of widthSteps(s)) { + // A lone surrogate the INPUT already held is dropped rather than carried: the old code-unit repair only + // ever looked at the last unit, so a leading or interior one survived it. An input that holds one and + // needs no cut at all still carries it, because `clip` returns early when the string fits: that is + // issue #402's ground rather than this one's. + if (step.orphan) continue; + if (used + step.cols > budget) break; + out += step.text; + used += step.cols; } // A TRAILING JOINER IS DANGLING: it joins this character to the next one, and the next one is what was // just cut away. Left in place it reaches the terminal ahead of the ellipsis and asks it to join a glyph - // to a horizontal bar. The same is true of a variation selector whose base was cut, which cannot happen - // here (a selector is only ever appended after its base) but costs nothing to drop with it. - // - // - // An input that already holds a lone surrogate and needs no cut at all still carries it: `clip` returns - // early when the string fits. That is issue #402's ground rather than this one's. - return out.replace(/[\u200d\ufe0e\ufe0f]+$/u, ""); + // to a horizontal bar. A selector cannot be stranded here any more, because a promoted base carries its + // selector inside one step, but a ZWJ is a step of its own and is exactly what this strips. + return out.replace(/\u200d+$/u, ""); } /** @@ -559,6 +594,26 @@ export function fmtCost(cost) { */ export const LINE_INPUT_CURSOR = ["\x01", "\x02"]; +/** Code units in the character ending at `at`, so a cursor move never lands between the halves of a pair. */ +function charBefore(s, at) { + const lo = s.charCodeAt(at - 1); + if (lo >= 0xdc00 && lo <= 0xdfff && at >= 2) { + const hi = s.charCodeAt(at - 2); + if (hi >= 0xd800 && hi <= 0xdbff) return 2; + } + return 1; +} + +/** Code units in the character starting at `at`, the forward twin of `charBefore`. */ +function charAfter(s, at) { + const hi = s.charCodeAt(at); + if (hi >= 0xd800 && hi <= 0xdbff && at + 1 < s.length) { + const lo = s.charCodeAt(at + 1); + if (lo >= 0xdc00 && lo <= 0xdfff) return 2; + } + return 1; +} + /** * A pure single-line text-input state machine. No key decoding lives here -- the caller decodes raw * input (keys.mjs) and calls the edit methods, which keeps this module free of the pi-tui resolver and @@ -581,19 +636,25 @@ export function makeLineInput(initial = "") { value = value.slice(0, cursor) + clean + value.slice(cursor); cursor += clean.length; }, + // ONE CHARACTER, NOT ONE CODE UNIT, in all four (issue #401). These moved and deleted by code unit, so + // two `left`s put the cursor between the halves of an astral pair and the next `backspace` deleted ONE + // HALF: the surviving half stayed in `value`, which is what `value()` hands to whatever saves it, so + // the broken character outlived the session. Rendering cannot repair that -- the damage is in the + // stored string, not in the view -- which is why the fix is here rather than in `render`. backspace() { if (cursor === 0) return; - value = value.slice(0, cursor - 1) + value.slice(cursor); - cursor -= 1; + const step = charBefore(value, cursor); + value = value.slice(0, cursor - step) + value.slice(cursor); + cursor -= step; }, del() { - if (cursor < value.length) value = value.slice(0, cursor) + value.slice(cursor + 1); + if (cursor < value.length) value = value.slice(0, cursor) + value.slice(cursor + charAfter(value, cursor)); }, left() { - if (cursor > 0) cursor -= 1; + if (cursor > 0) cursor -= charBefore(value, cursor); }, right() { - if (cursor < value.length) cursor += 1; + if (cursor < value.length) cursor += charAfter(value, cursor); }, home() { cursor = 0; @@ -607,46 +668,47 @@ export function makeLineInput(initial = "") { }, render(width, { focused = true } = {}) { const w = Math.max(1, Math.trunc(width) || 1); - // THE WINDOW IS CHOSEN IN COLUMNS AND ITS EDGES ARE WHOLE CHARACTERS (issue #401). This was + // THE WINDOW IS CHOSEN IN COLUMNS AND ITS EDGES ARE WHOLE STEPS (issue #401). This was // `value.slice(start, start + w).padEnd(w)` on UTF-16 indices, so it measured a CJK value at half // what the terminal draws, and a window edge landing between the halves of an astral pair emitted a - // BARE LOW SURROGATE into the live trigger editor -- the exact hazard the rest of this module cuts - // around. `cursor` is still a code-unit index, because every edit method moves it by one unit, so it - // is snapped to a character boundary here rather than trusted. - const chars = [...value]; + // BARE LOW SURROGATE into the live trigger editor. It walks `widthSteps` rather than characters for + // the reason that function exists: a per-character measure is one column short on an emoji-form + // sequence, so a window built from one overflowed its own pane. + const steps = [...widthSteps(value)].filter((step) => !step.orphan); const offs = []; let at = 0; - for (const c of chars) { + for (const step of steps) { offs.push(at); - at += c.length; + at += step.text.length; } offs.push(at); + // `cursor` is a code-unit index and the edit methods keep it on a character boundary, but a step can + // span three of them, so it is snapped to the step it falls inside rather than trusted to name one. let ci = offs.findIndex((o) => o >= cursor); - if (ci < 0) ci = chars.length; + if (ci < 0) ci = steps.length; // THE RESERVED CELL MOVES THE WINDOW'S START, not its length: the window is `w` columns wide, and // the cursor is kept at most `w - 1` columns past the start so its own cell is always inside it. let start = ci; let back = 0; - while (start > 0 && back + columnsOf(chars[start - 1]) <= w - 1) { + while (start > 0 && back + steps[start - 1].cols <= w - 1) { start -= 1; - back += columnsOf(chars[start]); + back += steps[start].cols; } let end = start; let used = 0; - while (end < chars.length && used + columnsOf(chars[end]) <= w) { - used += columnsOf(chars[end]); + while (end < steps.length && used + steps[end].cols <= w) { + used += steps[end].cols; end += 1; } - const head = chars.slice(start, Math.min(ci, end)).join(""); - const text = chars.slice(start, end).join(""); - if (!focused) return pad(text, w); - // The cursor wraps a WHOLE character, never one half of a pair, and sits on a space once it is past - // the last character the window shows. - const onChar = ci < end; - const under = onChar ? chars[ci] : " "; - const tail = onChar ? chars.slice(ci + 1, end).join("") : ""; - const shown = head + (onChar ? under : "") + tail; - const fill = Math.max(0, w - columnsOf(shown) - (onChar ? 0 : 1)); + const textOf = (a, b) => steps.slice(a, b).map((step) => step.text).join(""); + const head = textOf(start, Math.min(ci, end)); + if (!focused) return pad(textOf(start, end), w); + // The cursor wraps a WHOLE step, never one half of a pair and never half a keycap, and sits on a + // space once it is past the last step the window shows. + const onStep = ci < end; + const under = onStep ? steps[ci].text : " "; + const tail = onStep ? textOf(ci + 1, end) : ""; + const fill = Math.max(0, w - columnsOf(head + (onStep ? under : "") + tail) - (onStep ? 0 : 1)); return head + LINE_INPUT_CURSOR[0] + under + LINE_INPUT_CURSOR[1] + tail + " ".repeat(fill); }, }; diff --git a/admin/test/width.test.mjs b/admin/test/width.test.mjs index ccef3a5..3ae1f1b 100644 --- a/admin/test/width.test.mjs +++ b/admin/test/width.test.mjs @@ -152,6 +152,8 @@ test("the table is held to the renderer on every code point there is (#401)", as // line comes out short and the border holds. Under-counting cuts LATE: the line runs past the border, // which is the defect this issue names. const OVER = []; + let swept = 0; + let wide = 0; for (let cp = 0; cp <= 0x10ffff; cp++) { // A surrogate on its own is not a character and neither side promises anything about it. if (cp >= 0xd800 && cp <= 0xdfff) continue; @@ -163,6 +165,8 @@ test("the table is held to the renderer on every code point there is (#401)", as const ch = String.fromCodePoint(cp); const ours = columnsOf(ch); const theirs = visibleWidth(ch); + swept += 1; + if (theirs === 2) wide += 1; assert.ok(ours >= theirs, `U+${cp.toString(16).toUpperCase().padStart(4, "0")}: we say ${ours}, the renderer draws ${theirs}`); if (ours !== theirs) OVER.push(cp); } @@ -176,6 +180,61 @@ test("the table is held to the renderer on every code point there is (#401)", as ); } assert.ok(OVER.length > 0, "the unassigned departure is real, not a dead clause"); + + // THE SWEEP'S OWN SCOPE IS PINNED, because a sweep is only a bolt while it actually sweeps. A review pass + // showed that narrowing the loop bound to `0xffff` -- dropping the whole astral plane, which is exactly + // where the 139,820-code-point hole was -- left the suite green, as did widening the control skip and + // weakening the assertion above. These two counts fail on any of those. + assert.equal(swept, 1111999, "the sweep covers every code point outside the surrogates and the control class"); + assert.equal(wide, 182889, "and the renderer's double-width set is transcribed whole"); + + // WHAT THESE TWO COUNTS ARE FOR, and what they cannot do. They pin the sweep's SCOPE: narrowing the loop + // bound to the BMP -- which is exactly where the 139,820-code-point hole was -- or widening the control + // skip now fails here, where before it left the suite green. They do NOT make the two assertions above + // mutation-detectable, and nothing can: weakening a guard is unobservable while the thing it guards + // against is absent, so on a correct table `ours >= theirs` and `ours >= 0` pass the same inputs. Those + // assertions are shown to be live the only way a guard can be, by mutating the SOURCE: reverting one + // range of `WIDE` fails the first, and making the zero rule stop zeroing format characters fails the + // second. + +}); + +test("every character-with-U+FE0F pair, and every keycap, agrees with the renderer (#401)", async () => { + const visibleWidth = await loadVisibleWidth(); + assert.equal(typeof visibleWidth, "function"); + + // THE SWEEP THE FIRST REPAIR SAID IT DID AND DID NOT. Its commit, its docblock and its spec row all + // claimed "every code point, plus every character-with-U+FE0F pair"; the pair half was a six-entry list. + // A review pass found the hole one selector further along: a KEYCAP is a base, U+FE0F and U+20E3 drawn + // as one two-column glyph, and it measured 1 against the renderer's 2 -- an UNDER-count, the direction + // that runs a line past its pane. So both sequence shapes are swept, and the keycap one is why. + let pairs = 0; + let keycaps = 0; + for (let cp = 0x20; cp <= 0x10ffff; cp++) { + if (cp >= 0xd800 && cp <= 0xdfff) continue; + if (cp === 0x7f || (cp >= 0x80 && cp <= 0x9f)) continue; + const base = String.fromCodePoint(cp); + for (const seq of [base + "\ufe0f", base + "\ufe0f\u20e3"]) { + const ours = columnsOf(seq); + const theirs = visibleWidth(seq); + assert.ok(ours >= theirs, `${JSON.stringify(seq)}: we say ${ours}, the renderer draws ${theirs}`); + // THE TWO STATED DEPARTURES, and nothing else. An unassigned base is the table's declared guess of + // one column. The other is a keycap applied to a base that HAS no keycap form -- a heart in a key -- + // which the renderer collapses to one column and we count as the promoted base plus an enclosing + // mark. It is an over-count, so it draws short rather than overflowing, and it is a sequence no + // writer produces: the twelve real keycap bases are asserted exactly, just below. + const degenerateKeycap = seq.endsWith("\u20e3") && !/[#*0-9]/.test(base); + if (ours !== theirs) { + assert.ok(!/\p{Assigned}/u.test(base) || degenerateKeycap, `${JSON.stringify(seq)} over-counts an assigned base for no stated reason`); + } + } + pairs += 1; + // The keycap RULE, isolated: a base the selector alone leaves narrow and the enclosing key widens. A + // looser count would also catch the 201 the selector promotes on its own and prove nothing about it. + if (columnsOf(base) === 1 && columnsOf(base + "\ufe0f") === 1 && columnsOf(base + "\ufe0f\u20e3") === 2) keycaps += 1; + } + assert.equal(pairs, 1111999, "the pair sweep covers every base there is"); + assert.equal(keycaps, 12, "and exactly the twelve keycap bases are promoted, which is what the renderer does"); }); test("U+FE0F asks for the emoji form, and the table follows the renderer there too (#401)", () => { @@ -283,8 +342,12 @@ test("a cut drops a lone surrogate rather than passing it on (#401)", () => { // which is not the same as dropping one that was already in the input: a review pass measured `clip` // passing a lone high surrogate on where the old slice-then-`dropLoneSurrogate` had removed it. const lone = /[\ud800-\udbff](?![\udc00-\udfff])|(? w, `the fixture has to be cut at ${w} or this proves nothing`); assert.doesNotMatch(clip(s, w), lone, `clip(${JSON.stringify(s)}, ${w})`); } } @@ -353,3 +416,81 @@ test("the graph's hostile-string caps never leave half a character (#401)", asyn } } }); + +test("the divider clips the meta before the label, measured in columns (#401)", () => { + // TOTAL WIDTH IS NOT ENOUGH, which is why this asserts the CONTENT. `divider` has four column measures + // and two of them survived a revert to `.length` while the suite was green: `Math.max(1, ...)` absorbs + // the error into the rule length, so the line stays exactly its width and only the PRIORITY changes. + // That priority is the documented behaviour ("clip the META first, and the LABEL only if it alone still + // does not fit"), so it is what gets asserted. + const styler = makeStyler(PLAIN_THEME); + const wide = styler.divider("ジョブ番号", "会社/製品", 20); + assert.equal(visibleLen(wide), 20, "still exactly the width"); + assert.ok(wide.includes("ジョブ番号"), "the label is kept whole while the meta still has room to give"); + // A meta measured by `.length` is under-sized, so the label gets clipped in its place. + const marks = styler.divider("operator label here", "jób́márks", 28); + assert.equal(visibleLen(marks), 28, "still exactly the width"); + assert.ok(marks.includes("OPERATOR LABEL HERE"), "a combining-mark meta does not cost the label its tail"); +}); + +test("the line editor keeps the cursor on the step the caller moved it to (#401)", () => { + // THE SNAP THE REPAIR ADDED, pinned. Every earlier case built a fresh editor, so the cursor always sat + // past the end and the snap was never exercised: a mutation using the code-unit index as a step index + // survived the whole suite. These move the cursor with the object's own methods. + const li = makeLineInput("\u{1f600}\u{1f600}\u{1f600}"); + li.home(); + li.right(); + const out = li.render(10); + const before = out.slice(0, out.indexOf(LINE_INPUT_CURSOR[0])); + // ONE `right` IS ONE CHARACTER, so the cursor sits on the second emoji. Reading `cursor` as a step index + // would put it on the third, because an astral character is two code units. + assert.equal(columnsOf(before), 2, "the cursor sits on the second emoji, not the third"); + // And the promise still holds from every position the caller can reach. + for (const value of ["\u{1f600}\u{1f600}\u{1f600}", "会社/製品", "❤️❤️ab", "1️⃣1️⃣"]) { + const ed = makeLineInput(value); + ed.home(); + for (let i = 0; i <= value.length; i++) { + for (const w of [1, 2, 4, 8, 16]) { + const r = ed.render(w); + const plain = r.split(LINE_INPUT_CURSOR[0]).join("").split(LINE_INPUT_CURSOR[1]).join(""); + assert.equal(columnsOf(plain), w, `${JSON.stringify(value)} cursor ${i} width ${w}`); + } + ed.right(); + } + } +}); + +test("an edit never leaves half a character in the value itself (#401)", () => { + // THE STATE MACHINE, not the render. `cursor` moves one CODE UNIT at a time, so a backspace with the + // cursor between the halves of an astral pair used to delete one half and leave the other IN THE STORED + // VALUE -- which `value()` hands back to whatever saves it, so the half-character outlives the session. + // The render-side repair cannot reach that; this is the edit-side half of the same promise. + const lone = /[\ud800-\udbff](?![\udc00-\udfff])|(? { + // THE SECOND of the two graph caps. `clipName` is pinned above; this one was repaired in the same commit + // and nothing drove it, so a revert survived. The prefix is odd for the same reason as there: a cut + // through a run of astral characters at an even offset splits nothing. + const { parseSkillMeta } = await import("../src/graph-model.mjs"); + const lone = /[\ud800-\udbff](?![\udc00-\udfff])|(? `readBeforeArming`, `changedWhileArming`, `WATCH_DEBOUNCE_MS`. | | 2026-09-24 | Issue #404. **`DES-ADMIN-VIA-PI-EXTENSION` AMENDED**: the control-byte gate names FOUR funnels, not three. #382 closed the two that render a pane and the model-visible `send`, and its own wording -- "every finished line passes one gate" -- read as complete while pi's DIALOGS took strings straight from stored fields. Measured: a pause window whose `scope` carries an erase-display, an OSC-52 clipboard write and an OSC-8 link reaches a select option, an input prompt and a confirm body. (An earlier draft of this row said "with the TUI suspended" and named an input DEFAULT: neither is true at the pin -- these dialogs are drawn by the LIVE tui, and `ExtensionInputComponent` accepts a placeholder and never renders it.) `pause-windows.mjs` checks only `isNonEmptyString(w.scope)`, so the FILE is the whole validator; `secrets-command.ts` has the same shape. Not a regression -- it behaved identically before #382 -- but it IS model-reachable, which the first draft of this row denied: a tool's `execute` gets its own `ctx` from pi, never passing the command door, and `dispatch_trigger_edit`'s confirm body interpolates the model's own `flow`. The gate is one wrapper per DOOR -- six of them: the command handler, the exported dashboard action, `confirmedWrite` for every tool that confirms, the secrets command, the setup wizard and the `session_start` handler (the sixth found by a second review pass, its own notify a constant -- gated rather than excused, because a door left out for being currently safe is how the render gates were refuted) -- not at the 75 call sites: more than one is there because no single one is the only door, and a test driving the dashboard action directly would otherwise exercise the dialogs ungated -- the same covered-on-one-path shape the render gates were refuted for. `notify` is included, because an error message that moves the cursor is the same class as any other. `custom` is not, because it takes a factory whose output is the overlay. **AND THE SELECTION IS TRANSLATED BACK**: pi returns the exact option string it was handed, so a scrubbed copy made `labels.indexOf(picked)` fail and four edit/delete actions returned silently -- a gate that introduced the silent no-op it exists to prevent, caught by a review pass and pinned. **Code evidence**: `admin/src/dialog-gate.mjs` -> `gateDialogs`. | | 2026-09-24 | Issue #403. **`DES-ADMIN-VIA-PI-EXTENSION` AMENDED**: the unframed degrade honours the width it was given. It was a layout promise that held on one branch of an `if` -- the run rows are bounded to their own fixed 24-column pane, everything else was returned as built (the live tail's unframed branch had no clip at all, and said so) -- so a single render showed rows cut to the pane beside section headers, settings lines and hints running past it (measured: 28 of 59 lines over the width at `render(4)`, the widest 52 columns). Same shape as the logs viewer's #399, which this file had already fixed. **A width that is MISSING or non-finite is left alone**, deliberately: the degrade is what those get too, and `clip(x, NaN)` returns the empty string, so inventing a number there would blank the pane rather than degrade it. **The clip is by LENGTH and that rests on a measured assumption**, now pinned: this path is monochrome, zero of its lines carrying an SGR sequence under a real theme, because its own branches strip colour before returning. If that changes the result is worse than lost colour -- `clipData` substitutes the ESC and leaves `[31m` VISIBLE, charging four phantom columns against the width -- and the answer is an ANSI-aware cut. The width remains a UTF-16 count, wrong for CJK exactly as it is everywhere else in this module, which is #401 and is not made worse here. The TRUNCATION is centralised beside the comparison, not only the comparison: a review pass showed that splitting them is invisible, since `Math.round(7.9)` frames nothing and then clips to 8. **Code evidence**: `admin/src/dashboard.ts` -> `renderPanel`, `framedAt`, `degradeWidth`. | -| 2026-09-24 | Issue #401. **`DES-ADMIN-VIA-PI-EXTENSION` AMENDED**: the panel measures COLUMNS, and the entry's own sentence saying it does not is what changed. Every width promise in this module was a UTF-16 code-unit count -- `clip`, `pad`, `styler.cell`, `divider`, `clipPlain`, `visibleLen`, both frame builders' top rules, the line editor's window and the model-visible runs table -- so a CJK job id framed a pane this module reported as exactly 80 columns and the renderer drew at 90, and a combining mark ragged it the other way. Measured against the pinned renderer, as this module counted against what it draws: a CJK job id 5 against 10, fullwidth latin 3 against 6, Hangul 5 against 10, combining marks 5 against 3. **ONE TABLE, IN `panel.mjs`, AND THE STYLER COMES TO IT.** Importing pi-tui's `visibleWidth` into `style.mjs` was the alternative, and that module is overlay-only and already depends on pi: rejected because `panel.mjs` owns `clip` and the monochrome renderer, its purity pin forbids it reaching the world at all, and the two renderers draw the same geometry. A width rule holding in the framed pane and not in the plain one is the shape that holds on one branch of an `if`, which #403 had just been about -- and a mutation of the MONOCHROME top rule survived a suite pinning only the coloured one, which is that same shape appearing inside this very fix. **THE CORRECTION THAT MATTERS: the table is TRANSCRIBED FROM THE RENDERER, not written out of UAX #11.** A first version was hand-written from the standard and checked against ten strings. It passed, and it UNDER-counted 139,820 code points -- CJK Extension B through H, Tangut, the Kana supplement, the Hangul Jamo extensions, every emoji block added since Unicode 13 -- so the fix carried the defect it was fixing, on exactly the CJK repository name the issue is about; and in two classes (an astral character, and a character followed by U+FE0F) it was WORSE than the `.length` it replaced, which had been right there by accident because such a sequence is two code units and two columns. Ten examples cannot see that, which is why the bolt is **a sweep of every code point there is**, plus every character-with-U+FE0F pair: the table may never measure NARROWER than the renderer, and every place it measures wider must be an unassigned code point. The DIRECTION is the whole property, and "over-counting is harmless" would be too kind: BOTH directions rag a frame, differently. An over-count pads a body line as though it were wider than it is, so the line comes out SHORT and the right border sits left of the one above it, contained by the pane -- a review pass measured 200 such lines from ZWJ sequences, Hangul Jamo clusters and Devanagari. An under-count runs the line PAST the pane and past the terminal, where it wraps and takes the border with it, which is the defect this issue names. The sweep is one-sided on purpose, and what is left is the safer of two bad shapes rather than a harmless one. **WHAT IT COUNTS**: zero for a mark and for a format character plus the fillers, two for the renderer's own double-width set, one more when U+FE0F asks for the emoji form of a narrow base, one for everything else including an unassigned code point. Mc is no longer excepted: a first version reasoned from Unicode that a spacing mark occupies a column, and the renderer that draws the pane says otherwise, so the renderer wins. **ITS ONE REMAINING LIMIT, PINNED AS A DISAGREEMENT**: the table sums CODE POINTS, so an emoji ZWJ sequence counts every member where the renderer collapses it to one glyph, 6 against 2; terminals disagree there too, and over-counting is the safe direction. **THE ORACLE IS pi ITSELF**, loaded from the pinned installation per `CONST-PI-VERSION-PINNED` with its VERSION asserted, not just its presence, because pi depends on pi-tui by a range and a table pinned to the wrong renderer is not pinned; and asserted to be a function rather than skipped on failure, since a sibling pin made exactly that silent-fallback mistake once. **Also repaired, as the same defect class rather than as scope creep**: the line editor windowed and padded by code units, so a CJK value measured half what the terminal drew and a window edge put a BARE LOW SURROGATE into the live trigger editor; `renderRuns`, the model-visible table, sized its columns by `.length`, so one CJK character in a `local:` target shifted every later column of that row; a cut to zero returned a floating combining mark, and a cut through a ZWJ sequence left the joiner dangling before the ellipsis; and two hostile-string caps in the graph model could split a surrogate pair. **`DES-PANEL-SEPARATE-FROM-RECEIVER` UNCHANGED, checked**: nothing about where the panel runs or what it binds moves. **The bidi and zero-width residual is UNCHANGED and still open**: those code points sit outside a control-byte class defined by what a terminal INTERPRETS, and they are a reader-deception question rather than a width one. **Code evidence**: admin/src/panel.mjs -> columnsOf, WIDE, ZERO_WIDTH, sliceColumns, clip, pad, box, makeLineInput; admin/src/style.mjs -> visibleLen, cell, divider, frame, clipPlain; admin/src/render.mjs -> renderRuns; admin/src/graph-model.mjs -> clipName; admin/test/width.test.mjs. | +| 2026-09-24 | Issue #401. **`DES-ADMIN-VIA-PI-EXTENSION` AMENDED**: the panel measures COLUMNS, and the entry's own sentence saying it does not is what changed. Every width promise in this module was a UTF-16 code-unit count -- `clip`, `pad`, `styler.cell`, `divider`, `clipPlain`, `visibleLen`, both frame builders' top rules, the line editor's window and the model-visible runs table -- so a CJK job id framed a pane this module reported as exactly 80 columns and the renderer drew at 90, and a combining mark ragged it the other way. Measured against the pinned renderer, as this module counted against what it draws: a CJK job id 5 against 10, fullwidth latin 3 against 6, Hangul 5 against 10, combining marks 5 against 3. **ONE TABLE, IN `panel.mjs`, AND THE STYLER COMES TO IT.** Importing pi-tui's `visibleWidth` into `style.mjs` was the alternative, and that module is overlay-only and already depends on pi: rejected because `panel.mjs` owns `clip` and the monochrome renderer, its purity pin forbids it reaching the world at all, and the two renderers draw the same geometry. A width rule holding in the framed pane and not in the plain one is the shape that holds on one branch of an `if`, which #403 had just been about -- and a mutation of the MONOCHROME top rule survived a suite pinning only the coloured one, which is that same shape appearing inside this very fix. **THE CORRECTION THAT MATTERS: the table is TRANSCRIBED FROM THE RENDERER, not written out of UAX #11.** A first version was hand-written from the standard and checked against ten strings. It passed, and it UNDER-counted 139,820 code points -- CJK Extension B through H, Tangut, the Kana supplement, the Hangul Jamo extensions, every emoji block added since Unicode 13 -- so the fix carried the defect it was fixing, on exactly the CJK repository name the issue is about; and in two classes (an astral character, and a character followed by U+FE0F) it was WORSE than the `.length` it replaced, which had been right there by accident because such a sequence is two code units and two columns. Ten examples cannot see that, which is why the bolt is **two sweeps: every code point there is, and every character followed by U+FE0F and by a keycap**. The table may never measure NARROWER than the renderer, and every place it measures wider must be a declared departure. The pair sweep exists because a first version of this row CLAIMED it and the test shipped a six-entry list instead, and the hole was one selector further along: a KEYCAP, twelve bases plus U+FE0F plus U+20E3 drawn as one key, measured 1 against the renderer's 2. That was an UNDER-count, so it is fixed rather than stated. The DIRECTION is the whole property, and "over-counting is harmless" would be too kind: BOTH directions rag a frame, differently. An over-count pads a body line as though it were wider than it is, so the line comes out SHORT and the right border sits left of the one above it, contained by the pane -- a review pass measured 200 such lines from ZWJ sequences, Hangul Jamo clusters and Devanagari. An under-count runs the line PAST the pane and past the terminal, where it wraps and takes the border with it, which is the defect this issue names. The sweep is one-sided on purpose, and what is left is the safer of two bad shapes rather than a harmless one. **WHAT IT COUNTS**: zero for a mark and for a format character plus the fillers, two for the renderer's own double-width set, one more when U+FE0F asks for the emoji form of a narrow base, one for everything else including an unassigned code point. Mc is no longer excepted: a first version reasoned from Unicode that a spacing mark occupies a column, and the renderer that draws the pane says otherwise, so the renderer wins. **WHAT IS LEFT IS A CLASS, NOT ONE SHAPE, and saying otherwise was the other thing a review pass refuted**: the table sums steps, so every run the renderer collapses into one glyph comes out wider than it draws -- an emoji ZWJ sequence (6 against 2), a skin-tone modifier, a regional-indicator flag pair, a Hangul jamo cluster and a Devanagari cluster. All of them OVER-count, so all of them draw short inside the border rather than through it; terminals disagree with each other on all of them. **THE ORACLE IS pi ITSELF**, loaded from the pinned installation per `CONST-PI-VERSION-PINNED` with its VERSION asserted, not just its presence, because pi depends on pi-tui by a range and a table pinned to the wrong renderer is not pinned; and asserted to be a function rather than skipped on failure, since a sibling pin made exactly that silent-fallback mistake once. **Also repaired, as the same defect class rather than as scope creep**: the line editor windowed and padded by code units, so a CJK value measured half what the terminal drew and a window edge put a BARE LOW SURROGATE into the live trigger editor; `renderRuns`, the model-visible table, sized its columns by `.length`, so one CJK character in a `local:` target shifted every later column of that row; a cut to zero returned a floating combining mark, and a cut through a ZWJ sequence left the joiner dangling before the ellipsis; and two hostile-string caps in the graph model could split a surrogate pair. **The line editor's EDIT side needed the same fix as its render side**, which is the sharper half: `cursor` moved one CODE UNIT at a time, so two `left`s put it between the halves of an astral pair and the next `backspace` deleted ONE HALF, leaving the other in `value()` -- in the string that gets SAVED, where no amount of rendering can repair it. **`DES-PANEL-SEPARATE-FROM-RECEIVER` UNCHANGED, checked**: nothing about where the panel runs or what it binds moves. **The bidi and zero-width residual is UNCHANGED and still open**: those code points sit outside a control-byte class defined by what a terminal INTERPRETS, and they are a reader-deception question rather than a width one. **AND THE COUNT AND THE CUT NOW WALK ONE STEPPER**, which is the structural half of the repair: a first version put the U+FE0F state inside `columnsOf` alone, and `sliceColumns` and the line editor call it ONE CHARACTER AT A TIME, where a base is 1 and its selector is 0 -- so the cut spent a budget of 2 on a glyph drawn 3 wide and the pane overflowed. One generator yields `{text, cols}` steps, the count sums them and the cut takes whole ones, so the two cannot disagree. **Code evidence**: admin/src/panel.mjs -> columnsOf, widthSteps, WIDE, ZERO_WIDTH, sliceColumns, clip, pad, box, makeLineInput; admin/src/style.mjs -> visibleLen, cell, divider, frame, clipPlain; admin/src/render.mjs -> renderRuns; admin/src/graph-model.mjs -> clipName; admin/test/width.test.mjs. | From 596e86661ec32de2ceffd16e89e969a9026eceed Mon Sep 17 00:00:00 2001 From: Rob Boerman Date: Thu, 24 Sep 2026 19:00:54 +0200 Subject: [PATCH 4/5] fix(admin): remove half a character where text enters, not where it is cut (#401) A third adversarial pass found the defect class again, in the function introduced to end it. `widthSteps` emitted a lone surrogate as a step of its own, and both consumers DROPPED it and re-joined what was left. Dropping it splices its neighbours back together: a base and the selector on the other side of it become one emoji-form sequence, so the cut emitted two columns for a budget of one. Measured through the real pane, a 24-column box drew at 43 by this module's own count. That is the failure the previous commit's body describes, one level down. ONE RULE, AT THE TWO DOORS WHERE TEXT ENTERS. A lone surrogate is not a character, so `widthSteps` removes them BEFORE it steps. There is then no orphan step, nothing to drop and nothing to splice, and the count describes what will be DRAWN rather than what arrived, which is the property every consumer actually needs. The line editor gets the same rule at its constructor, `insert` and `setValue`. `backspace` and `del` were fixed to step by character in the previous commit, and this pass pointed out that the three doors admitting text were left open: `stripControls` removes C0 and C1, not half a character, and `insert` takes a whole paste. The argument that made the edit-side fix necessary is that `value()` is the string that gets SAVED, and it applies to those doors unchanged. It also makes the window's arithmetic sound, because `cursor` indexes that string. Newly pinned, having survived the previous commit: the keycap's ADVANCE, which no width assertion can see because the enclosing mark is zero columns, so a step that advances by one instead of two emits U+20E3 again and corrupts the glyph while the count stays right. Corrected, all false when written: two comments above ZERO_WIDTH saying Mc is excluded because a spacing mark occupies a column, where the code includes it and the revision row says the renderer's answer wins; and the cursor comment, which described a snap to the step the cursor falls inside where it rounds forward to the next one. FILED RATHER THAN CHASED, under this round's cap of three review rounds: - #417. The never-narrower guarantee is per code point and per the swept pairs, not for every string. The renderer computes a cluster's base AFTER stripping leading non-printing characters, so a cluster beginning with one of 2,616 zero-width characters and continuing with one of exactly four tails has that tail counted twice. It is an under-count and it amplifies with a repeated breaker. The claim is narrowed to what the sweeps actually assert. - #418. Two HTML views still size an SVG chip and cut their text by code unit, the same class this issue answered everywhere else. Thirty-six mutations checked, zero survivors. Full suite in the CI posture: 4214 tests, 0 fail, 1 skipped. Same under the +399 day clock shift. All four guards pass and a run under a fresh TMPDIR leaves nothing behind. No version moved. Signed-off-by: Rob Boerman --- admin/src/panel.mjs | 60 +++++++++++++++++++++++++-------------- admin/test/width.test.mjs | 57 +++++++++++++++++++++++++++++++++++++ specs/design.md | 13 ++++++--- 3 files changed, 104 insertions(+), 26 deletions(-) diff --git a/admin/src/panel.mjs b/admin/src/panel.mjs index 52ed16f..ac28939 100644 --- a/admin/src/panel.mjs +++ b/admin/src/panel.mjs @@ -258,14 +258,19 @@ export function columnsOf(s) { * it can no longer strand a selector without its base. */ function* widthSteps(s) { - const chars = [...String(s ?? "")]; + // HALF A PAIR IS NOT A CHARACTER, and it is removed HERE, once, before anything is measured. A first + // version made it a step of its own and let each consumer drop it, which a review pass showed is the same + // defect one level down: dropping it SPLICES ITS NEIGHBOURS TOGETHER, so a base and the selector on the + // other side of it become one emoji-form sequence, and the cut emitted two columns for a budget of one. + // Measured through the real pane: a 24-column box drew at 43, by this module's own count. + // + // Normalising first also makes the count describe WHAT WILL BE DRAWN rather than what arrived, which is + // the property every consumer actually needs. It costs an over-count of one against the renderer on the + // raw input, in the safe direction, because the renderer measures the orphan as a sequence break and we + // measure the text we are about to hand it. + const chars = dropOrphans([...String(s ?? "")]); for (let i = 0; i < chars.length; i++) { const ch = chars[i]; - // HALF A PAIR IS NOT A CHARACTER. It is a step of its own so a cut can drop it by name. - if (ch.length === 1 && ch.charCodeAt(0) >= 0xd800 && ch.charCodeAt(0) <= 0xdfff) { - yield { text: ch, cols: 0, orphan: true }; - continue; - } if (ZERO_WIDTH.test(ch)) { yield { text: ch, cols: 0 }; continue; @@ -296,6 +301,11 @@ function* widthSteps(s) { } } +/** Drop every unpaired surrogate from a character array. Half a pair reaches a terminal as U+FFFD at best. */ +function dropOrphans(chars) { + return chars.filter((c) => !(c.length === 1 && c.charCodeAt(0) >= 0xd800 && c.charCodeAt(0) <= 0xdfff)); +} + /** VARIATION SELECTOR-16 asks for the emoji form; U+20E3 encloses the keycap bases below in a key. */ const KEYCAP = "\u20e3"; const KEYCAP_BASE = /[#*0-9]/; @@ -303,12 +313,12 @@ const KEYCAP_BASE = /[#*0-9]/; /** VARIATION SELECTOR-16, which asks for the emoji form of the character before it. */ const VS16 = "\ufe0f"; -// Mn and Me: a mark that draws on the character before it. `\p{M}` would also take Mc (spacing marks), which -// DO occupy a column in the Indic scripts that use them. -// ZERO COLUMNS. Mn and Me draw on the character before them; Mc (a SPACING mark) is deliberately NOT here, -// because it does occupy a column -- the one place this table knowingly departs from the renderer below. -// Cf covers the zero-width and bidi format characters, and the bracketed tail is the Hangul fillers and the -// noncharacters the renderer also draws as nothing. +// ZERO COLUMNS. `\p{M}` is every mark, SPACING MARKS INCLUDED, and that is deliberate: two earlier versions +// of this comment said the opposite, that Mc is excluded because it occupies a column, and both were false +// when written. The argument for excluding it came from Unicode; the renderer that draws this pane gives Mc +// zero, and between a standard and the thing painting the characters the painter wins. Cf covers the +// zero-width and bidi format characters, and the bracketed tail is the Hangul fillers and the noncharacters +// the renderer also draws as nothing. const ZERO_WIDTH = /\p{M}|\p{Cf}|[ᅟᅠ᠎ㅤﾠ￰-]/u; // A TEXT-PRESENTATION EMOJI: one that draws narrow on its own and WIDE once U+FE0F asks for the emoji @@ -377,11 +387,9 @@ export function sliceColumns(s, w) { let out = ""; let used = 0; for (const step of widthSteps(s)) { - // A lone surrogate the INPUT already held is dropped rather than carried: the old code-unit repair only - // ever looked at the last unit, so a leading or interior one survived it. An input that holds one and - // needs no cut at all still carries it, because `clip` returns early when the string fits: that is - // issue #402's ground rather than this one's. - if (step.orphan) continue; + // Orphans are already gone: `widthSteps` removes them before it steps, so this loop cannot drop a step + // and splice its neighbours together. An input that holds one and needs no cut at all still carries it, + // because `clip` returns early when the string fits: that is issue #402's ground rather than this one's. if (used + step.cols > budget) break; out += step.text; used += step.cols; @@ -624,14 +632,21 @@ function charAfter(s, at) { * when focused, the cursor cell is wrapped in `LINE_INPUT_CURSOR` (see above). */ export function makeLineInput(initial = "") { - let value = stripControls(initial); + // THE SAME RULE AT THE VALUE'S OWN DOORS. `backspace` and `del` were fixed to step by character, and a + // review pass pointed out that the constructor, `insert` (which takes a whole PASTE) and `setValue` were + // left open: `stripControls` removes C0 and C1, not half a character. The argument that made the edit-side + // fix necessary applies unchanged here, because `value()` is what gets SAVED, and it also keeps `render` + // honest -- `cursor` is an index into this string, so a value with no orphans in it means the window's + // offsets and the cursor cannot disagree about what they are counting. + const enter = (text) => dropOrphans([...stripControls(text)]).join(""); + let value = enter(initial); let cursor = value.length; return { value: () => value, cursor: () => cursor, /** Insert a printable char -- or a whole pasted string -- at the cursor; control chars are stripped first. */ insert(ch) { - const clean = stripControls(ch); + const clean = enter(ch); if (clean.length === 0) return; value = value.slice(0, cursor) + clean + value.slice(cursor); cursor += clean.length; @@ -663,7 +678,7 @@ export function makeLineInput(initial = "") { cursor = value.length; }, setValue(s) { - value = stripControls(s); + value = enter(s); cursor = value.length; }, render(width, { focused = true } = {}) { @@ -674,7 +689,7 @@ export function makeLineInput(initial = "") { // BARE LOW SURROGATE into the live trigger editor. It walks `widthSteps` rather than characters for // the reason that function exists: a per-character measure is one column short on an emoji-form // sequence, so a window built from one overflowed its own pane. - const steps = [...widthSteps(value)].filter((step) => !step.orphan); + const steps = [...widthSteps(value)]; const offs = []; let at = 0; for (const step of steps) { @@ -683,7 +698,8 @@ export function makeLineInput(initial = "") { } offs.push(at); // `cursor` is a code-unit index and the edit methods keep it on a character boundary, but a step can - // span three of them, so it is snapped to the step it falls inside rather than trusted to name one. + // span three of them, so it is resolved to a step rather than trusted to name one. It rounds FORWARD: + // a cursor inside a keycap names the step after it, never a position inside a glyph. let ci = offs.findIndex((o) => o >= cursor); if (ci < 0) ci = steps.length; // THE RESERVED CELL MOVES THE WINDOW'S START, not its length: the window is `w` columns wide, and diff --git a/admin/test/width.test.mjs b/admin/test/width.test.mjs index 3ae1f1b..ebe7e43 100644 --- a/admin/test/width.test.mjs +++ b/admin/test/width.test.mjs @@ -494,3 +494,60 @@ test("the skill frontmatter cap never leaves half a character either (#401)", as assert.doesNotMatch(String(meta?.description ?? ""), lone, `a description with a ${n}-character prefix left half a pair`); } }); + +test("a cut cannot splice its input into something wider than the budget (#401)", () => { + // THE DEFECT CLASS, ONE LEVEL DOWN, and the reason `widthSteps` normalises before it steps rather than + // letting each consumer drop an orphan. A lone surrogate BREAKS a sequence: the renderer measures + // `heart + orphan + selector` as one column, because the selector no longer follows its base. Dropping + // the orphan and re-joining puts them back together as a two-column glyph, so the cut emitted two columns + // for a budget of one and a 24-column pane drew at 43 by this module's own count. + const lone = /[\ud800-\udbff](?![\udc00-\udfff])|(? { + // THE ADVANCE, which is the half of the keycap rule no width assertion can see: the enclosing mark is + // zero columns, so a step that yields the keycap and then advances by one instead of two emits U+20E3 + // AGAIN as its own step. The count is unchanged and the glyph is corrupt, which is exactly the shape that + // survived a suite pinning only the count. + const cut = sliceColumns("1️⃣ab", 4); + assert.equal([...cut].filter((c) => c === "⃣").length, 1, "one enclosing mark, not two"); + assert.equal(cut, "1️⃣ab", "and the keycap is followed by what followed it"); + assert.equal(columnsOf(clip("1️⃣abcdef", 5)), 5, "the clipped form is still exactly the budget"); + assert.equal([...clip("1️⃣abcdef", 5)].filter((c) => c === "⃣").length, 1, "and still one mark"); +}); + +test("the editor's value never admits half a character, through any door (#401)", () => { + // `backspace` and `del` were fixed to step by character; a review pass found the three doors left open. + // `stripControls` removes C0 and C1, not half a character, and `insert` takes a whole PASTE. The argument + // that made the edit-side fix necessary is that `value()` is the string that gets SAVED, and it applies + // here unchanged. + const lone = /[\ud800-\udbff](?![\udc00-\udfff])|(? `readBeforeArming`, `changedWhileArming`, `WATCH_DEBOUNCE_MS`. | | 2026-09-24 | Issue #404. **`DES-ADMIN-VIA-PI-EXTENSION` AMENDED**: the control-byte gate names FOUR funnels, not three. #382 closed the two that render a pane and the model-visible `send`, and its own wording -- "every finished line passes one gate" -- read as complete while pi's DIALOGS took strings straight from stored fields. Measured: a pause window whose `scope` carries an erase-display, an OSC-52 clipboard write and an OSC-8 link reaches a select option, an input prompt and a confirm body. (An earlier draft of this row said "with the TUI suspended" and named an input DEFAULT: neither is true at the pin -- these dialogs are drawn by the LIVE tui, and `ExtensionInputComponent` accepts a placeholder and never renders it.) `pause-windows.mjs` checks only `isNonEmptyString(w.scope)`, so the FILE is the whole validator; `secrets-command.ts` has the same shape. Not a regression -- it behaved identically before #382 -- but it IS model-reachable, which the first draft of this row denied: a tool's `execute` gets its own `ctx` from pi, never passing the command door, and `dispatch_trigger_edit`'s confirm body interpolates the model's own `flow`. The gate is one wrapper per DOOR -- six of them: the command handler, the exported dashboard action, `confirmedWrite` for every tool that confirms, the secrets command, the setup wizard and the `session_start` handler (the sixth found by a second review pass, its own notify a constant -- gated rather than excused, because a door left out for being currently safe is how the render gates were refuted) -- not at the 75 call sites: more than one is there because no single one is the only door, and a test driving the dashboard action directly would otherwise exercise the dialogs ungated -- the same covered-on-one-path shape the render gates were refuted for. `notify` is included, because an error message that moves the cursor is the same class as any other. `custom` is not, because it takes a factory whose output is the overlay. **AND THE SELECTION IS TRANSLATED BACK**: pi returns the exact option string it was handed, so a scrubbed copy made `labels.indexOf(picked)` fail and four edit/delete actions returned silently -- a gate that introduced the silent no-op it exists to prevent, caught by a review pass and pinned. **Code evidence**: `admin/src/dialog-gate.mjs` -> `gateDialogs`. | | 2026-09-24 | Issue #403. **`DES-ADMIN-VIA-PI-EXTENSION` AMENDED**: the unframed degrade honours the width it was given. It was a layout promise that held on one branch of an `if` -- the run rows are bounded to their own fixed 24-column pane, everything else was returned as built (the live tail's unframed branch had no clip at all, and said so) -- so a single render showed rows cut to the pane beside section headers, settings lines and hints running past it (measured: 28 of 59 lines over the width at `render(4)`, the widest 52 columns). Same shape as the logs viewer's #399, which this file had already fixed. **A width that is MISSING or non-finite is left alone**, deliberately: the degrade is what those get too, and `clip(x, NaN)` returns the empty string, so inventing a number there would blank the pane rather than degrade it. **The clip is by LENGTH and that rests on a measured assumption**, now pinned: this path is monochrome, zero of its lines carrying an SGR sequence under a real theme, because its own branches strip colour before returning. If that changes the result is worse than lost colour -- `clipData` substitutes the ESC and leaves `[31m` VISIBLE, charging four phantom columns against the width -- and the answer is an ANSI-aware cut. The width remains a UTF-16 count, wrong for CJK exactly as it is everywhere else in this module, which is #401 and is not made worse here. The TRUNCATION is centralised beside the comparison, not only the comparison: a review pass showed that splitting them is invisible, since `Math.round(7.9)` frames nothing and then clips to 8. **Code evidence**: `admin/src/dashboard.ts` -> `renderPanel`, `framedAt`, `degradeWidth`. | -| 2026-09-24 | Issue #401. **`DES-ADMIN-VIA-PI-EXTENSION` AMENDED**: the panel measures COLUMNS, and the entry's own sentence saying it does not is what changed. Every width promise in this module was a UTF-16 code-unit count -- `clip`, `pad`, `styler.cell`, `divider`, `clipPlain`, `visibleLen`, both frame builders' top rules, the line editor's window and the model-visible runs table -- so a CJK job id framed a pane this module reported as exactly 80 columns and the renderer drew at 90, and a combining mark ragged it the other way. Measured against the pinned renderer, as this module counted against what it draws: a CJK job id 5 against 10, fullwidth latin 3 against 6, Hangul 5 against 10, combining marks 5 against 3. **ONE TABLE, IN `panel.mjs`, AND THE STYLER COMES TO IT.** Importing pi-tui's `visibleWidth` into `style.mjs` was the alternative, and that module is overlay-only and already depends on pi: rejected because `panel.mjs` owns `clip` and the monochrome renderer, its purity pin forbids it reaching the world at all, and the two renderers draw the same geometry. A width rule holding in the framed pane and not in the plain one is the shape that holds on one branch of an `if`, which #403 had just been about -- and a mutation of the MONOCHROME top rule survived a suite pinning only the coloured one, which is that same shape appearing inside this very fix. **THE CORRECTION THAT MATTERS: the table is TRANSCRIBED FROM THE RENDERER, not written out of UAX #11.** A first version was hand-written from the standard and checked against ten strings. It passed, and it UNDER-counted 139,820 code points -- CJK Extension B through H, Tangut, the Kana supplement, the Hangul Jamo extensions, every emoji block added since Unicode 13 -- so the fix carried the defect it was fixing, on exactly the CJK repository name the issue is about; and in two classes (an astral character, and a character followed by U+FE0F) it was WORSE than the `.length` it replaced, which had been right there by accident because such a sequence is two code units and two columns. Ten examples cannot see that, which is why the bolt is **two sweeps: every code point there is, and every character followed by U+FE0F and by a keycap**. The table may never measure NARROWER than the renderer, and every place it measures wider must be a declared departure. The pair sweep exists because a first version of this row CLAIMED it and the test shipped a six-entry list instead, and the hole was one selector further along: a KEYCAP, twelve bases plus U+FE0F plus U+20E3 drawn as one key, measured 1 against the renderer's 2. That was an UNDER-count, so it is fixed rather than stated. The DIRECTION is the whole property, and "over-counting is harmless" would be too kind: BOTH directions rag a frame, differently. An over-count pads a body line as though it were wider than it is, so the line comes out SHORT and the right border sits left of the one above it, contained by the pane -- a review pass measured 200 such lines from ZWJ sequences, Hangul Jamo clusters and Devanagari. An under-count runs the line PAST the pane and past the terminal, where it wraps and takes the border with it, which is the defect this issue names. The sweep is one-sided on purpose, and what is left is the safer of two bad shapes rather than a harmless one. **WHAT IT COUNTS**: zero for a mark and for a format character plus the fillers, two for the renderer's own double-width set, one more when U+FE0F asks for the emoji form of a narrow base, one for everything else including an unassigned code point. Mc is no longer excepted: a first version reasoned from Unicode that a spacing mark occupies a column, and the renderer that draws the pane says otherwise, so the renderer wins. **WHAT IS LEFT IS A CLASS, NOT ONE SHAPE, and saying otherwise was the other thing a review pass refuted**: the table sums steps, so every run the renderer collapses into one glyph comes out wider than it draws -- an emoji ZWJ sequence (6 against 2), a skin-tone modifier, a regional-indicator flag pair, a Hangul jamo cluster and a Devanagari cluster. All of them OVER-count, so all of them draw short inside the border rather than through it; terminals disagree with each other on all of them. **THE ORACLE IS pi ITSELF**, loaded from the pinned installation per `CONST-PI-VERSION-PINNED` with its VERSION asserted, not just its presence, because pi depends on pi-tui by a range and a table pinned to the wrong renderer is not pinned; and asserted to be a function rather than skipped on failure, since a sibling pin made exactly that silent-fallback mistake once. **Also repaired, as the same defect class rather than as scope creep**: the line editor windowed and padded by code units, so a CJK value measured half what the terminal drew and a window edge put a BARE LOW SURROGATE into the live trigger editor; `renderRuns`, the model-visible table, sized its columns by `.length`, so one CJK character in a `local:` target shifted every later column of that row; a cut to zero returned a floating combining mark, and a cut through a ZWJ sequence left the joiner dangling before the ellipsis; and two hostile-string caps in the graph model could split a surrogate pair. **The line editor's EDIT side needed the same fix as its render side**, which is the sharper half: `cursor` moved one CODE UNIT at a time, so two `left`s put it between the halves of an astral pair and the next `backspace` deleted ONE HALF, leaving the other in `value()` -- in the string that gets SAVED, where no amount of rendering can repair it. **`DES-PANEL-SEPARATE-FROM-RECEIVER` UNCHANGED, checked**: nothing about where the panel runs or what it binds moves. **The bidi and zero-width residual is UNCHANGED and still open**: those code points sit outside a control-byte class defined by what a terminal INTERPRETS, and they are a reader-deception question rather than a width one. **AND THE COUNT AND THE CUT NOW WALK ONE STEPPER**, which is the structural half of the repair: a first version put the U+FE0F state inside `columnsOf` alone, and `sliceColumns` and the line editor call it ONE CHARACTER AT A TIME, where a base is 1 and its selector is 0 -- so the cut spent a budget of 2 on a glyph drawn 3 wide and the pane overflowed. One generator yields `{text, cols}` steps, the count sums them and the cut takes whole ones, so the two cannot disagree. **Code evidence**: admin/src/panel.mjs -> columnsOf, widthSteps, WIDE, ZERO_WIDTH, sliceColumns, clip, pad, box, makeLineInput; admin/src/style.mjs -> visibleLen, cell, divider, frame, clipPlain; admin/src/render.mjs -> renderRuns; admin/src/graph-model.mjs -> clipName; admin/test/width.test.mjs. | +| 2026-09-24 | Issue #401. **`DES-ADMIN-VIA-PI-EXTENSION` AMENDED**: the panel measures COLUMNS, and the entry's own sentence saying it does not is what changed. Every width promise in this module was a UTF-16 code-unit count -- `clip`, `pad`, `styler.cell`, `divider`, `clipPlain`, `visibleLen`, both frame builders' top rules, the line editor's window and the model-visible runs table -- so a CJK job id framed a pane this module reported as exactly 80 columns and the renderer drew at 90, and a combining mark ragged it the other way. Measured against the pinned renderer, as this module counted against what it draws: a CJK job id 5 against 10, fullwidth latin 3 against 6, Hangul 5 against 10, combining marks 5 against 3. **ONE TABLE, IN `panel.mjs`, AND THE STYLER COMES TO IT.** Importing pi-tui's `visibleWidth` into `style.mjs` was the alternative, and that module is overlay-only and already depends on pi: rejected because `panel.mjs` owns `clip` and the monochrome renderer, its purity pin forbids it reaching the world at all, and the two renderers draw the same geometry. A width rule holding in the framed pane and not in the plain one is the shape that holds on one branch of an `if`, which #403 had just been about -- and a mutation of the MONOCHROME top rule survived a suite pinning only the coloured one, which is that same shape appearing inside this very fix. **THE CORRECTION THAT MATTERS: the table is TRANSCRIBED FROM THE RENDERER, not written out of UAX #11.** A first version was hand-written from the standard and checked against ten strings. It passed, and it UNDER-counted 139,820 code points -- CJK Extension B through H, Tangut, the Kana supplement, the Hangul Jamo extensions, every emoji block added since Unicode 13 -- so the fix carried the defect it was fixing, on exactly the CJK repository name the issue is about; and in two classes (an astral character, and a character followed by U+FE0F) it was WORSE than the `.length` it replaced, which had been right there by accident because such a sequence is two code units and two columns. Ten examples cannot see that, which is why the bolt is **two sweeps: every code point there is, and every character followed by U+FE0F and by a keycap**. On every one of those the table may never measure NARROWER than the renderer, and every place it measures wider must be a declared departure. **That is per code point and per those pairs, not for every string**, and a third round measured where it stops: the renderer computes a cluster's base after stripping leading non-printing characters, so a cluster BEGINNING with one of 2,616 zero-width characters and continuing with one of exactly four tails has that tail counted twice. It is an UNDER-count and it amplifies with a repeated breaker, so it is filed as #417 rather than left implied. The pair sweep exists because a first version of this row CLAIMED it and the test shipped a six-entry list instead, and the hole was one selector further along: a KEYCAP, twelve bases plus U+FE0F plus U+20E3 drawn as one key, measured 1 against the renderer's 2. That was an UNDER-count, so it is fixed rather than stated. The DIRECTION is the whole property, and "over-counting is harmless" would be too kind: BOTH directions rag a frame, differently. An over-count pads a body line as though it were wider than it is, so the line comes out SHORT and the right border sits left of the one above it, contained by the pane -- a review pass measured 200 such lines from ZWJ sequences, Hangul Jamo clusters and Devanagari. An under-count runs the line PAST the pane and past the terminal, where it wraps and takes the border with it, which is the defect this issue names. The sweep is one-sided on purpose, and what is left is the safer of two bad shapes rather than a harmless one. **WHAT IT COUNTS**: zero for a mark and for a format character plus the fillers, two for the renderer's own double-width set, one more when U+FE0F asks for the emoji form of a narrow base, one for everything else including an unassigned code point. Mc is no longer excepted: a first version reasoned from Unicode that a spacing mark occupies a column, and the renderer that draws the pane says otherwise, so the renderer wins. **WHAT IS LEFT IS A CLASS, NOT ONE SHAPE, and saying otherwise was the other thing a review pass refuted**: the table sums steps, so every run the renderer collapses into one glyph comes out wider than it draws -- an emoji ZWJ sequence (6 against 2), a skin-tone modifier, a regional-indicator flag pair, a Hangul jamo cluster and a Devanagari cluster. All of them OVER-count, so all of them draw short inside the border rather than through it; terminals disagree with each other on all of them. **THE ORACLE IS pi ITSELF**, loaded from the pinned installation per `CONST-PI-VERSION-PINNED` with its VERSION asserted, not just its presence, because pi depends on pi-tui by a range and a table pinned to the wrong renderer is not pinned; and asserted to be a function rather than skipped on failure, since a sibling pin made exactly that silent-fallback mistake once. **Also repaired, as the same defect class rather than as scope creep**: the line editor windowed and padded by code units, so a CJK value measured half what the terminal drew and a window edge put a BARE LOW SURROGATE into the live trigger editor; `renderRuns`, the model-visible table, sized its columns by `.length`, so one CJK character in a `local:` target shifted every later column of that row; a cut to zero returned a floating combining mark, and a cut through a ZWJ sequence left the joiner dangling before the ellipsis; and two hostile-string caps in the graph model could split a surrogate pair. **The line editor's EDIT side needed the same fix as its render side**, which is the sharper half: `cursor` moved one CODE UNIT at a time, so two `left`s put it between the halves of an astral pair and the next `backspace` deleted ONE HALF, leaving the other in `value()` -- in the string that gets SAVED, where no amount of rendering can repair it. **`DES-PANEL-SEPARATE-FROM-RECEIVER` UNCHANGED, checked**: nothing about where the panel runs or what it binds moves. **The bidi and zero-width residual is UNCHANGED and still open**: those code points sit outside a control-byte class defined by what a terminal INTERPRETS, and they are a reader-deception question rather than a width one. **AND THE COUNT AND THE CUT NOW WALK ONE STEPPER**, which is the structural half of the repair: a first version put the U+FE0F state inside `columnsOf` alone, and `sliceColumns` and the line editor call it ONE CHARACTER AT A TIME, where a base is 1 and its selector is 0 -- so the cut spent a budget of 2 on a glyph drawn 3 wide and the pane overflowed. One generator yields `{text, cols}` steps, the count sums them and the cut takes whole ones, so the two cannot disagree. **Code evidence**: admin/src/panel.mjs -> columnsOf, widthSteps, WIDE, ZERO_WIDTH, sliceColumns, clip, pad, box, makeLineInput; admin/src/style.mjs -> visibleLen, cell, divider, frame, clipPlain; admin/src/render.mjs -> renderRuns; admin/src/graph-model.mjs -> clipName; admin/test/width.test.mjs. | From 257ebe63d442a98160b98f6bce1b1a88b6d99665 Mon Sep 17 00:00:00 2001 From: Rob Boerman Date: Thu, 24 Sep 2026 19:33:38 +0200 Subject: [PATCH 5/5] fix(admin): step the fitted line too, so an orphan never reaches the terminal (#401) The final review pass found one more instance of the leading-mark class filed as #417, with a different leader and a one-line containment, so it is fixed here rather than filed. A LONE SURROGATE breaks a cluster for the renderer exactly as one of #417's 2,616 zero-width leaders does: it skips the break, computes the next cluster's base, and counts that base twice. `clip` returned its input unchanged whenever it fitted, so an orphan the COUNT had already removed was still in the TEXT handed to the renderer. Measured: `("\ud800\uff9f").repeat(12)` counted 12 here and 24 there, and a 24-column pane drew at 36, a 40-column one at 52. `clip` steps its fitted line now. That makes the rule this module already states true of the text rather than only of the count: half a character is removed where text ENTERS. The #417 leader set is unchanged, all 2,616 of them zero-width characters. Three issue texts corrected with it, each false at HEAD rather than merely incomplete: - #418 said both HTML views already import from `panel.mjs`. `graph-html.mjs` imports nothing, deliberately, with a comment saying it is allowed no dependencies at all and a parity test standing in for the import. That was the suggested route, so the correction matters. - #402's table said a zero-width space is counted as a column and breaks the width math. It is zero columns now, which is what the renderer draws, so the width argument is answered and only the deception argument is left. - #417 gains two measured notes: the line editor's window can begin on a zero-width character and manufacture the shape out of a well-formed value, and both its shapes are a REGRESSION against main for those inputs, which is worth saying so the fix is not deprioritised as a pre-existing residual. Verified at scale against the pinned renderer by the review pass: 64,000 hostile strings through every cutter at every width 0 to 40, and 1,500 random title/body/footer triples through both panes at twelve widths, with no width failure by either measure; and 15,000 random 24-operation editor sequences with no lone surrogate in the value, no wrong render width and no new class. Full suite in the CI posture: 4215 tests, 0 fail, 1 skipped. Same under the +399 day clock shift. All four guards pass, a run under a fresh TMPDIR leaves nothing behind, and reverting the containment turns the new pin red. No version moved. Signed-off-by: Rob Boerman --- admin/src/panel.mjs | 14 ++++++++++---- admin/test/width.test.mjs | 24 ++++++++++++++++++++++++ specs/design.md | 4 ++-- 3 files changed, 36 insertions(+), 6 deletions(-) diff --git a/admin/src/panel.mjs b/admin/src/panel.mjs index ac28939..07d5350 100644 --- a/admin/src/panel.mjs +++ b/admin/src/panel.mjs @@ -363,7 +363,13 @@ const WIDE = new RegExp( */ export function clip(line, w) { const width = Math.max(0, Math.trunc(w) || 0); - const clean = stripControls(line); + // THE TEXT, NOT THE INPUT, on both paths. Returning the input unchanged when it fits left a lone + // surrogate in the DRAWN line, and the renderer treats one as a cluster break: it then computes the next + // cluster's base after skipping it and counts that base twice, so `\ud800\uff9f` repeated twelve times + // measured 12 here and 24 there, and a 24-column pane drew at 36. Stepping the fitted line too makes the + // rule this module already states -- half a character is removed where text ENTERS -- true of the text + // rather than only of the count. + const clean = sliceColumns(stripControls(line), Number.MAX_SAFE_INTEGER); if (columnsOf(clean) <= width) return clean; const ell = active.ellipsis; if (width <= columnsOf(ell)) return sliceColumns(ell, width); @@ -387,9 +393,9 @@ export function sliceColumns(s, w) { let out = ""; let used = 0; for (const step of widthSteps(s)) { - // Orphans are already gone: `widthSteps` removes them before it steps, so this loop cannot drop a step - // and splice its neighbours together. An input that holds one and needs no cut at all still carries it, - // because `clip` returns early when the string fits: that is issue #402's ground rather than this one's. + // Orphans are already gone: `widthSteps` removes them before it steps, so this loop cannot drop a step + // and splice its neighbours together. `clip` steps its fitted line through here too, so a cut is no + // longer the only path that removes one. if (used + step.cols > budget) break; out += step.text; used += step.cols; diff --git a/admin/test/width.test.mjs b/admin/test/width.test.mjs index ebe7e43..0956a60 100644 --- a/admin/test/width.test.mjs +++ b/admin/test/width.test.mjs @@ -551,3 +551,27 @@ test("the editor's value never admits half a character, through any door (#401)" const ed = makeLineInput("ab\ud800cd"); for (const w of [1, 3, 6, 10]) assert.doesNotMatch(ed.render(w), lone, `render at ${w}`); }); + +test("a lone surrogate cannot reach the drawn line and break a cluster (#401)", async () => { + // THE LAST VARIANT, and it is the same mechanism as #417 with a different leader. The renderer treats a + // lone surrogate as a cluster BREAK, then computes the next cluster's base after skipping it and counts + // that base twice. So an orphan the count had already removed was still in the text handed to the + // renderer, and it made the renderer measure a line wider than we did: measured at 12 columns here and + // 24 there, with a 24-column pane drawing at 36. + // + // `clip` used to return its input unchanged when it fitted. It steps the fitted line too now, which makes + // the rule this module states -- half a character is removed where text ENTERS -- true of the text rather + // than only of the count. + const visibleWidth = await loadVisibleWidth(); + assert.equal(typeof visibleWidth, "function"); + const lone = /[\ud800-\udbff](?![\udc00-\udfff])|(? `readBeforeArming`, `changedWhileArming`, `WATCH_DEBOUNCE_MS`. | | 2026-09-24 | Issue #404. **`DES-ADMIN-VIA-PI-EXTENSION` AMENDED**: the control-byte gate names FOUR funnels, not three. #382 closed the two that render a pane and the model-visible `send`, and its own wording -- "every finished line passes one gate" -- read as complete while pi's DIALOGS took strings straight from stored fields. Measured: a pause window whose `scope` carries an erase-display, an OSC-52 clipboard write and an OSC-8 link reaches a select option, an input prompt and a confirm body. (An earlier draft of this row said "with the TUI suspended" and named an input DEFAULT: neither is true at the pin -- these dialogs are drawn by the LIVE tui, and `ExtensionInputComponent` accepts a placeholder and never renders it.) `pause-windows.mjs` checks only `isNonEmptyString(w.scope)`, so the FILE is the whole validator; `secrets-command.ts` has the same shape. Not a regression -- it behaved identically before #382 -- but it IS model-reachable, which the first draft of this row denied: a tool's `execute` gets its own `ctx` from pi, never passing the command door, and `dispatch_trigger_edit`'s confirm body interpolates the model's own `flow`. The gate is one wrapper per DOOR -- six of them: the command handler, the exported dashboard action, `confirmedWrite` for every tool that confirms, the secrets command, the setup wizard and the `session_start` handler (the sixth found by a second review pass, its own notify a constant -- gated rather than excused, because a door left out for being currently safe is how the render gates were refuted) -- not at the 75 call sites: more than one is there because no single one is the only door, and a test driving the dashboard action directly would otherwise exercise the dialogs ungated -- the same covered-on-one-path shape the render gates were refuted for. `notify` is included, because an error message that moves the cursor is the same class as any other. `custom` is not, because it takes a factory whose output is the overlay. **AND THE SELECTION IS TRANSLATED BACK**: pi returns the exact option string it was handed, so a scrubbed copy made `labels.indexOf(picked)` fail and four edit/delete actions returned silently -- a gate that introduced the silent no-op it exists to prevent, caught by a review pass and pinned. **Code evidence**: `admin/src/dialog-gate.mjs` -> `gateDialogs`. | | 2026-09-24 | Issue #403. **`DES-ADMIN-VIA-PI-EXTENSION` AMENDED**: the unframed degrade honours the width it was given. It was a layout promise that held on one branch of an `if` -- the run rows are bounded to their own fixed 24-column pane, everything else was returned as built (the live tail's unframed branch had no clip at all, and said so) -- so a single render showed rows cut to the pane beside section headers, settings lines and hints running past it (measured: 28 of 59 lines over the width at `render(4)`, the widest 52 columns). Same shape as the logs viewer's #399, which this file had already fixed. **A width that is MISSING or non-finite is left alone**, deliberately: the degrade is what those get too, and `clip(x, NaN)` returns the empty string, so inventing a number there would blank the pane rather than degrade it. **The clip is by LENGTH and that rests on a measured assumption**, now pinned: this path is monochrome, zero of its lines carrying an SGR sequence under a real theme, because its own branches strip colour before returning. If that changes the result is worse than lost colour -- `clipData` substitutes the ESC and leaves `[31m` VISIBLE, charging four phantom columns against the width -- and the answer is an ANSI-aware cut. The width remains a UTF-16 count, wrong for CJK exactly as it is everywhere else in this module, which is #401 and is not made worse here. The TRUNCATION is centralised beside the comparison, not only the comparison: a review pass showed that splitting them is invisible, since `Math.round(7.9)` frames nothing and then clips to 8. **Code evidence**: `admin/src/dashboard.ts` -> `renderPanel`, `framedAt`, `degradeWidth`. | -| 2026-09-24 | Issue #401. **`DES-ADMIN-VIA-PI-EXTENSION` AMENDED**: the panel measures COLUMNS, and the entry's own sentence saying it does not is what changed. Every width promise in this module was a UTF-16 code-unit count -- `clip`, `pad`, `styler.cell`, `divider`, `clipPlain`, `visibleLen`, both frame builders' top rules, the line editor's window and the model-visible runs table -- so a CJK job id framed a pane this module reported as exactly 80 columns and the renderer drew at 90, and a combining mark ragged it the other way. Measured against the pinned renderer, as this module counted against what it draws: a CJK job id 5 against 10, fullwidth latin 3 against 6, Hangul 5 against 10, combining marks 5 against 3. **ONE TABLE, IN `panel.mjs`, AND THE STYLER COMES TO IT.** Importing pi-tui's `visibleWidth` into `style.mjs` was the alternative, and that module is overlay-only and already depends on pi: rejected because `panel.mjs` owns `clip` and the monochrome renderer, its purity pin forbids it reaching the world at all, and the two renderers draw the same geometry. A width rule holding in the framed pane and not in the plain one is the shape that holds on one branch of an `if`, which #403 had just been about -- and a mutation of the MONOCHROME top rule survived a suite pinning only the coloured one, which is that same shape appearing inside this very fix. **THE CORRECTION THAT MATTERS: the table is TRANSCRIBED FROM THE RENDERER, not written out of UAX #11.** A first version was hand-written from the standard and checked against ten strings. It passed, and it UNDER-counted 139,820 code points -- CJK Extension B through H, Tangut, the Kana supplement, the Hangul Jamo extensions, every emoji block added since Unicode 13 -- so the fix carried the defect it was fixing, on exactly the CJK repository name the issue is about; and in two classes (an astral character, and a character followed by U+FE0F) it was WORSE than the `.length` it replaced, which had been right there by accident because such a sequence is two code units and two columns. Ten examples cannot see that, which is why the bolt is **two sweeps: every code point there is, and every character followed by U+FE0F and by a keycap**. On every one of those the table may never measure NARROWER than the renderer, and every place it measures wider must be a declared departure. **That is per code point and per those pairs, not for every string**, and a third round measured where it stops: the renderer computes a cluster's base after stripping leading non-printing characters, so a cluster BEGINNING with one of 2,616 zero-width characters and continuing with one of exactly four tails has that tail counted twice. It is an UNDER-count and it amplifies with a repeated breaker, so it is filed as #417 rather than left implied. The pair sweep exists because a first version of this row CLAIMED it and the test shipped a six-entry list instead, and the hole was one selector further along: a KEYCAP, twelve bases plus U+FE0F plus U+20E3 drawn as one key, measured 1 against the renderer's 2. That was an UNDER-count, so it is fixed rather than stated. The DIRECTION is the whole property, and "over-counting is harmless" would be too kind: BOTH directions rag a frame, differently. An over-count pads a body line as though it were wider than it is, so the line comes out SHORT and the right border sits left of the one above it, contained by the pane -- a review pass measured 200 such lines from ZWJ sequences, Hangul Jamo clusters and Devanagari. An under-count runs the line PAST the pane and past the terminal, where it wraps and takes the border with it, which is the defect this issue names. The sweep is one-sided on purpose, and what is left is the safer of two bad shapes rather than a harmless one. **WHAT IT COUNTS**: zero for a mark and for a format character plus the fillers, two for the renderer's own double-width set, one more when U+FE0F asks for the emoji form of a narrow base, one for everything else including an unassigned code point. Mc is no longer excepted: a first version reasoned from Unicode that a spacing mark occupies a column, and the renderer that draws the pane says otherwise, so the renderer wins. **WHAT IS LEFT IS A CLASS, NOT ONE SHAPE, and saying otherwise was the other thing a review pass refuted**: the table sums steps, so every run the renderer collapses into one glyph comes out wider than it draws -- an emoji ZWJ sequence (6 against 2), a skin-tone modifier, a regional-indicator flag pair, a Hangul jamo cluster and a Devanagari cluster. All of them OVER-count, so all of them draw short inside the border rather than through it; terminals disagree with each other on all of them. **THE ORACLE IS pi ITSELF**, loaded from the pinned installation per `CONST-PI-VERSION-PINNED` with its VERSION asserted, not just its presence, because pi depends on pi-tui by a range and a table pinned to the wrong renderer is not pinned; and asserted to be a function rather than skipped on failure, since a sibling pin made exactly that silent-fallback mistake once. **Also repaired, as the same defect class rather than as scope creep**: the line editor windowed and padded by code units, so a CJK value measured half what the terminal drew and a window edge put a BARE LOW SURROGATE into the live trigger editor; `renderRuns`, the model-visible table, sized its columns by `.length`, so one CJK character in a `local:` target shifted every later column of that row; a cut to zero returned a floating combining mark, and a cut through a ZWJ sequence left the joiner dangling before the ellipsis; and two hostile-string caps in the graph model could split a surrogate pair. **The line editor's EDIT side needed the same fix as its render side**, which is the sharper half: `cursor` moved one CODE UNIT at a time, so two `left`s put it between the halves of an astral pair and the next `backspace` deleted ONE HALF, leaving the other in `value()` -- in the string that gets SAVED, where no amount of rendering can repair it. **`DES-PANEL-SEPARATE-FROM-RECEIVER` UNCHANGED, checked**: nothing about where the panel runs or what it binds moves. **The bidi and zero-width residual is UNCHANGED and still open**: those code points sit outside a control-byte class defined by what a terminal INTERPRETS, and they are a reader-deception question rather than a width one. **AND THE COUNT AND THE CUT NOW WALK ONE STEPPER**, which is the structural half of the repair: a first version put the U+FE0F state inside `columnsOf` alone, and `sliceColumns` and the line editor call it ONE CHARACTER AT A TIME, where a base is 1 and its selector is 0 -- so the cut spent a budget of 2 on a glyph drawn 3 wide and the pane overflowed. One generator yields `{text, cols}` steps, the count sums them and the cut takes whole ones, so the two cannot disagree. **Code evidence**: admin/src/panel.mjs -> columnsOf, widthSteps, WIDE, ZERO_WIDTH, sliceColumns, clip, pad, box, makeLineInput; admin/src/style.mjs -> visibleLen, cell, divider, frame, clipPlain; admin/src/render.mjs -> renderRuns; admin/src/graph-model.mjs -> clipName; admin/test/width.test.mjs. | +| 2026-09-24 | Issue #401. **`DES-ADMIN-VIA-PI-EXTENSION` AMENDED**: the panel measures COLUMNS, and the entry's own sentence saying it does not is what changed. Every width promise in this module was a UTF-16 code-unit count -- `clip`, `pad`, `styler.cell`, `divider`, `clipPlain`, `visibleLen`, both frame builders' top rules, the line editor's window and the model-visible runs table -- so a CJK job id framed a pane this module reported as exactly 80 columns and the renderer drew at 90, and a combining mark ragged it the other way. Measured against the pinned renderer, as this module counted against what it draws: a CJK job id 5 against 10, fullwidth latin 3 against 6, Hangul 5 against 10, combining marks 5 against 3. **ONE TABLE, IN `panel.mjs`, AND THE STYLER COMES TO IT.** Importing pi-tui's `visibleWidth` into `style.mjs` was the alternative, and that module is overlay-only and already depends on pi: rejected because `panel.mjs` owns `clip` and the monochrome renderer, its purity pin forbids it reaching the world at all, and the two renderers draw the same geometry. A width rule holding in the framed pane and not in the plain one is the shape that holds on one branch of an `if`, which #403 had just been about -- and a mutation of the MONOCHROME top rule survived a suite pinning only the coloured one, which is that same shape appearing inside this very fix. **THE CORRECTION THAT MATTERS: the table is TRANSCRIBED FROM THE RENDERER, not written out of UAX #11.** A first version was hand-written from the standard and checked against ten strings. It passed, and it UNDER-counted 139,820 code points -- CJK Extension B through H, Tangut, the Kana supplement, the Hangul Jamo extensions, every emoji block added since Unicode 13 -- so the fix carried the defect it was fixing, on exactly the CJK repository name the issue is about; and in two classes (an astral character, and a character followed by U+FE0F) it was WORSE than the `.length` it replaced, which had been right there by accident because such a sequence is two code units and two columns. Ten examples cannot see that, which is why the bolt is **two sweeps: every code point there is, and every character followed by U+FE0F and by a keycap**. On every one of those the table may never measure NARROWER than the renderer, and every place it measures wider must be a declared departure. **That is per code point and per those pairs, not for every string**, and a third round measured where it stops: the renderer computes a cluster's base after stripping leading non-printing characters, so a cluster BEGINNING with one of 2,616 zero-width characters and continuing with one of exactly four tails has that tail counted twice. It is an UNDER-count and it amplifies with a repeated breaker, so it is filed as #417 rather than left implied. A LONE SURROGATE is a leader of the same kind and that one is fixed here, because the containment was one line: `clip` steps its fitted line rather than returning it unchanged, so an orphan can no longer reach the drawn text after the count has already removed it. The pair sweep exists because a first version of this row CLAIMED it and the test shipped a six-entry list instead, and the hole was one selector further along: a KEYCAP, twelve bases plus U+FE0F plus U+20E3 drawn as one key, measured 1 against the renderer's 2. That was an UNDER-count, so it is fixed rather than stated. The DIRECTION is the whole property, and "over-counting is harmless" would be too kind: BOTH directions rag a frame, differently. An over-count pads a body line as though it were wider than it is, so the line comes out SHORT and the right border sits left of the one above it, contained by the pane -- a review pass measured 200 such lines from ZWJ sequences, Hangul Jamo clusters and Devanagari. An under-count runs the line PAST the pane and past the terminal, where it wraps and takes the border with it, which is the defect this issue names. The sweep is one-sided on purpose, and what is left is the safer of two bad shapes rather than a harmless one. **WHAT IT COUNTS**: zero for a mark and for a format character plus the fillers, two for the renderer's own double-width set, one more when U+FE0F asks for the emoji form of a narrow base, one for everything else including an unassigned code point. Mc is no longer excepted: a first version reasoned from Unicode that a spacing mark occupies a column, and the renderer that draws the pane says otherwise, so the renderer wins. **WHAT IS LEFT IS A CLASS, NOT ONE SHAPE, and saying otherwise was the other thing a review pass refuted**: the table sums steps, so every run the renderer collapses into one glyph comes out wider than it draws -- an emoji ZWJ sequence (6 against 2), a skin-tone modifier, a regional-indicator flag pair, a Hangul jamo cluster and a Devanagari cluster. All of them OVER-count, so all of them draw short inside the border rather than through it; terminals disagree with each other on all of them. **THE ORACLE IS pi ITSELF**, loaded from the pinned installation per `CONST-PI-VERSION-PINNED` with its VERSION asserted, not just its presence, because pi depends on pi-tui by a range and a table pinned to the wrong renderer is not pinned; and asserted to be a function rather than skipped on failure, since a sibling pin made exactly that silent-fallback mistake once. **Also repaired, as the same defect class rather than as scope creep**: the line editor windowed and padded by code units, so a CJK value measured half what the terminal drew and a window edge put a BARE LOW SURROGATE into the live trigger editor; `renderRuns`, the model-visible table, sized its columns by `.length`, so one CJK character in a `local:` target shifted every later column of that row; a cut to zero returned a floating combining mark, and a cut through a ZWJ sequence left the joiner dangling before the ellipsis; and two hostile-string caps in the graph model could split a surrogate pair. **The line editor's EDIT side needed the same fix as its render side**, which is the sharper half: `cursor` moved one CODE UNIT at a time, so two `left`s put it between the halves of an astral pair and the next `backspace` deleted ONE HALF, leaving the other in `value()` -- in the string that gets SAVED, where no amount of rendering can repair it. **`DES-PANEL-SEPARATE-FROM-RECEIVER` UNCHANGED, checked**: nothing about where the panel runs or what it binds moves. **The bidi and zero-width residual is UNCHANGED and still open**: those code points sit outside a control-byte class defined by what a terminal INTERPRETS, and they are a reader-deception question rather than a width one. **AND THE COUNT AND THE CUT NOW WALK ONE STEPPER**, which is the structural half of the repair: a first version put the U+FE0F state inside `columnsOf` alone, and `sliceColumns` and the line editor call it ONE CHARACTER AT A TIME, where a base is 1 and its selector is 0 -- so the cut spent a budget of 2 on a glyph drawn 3 wide and the pane overflowed. One generator yields `{text, cols}` steps, the count sums them and the cut takes whole ones, so the two cannot disagree. **Code evidence**: admin/src/panel.mjs -> columnsOf, widthSteps, WIDE, ZERO_WIDTH, sliceColumns, clip, pad, box, makeLineInput; admin/src/style.mjs -> visibleLen, cell, divider, frame, clipPlain; admin/src/render.mjs -> renderRuns; admin/src/graph-model.mjs -> clipName; admin/test/width.test.mjs. |