feat!: replace punkt-segmenter with a swappable segmenter seam - #11
Merged
Merged
Conversation
punkt-segmenter trains an unsupervised abbreviation model on the single text node it is about to segment, which teaches it nothing and degrades to bare period-space-capital rules -- measured against real input, it false-splits abbreviations and initials in most of the cases that matter (Проф., Dr., St., No., fig., рис., initials like А. С. Пушкин). It also drags in unicode_utils, eleven Unicode versions behind, for three methods Ruby has done natively since 2.4. TranslationDiff::Segmenter replaces it: split after a terminator run followed by whitespace (or a CJK terminator, no whitespace needed) unless a guard fires -- lowercase follows, a known abbreviation or initial precedes, digits sit on both sides, or the period is inside a URL/email. Every guard only prevents a split, never adds one, because a missed sentence boundary just makes a bigger cache unit while a false one sends half a sentence to the translation provider on its own. split_offsets returns character offsets rather than punkt's [start, end] pairs, which makes full text coverage true by construction and removes Tokenizer's boundary-extension patch-up. TranslationDiff.segmenter is swappable the way .api and .cache_store already are. The thirteen existing tokenizer tests, written against punkt, pass unchanged. ox is now the gem's only runtime dependency.
Measured against the Golden Rules corpus, the in-house segmenter scores 47/80, worst on languages with no letter case at all (Arabic, Hindi, Armenian, Greek) since its central rule depends on case. pragmatic_segmenter scores 78/80 on the same corpus, so it becomes the default. - Rename TranslationDiff::Segmenter to TranslationDiff::Segmenters::Simple, moved to its own namespace alongside the new TranslationDiff::Segmenters::Pragmatic. All 21 of Simple's tests move with it unchanged; it stays available as the zero-dependency option. - Add TranslationDiff::Segmenters::Pragmatic, wrapping pragmatic_segmenter's per-language rule sets. It recovers offsets by locating each returned sentence in the source, in order, and raises rather than guessing if a sentence cannot be found -- verified against a real trigger (a Japanese cleaner rule that deletes a newline after "の"). - split_offsets gains a language: keyword, threaded from Request through Tokenizer, since pragmatic_segmenter picks its rules by language and otherwise falls back to English (which mis-segments, e.g., Russian abbreviations). - Add pragmatic_segmenter as a runtime dependency (MIT, zero dependencies of its own). - Add a Golden Rules regression sample and document the segmenter contract, both implementations, and the language argument in the README. Two of the thirteen existing tokenizer tests now fail under the new default and are left unedited, as instructed: one is a defensible missed-boundary difference (a bare "!" no longer treated as its own sentence), the other is a genuine false-split risk (pragmatic_segmenter treats a bare newline not preceded by whitespace as a sentence boundary even with no punctuation at all). Full analysis in .superpowers/sdd/2026-09-07-translation-diff-plan/pragmatic-report.md.
pragmatic_segmenter treats almost any single newline as a sentence boundary candidate, even with no punctuation at all -- a false split, the harmful kind, and a common one since HTML text nodes routinely carry incidental newlines from source formatting. Fix the segmenter rather than the input: replace single newlines (not part of a \n\n+ run) with a space in a shadow copy, segment the shadow, and recover offsets against it, then slice the original text -- "\n" and " " are both one character, so every recovered offset is valid in the source too, guarded by an explicit length check rather than an assumption. This costs one Golden Rules point (76/80 -> 75/80): a bare list of items separated by single newlines, with no punctuation, now segments as one unit instead of three. That shape doesn't arise in this gem's actual input -- HTML list items are already separated into distinct text nodes by markup -- so it's a deliberate trade for closing a false-split class that does occur in real input. It also fixes the Japanese trigger for the offset-recovery raise found previously; a new, non-newline-related real trigger (pragmatic_segmenter's InlineFormattingRule, which deletes a PDF/OCR artefact unconditionally) replaces it in the test suite, since the raise is narrowed, not removed. Also accept the controller's ruling on the other tokenizer disagreement: a bare "!" no longer starts its own sentence under Pragmatic, which is a missed boundary (the safe direction), so test_tokenizes_text_split_into_sentences now expects the merged text with a comment recording why.
H1: pragmatic_segmenter's cleaner rewrites sentences in ways newline shadowing doesn't cover -- it collapses runs of three or more spaces and respaces abbreviations like "Ph.D." into "Ph. D.", among other things -- so recovery could still fail to find a sentence verbatim, and it raised. That was wrong: raising is not the only alternative to guessing. Recover every offset that can be verified, stop at the first sentence that cannot be located, and let the remainder of the text stand as one final unit. [0] is always a legal answer -- it means the node isn't sliced further -- so this can never corrupt a document, only coarsen a cache unit. Also close a related case: an empty sentence from upstream is skipped rather than emitting a duplicate, non-increasing offset. The one remaining raise is a postcondition assertion (offsets start at 0, strictly increase, stay within the text) that #recover_offsets guarantees by construction and should never actually fire. M1: language codes reaching Segmenters::Pragmatic were not normalised, so "RU" and "ru-RU" fell through to English rules -- and DeepL, this gem's own adapter, sends exactly those shapes. Downcase and drop the region subtag before checking the code against PragmaticSegmenter::Languages::LANGUAGE_CODES, falling back to English explicitly for anything unrecognised. Also: reworded the class comment to describe today's pinned-version cleaner behaviour as present-tense fact rather than a hypothetical future one; corrected golden_rules_test.rb's comment (eleven exemplars, one Japanese, two Arabic, not "a dozen" and "two Japanese"); added a test for Simple's language: keyword being accepted and inert; and added a reconstruction case with multiple blank lines (more than one, and more than one gap) for both segmenters. Golden Rules: 75/80 holds (per-language breakdown unchanged from the previous round -- neither fix touches any exemplar in the sample).
The scratchpad corpus used to score the previous two rounds was a reconstruction from a leftover JSON dump and was off by one exemplar, so every derived figure (README, CHANGELOG, the class comment) was one point low. Rebuilt from the authoritative source instead -- the `context "Golden Rules" do` block of each of the ten per-language spec files on diasks2/pragmatic_segmenter -- which yields exactly 80 exemplars and the canonical per-language counts. Verified directly on three commits of this branch: 48e5659 scores 77/80, 229b2e3 (newline shadowing) 76/80, and the current code (H1 + M1 fixes) 76/80. Every delta already documented still holds -- shadowing costs exactly one point, the H1 fix costs nothing -- only the absolute numbers move, from 75/80 to 76/80. Also, in the three places that stated the number, name where it comes from, so a bare figure with no provenance cannot drift unnoticed again the way this one did. And made the README honest about what the remaining misses actually are: of the 4 exemplars Pragmatic misses, 3 (two English, one Japanese) are not boundary disagreements -- pragmatic_segmenter's own expected value has an incidental newline rewritten into a space, which this gem's reconstruction invariant forbids doing to the source -- leaving exactly one real boundary disagreement, the newline-separated list shadowing already documents as a deliberate trade. Re-measured punkt on the corrected corpus: still 38/80, unchanged from what CHANGELOG.md already claimed. golden_rules_test.rb's own citation was also wrong (it named a spec/lib/ file that does not exist upstream) and its "79 exemplars" is now 80; corrected both. No behaviour changed; the suite stays at 123 runs.
N3: recover_offsets discarded a boundary it had already proved. For "First is fine. Hello world mid. Third one here.", "First is fine." is matched verbatim but the walk still returned [0], throwing away the fact that we know exactly where it ends -- the algorithm only emitted the starts of located sentences, and the second sentence's start was the thing that couldn't be found. Now the cursor left by the last successfully located sentence is emitted too when the walk stops early, guarded so it can never equal the text's length or fail to advance past the last offset already emitted (the one place this round put the offsets invariant at risk). Split the walk out of recover_offsets into its own method to keep both under RuboCop's method-length limit without disabling the cop. N1: test_an_unrecognised_language_code_falls_back_to_english_rules passed whether the fallback worked or not -- its Russian fixture segments identically under English rules and under Common, so the assertion couldn't tell a working fallback from a broken one. Replaced it with a fixture where they genuinely differ (English's cleaner disables its no-space-between- sentences abbreviation guard, Common's doesn't), verified to fail if the fallback is reverted to passing the code straight through. N2: locate's dead ternary arm (unreachable once the empty-sentence check above it holds, since a located non-empty sentence always advances the cursor by construction) is removed, and its comment no longer claims a branch exists that cannot be taken. Golden Rules: 76/80 holds -- the corpus contains no multi-sentence text where a prefix is verified and a later sentence in the same node fails, so N3 has nothing to move here; it changes granularity only on the kind of input the corpus doesn't happen to contain.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the abandoned
punkt-segmenterwith two segmenters behind one swappable seam, and makes the better of the two the default.Why
punkt-segmenterwas last released in 2010 and last committed in 2018. It exists to run an unsupervised training pass over a corpus and learn abbreviations, and this gem was calling it asPunkt::SentenceTokenizer.new(text).sentences_from_text(text)— training on the single text node it was about to segment, on every call. It learned nothing useful that way. It also dragged inunicode_utils, whose Unicode tables are eleven versions behind what Ruby has shipped natively since 2.4.What changed
TranslationDiff.segmenteris now swappable the way the adapter and cache store already are, with two implementations:Segmenters::Pragmatic— the default. Wrapspragmatic_segmenter, which picks its rule set per language.Segmenters::Simple— the in-house, zero-dependency alternative, kept for callers who want no extra gem and translate only from cased scripts.The contract is
split_offsets(text, language: nil), returning split points rather than strings, so the tokenizer reconstructs the source character for character by construction.Requestthreads the source language down to the segmenter when the caller passedfrom:; in the auto-detection path it is nil, because the detection sample is built from the segmented text and asking earlier would be circular.Runtime dependencies stay at two:
oxandpragmatic_segmenter.Segmentation quality
Golden Rules corpus, 80 exemplars across 10 languages, taken from the
context "Golden Rules"blocks of the upstream per-language specs:SimplePragmatic(default)The gain is concentrated on scripts with no letter case — Arabic, Hindi, Armenian, Greek — where
Simple's central rule ("the next visible character is lowercase, so do not split") has nothing to work with. That is a limit of the approach, not of its tuning.Two of the three remaining English misses are not boundary disagreements: the corpus expects an errant newline rewritten into a space, and this gem never rewrites the source. The only real disagreement is a lowercase list separated by bare newlines.
Two things
pragmatic_segmenterdoes that had to be handledIt treats any single newline as a sentence boundary.
"The cat sat on the mat\nand looked at the moon."came back as two sentences. That is a false split, which is the harmful kind of error here — each half would reach the provider without the other. Single newlines are now shadowed as spaces before segmentation and the offsets applied to the original, so the source is untouched. This knowingly costs one Golden Rules point, on the bare-newline list case; inside this gem, list items are separated by markup and are already distinct text nodes, so a newline within one node is almost always incidental formatting.It rewrites the text it hands back. It collapses runs of three or more spaces and respaces
Ph.D.intoPh. D., so the returned sentence is not always findable in the source. Offset recovery keeps every boundary it can verify character for character, stops at the first it cannot, and lets the remainder stand as one unit. It never emits an offset it did not prove, and a coarser cache unit is the harmless direction — the text still translates correctly.Verification
124 tests, RuboCop clean with no cop disabled. The reassembly invariant was fuzzed at 500k inputs across 24 language codes with zero violations, and the guards on the recovery path were each shown to have a real triggering input rather than only a synthetic one.