uax14

uax14 - Unicode Line Breaking Algorithm

Implementation of UAX #14: Unicode Line Breaking Algorithm in Go.

This code was originally implemented in github.com/SCKelemen/layout and has been extracted to a standalone package for reusability.

Features

• Finds valid line break opportunities in text according to UAX #14
• Supports multiple hyphenation modes (none, manual, auto)
• Handles word boundaries and spaces
• Supports CJK ideographic text
• Handles mandatory breaks (newlines, paragraph separators)
• Respects punctuation and numeric sequence rules
• Returns byte positions for direct string slicing

Installation

go get github.com/SCKelemen/unicode/v6/uax14

Usage

package main

import (
    "fmt"
    "github.com/SCKelemen/unicode/v6/uax14"
)

func main() {
    text := "Hello world! This is a test."
    breaks := uax14.FindLineBreakOpportunities(text, uax14.HyphensManual)

    fmt.Println("Break positions:", breaks)
    // Output: Break positions: [0 6 13 17 21 28]

    // Use break positions to wrap text
    for i := 1; i < len(breaks); i++ {
        segment := text[breaks[i-1]:breaks[i]]
        fmt.Printf("Segment %d: %q\n", i, segment)
    }
}

Hyphenation Modes

The package supports three hyphenation modes:

HyphensNone

Disables all hyphenation. No breaks are allowed at hyphens (hard or soft).

breaks := uax14.FindLineBreakOpportunities("super-cali", uax14.HyphensNone)
// No break at hyphen

HyphensManual (Recommended default)

Only allows breaks at U+00AD soft hyphens. Hard hyphens (regular -) are treated as regular characters.

// Soft hyphen: U+00AD
breaks := uax14.FindLineBreakOpportunities("super\u00ADcali", uax14.HyphensManual)
// Break allowed at soft hyphen

HyphensAuto

Allows breaks at all hyphens. Dictionary-based automatic hyphenation is not yet implemented.

breaks := uax14.FindLineBreakOpportunities("super-cali", uax14.HyphensAuto)
// Break allowed at hard hyphen

Implementation Details

This is a simplified implementation focusing on practical line breaking for common text layout scenarios:

• Simplified break classes: Uses the core UAX #14 break classes but focuses on common cases
• Word-boundary focused: Optimized for breaking at word boundaries (spaces)
• Ideographic support: Properly handles CJK text with breaks between ideographic characters
• Byte positions: Returns byte offsets (not rune indices) for direct string slicing
• Zero dependencies: Uses only the Go standard library

Limitations

This is a simplified implementation focusing on practical line breaking. Some UAX #14 features are not fully implemented:

Not Implemented

• Dictionary-based automatic hyphenation (for HyphensAuto mode)

What Works

• Regular newlines (\n, \r, \r\n)
• Unicode line/paragraph separators (U+2028, U+2029, U+0085)
• Zero-width spaces (U+200B)
• Word boundaries (spaces and tabs)
• Soft hyphens (U+00AD)
• Hard hyphens with appropriate mode settings
• CJK ideographic text
• Non-breaking spaces (U+00A0)
• Word joiners (U+2060)
• Combining marks and emoji
• Mixed scripts
• Punctuation and numeric sequences

Examples

See example_test.go for more usage examples, including:

• Basic text wrapping
• Soft hyphen handling
• CJK text processing
• Line wrapping algorithms

Performance

The implementation uses a rule-based architecture (v5.x) with extensive optimizations:

ASCII Fast Path (Phase 7d)

• Simple ASCII text: 10x faster than original implementation! (~42ns for short strings)
• Applies to: alphanumeric characters, spaces, CR, LF (no punctuation or tabs)
• Use case: Variable names, identifiers, simple English prose
• 30-40x speedup over Unicode path for qualifying text

Unicode Text

• 1.28x faster than baseline rule-based implementation (through Phase 9)
• 1.95x slower than original inline state machine for complex Unicode
• Trade-off: Maintainability and correctness over raw performance
• Full UAX #14 compliance: 100% conformance (19,338/19,338 official tests)

Optimizations Applied

• Phase 1-5: Bitpacking, flat arrays, streaming parser, dense enums (1.22x)
• Phase 7d: ASCII fast path with SIMD-style detection (30-40x for ASCII)
• Phase 9: Hybrid dispatch - pair table first for 82.59% of cases (1.05x)

See PERFORMANCE_OPTIMIZATION.md for detailed optimization history.

Testing

go test -v
go test -bench=.

References

• UAX #14: Unicode Line Breaking Algorithm
• CSS Text Module Level 3 - Hyphenation
• Unicode Line Breaking Properties

License

MIT