Replace Handlebars parser with Peggy grammar for @glimmer/syntax

[NullVoxPopuli-ai-agent]

Summary

Replaces the entire Handlebars parsing pipeline in @glimmer/syntax with a single Peggy grammar that parses full Glimmer templates (HTML + Handlebars expressions) and produces ASTv1 nodes directly.

Before

• @handlebars/parser (Jison-generated, 2032 lines) → flat HBS AST
• whitespace-control.js → standalone stripping pass
• handlebars-node-visitors.ts → HBS AST → ASTv1 conversion
• simple-html-tokenizer → HTML tokenization (interleaved with HBS visitor)
• tokenizer-event-handlers.ts → HTML event handlers building ASTv1

After

• hbs.peggy (1048 lines) → the grammar file, single source of truth
• parser.js → generated from grammar (gitignored, built by rollup plugin)
• tokenizer-event-handlers.ts → preprocess() calls Peggy parser, does entity decoding + whitespace stripping + SourceSpan conversion

What was removed

• @handlebars/parser package (entire directory)
• simple-html-tokenizer dependency
• handlebars-node-visitors.ts (585 lines)
• parser.ts (203 lines — base Parser class)
• whitespace-control.js (178 lines)
• visitor.js (105 lines)
• Jison grammar source files (.yy, .l)

Key design decisions

• Peggy was chosen as the parser generator — grammar file in, standalone JS out, dev dependency only, zero runtime cost
• The grammar handles both HTML and {{...}} expressions in a single pass
• Path validation (reject ./, ../, mixed separators) is done in the grammar, not downstream
• Partials, decorators, and partial blocks are rejected at parse time
• The grammar produces structured head/tail on PathExpression directly
• Entity decoding and whitespace stripping are post-processing passes

Build

• pnpm build:hbs-parser regenerates parser.js from the grammar
• Rollup plugin auto-generates parser.js if grammar is newer
• prepare script generates on install
• parser.js is gitignored (generated artifact)

Test plan

• pnpm build:js succeeds
• pnpm test:node — all 20 tests pass
• Type checking passes
• ESLint clean
• Prettier formatted
• Full browser test suite
• Prettier handlebars smoke test

🤖 Generated with Claude Code

[NullVoxPopuli-ai-agent]

Addressing review comments:

exception.js — "why not subclass error?"
Good call — converted to class Exception extends Error in 32bfd3c.

handlebars-node-visitors.ts — "why do we need this? can we parse correctly the first time?"
This file exists because the Jison-generated parser produces a flat Handlebars AST (a list of statements: MustacheStatement, ContentStatement, BlockStatement, etc.), but Glimmer needs an HTML-aware AST with ElementNode, attributes, modifiers, etc. The two can't be unified into one pass because the HTML tokenizer (simple-html-tokenizer) and the Handlebars parser operate on different levels — ContentStatement nodes contain raw HTML text that gets fed back into the HTML tokenizer via tokenizePart().

The long-term fix would be replacing the Jison parser entirely with a single-pass Glimmer-native parser that handles both HTML and expression syntax together. This PR is a stepping stone toward that — by inlining the parser we can now iteratively replace pieces of it.

[NullVoxPopuli-ai-agent]

Current state: 9057/9138 passing (99.1%)

32 remaining failures — many may be pre-existing (main branch CI also fails).

Breakdown:

• 7 fn/on keyword scope tests — fn and on aren't in STRICT_MODE_KEYWORDS list. May be pre-existing.
• 7 block params HTML syntax errors — need custom error messages for invalid as |...| patterns in elements
• 5 compile error messages — unclosed elements, void tags, unmatched end tags
• 5 parser AST tests — disallowed chars in element space, mustache in tag name
• 6 location info tests — column precision after whitespace stripping
• 1 code generation — entity encoding
• 1 named blocks — error message format

The Peggy grammar (hbs.peggy) is now the single source of truth for the full template syntax. tokenizer-event-handlers.ts is down to post-processing: entity decoding, whitespace stripping, location conversion, and AST plugin execution. All the old pipeline code (simple-html-tokenizer, handlebars-node-visitors, parser.ts, whitespace-control, visitor) is deleted.

[NullVoxPopuli]

Since the errors are going to be different, I think I want to explore @glimmer/syntax being rust/wasm, like how content-tag is

options:

• https://github.com/lalrpop/lalrpop
  • error recovery: https://lalrpop.github.io/lalrpop/tutorial/008_error_recovery.html
• https://github.com/kevinmehall/rust-peg
• https://github.com/pest-parser/pest
  • they call out their error reporting: https://github.com/pest-parser/pest?tab=readme-ov-file#meaningful-error-reporting
  • better perf than nom and serde? https://pest.rs/
    • https://github.com/serde-rs/serde
    • https://github.com/rust-bakery/nom
• https://ptal.github.io/oak/learn-oak.html

[NullVoxPopuli]

this should not need changing. changing original should mean that the other nodes updates

[NullVoxPopuli]

no, we shoould remove this. it's internal, and we don't want it. this grammar work should negate the neat for simple-html-tokenizer entirely