Repository navigation
Conversation
Adds tool/src/org/antlr/v4/analysis/DecisionClassifier.java: an exhaustive static SLL DFA constructor that classifies every parser decision by the lookahead machinery it requires to predict without adaptivePredict at runtime. Classification categories: - LL1 existing fast path (delegated to AnalysisPipeline.disjoint) - LLK acyclic SLL DFA, decidable in <=k tokens (k<=5 across SQL grammars) - LLSTAR cyclic SLL DFA, regular unbounded lookahead (ANTLR3-style) - EXACT_AMBIG exact+untainted ambiguity; min-alt is behavior-preserving - CONTEXT_SENSITIVE non-exact conflicts OR conflicts tainted by widening - PREDICATED lookahead traverses a semantic predicate - NON_GREEDY never statically predicted - LR_PRECEDENCE precedence loop of a left-recursive rule - OVERFLOW DFA exceeded the state/budget cap The classifier exhaustively constructs the SLL DFA the runtime would otherwise build lazily, reusing the runtime's ATNConfig/ATNConfigSet/ PredictionMode conflict machinery so verdicts match runtime SLL semantics. Token-interval equivalence classes serve as the alphabet (never per-token). Iterative deepening: exact construction first (widening only at direct call-site recursion, which yields cyclic LL(*) DFAs), falling back to a fully context-free approximation on overflow. A WIDENED_TAINT bit in ATNConfig.reachesIntoOuterContext marks configs whose calling context was widened; conflict states involving any tainted config are untrusted (demoted from EXACT_AMBIG to CONTEXT_SENSITIVE), because widened conflicts may be analysis artifacts that the runtime never encounters (cf. PERMUTE(a, b) in Bigquery's patternPrimary). Wiring: new -Xdecision-report flag in Tool.java prints the report after AnalysisPipeline. Default off; zero impact otherwise. Output includes per-decision category, k, dfaStates, conflict alt sets (exactAmbigAlts / ctxSensitiveAlts / approxConflictAlts), taint flags, an LLK k-histogram, and split max-k (LLK vs any acyclic non-LL(1) including conflicted). Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
Generate table-driven prediction for every parser decision whose static resolution is provably behavior-preserving, eliminating adaptivePredict from ~82-94% of non-LR decisions in real-world SQL grammars. Covered classes: LL(k) decisions (k<=5 in practice), cyclic LL(*) decisions (e.g. dotted-name scanning), and trusted exact ambiguities (resolved min-alt, identical to runtime SLL/LL). Unsafe residues (context- sensitive, overflow, predicated, non-greedy, LR-precedence) keep adaptivePredict. Tool: - DecisionClassifier now captures fully-labeled prediction DFAs (StaticDFA) during classification and exposes buildTables(), which aborts a decision early at the first untrusted conflict; wired behind the opt-in -Xstatic-dfa flag (-Xforce-atn overrides). - New choice models DFAAltBlock/DFAOptionalBlock/DFAStarBlock/ DFAPlusBlock mirror the adaptive shapes with the prediction call swapped; gated by Target.supportsStaticDFA() (Java, Rust). - Tables serialize compactly via CompactSerializer: zigzag LEB128 varints, base64, 80-column line segments. Rust target: - The serialized ATN switches to the same compact encoding (SerializedBase64ATN), replacing the decimal int list; static DFA tables are appended inside the same blob. Both are decoded in one streaming pass (serialized_ints::Decoder, ATNDeserializer::deserialize_compact) into ATN::static_dfas, with no intermediate or retained Vec<i32>. - Prediction runs through BaseParser::dfa_predict(decision): a plain table walk over LA(1..n) - no ATN simulation, no per-prediction allocation, no shared DFA cache - with NoViableAlt semantics matching adaptive_predict. Java target (testing parity): - Separate _serializedStaticDFA blob deserialized at class init (atn.StaticDFATables) + Parser.dfaPredict walker; the standard serialized ATN format is unchanged. Generated output is byte-identical when the flag is off; with it on, parse trees and error messages are identical to adaptivePredict in all differential tests. New test grammar StaticDFA.g4 (LL(k), cyclic LL(*), non-LL(1) loop/optional decisions, exact ambiguity) is wired into gen_test_grammars.sh with -Xstatic-dfa, and tests/static_dfa_tests.rs exercises the full decode+predict path, asserting zero adaptive_predict call sites in the generated parser. Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
Add rust.RustStaticDFARuntimeTests: runs the full runtime-test descriptor corpus against the Rust target with -Xstatic-dfa (via RustStaticDFARunner). Expected outputs were authored against adaptive_predict, so the corpus acts as a broad differential test of the table-driven path. Descriptors that introspect the adaptive engine itself (showDiagnosticErrors, traceATN) are excluded through a new overridable RuntimeTests.skipDescriptor() hook: ambiguity and full-context diagnostics are reported from inside adaptive_predict, which statically-decided decisions bypass by design. The suite caught two real bugs, both fixed here: 1. Dangling-else mis-parse (IfIfElseNonGreedyBinding2): an exact SLL conflict whose epsilon-alternative reaches the conflict only by popping through the decision-entry wildcard into phantom FOLLOW contexts is NOT a context-independent ambiguity - full-context prediction kills that alternative under the real stack. Reinstate BOUNDARY_TAINT (marked when closure pops through the decision boundary, mirroring the runtime's reachesIntoOuterContext bookkeeping) and require uniform boundary usage for a trusted exact ambiguity: either no conflicting config crossed the boundary or all did. Mixed usage falls back to adaptivePredict. Taint bits are ordered so ATNConfigSet's max-merge can only lose the boundary bit in favor of hard bits, which demote regardless. Costs only 1-3 tables per real-world SQL grammar. 2. Wrong error class (NoViableAltAvoidance): on a failed reach, adaptivePredict returns the minimum alternative that already finished the decision entry rule, letting the parser report a precise mismatched-input error later, instead of throwing NoViableAlt immediately. Tables now carry a per-state fallbacks[] array (the static analogue of getAltThatFinishedDecisionEntryRule, derived from boundary-tainted configs) consulted by both walkers on dead edges. Static DFA wire format bumped to v3. Verified: RustStaticDFARuntimeTests 345/345, RustRuntimeTests baseline 362/362, full cargo test, Java differential probe 24/24, tool-testsuite 694 green. Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
The operator loop decision of a left-recursive rule (the
StarLoopEntryState the LR transformer generates) cannot be served by a
single static DFA because its behavior depends on the current
precedence: each operator alternative is guarded by precpred(_ctx, n),
which holds iff n >= _p. The adaptive runtime handles this by keeping a
separate lazily-built DFA start state per precedence value
(DFA.isPrecedenceDfa). Statically, the guard constants n1<...<nk
partition all precedence values into at most k+1 equivalence classes
with identical guard outcomes, so the complete behavior is captured by
one StaticDFA per class - the static analogue of the runtime's
per-precedence start states.
Classifier (DecisionClassifier.classifyPrecedence):
- A frame-balance depth counter (+1 per rule invocation while
non-negative, -1 per return, latched negative on boundary FOLLOW
chases via min(depth-1, -1)) identifies frame-0 precedence
predicates. During a precedence class's start-state closure,
precpred(n) is evaluated against the class representative
(precEvalValue) exactly when depth==0 - mirroring the runtime, which
only evaluates precedence predicates while computing a precedence DFA
start state. Deeper predicates continue to be traversed as epsilon
(assumed-true) with PRECPRED_TAINT, matching every runtime mode.
- applyPrecedenceFilter ports ParserATNSimulator's pass-2: eliminate
loop-exit configurations shadowed by a loop-entry configuration at
the same ATN state with an equal context, except those whose
suppression mark identifies them as re-entries through 0-precedence
call sites (e.g. e '[' e ']'). The mark is re-derived on the tool
ATN from RuleTransition.precedence==0 self-call FOLLOW links - the
tool-side equivalent of ATNDeserializer's outermostPrecedenceReturn.
Because the suppression bit lives above the taint bits in
ATNConfig.reachesIntoOuterContext and merging takes the max, a merge
with a suppressed config can swallow taint bits; they are
conservatively restored from pre-merge configurations keyed by
(state, alt).
- Tables are emitted only if every class is statically decidable
(all-or-nothing: the runtime dispatch must be total). Exact
ambiguities are not trusted inside precedence classes: the min-alt
substitution theorem has not been re-established under precedence
filtering, so such classes conservatively demote the whole decision.
Wire format v4 (SerializedStaticDFAs):
- Tables may be anonymous (decision=-1) when referenced only via a
dispatch; identical class tables are deduplicated by content and
shared through their table index. New dispatch section after the
tables: {decision, numCutoffs, cutoffs[], tableIndex[numCutoffs+1]}.
- The decision_to_table entry encodes three cases: >=0 (plain table
index), -1 (no table), -(dispatch offset)-2 (precedence-dispatched).
Class dispatch: class(p) = #cutoffs < p, a linear scan over <= a
handful of entries.
Runtimes (Rust static_dfa.rs + Java StaticDFATables): both walkers
fetch parser.get_precedence() once at dfa_predict entry and resolve
the table via the dispatch; plain decisions ignore the precedence.
Zero template changes - the LR loop is a StarBlock selected as
DFAStarBlock by the existing hasStaticDFA gate, now extended to
recognize precedence groups.
Tests: new runtime/Rust/grammars/LrDfa.g4 exercises every shape that
makes precedence prediction hard - multiple binary precedence levels,
right-associative operators (^ and ternary), a prefix operator, a
postfix operator, and array indexing (e '[' e ']', whose nested
0-precedence re-entry exercises the suppression path). Generated with
-Xstatic-dfa; the operator loop is fully table-driven (zero
adaptive_predict call sites) with per-precedence class dispatch.
9 Rust unit tests cover associativity, suppression, ternary, mixed
expressions, and error recovery. Verified: both Rust runtime suites
707/707 (the LeftRecursion/* descriptors now run through the
dispatch), tool-testsuite 694, Java differential probe 29/29 against
an adaptive-prediction build of the same grammar including malformed
inputs, p1probe v3-era regression 24/24 under v4, full cargo test +
fmt clean.
Effect on SQL grammars: 10 of 34 LR loop decisions go static (those
whose SLL prediction is context-independent per class). The remaining
24 demote with approx[{1,2}] conflicts that are genuinely context-
sensitive - isolated to inputs like 'FRAME BETWEEN b AND b' where AND
cannot be resolved without the real stack (BETWEEN separator vs
boolean operator). The classifier demotes precisely the decisions that
are theoretically non-static; the adaptive runtime pays a full-context
LL retry on every such token, which no SLL table can precompute.
Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
The 17 OVERFLOW decisions per SQL grammar were not wrong analyses but impractically large ones: decisions like 'expression (AS? alias)?' or 'identifier (DOT pathComponent)* ' have tiny alternatives that pop through the decision boundary immediately, so all their lookahead beyond a couple of tokens lives in the grammar-wide phantom FOLLOW space (instrumentation showed >99% boundary-tainted configurations concentrated in unrelated rules). The complete SLL DFA over that space is effectively unbounded - the adaptive runtime only wins by building it lazily per input. No pruning fixes an enumeration problem, so stop enumerating: cap the table and keep the reference engine for the rest. A hybrid table is the BFS prefix of the exact-context SLL DFA (depth <= 4, <= 64 states by default) whose remaining states are escapes: untrusted conflicts and the beyond-budget frontier get accepts[s] = -1, where the walker abandons the table and the call site reruns the prediction through adaptivePredict. dfa_predict never consumes input, so the rescan is trivially sound: the table only short-circuits lookahead prefixes it fully proved, the adaptive engine handles everything else. BFS order means the emitted prefix is exactly the shallow states where dynamic predictions concentrate; a reachability-based coverage metric (share of start-state token space that can complete some prediction in-table) gates emission so tables that could never resolve anything are not generated at all. This also generalizes the per-precedence dispatch: classes are no longer all-or-nothing. A precedence class may be fully static, hybrid, or table-less (dispatch entry -1 = straight to adaptivePredict), so the operator loops of booleanExpression/valueExpression go static at 98-100% coverage with only the genuinely context-sensitive corners escaping (the BETWEEN...AND collision - verified differentially on a minimal repro grammar, 12/12 including nested BETWEENs). Also included: precedence guards in lookahead are now evaluated against the constant call-site precedence recovered from the configuration's context top (callByFollowState), instead of assumed-true, whenever the entry precedence is statically known - pruning the redundant derivations that inflate precedence-ladder closures. Unknown or ambiguous entry precedence falls back to assumed-true + taint as before. A -Dantlr.dfa.debug dump of overflow config populations is kept for future tuning. Wire format v5: accepts may be -1 (escape), dispatch table indices may be -1 (adaptive class). Rust walker returns INVALID_ALT to defer; the staticDFAPredict template inlines the fallback at each call site. Java defers inside Parser.dfaPredict via getInterpreter().adaptivePredict. New permanent differential fuzz harness: grammars/FuzzExpr.g4 (precedence ladder + scan-past-expression hybrid decision + a soft keyword making 'as' a valid identifier, so escapes are exercised dynamically) is generated twice - with -Xstatic-dfa into tests/gen and adaptive into tests/gen/adaptive - and tests/fuzz_differential_tests.rs drives 800 fixed-seed random inputs (grammar-directed and token soup) plus pinned soft-keyword corners through both parsers, asserting identical trees. Results across the six SQL grammars: OVERFLOW and CONTEXT_SENSITIVE residue eliminated entirely; every LR precedence loop static; zero bare adaptive_predict call sites remain (every prediction is table-first); generated parser size +3.7% to +7.3%. Verified: full cargo test incl. fuzz, RustStaticDFARuntimeTests + RustRuntimeTests 707/707, tool-testsuite 694, Java probes 24/24 + 8/8 + 12/12 escape corners. Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
…bles
Diagnosis of why simple arithmetic (SELECT 1 + 1) stayed on the
adaptive path while multi-column selects got their speedup: every
prediction was table-first, but the hot class-0 states of the operator
loops escaped. Two independent causes, both fixed:
1. Escape-by-truncation: the hybrid state budget starved sibling
states of wide first ranks, so single-token predictions - the
dynamic majority - escaped on sheer decision fanout. Depth 0/1
states are now exempt from the state cap (rank 1 is bounded by the
token alphabet); the cap applies from depth 2.
2. Escape-by-phantom-conflict: the loop's exit alternative popped into
grammar-wide FOLLOW, where a phantom enclosing loop with
assumed-true guards could always eat the operator, manufacturing an
enter/exit conflict on every operator token. Addressed on two
fronts:
- Boundary pops from the decision's own frame (depth >= 0) now
chase only call sites whose precedence argument lies in the class
interval - the only invocations the class table serves. Pops from
phantom enclosing frames (depth < 0) still chase everything: the
closure busy set is keyed by lineage (pre/post boundary) so the
filtered first pop cannot swallow the unfiltered deeper pop of an
identical configuration (that dedup once dropped the exit-two-
loops-then-BETWEEN-takes-the-AND derivations). Entry invocations
of the left-recursive rule itself (no caller frame) defer to
adaptivePredict at runtime - the tables deliberately prune the
empty-stack FOLLOW space such invocations predict against.
- Class-0 enter/exit exact conflicts resolve statically to iterate
by a substitution argument: an exit derivation whose scanned
tokens are consumed by an enclosing invocation of the same loop
maps to an iterate derivation of the identical sentence (all
guards pass at class-0 entry precedences), so exit never becomes
uniquely viable and the adaptive engine answers iterate whenever
it terminates. Trust requires: exactness (config pairing - the
dangling shapes fail it), no FOREIGN_CONSUME taint on exit (a new
taint marking post-pop consumption outside the loop block without
a loop frame in context: the genuinely context-sensitive
BETWEEN e AND e collision is foreign and keeps escaping), and no
unmet obligation in any phantom same-rule frame (a new epsilon-
completability check: a plain binary loop is completable right
after its operand, but expr (',' expr)* '>>' expr still owes its
'>>', and there the runtime kills iterate with deeper lookahead -
caught by the ReturnValueAndActionsList descriptors).
Assumed-true precedence guards (PRECPRED taint) no longer block
exact-ambiguity trust in plain decisions: the runtime's own SLL
closure traverses non-frame-0 precedence predicates as epsilon
under the same assumption, so such configs mirror the runtime;
only widening remains an analysis-only artifact.
Both unsoundness candidates in the development of this change were
caught by differential tests (B4 escape corners, runtime suite), and
the final form passes all of them.
Rust runtime: dfa_predict escape tracing behind ANTLR_DFA_TRACE
(OnceLock-cached), is_precedence_dispatched + entry-frame defer;
Java runtime: same defer in Parser.dfaPredict.
Measured on Trino (Rust, 30k parses each):
- SELECT T1.C1, T2.C2, T3.C3, T4.C4 FROM MY_TABLE:
17.6us -> 3.8us (4.6x)
- SELECT 1 + 1 FROM MY_TABLE: one residual escape (class-0 operator
conflict where phantom foreign/obligated exit paths remain
possible), ~parity with adaptive
- operator-loop classes now 98-100% coverage across all six SQL
grammars, all LR loops static, zero bare adaptive call sites,
OVERFLOW/CONTEXT_SENSITIVE still zero
Verified: cargo tests incl. fuzz differential (53), runtime suites
707/707, tool-testsuite 694, B4 17/17 escape corners, p1probe 24/24,
lrdfa 8/8.
Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
Isolates the production escape pattern in a small differential grammar (static + adaptive twins, pinned corners): BETWEEN as a suffix predicate on a pure-arithmetic value ladder, with a boolean AND/OR loop above - the dbt SQL shape. Findings recorded in the grammar and test comments: - Without value-position recursion into the boolean rule, the collision is an illusion: the BETWEEN separator AND is consumed by the predicate rule, the boolean loop's LA=AND is always iterate, and every decision is statically k=1 (verified with -Xdecision-report). Function calls, IN lists, subscripts and LIKE..ESCAPE suffixes do not change this. - The trigger is a value-position recursion back into the boolean rule: adding a lambda production (ID '->' b, Trino's identifier '->' expression primary) makes the collision genuinely context-sensitive. A BETWEEN bound can then end with a lambda whose body absorbs AND tokens: in `x BETWEEN y -> a AND b AND c` the first AND belongs to the lambda body and the second is the separator - which is which depends on the whole rest of the input. The boolean loop and the predicate decision become hybrid tables escaping exactly on these inputs (ANTLR_DFA_TRACE confirms escapes only on AND/OR at the loop and BETWEEN at the predicate decision). - Both engines resolve the ambiguous corners identically, maximal munch: `x BETWEEN y -> a AND b AND c` = BETWEEN(x, y -> (a AND b), c), while `x BETWEEN f -> a AND b` keeps the body minimal so the single AND can separate. Expected trees are pinned in test_semantics. This is also the explanation for the residual Trino escapes: they are not BETWEEN-specific, they are lambda-shaped (value-position boolean recursion), and their resolution must stay with the adaptive engine. Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
Mirror of the Rust runtime's ANTLR_DFA_TRACE: with -Dantlr.dfa.trace, every prediction the static tables defer to the adaptive engine is logged (decision, precedence, table state) - the tool for finding which decisions a given input keeps escaping on. Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
arithmetic resolves in-table (Trino 1+1: 0 escapes, 2.7x)
The class-0 operator-loop escape behind "SELECT 1 + 1" falling back to
adaptivePredict has been eliminated through three analysis refinements,
each validated against the differential oracles:
1. The loop block is now rooted at the rule start, covering the
base-operand descent. Previously the BFS started at the loop's
iterate branch, so the follow state of the base operand's rule
reference was outside the block and consumption inside a nested
invocation's operand was misclassified FOREIGN, vetoing the
enter/exit substitution on every plain binary-operator conflict.
2. Obligation checking uses pending-token states: positions that can
reach a token match by epsilon without passing through the loop
decision (reverse-epsilon closure of token positions, blocked at
the decision). A phantom frame owing such a token - Send's list
element returning into a pending ','/'>>' - has no iterate-side
counterpart and keeps the veto; a frame blocked only by the
decision owes exactly the next operator, which the iterate reading
owes too. The own-state veto fires only when the operand's
invocation returns into pending tokens (compound alternatives);
the operand of a simple binary/unary operator returns to the
alternative tail and is shared with iterate.
3. The enter/exit substitution now requires every guarded alternative
of the loop rule to be simple: at most one rule invocation and no
internal iteration. This was forced by a demonstrated unsoundness:
for expr (',' expr)* '>>' expr the iterate reading of "a,c>>x"
completes the inner Send as a list element and dies (the outer
Send's '>>' is missing) while the exit reading completes the
sentence - accepting iterate there produced the nested tree
(expr a) , ((expr c) >> (expr x)) instead of the flat
(expr a) , (expr c) >> (expr x). Compound loops keep escaping to
the adaptive engine.
An additional mechanism was built and deliberately removed during
development: pruning boundary-popped re-invocations of the precedence
rule from outside the loop block. It was unsound - the BetweenExpr
differential (a < b OR c > d BETWEEN 1 AND 2) caught it accepting a
second predicate - and turned out unnecessary: refinements 1-3
suffice. The diagnostics that guided the work stay (property-gated):
-Dantlr.dfa.debug=states/veto decision dumps with rule-decoded
contexts.
Results (Trino, Rust, 30k parses each):
- SELECT 1 + 1 FROM MY_TABLE: 5.6us -> 2.1us (2.7x), zero escapes
- SELECT T1.C1, ..., T4.C4 FROM MY_TABLE: 17.3us -> 3.6us (4.8x)
- valueExpression class 0 now 100% table coverage; all six SQL
grammars keep all LR loops static with zero bare adaptive sites
Verified: cargo 55/55 (fuzz + BetweenExpr + lr_dfa differentials),
runtime suites 707/707 (incl. ReturnValueAndActionsList), tool
-testsuite 694, B4 17/17 escape corners, Send 9/9, p1probe 24/24,
lrdfa 8/8.
Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
The pending-token obligation check treated every epsilon-reachable token as owed, but an OPTIONAL trailing token is not an obligation - the phantom frame can skip it. Duckdb's valueExpression base alternative (primaryExpression '(+)'?) was exactly that shape, vetoing the enter/exit substitution on every simple arithmetic expression. Pending is now computed as unskippability: positions that cannot reach the loop decision by epsilon alone. An optional marker position skips freely (benign); Send's list-star exit still cannot proceed without its '>>' (veto preserved). Duckdb, Rust, 20k parses each: - SELECT 1 + 1 FROM MY_TABLE: 6.1us -> 2.6us (2.3x), zero escapes (was: parity with adaptive, 1 escape/parse) - SELECT T1.C1, ..., T4.C4 FROM MY_TABLE: still parity - 12 escapes/parse remain in the function-call postfix loop (guarded alternatives with argument-list iteration: the substitution stays conservatively off per the Send evidence), the selectItem alias decision, and the primary base decision's identifier ambiguity. Verified: cargo 55/55, runtime suites 707/707, tool-testsuite 694, B4 17/17, Send 9/9, p1probe 24/24, lrdfa 8/8, Trino 1+1 still 0 escapes, all six SQL grammars fully table-first. Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
The category summary was deceptive under the hybrid scheme: a decision marked static could still carry hybrid escape states or adaptive precedence classes that hand individual predictions back to the adaptive engine at runtime. The report now shows the true fallback structure: - per decision, an fb column: 0 (no prediction can reach adaptivePredict: pure LL(1) or complete table), all (no table), N (escape states of a hybrid table), Ne+Mc (precedence decisions: escape states + classes without tables); - a grammar-level fallbacks line: static-only decisions/total, total escape states and how many hybrid decisions carry them, adaptive precedence classes, and decisions with no table at all. Current picture across the six SQL grammars: grammar decisions static-only escapes (hybrid decisions) adaptive-only Trino 415 355 1367 (60) 0 Bigquery 450 420 746 (30) 0 Databricks 520 424 4394 (96) 0 Duckdb 529 429 4932 (100) 0 Redshift 486 397 3663 (89) 0 Snowflake 786 695 3829 (91) 0 Every prediction in every grammar is table-first (zero adaptive-only decisions, zero adaptive precedence classes); all fallbacks are per-state hybrid escapes. Escape-heavy decisions are the known cold corners: Trino valueExpression fb=90e, primaryExpression fb=56e, booleanExpression fb=2e - the lambda/BETWEEN shapes the adaptive engine owns by design, invisible to simple arithmetic (0 escapes on SELECT 1 + 1 despite fb=90e on the loop). Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
Generated parsers wrapped every dfa_predict call site with an adaptive_predict fallback, even for decisions whose tables are complete (no escape states, no adaptive classes) - dead code at every such site, and it made the generated files misleading as an audit of the fallback structure (betweenexprparser.rs showed 8 adaptivePredict sites while only 3 decisions can actually defer). The Choice model now carries tableCanDefer, computed from the tables: true for precedence-dispatched decisions (they additionally defer at the entry frame - when the left-recursive rule itself is the parse entry - regardless of escapes) and for any table with escape states or classes without tables; false for plain complete tables. The Rust staticDFAPredict template emits the bare walker for the latter. BetweenExpr: 5 of 8 sites keep the fallback (d=0 hybrid; d=3, d=7 dispatched with hybrid/adaptive classes), d=2/d=6/d=8 go bare. Trino: 82 of 206 sites keep it (124 dead wrappers removed). A/B benchmark of the two template forms on identical tables: 4.32-4.54us vs 4.29-4.32us on the multi-column query - perf-neutral. Verified: cargo 55/55 (incl. both differential suites), runtime suites 707/707, tool-testsuite 694, B4 17/17, Send 9/9. Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
… operator loops resolve in-table The enter/exit substitution (resolve a precedence loop's iterate/exit conflict statically to iterate) no longer requires an exact conflict and is no longer restricted to class 0. Iterate viability on the shared prefix attests the operator guard passes at the current precedence, and the iterate lineage can always take the free loop-exit and rejoin exactly the caller continuation the exit lineage heads for, so exit never becomes uniquely viable in any class; the foreign- consumption, pending-obligation, and attested-real-iterate vetoes remain the load-bearing conditions. ExprLoops repro: value loop fully static in all 6 classes. All seven SQL dialects improve (Duckdb -263, Databricks -275, Redshift -267, Snowflake -87, Trino -54, Bigquery -32 escapes). Operator chains that escaped on every operator now parse in-table: A || B * C 30us -> 2.7us, A + B*C - D/E 63us -> 3.4us (11-19x vs adaptive). Send-T witness and postfix compound alternatives stay vetoed, by design. Divergence note: on invalid inputs where adaptivePredict throws NoViableAltException at a substituted conflict state, the static table commits to iterate and the parser reports at the match site instead - same error position, different message/recovery tree. Valid-input behavior is unchanged (the resolution above); this divergence class was already inherent to the class-0 substitution. Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
PrefixFactorAnalyzer detects prefix-factorable decision groups:
alternatives sharing an identical leading element sequence (token
matches and rule invocations) that ends at a rule invocation, with
non-nullable pairwise LL(1)-disjoint tails and no predicates/actions in
the shared region. For such groups a static prediction walker may
accept with an alternative mask instead of a unique alt or an adaptive
escape ("implied left-factoring"): the parser executes the shared
prefix once and resolves the group with a trivial tail decision.
Dry-run only: buildDFA counts the states the mask protocol would
resolve and the escapes it would cure; -Xdecision-report prints
per-decision and total coverage. No emitted table changes (verified:
generated parsers byte-identical, cargo test green).
Coverage across the seven SQL dialects: the ('(' expression) group
(rowConstructor vs parenthesizedExpression, tails ',' vs ')') appears
in every one; ~78 escape states curable overall, including the hot
shallow states (Trino d=316's state 226, hit by every typical
parenthesized expression). Debug aids under -Dantlr.dfa.debug=factor:
group extraction, cure lines, terminal simulation.
Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
…alkers
Tables may now mark a state as mask-accept: its live alternatives are
covered by one prefix-factor group, so the walker returns the group's
alternative bitmask instead of a unique alt or an escape. Mask accepts
are emitted only where a state would otherwise escape (conflict or
budget frontier), so this strictly enlarges what tables resolve; Trino
d=316 goes from 89 escapes to 71 escapes + 18 mask accepts.
Wire format: v6 appends a per-table mask section (state, alts...) after
each table's edges; decoders reconstruct per-state bitmasks. Runtime
protocol: dfa_predict_mask(decision) -> u64 (bit 1<<(alt-1) per live
alt; 0 = defer to adaptive) shares the walker core with dfa_predict,
which degrades mask states to adaptive deferral for call sites without
a factored alternative path. The Java runtime (no factored codegen)
likewise defers mask states and traces them as 'mask-defer'.
Verified: the Rust walker returns mask {2,25} (0x1000002) at Trino
d=316's hot state 226 on 'WHERE COL = (1 + 2)'; Java defers the same
state and parses identically; cargo test green; ExprLoops differential
1690/1690 identical. Factored codegen (mask arms in generated parsers)
is M2.
Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
…cisions
Decisions with a validated prefix-factor plan (M0/M1) now get a
FactoredAltBlock in the Rust codegen: the prediction call uses the
mask protocol (dfa_predict_mask), singleton masks take the ordinary
alternative arms, and each group mask takes a factored arm that
executes the shared prefix once and resolves the choice with the
group's LL(1) tail switch, specializing the label context in place
(copy_from). The adaptive fallback widens its unique answer to a
singleton bit and routes through the same arms.
The shared rule invocation pushes a synthetic call-site state (a
BasicState with a RuleTransition to the shared rule) whose follow is a
synthetic tail-decision state spanning every member's tail (epsilon
transitions to each tail start). Without it, nested adaptive
simulations inside the shared invocation prune against one member's
alternative continuation - e.g. 'SELECT (-1)' dying because the
rowConstructor continuation wants ',' instead of ')'. The states are
appended to the grammar ATN at table-build time and are unreachable
from any rule start.
Verified: Trino d=316's hot state resolves in-table via mask {2,25};
1200 generated paren-heavy queries plus the 1690-input corpus are
static/adaptive identical (the two pre-existing '::' error-path
divergences excepted, from the enter/exit substitution work); the new
ParenExpr witness grammar's differential corners (nested parens,
subqueries, row constructors, unary minus, error inputs) all pass;
cargo test green (11 suites); Send witness unchanged; Q1 perf stable
(Trino 2.2us, Duckdb 2.4us).
Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
OptionalPostfixAnalyzer detects optional-postfix (X Y?) decisions and DecisionClassifier's dry run counts the conflict states where the take rule would fire: skip is never uniquely viable when every skip-side token consumption mirrors into the take structure, so the conflict statically resolves to take. Conditions: optional-postfix shape (2-alt, alt 1 postfix, alt 2 skip), no starvation (a mandatory following element whose FIRST intersects the optional's would let skip serve it - kills TRIM's trimChar? and namedParameter's identifier?), mirror (each descended skip config's deepest context frame is a return into the decision block's end - re-entry through the take path), no foreign consumption, attested real take. Dry run only: no accept changes (checked-in parsers byte-identical). Report: per-decision [postfix: takeFires=N] and a take-rule summary; TAKE-FIRE/TAKE-VETO diagnostics under -Dantlr.dfa.debug=postfix. Coverage: 158 conflict states across the six dialects, all in the pred? family (Trino d=235, Bigquery d=309, Databricks d=382, Duckdb d=337, Redshift d=314, Snowflake d=603) plus joinCriteria dangling else; the BetweenExpr witness decision fires at all 3 of its escapes. The alias family vetoes on mirror (clause continuations let skip win: SELECT a LIMIT 10 / SELECT a FROM T). Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
…zable synthetic tail decision state Two Java-compatibility fixes for the alt-mask machinery. First, the controller's factored-block replacement now runs only for targets with mask codegen (new Target.supportsFactoredAltMask, true for Rust); other targets keep the ordinary DFAAltBlock, whose table's mask-accept states defer to adaptivePredict - Java generation on factorable grammars (Trino d=316) was failing with 'missing code generation template FactoredAltBlock' since the factored work landed. Second, the synthetic tail-decision state appended to the ATN for each factor group is now a BasicBlockStartState (with a paired BlockEndState and a defined decision) instead of a BasicState: it fans out epsilon-wise to every group member's tail start, and the runtime ATN verifier (ATNDeserializer.verifyATN) rejects non-decision states with more than one transition. The Rust deserializer accepted the old form; the Java one did not. Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
The take/skip conflict of an optional-postfix decision (pred? in booleanExpression, joinCriteria? in joinedRelation) is a dangling-else shape: when every token the skip reading consumes is mirrored into the take structure itself (each descended skip configuration's deepest context frame is a return into the decision's block end, nothing was consumed in a foreign frame, and take viability is attested by a non-widened, non-foreign config), skip is never uniquely viable, so the conflict statically resolves to take (alt 1). The starvation check in the analyzer excludes X Y? Z shapes where a mandatory Z can be served by the skipped input (TRIM trimChar, namedParameter identifier); the mirror check excludes the alias family, whose clause continuations (LIMIT, FROM, ...) let skip genuinely win - those stay escaped, as do the context-free divergences between the two. Fires at 158 conflict states across the seven SQL dialects, all in the pred? family plus the joinCriteria dangling-else; BetweenExpr's witness decision goes fully static (fb=3 -> 0) with all pinned corners green. Verified: the original user query is now fully static on Trino; alias and TRIM witnesses parse identically to adaptive; 312-query pred-family corpus and the 1690+1200+10 corpora are static/adaptive identical; cargo (11 suites) and runtime-testsuite rust.** (707 tests) green; escape counts drop in every dialect (Duckdb -56, Redshift -48, Databricks -40, Snowflake -32, Bigquery -19, Trino -6). Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
…ing events
Debug-only (-Dantlr.dfa.debug=widening), zero-cost when off: counts the
closure's context-widening events per decision and classifies each by
whether the widened call is guard-reachable - dominated by a
precedence-guard head of its alternative in the left-recursion
transform's shape (epsilon head walk to the guard, then domination up
to the alternative's own block end, crossing tokens, not descending
into callees). Motivation: sizing a proposed precedence-bounded
wildcard root for widened contexts. The measurement answered it in the
negative: only ~5-11% of widening events are guard-reachable
(ExprLoops 5/170, Trino 12983/120869); the dominant source is the
unguarded operand-descent cycle (valueExpression -> primaryExpression
-> '(' expression ')' -> expression -> booleanExpression -> ...), on
which no static precedence constraint exists. Kept as analysis
tooling; no accept logic, table output unchanged.
Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
… min attestation A conflict in a static-table construction resolves to min(U), the minimum of the conflicting union, when min(U) is the runtime's answer on every real stack. That holds when (a) every (state, ctx) entry of the conflict has min(U) assured: a same-state min(U) configuration whose derivation is tail-independent - runtime-faithful (no widening joint in its context), boundary-pop-free and foreign-consume-free, so it is realizable on every stack matching its context - and whose context covers the entry's, exactly (common-min) or as a wildcard-tailed prefix (shadow); and (b) boundary usage is uniform (all configs popped through the decision boundary, or none), the same fate-equivalence condition the exact-ambiguity trust relies on. Two soundness holes were found and fixed by the witness battery: merged contexts must be covered branch-by-branch (covering only the 16-29-$ branch of [16 29 $, 40 $] misparsed 'x BETWEEN y -> a AND b AND c' in BetweenExpr: a stack arriving via the uncovered branch is matched by the entry but not by the shadow), and mixed boundary usage vetoes (alt 1 boundary-free vs alt 2 boundary-tainted misparsed 'a,c>>x' in LeftRecursion/ReturnValueAndActionsList1_2: the iterate lineage is closure-live but dies in the outer frames at EOF - closure-liveness is not input-viability). Support machinery: widening joints are identity-tracked (SingletonPredictionContext.create allocates per call) so runtime-faithfulness is exact rather than the sticky WIDENED_TAINT over-approximation (which vetoes real resolutions); faithfulness is memoized per attempt. The rule fires in both the full pass and hybrid mode; a full-pass resolution can un-hybridize an entire decision. Results: 21 conflict states resolved across the eight grammars, -837 escape states total (Duckdb -203, Redshift -358, Snowflake -214, Databricks -36, Bigquery -13, Trino -11, ExprLoops -2; BetweenExpr's two escapes correctly retained, as are Send2's). Battery: cargo 11 suites, tool-testsuite 694, runtime-testsuite rust.** 707, 3202-query differential corpus (only the 2 known '::' error-path divergences), 7/7 pinned witnesses, perf unchanged (Trino Q1 2.5us, user query 2.3x vs adaptive). Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
…escapes Generalizes the prefix-factor mask from token-prefix groups to semantic shared-descent groups: a set of a decision's alternatives whose derivations all transit a common rule R with zero token consumption (epsilon descent from the decision). At an escape state (cap or conflict) whose live alternatives are covered by such a group and whose live configurations are all inside an R invocation (R is their rule or a frame's return-state rule), the walker accepts with the group's alternative mask instead of escaping. Per-state prefix consistency is what makes the inside-R proof sufficient: a state is reached by a single consumed prefix, so configurations that entered R through a token-consuming call site land in different states than configurations that entered R through the decision's epsilon descent. The masks defer to adaptivePredict at runtime for now (the factored codegen is the follow-up), so this changes only table markers and sizes: 935 escape states across the eight grammars become mask accepts (ExprLoops 69->42... escapes 69->27, Trino -157, Duckdb -244, Redshift -178, Snowflake -99, Databricks -151, Bigquery -64). Three wiring fixes fell out of the battery: staticDFACanDefer must also treat mask-accept tables as defer-capable (an EXACT_AMBIG table with only masks previously got no adaptive fallback and silently fell through its match arms - the FuzzExpr degenerate-parse failure); mask-resolved conflicts are removed from approxConflicts/ contextSensitiveConflicts so an all-masked decision emits its table instead of collapsing to CONTEXT_SENSITIVE (same bookkeeping class as the utrust fix); and the stale-plan field is nulled per decision (classifyPrecedence runs before the plans are built). The FuzzExpr shape test now pins mask accepts + precedence dispatch. Battery: cargo 11 suites, tool-testsuite 694, runtime-testsuite rust.** 707, 3202-query differential corpus (2 known '::' error-path divergences), 7/7 pinned witnesses, perf unchanged. Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
…e, re-descend top-down The shared-descent group's factored block parses the common rule R once (neutral, muted error listeners, invoking state set to a valid call site), then start_resume un-grafts the parsed node and rewinds the token stream to the prefix start. The ordinary alternatives then re-descend with their natural prediction machinery on the prefix tokens - consuming nothing, since the group's first-consumer proof requires an epsilon descent - until the stored rule short-circuits via resume_take, which grafts the node under the caller's context and restores the stream to the tail position. This replaces the rejected alternatives: probe+reparse (provably exponential on nested prefixes) and bottom-up resume functions (heavy code duplication against ANTLR's top-down flow). No forcing is needed anywhere: the rewound stream positions each prediction on the right tokens, the descent is consumption-free, and every alt body is the ordinary one, verbatim. Soundness rests on the rigorous first-consumer proof: every live configuration of a mask-accept state must contain an R invocation as the decision's first consuming invocation (a deepest R segment on its call chain whose below-segments all lie on the epsilon descent from the decision), so the neutral R call can never fail on a valid mask-covered input. On invalid inputs the neutral parse is muted and its result discarded before deferring to adaptivePredict, so error reporting matches the adaptive engine; nested descent blocks defer when resume is already active. Machine: ParserBase resume state + start_resume/resume_take/ resume_active/abort_resume/begin_mute/end_mute/discard_graft/ syntax_error_count; resume_take hooks at the top of resume-target rule functions (public entries and _rec alike, since left-recursive alt bodies and operands call _rec); DescentAltBlock/DescentGroup codegen emitted only when the table actually carries a group's mask (dead plans add no adaptive widen paths); dispatch plan (explicit tail arms + nullable default) with empty-mandatory tails treated as nullable. The walker itself is unchanged (Stage 1 masks). Fixes caught by the battery: resume_take must match the stored rule (any-rule take livelocked the recursion-context machinery); the tail token must be read before start_resume rewinds the stream; mark() is a stubbed bookmark API - positions use index()/seek(); the neutral node needs a real call-site invoking state (error recovery re-reads it); DeadPlan blocks must not be emitted. Battery: cargo 11 suites, tool-testsuite 694, runtime-testsuite rust.** 707, 3202-query differential corpus (only the 2 known '::' error-path divergences), 7/7 pinned witnesses, perf: Q1 2.28us vs 5.20us adaptive (2.3x), user query 2.5x, star/alias family 3.3x (6.6us vs 21.9us). Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
…n one model
Extends the shared-descent machinery into a unified factor analysis:
a group is a shared leading token prefix (zero or more tokens) plus a
shared first-consuming rule R - the prefix-factor analyzer's token
prefixes and the shared-descent analyzer's rule descents as one model
('(' expression for the paren family is a one-token prefix; wrapper
chains like selectItem->...->primaryExpression are the zero-token
case). The factored block matches the prefix tokens, parses R once,
then resume mode re-descends: the re-descent re-matches the prefix
naturally and short-circuits R with the stored node. So the nested
parenthesized family (the last escape family at Trino d=316 and its
ParenExpr witness) is statically resolved: the common prefix case of
'(' query ')' vs '(' expression ')' is decided by trying R and
dispatching on the tail, and deep nesting parses through with one
parse per level, no speculation, no exponential paths.
Analysis: one DFS per alternative yields every (rule, prefix)
candidate, clustered by (rule, identical prefix); blockAlts widen the
mask to prefix-sharing non-members (they fall through to the widen
arm, accepted without conflict resolution); the first-consumer proof
is boundary-aware (wrapper segments validated down to the R-call
follow state, the prefix consumed below); and a precise ambiguity
disjointness check (a collision is fatal only when a token completes
both a non-member alternative and R and min-alt would prefer the
non-member - ParenExpr's '(ID)' collision is harmless since parens
wins min-alt, and Trino's d=316 group passes since '(TABLE)' cannot
complete a query).
Codegen: the descent block arm fires for any multi-alt subset of a
group's block alternatives; the block is emitted only when the table
actually carries a group's mask (shared SharedDescentAnalyzer.
hasDescentMask helper, also used to let the inline factor executor
yield); start_resume/discard_graft un-match the block's prefix tokens
(the re-descent re-matches them); and the walker's returned live-set
mask (not the group's bits) is what the table stores. Prefixes are
capped at two tokens (real factor prefixes are one or two).
Fixes caught by the battery: prefix marks must be taken before the
prefix matches (the re-descent re-matches them); the R-call branch of
the path search is itself the reach (no self-descent); mandatoryTail
must start post-prefix; and the errorful path must discard the block's
prefix children too.
Battery: cargo 11 suites (paren corners now include (((((x))))),
((((((1, 2)))))), ((((SELECT 1)))) deep nesting), tool-testsuite 694,
runtime-testsuite rust.** 707, 3202-query corpus (2 known '::'
error-path divergences), 7/7 witnesses, perf unchanged. Escapes:
ExprLoops 27, Trino 1119, Bigquery 615, Databricks 3881, Duckdb 4123,
Redshift 2811, Snowflake 3339 (Snowflake -157, Duckdb -52).
Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
Soundness wrap-up of the unified factoring work, fixing three holes that let the static path mis-parse valid inputs (Trino TRIM from-form, parenthesized subqueries) and one tree-integrity hazard: - SharedDescentAnalyzer.firstConsumerUnique: a member alternative is admitted only when R is its unique first consuming position after the prefix (epsilon-only reachability, no tokens, no other calls, no alt end before R). The TRIM group (TRIM LPAREN (trimsSpec? ve? FROM) ve RPAREN) failed this: the body's optional trimsSpecification can consume BOTH while the neutral valueExpression parses the same token as identifier content, so the resume graft attached at the wrong place. - SharedDescentAnalyzer.widenHarmless: a prefix-sharing non-member is admitted to the block mask only when it cannot collide with the neutral-R span + tail dispatch: either every member beats it on adaptivePredict's min-alt rule, or its post-prefix FIRST set is disjoint from FIRST(R). The old unconditional widening plus the single-token disjointOk check missed multi-token span collisions: '(SELECT 1)' is both '( expression )' over a subquery and '( query )', and the tail dispatch committed to the wrong one (68/1200 paren-corpus divergences; now 0). - Rust.stg DescentGroup: the tail-default arm now rewinds to the decision start and defers to adaptivePredict without arming resume (it previously predicted from the post-R tail position, and armed resume for a body that re-parses everything, mis-grafting the neutral node at the first R call). - BaseParser.begin/end_neutral_parse saves and restores the parse context: an error unwinding out of the neutral parse left the context pointer stale (enter_rule grafts eagerly), corrupting both the tree and the following graft discard. Verified: cargo suites; tool-testsuite 694/0; runtime-testsuite rust.** 707/0; Trino static-vs-adaptive differential on Rust and Java twins over 3208 queries (only the 2 known '::' error-path divergences); TRIM from-form family and nested-paren subquery corners now agree. Trino Q1 static 2.4x adaptive. Descent mask-accept counts drop (the old counts included the unsound accepts); the widen gate itself costs 11 escape states on Trino. The chain-proof helpers (allStartViaR/startsViaR/epsilonCallSites) remain but are currently unused; re-admitting chain-descended groups soundly is future work. Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
Patch bump across all lockstep version locations: - Maven parent + 5 modules (runtime/Java, tool, antlr4-maven-plugin, tool-testsuite, runtime-testsuite) - Rust crate (Cargo.toml + Cargo.lock) - runtime/Rust/gen_test_grammars.sh TOOL_VERSION Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
…ight-sized overflow cap) Static-DFA table construction was dominated by DecisionClassifier.closure; profiling attributed ~91% of -Xstatic-dfa build time to it, driven by raw closure count (Trino 40M, Databricks 473M) concentrated in a few dozen hard decisions per grammar. Four fixes, none of which change results: - maxDfaStates 2000 -> 1000 (now -Dantlr.dfa.maxDfaStates=N tunable). The cap is an overflow trigger, not a coverage target: decisions needing unbounded lookahead expand to the full cap before being abandoned to a hybrid/adaptive fallback, so the cap is a direct linear factor on worst-case time (and precedence rules pay it per precedence class). The largest table any decision actually resolves to across the SQL dialect grammars is ~700 states, so 1000 leaves headroom while halving the doomed frontier. Lowering the cap is sound by construction - it only shifts hard decisions to adaptivePredict, never mis-predicts - and emits an identical set of static prediction sites on every tested grammar (zero coverage loss). - hasReturnState: iterative singleton fast-path. It ran once per rule transition per closure step and allocated a fresh IdentityHashMap each call for an O(depth) DAG walk; contexts are overwhelmingly cycle-free singleton chains, so walk parent pointers directly and fall back to the visited-set DAG walk only when a merged (array) node appears. - ConfigSetKey: cache the config-set hash used to key the DFA-state dedup map and the successor-merge map. A raw Set key recomputed AbstractSet.hashCode - O(|set|), summing every member's hash - on every get/put; caching collapses that to once per set (equality unchanged). - Presize the per-class/per-start closure busy sets to cut HashMap resizes (previously the top self-time frame). Static gen wall time: Trino 5.0->3.8s, Databricks 38.7->26.6s, Duckdb 25.1->18.0s, Snowflake 27.3->21.1s. Differential (static vs adaptive) unchanged (2 known :: error-path divergences), tests/gen byte-identical, cargo 11/11, tool-testsuite 694/0, runtime-testsuite rust.** 707/707. Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
…-report (lossless two-pass) The decision report (-Xdecision-report) ran classifyAll with no early termination, so every decision with unbounded lookahead expanded its exact-context attempt to the full state cap before overflowing - work that classify() then discards, since the reported Result for such a decision comes from the subsequent bound-0 / hybrid attempt, not the doomed exact one. That made the report many times slower than table building and dominated its wall time (Trino ~86s, Redshift ~330s). Add abortInfAsOverflow: in a non-hybrid attempt, treat the first untrusted conflict as a state overflow (overflow=true; break) rather than exploring to completion. Unlike the table builder's abortOnUntrustedConflict (which returns CONTEXT_SENSITIVE and skips the remaining widening depths), this keeps classify()'s full widening-then-hybrid flow intact. This is exactly transparent for any decision that resolves to a static or hybrid table: clean tables never hit an untrusted conflict (so are never short-circuited), and whether a decision gets a hybrid table - and its escape-state detail - is decided by the hybrid pass, which is never short-circuited (the flag is gated out under hybridMode). It is NOT transparent for a decision that ends on the adaptive engine: there the report would display the short-circuited attempt's category/conflict inventory instead of the true full-exploration one - and those are precisely the "remaining adaptive fallback" decisions the report exists to explain. So report() runs two passes: the fast pass with the flag, then a re-classification of exactly the adaptive-fallback decisions (non-precedence with no table; precedence with an adaptive class) without the flag, restoring their exact category and complete conflict inventory. That set is empty on well-covered grammars (the report stays fast) and small otherwise (a grammar with many is barely statically covered). Report output is byte-identical to the full no-abort report on all six SQL dialect grammars, and - stress-tested at maxDfaStates=60, which forces 58 fully-adaptive Trino decisions through the re-run path - byte-identical there too. Table building is untouched (it never sets the flag), so emitted tables are unchanged. Wall time: Trino 210s->5s, Redshift 330s->17s (the 210->86s part came from the prior maxDfaStates commit). tests/gen byte-identical, static prediction sites unchanged, cargo 11/11, tool-testsuite 694/0, runtime-testsuite rust.** 707/707. Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
The table-walk edge lookup was a binary search over the serialized (lo, hi, target) interval triples - O(log E) per token, with /3,*3 index math and multiple loads per probe. On 0-escape decisions that made the static walk ~1.5x slower per decision than the warm adaptive SLL DFA (which indexes its edge array in O(1)), so a long all-static parse lost to the adaptive engine it is meant to beat. Expand each table's edges into per-state direct-indexed rows at load time (from_int_stream), leaving the serialized blob unchanged (no code-size bloat, a memory-only expansion): edge_min[s] is the state's base token, edge_row_at is CSR offsets, edge_targets the flat rows (target per token in [min, min+len), -1 = no edge). edge() is now a bounds check plus one indexed load. Measured (Trino, Rust release, warm, SELECT A, A, ..., A FROM T, all 0-escape), same runtime with/without the change: cols static before -> after S/A before -> after 300 133.3us -> 97.2us 1.50 -> 1.10 Per-decision static cost 441ns -> 320ns (-27%), closing most of the gap to warm adaptive (289ns/decision). The residual 1.10x is the per-state metadata (accepts loaded twice + escape/mask/accept checks before the edge), a separate follow-up. Behavior unchanged: cargo 11/11, runtime-testsuite rust.** 707/707, parsebench static-vs-adaptive diff over corpus-pred 318/0 mismatches. Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
…acing Two small follow-ups to the O(1)-edge change: - dfa_walk loaded accepts[s] twice per state (is_escape() then accept()). Load it once and drive both the escape (== ESCAPE) and accept (> 0) checks from it. ~2% off the per-decision static walk on 0-escape input. - Gate the ANTLR_DFA_TRACE machinery behind #[cfg(debug_assertions)]: in release dfa_trace() is a compile-time `false`, so every trace branch in the walk is eliminated; the debug build keeps the env-var check (now a LazyLock). Keeps diagnostics out of the shipped hot path. Note: the originally-planned advance-first reorder (try the edge before the accept/escape/mask checks) was measured and *reverted* - it regressed the 0-escape walk ~11% (133us baseline: step1 O(1)-edge -> 97us, reorder -> 109us). The three metadata checks are effectively free: the tables are L1-hot and the branches are perfectly predicted on intermediate states, so moving la()+edge() ahead only perturbed codegen. The residual ~1.08x vs warm adaptive on 0-escape input is inherent per-state cost, not the metadata checks. Measured (Trino, Rust release, warm, SELECT A,...,A FROM T, 300 cols): static 133us (pre-step1) -> 97us (step1) -> 96us (this); S/A 1.50 -> 1.08. cargo 11/11, runtime-testsuite rust.** 707/707, parsebench static-vs- adaptive diff over corpus-pred 318/0. Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
Running mvn -Dtest='rust.**' test executed RustRuntimeTests and RustStaticDFARuntimeTests as concurrent JUnit classes in one forked JVM (junit.jupiter.execution.parallel.mode.classes.default=concurrent), which made the combined command non-deterministic: - surefire 2.22.0 mis-attributes dynamic tests of interleaved classes: every combined run reported arbitrary splits (706/1, 700/7, 703/4) instead of 362/345, and any failure is charged to the wrong suite. - Both suites shared one on-disk build cache (<tmp>/ANTLR-runtime-testsuite-cache/Rust): cargo clean during one configuration's runtime initialization deletes the rlib out from under the other configuration's rustc compiles whenever they overlap (concurrent classes, or concurrent Maven JVMs). Fixes: - @execution(SAME_THREAD) on both Rust test classes so the suites never overlap in the forked JVM; tests within each suite still run concurrently via the @testfactory's own @execution(CONCURRENT). Other targets' class-level parallelism is unchanged. - New RuntimeRunner.getRuntimeConfigurationName() hook (defaults to the language) keying both the cache directory and the runtime initialization lock; RustStaticDFARunner overrides it to "Rust-static-dfa", giving each configuration its own cache and init. Verified: 4 consecutive combined runs all report exactly 362 + 345 tests with zero failures (previously every combined run mis-attributed). Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
…fix conflicts
The optional-postfix take rule (X Y?, dangling-else) vetoes when a skip
configuration leaves the mirror discipline - a boundary follow-chase
whose consumption is not funneled through the decision's own take path.
Some vetoes are genuinely context-sensitive yet almost always resolvable:
Bigquery's pred=predicate? decision (d=309) must answer skip inside
MERGE ... UPDATE SET targets+=expression EQ values+=expression (verified
empirically: adaptivePredict parses SET a = b as target/EQ/value) but
take everywhere else, so no context-free static answer exists - the
vetoed states escaped to adaptivePredict on every WHERE/HAVING/ON
comparison, costing the dialect its entire static-DFA win.
This change resolves such conflicts to take guarded by a runtime stack
test (table format v7):
- DecisionClassifier marks skip-lineage consumption outside the
dangling-else discipline with ROOTED_TAINT and records guard roots
(the consumption position for EMPTY-bottomed contexts, the call site
of a non-mirror deepest context frame otherwise), propagated per DFA
state to a fixpoint. A vetoed conflict whose skip configs are all
in-discipline, mirror-bottomed, or rooted - none widening-born -
resolves to take, emitting the root set as the decision's guard.
Bails: clean out-of-discipline skip config, always-firing guard
(root epsilon-reachable from the block end), or never-firing guard
(block end cannot reach the rule stop; downgrades to plain takes).
Rooted-tainted conflicts are never trusted as exact ambiguities (a
max-merge may hide a hard bit beneath ROOTED_TAINT).
- The guard evaluates the epsilon-pop chase from the decision's block
end over the real parse stack: defer to adaptivePredict iff some
invoking state's follow region epsilon-reaches a guard root (danger
set), stopping the walk at the first invoking state whose follow
cannot reach its own rule's stop (pass set). Both sets are
precomputed per decision and serialized as sorted arrays.
- Java and Rust runtimes implement the GUARDED accept (-3) in their
table walkers; format v7 serializes per-state guarded alts plus the
danger/pass sets. Only Java and Rust targets support static tables.
- Witness: grammars/GuardExpr.g4 + guard_differential_tests.rs - the
assignment/comparison dangling-else (37 corners), exercising both
guard-take and guard-defer directions (10 each under ANTLR_DFA_TRACE).
Census: 380 guarded takes across six SQL dialects (Duckdb 157,
Redshift 141, Snowflake 64, Bigquery 12, Databricks 5). Bigquery d=309:
12 of 13 vetoed states guarded, roots = {mergeUpdateClause EQ sites,
pivotOperator IN site}.
Verification: cargo 12/12 (witnesses regenerated byte-deterministically
to v7), tool-testsuite 694/694, runtime-testsuite rust.** 707/707,
corpus differentials 3208 queries/dialect - Duckdb 0 mismatches, Trino
2 and Bigquery 3 (all proven pre-existing '::' error-path divergences
by re-running the pre-guard tool). Rust MERGE/CASE/ORDER-BY/parens
parity battery 10/10. Perf (Rust, release): Bigquery predicate-heavy
queries 1.6-2.6x static vs adaptive (was ~1.0x), Duckdb 1.1-2.2x.
Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
Two refinements that close out the remaining take-vetoed escapes, plus
a critical emission fix for fully-guarded decisions:
- Budget escapes (hybrid depth/state cap) of optional-postfix decisions
are now guard candidates: they were never conflict-analyzed, but the
guarded-take eligibility check runs on their configuration sets at
finalize time just as for conflict escapes (rooted taint and root
sets already propagate through them). Bigquery d=309's ten remaining
budget escapes - the hot '= (', '= CASE', '= CAST', '= ARRAY'
comparison paths - resolve statically now.
- The widening bail is relaxed from any-WIDENED to WIDENED-and-not-
ROOTED: a widened skip configuration's bottom frame is still a real
follow state, so a rooted one's realizability is covered by its
recorded guard roots exactly like a non-widened one's; only
non-rooted widened configurations remain unguardable. Also adds the
missing real-take attestation and skip-foreign checks to the
finalize-time eligibility (previously pre-filtered at veto time).
- Emission fix: when every escape of a decision resolves to a guarded
take, the escape count reaches zero and the hybrid-table emission
branch was skipped entirely - the decision silently lost its table
(reported CONTEXT_SENSITIVE). The emission gate now also fires on
guarded accepts, the candidates carry their conflicting alternative
sets through to finalize, and a resolved conflict moves from
approx/contextSensitive to exactAmbigConflicts like the factor and
descent cures.
Census after extension: 1797 guarded takes across six dialects
(Duckdb 755, Redshift 420, Databricks 306, Snowflake 272, Bigquery 23,
Trino 21). Bigquery d=309 closes fully: escapes=0, coverage=100%.
Verification: cargo 12/12 (witness now exercises 13 guard-takes and 18
guard-defers, 1 escape left), tool-testsuite 694/694, runtime-testsuite
rust.** 707/707, corpus differentials 3208/dialect - Duckdb 0
mismatches, Trino 2 and Bigquery 3 (all previously proven pre-existing
'::' error-path divergences), Rust MERGE/CAST/CASE/unary/parens parity
battery 12/12. Perf (Rust, release): Bigquery predicate-heavy queries
remain 1.5-2.5x static vs adaptive.
Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
…ng (the alias family) cfbed10 narrowed the member gate so no chain-descended group could form ('re-admitting chain-descended groups soundly is future work'). The selectItem alias family - expression (AS? alias)? vs multiSelect -> selectStar -> primaryExpression DOT ASTERISK - is exactly that shape: both alternatives descend to primaryExpression with zero token consumption, alt1 through expression/booleanExpression/valueExpression wrappers, alt2 through multiSelect/selectStar. The CAND candidates existed but every deep one was vetoed, so e.g. Trino's d=153 kept 133 escape states (its hottest: qualified names, aliases, function calls, subscripts, lambdas, windows). SharedDescentAnalyzer: - firstConsumerUnique generalizes to a wrapper-chain descent verdict (checkDescent): side exits before the body's first R call must be unreachable on the commit region - disjoint from FIRST(R), or, for tokens, preceded by the enclosing block's R-ward alternative (the min-alt rule then keeps the body's first R call aligned with the neutral span; e.g. valueExpression's primaryExpression base beats the MINUS/PLUS unary, which overlaps only on signed numerics). Overlapping side calls (TRIM's trimsSpecification) and R-less completions still veto; Snowflake's CONNECT_BY_ROOT? prefix vetoes d=454 (adaptive prefers the prefix take on the overlap - a genuine misalignment). - The dispatch's tail analysis pops along the recorded descent chain (mandatoryTailChain) instead of grammar-wide pops, which dragged the common rule's own follow positions and sibling members' call sites into the continuation sets (pe's deref DOT and selectStar's DOT arm). The selectItem dispatch is default=1, arms={2:{DOT}}. - The per-config first-consumer proof is re-admitted at both mask- accept sites (allStartViaR), extended for chain-descended groups: V_NOCLEAN inert positions, pre-R positions with approved side exits, post-R positions (wrapper stops, the alias block after the expression invocation), and consumption-based post-first-R exemption. Rust codegen (Rust.stg): - descentMaskPredict: a multi-bit mask no group arm covers now defers to adaptivePredict instead of falling through to an empty default arm. Tables carrying both prefix-factor masks and descent masks (now common) previously produced empty parse bodies on the factor mask states. - copy_from: replace + forget instead of *src = ctx. Assignment drop_in_place's the old value, deallocating the children Vec buffer the new value aliases; with the descent flow that buffer is the arena's most recent allocation (the neutral parse's graft push), so the arena hands it to the body's first Vec push and the decision ctx's subtree is silently replaced (selectItem ended up holding the neutral primaryExpression node instead of the expression wrappers). Verified: Trino Java difftest 1690 checked / 2 known '::' error-path divergences; Rust tree differential corpus 1680/2 (same 2), paren 1200/0, pred 318/0, corpus-err 1 (pre-existing on base2); 12 star/alias smoke cases (T.*, A.B.*, F(A).*, -5, nested subquery stars); cargo 11/11; tool-testsuite 694/0; runtime-testsuite rust.** 707/707; Duckdb static/adaptive corpus pass-fail agreement. Report deltas (escape states): Trino d=153 133 -> 39 (cover 100%); Trino 1263 -> 979; Bigquery 656 -> 546; Duckdb 3726 -> 2742; Redshift 2597 -> 1965. Alias-heavy query (release): 11.6us chain vs 13.4us no-chain static vs 41.0us adaptive. Not covered (sound vetoes or separate shapes): Snowflake d=454 (CONNECT_BY_ROOT overlap), the windowFrame continuation family (wildcard-truncated chains), Databricks pivotAsAlias (optional postfix, not a descent), Databricks LPAREN selectStar RPAREN (fails closed to adaptive). Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
…, FIRST pre-check
Three shared-descent extensions that close the identifier/keyword fanout
family of hybrid escapes (the 'primaryExpression widened' hotspot: fat
depth-2 budget escapes where functionCall/measure/lambda/columnRef all
descend through the same identifier chain), plus the ordering rule that
keeps the analysis and the generated dispatch code in agreement.
1. Guarded tail dispatch (the boundary-collision family). The plan-time
dispatch proof required explicit tail arms disjoint from a nullable
member's grammar-wide FOLLOW - hopeless in these grammars, where
every alias position accepts every keyword, so the identifier groups
never got arms. Worse, the default arm's 'viable set' never actually
contained the rule's FOLLOW (LOOK(stop, null) yields only epsilon),
so single-nullable groups passed the collision check unsoundly -
'F(A, 2)' committed columnLambdaExpr on the arg-list comma on
ClickHouse. mandatoryTailChain now chases the stop's grammar-wide
follow links for the default set, and groups that fail the check get
guarded arms instead: arm = the member's in-decision tail FIRST set
minus every other member's, committing only when the real parse
stack's postfix chase cannot consume the dispatch token. The chase
(BaseParser.follow_contains) walks the actual context frames against
per-invocation-site continuation sets - context-free fixpoints over
the ATN computed lazily once per grammar (follow_sets.rs), exact
where the static analysis only had the chase-wide boundary
approximation. Commits are then sound: every other member is dead on
that token, and a dying arm alternative is the accepted error-path
class.
2. Conditional members with head guards. firstConsumerUnique vetoes any
member whose body can consume before its first shared-rule call -
which excluded functionCall (its optional processingMode head) from
the identifier groups, the single hottest escape shape ('x (').
The gate now collects the side exits' gating tokens instead: on
lookahead outside the set no pre-R consumption is possible, so the
neutral span is aligned; on lookahead inside, the block defers to
adaptivePredict. altPaths also continues past nullable wrappers, so
the candidate chains through functionCallHead/functionName form at
all. Per alternative, a clean chain erases conditional ones (their
distinct frames fail the mask-time per-config proof); the group's
head guard is the union of what remains. Trino d=316: 52 -> 14
escape states; corpus escapes 1191 -> 959 (the remaining are the
alias and SUBSTRING families, where the dispatch would defer anyway).
3. FIRST(R) pre-check. The neutral parse is attempted only when the
post-prefix lookahead can start the shared rule - a doomed attempt
costs an errorful muted parse (16x regression on plain TRIM(x),
where the neutral trimsSpecification always fails; now 2.6us vs
2.3us adaptive).
4. Dispatch ordering and consistent pairing. Conditional admission made
a head-guarded primaryExpression group shadow the unconditional
strictNonReserved group at the alias family's selectItem decisions
(Databricks d=289/292/464/421, Snowflake d=454 flipped EXACT_AMBIG
-> HYBRID, ~+400 escape states): first-match covering won while its
first-consumer proof failed on the paren-lambda configs. Plans now
order unconditional (head-guard-free) groups before conditional
ones, and mask acceptance picks exactly the group the generated code
would fire - the first codegenable group covering the mask - and
accepts only when THAT group passes the proof, so a mask's runtime
arm is never unproven. When no codegenable group covers, any proven
group accepts (the runtime has no arm to fire and defers). All five
decisions restored: Databricks 2732 (-88 vs baseline), Snowflake
2012 (-183).
Also: begin_neutral_parse fetches the resume position before seeking
(the walk buffers only the lookahead it consumed; prefix>=2 groups
resumed one past the buffered range and hit the stream's seek panic).
Verification: cargo 57/0 (regen'd witnesses), tool-testsuite 694/0,
runtime-testsuite rust.** 707/707; static-vs-adaptive tree diffs:
Trino 2880/2 (known :: error-path class), Snowflake 2880/0,
Databricks 2880/0, Duckdb 2880/0, Bigquery 2880/3 (same class),
ClickHouse 1680/1 (error-path recovery shape on invalid input);
Trino Rust twins corpus 1680/2, paren 1200/0, pred 318/0, err 10/0.
Perf: SELECT A + F(A,2) X 3.3us vs 6.2us adaptive; A.*, B.C 2.3us vs
5.6us; corpus 73ms vs 90ms.
Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
Move per-rule boilerplate from the codegen template into runtime macros: - impl_ctx!: context type alias, extension struct, create() and the CustomRuleContext impl (plain/labeled/alt-label arms) - parse_rule!: rule-body scaffolding (enter_rule/exit_rule, error recovery, stack-growth guard; rec/catch variants) - impl_parser_recog!/impl_lexer_recog!: ParserRecog/LexerRecog/Actions impls including sempred/action dispatch - dbt_antlr4::prelude: single glob import replacing the per-file import wall Generated parser files shrink ~28% (tests/gen: 18638 -> 13479 lines); CSV parser 700 -> 414 lines. Verified with cargo test (default and all features) and the rust runtime-testsuite (707 tests, 0 failures). Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
…etoed only on foreign consumption The enter/exit substitution of a precedence-class build resolves an iterate/exit conflict to iterate whenever the exit lineage's consumption stays in the loop discipline; it vetoes when exit consumed in a foreign frame (FOREIGN_CONSUME_TAINT). The canonical shape is Trino's BETWEEN separator AND colliding with the booleanExpression loop's own AND (d=238): the phantom exit descends into predicate through a re-entered booleanExpression, a frame no real stack can carry at that decision (BETWEEN's operands are valueExpression, and every nesting path owes tokens the lookahead does not offer). This change gives such conflicts the guarded-take treatment, generalized from optional postfix to precedence loops. When the substitution vetoes solely on foreign consumption (iterate attested real, no obligations), the foreign-marking move records the consumption's root positions (the consuming state, plus call sites of non-loop context frames - loop-block frames play the mirror role the block end plays for the postfix guard) and the state is admitted as a guarded-iterate candidate. A new finalizeGuardedIterates resolves eligible states (every foreign exit config rooted, no unrooted widened exits, iterate attested) to GUARDED accepts with the same danger/pass stack chase as guarded-take - defer is adaptivePredict, so resolution is sound by construction - and upgrades to plain iterate accepts when no invoking state's follow region reaches a root (the foreign consumer is unreachable on every stack). The guard chase is precise about token obligations: nested boolean frames behind ')' / THEN / END never trip it, while BetweenExpr's lambda corners (ID '->' b, a value-position recursion into the boolean rule) trip it exactly where exit genuinely wins. Effect (full-dialect sweep vs rpt4): - Trino booleanExpression d=238: 2e -> 0 (every loop-continuation AND was one adaptivePredict; pred-corpus d=238 fires 190 -> 0, total 370 -> 180); static-only 360 -> 361 - Bigquery d=312: 2e -> 0; Databricks d=380: 164e -> 90e; Duckdb d=335: 118e -> 57e; Redshift d=312: 125e -> 57e; Snowflake d=600: 70e -> 18e - total escapes 10488 -> 10174 Perf: WHERE A=1 AND B=2 AND C=3 AND D=4 parses 5.25us static vs 21.3us adaptive (4.1x; was one escape per AND). Verification: cargo 57/0 (witness corners incl. lambda defers), tool-testsuite 694/0, runtime-testsuite rust.** 707/707; Java tree diffs: Trino 3208/2, Bigquery 2890/3 (both the known :: error-path class), Duckdb/Redshift/ClickHouse 2890/0; Rust twins: Databricks 1998/0, Snowflake 1998/2 (pre-existing error-recovery count class, oracle-proven with the pre-change tool). Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
…edence loops by substitution attestation
The enter/exit substitution of a precedence-class build resolves an
iterate/exit conflict to iterate when the exit lineage's consumption
stays in the loop discipline (no foreign frames), owes no pending
tokens the iterate side does not also owe, and iterate viability is
attested by a non-widened, non-foreign config. That certificate is a
property of the exit LINEAGE, not of any collision - but it was only
ever evaluated inside the conflict branch, so multi-alt states whose
two sides never share an exact stack escaped at the budget frontier
with both alts live. The canonical shape is Trino's valueExpression
(d=254): after 'expr +|-', a reserved keyword-led operand (CAST,
EXISTS, EXTRACT, GROUPING/LEFT/RIGHT, JSON_*, LISTAGG, NORMALIZE,
TRIM - anything not in nonReserved, so no identifier pop-back creates
a colliding $-rooted pair) leaves exactly two parallel configs, real
iterate vs widened phantom exit, that never trigger
hasSLLConflictTerminatingPrediction and die at the state cap.
This change evaluates the same attestations at the budget frontier:
a {1,2}-alt state of a substitutable simple-loop class build whose
configs attest exactly ITERATE_REAL accepts iterate outright - the
adaptive engine would answer iterate whenever it terminates (unique
iterate, or an ambiguity resolved to the minimum alternative), so
the resolution matches the runtime; on inputs where neither alt is
viable the commit-vs-NoViableAlt recovery divergence is the accepted
error-path class. The attestation evaluation is factored out of the
conflict branch and shared verbatim.
Effect (full-dialect sweep vs HEAD):
- Trino d=254: 36e -> 0, all 6 classes clean k=2; d=318: 56e -> 32e
- Bigquery d=330: 12e -> 0 (k=2); d=370: 44e -> 32e
- Snowflake d=623: 24e -> 0 (k=2); d=687: 35e -> 31e
- Databricks d=401: 149e -> 130e; Duckdb d=356: 57e -> 9e;
Redshift d=333: 38e -> 8e
- total escapes 11224 -> 11013 (-209); three decisions flip fully
static (Trino d=254, Bigquery d=330, Snowflake d=623); no other
decision touched anywhere
Perf (Java twins): SELECT A + CAST(B AS INT) FROM T 11.2us static
vs 13.7us adaptive; plain operator chains 11.7us vs 61.4us (5.2x).
Verification: cargo 57/0 (witness battery), tool-testsuite 694/0,
runtime-testsuite rust.** 707/707; 25-query targeted corpus of the
cured shapes 25/0; tree diffs static-vs-adaptive: Trino 3208/2 and
Bigquery 3208/3 (known :: error-path class), Duckdb 3208/0,
Redshift 3208/0, ClickHouse 3208/4 (pre-existing ::/error-recovery
class; its tables are provably untouched - zero frontier
resolutions); Rust twins: Snowflake 1998/2 (pre-existing
error-recovery count class), Databricks 1998/240 (>> error-recovery
class, A/B-proven identical at HEAD - pre-existing, not from this
change).
Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
…wildcard re-descents
The iterate/exit substitution of precedence-loop decisions vetoed on any
exit lineage owing pending tokens (a compound-alternative tail the
iterate side does not owe). In expression-grammar postfix loops (the
primaryExpression subscript/dereference family) every such obligation
is an artifact of the wildcard closure: the exit side pops through the
decision boundary, and the SLL caller-graph closure re-descends into a
fresh expression juxtaposed after the completed primary - a continuation
no real stack offers (real continuations consume a separator first,
which the unmarked follow-position configurations cover). The phantom
frames owe the tails of whatever compound alternatives the cascade
invents (')', '(', '->', OVER, ','...), which the iterate side's own
descent does not mirror, so the veto stood and 16 states per precedence
class escaped.
Track the birth of every exit-lineage (alt-2) context frame by 64-bit
structural (value) key: frames born from a boundary-chased or
widened/phantom-stacked lineage go to phantomFrameKeys, frames born
clean to realFrameKeys; the obligation checks (hasObligatedReturn,
hasPendingPrecReturn) skip a frame only when its value was born
exclusively by wildcard lineages. Alt-2-scoping keeps clean iterate
births of the same value from poisoning the classification, and the
real-keys guard preserves the genuine obligation (the Send/list shape:
an unchased in-rule list-element frame owing ',' or '>>' still vetoes).
Value keys (with per-slot synthesis for merged contexts) survive the
equal-value dedup that defeated identity-based tracking.
Coverage (7-dialect sweep): Trino d=318 fully static (32e -> 0, both
hybrid classes now k=2; escapes 878 -> 846), Bigquery d=370 (32e -> 0),
Redshift d=409 (41e -> 1e), Snowflake d=687 (31e -> 1e), Databricks
d=460 (191e -> 162e); Duckdb/ClickHouse unchanged. No decision
regressed anywhere.
Verification: cargo all suites, tool-testsuite 694/0, runtime-testsuite
rust.** 707/707; tree diffs Trino 1680/2 + paren 1200/0 + err 10/0 +
pred 318/0 (the 2 are the known '::' error-path class), Duckdb 2880/0,
Redshift 2880/0, Databricks 2880/0. The Trino 172 and Bigquery 230
'>>' mismatches and the Snowflake 2 (q1532/q1581 error-count) reproduce
identically with the pre-change tool - pre-existing divergences from
the d=238/d=254 closure era on inputs where '>>' lexes as two GT
tokens, flagged for separate investigation.
Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
…skip regression) e997256 made all remaining escapes of optional-postfix decisions resolve to guarded takes, driving their escape count to zero. The tableCanDefer gate (339d095: emit the adaptivePredict fallback only where it can fire) only consulted hasEscapes()/hasMasks(), so the generated Rust parser lost its adaptive fallback for these all-guarded decisions. But a guarded-take state still defers at runtime: when the real-stack epsilon-pop chase trips a guardDanger state, dfa_predict returns INVALID_ALT - which, with no fallback arm, silently fell into the skip arm of the generated match. Symptom (Snowflake): functionCallTail d=691 guard-defer on 'f(x) OVER (PARTITION BY k)' was misread as skip, OVER got consumed as the select-item alias (over_alias=OVER), and the window spec then failed at top level - static/adaptive PARSE-DIVERGENCE on the differential corpus. StaticDFA.hasGuards() already documents exactly this deferral; it just was never consulted. The fallback wrapper is now emitted for any table with guarded states. tableCanDefer is consumed only by Rust.stg, so no other target is affected. Verification: Snowflake differential corpus (2893 files, static vs adaptive) 0 divergences; DFA-TRACE confirms guard-defer d=691 now routes to adaptivePredict. Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
8ff68e9's uniform-widening trust resolved Snowflake property d=387's start-state conflict to alt 1, collapsing the table to a single k=0 accept. A depth-0 accept answers the decision without consuming a single token, but the runtime never reports a conflict before consuming input either - alt-viability pruning comes first, so min(U) is the runtime's answer only for inputs that actually reach the conflict. The depth-0 collision is a widening artifact of the shared closure: both alts of property: propertyKey EQ LPAREN property* RPAREN #nestedProperty | propertyKey EQ propertyValue #defaultProperty meet on (state, $) inside the common propertyKey/identifier prefix before any input disambiguates them, and utrust's per-entry assured-min attestations (ctx=$ shadows, faithful under the identity-tracked joints) covered every entry. Result: HANDLER = 'MyClass.myMethod' in CREATE FUNCTION ... LANGUAGE JAVA statically entered nestedProperty and failed expecting '(', where adaptive picks defaultProperty - a static/adaptive PARSE-DIVERGENCE. Depth >= 1 resolutions are unaffected: there the walker consumes input before accepting, so only inputs that genuinely reach the conflict get the min-alt answer (statement d=45/d=90 keep their utrust tables). Start-state conflicts now fall through to the escape/adaptive path, the pre-8ff68e904 behavior; with the doomed-attempt pruning of 280597b d=387's exact attempt completes outright (LLK, k=4). Verification: Snowflake differential corpus 2894 files, 0 divergences (incl. the cov3 CREATE FUNCTION HANDLER witness); Rust unit tests; tool-testsuite 694/694; runtime-testsuite rust.** 707/707. Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
…flowing 40b145d's v6 table format funnels every walker result through u64 alternative masks (1<<(alt-1) per alt), but nothing stopped the tool from emitting tables for decisions with more than 64 alternatives. Redshift's 80-alt statement decision statically accepted alt 68 for 'CREATE LIBRARY foo' and the walker's accept shift overflowed (panic in debug builds; a silently wrong bit in release). Mask accepts were worse: altBits() computes 1L<<(a-1) in Java, which wraps mod 64, so an over-64 group would have produced a silently corrupted mask with no panic at all. Same hazard in the descent-codegen adaptive-fallback arms (wrapping_shl of an adaptive alt). Rather than demoting such decisions to fully adaptive (which cost Snowflake statement d=310 its 97%-coverage hybrid table), the fix is per-state: over-64 accepts become escapes in hybrid mode, deferring exactly those inputs to adaptivePredict; in a full-table attempt they fail the construction, which then retries in hybrid mode. Mask acceptance (prefix-factor and shared-descent) is skipped for groups reaching past alt 64, and descent codegen falls back to the plain DFAAltBlock for >64-alt decisions. The Rust walker also defers on over-64 accepts/fallbacks as stale-table defense (returning 0 instead of panicking). Verified: Redshift differential corpus 2894 files 0 divergences ('CREATE LIBRARY foo' parses; statement d=114 keeps a hybrid table, cover=84%); Snowflake d=310 stays hybrid (escapes 19->30); Rust unit tests; tool-testsuite 694/694; runtime-testsuite rust.** 707/707. Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
…ischarge obligations 37c57a4's birth predicate marked every boundary-chased or wildcard-context re-descent frame phantom and discharged its token obligations. The premise - phantom frames hypothesize a juxtaposed fresh descent no real stack offers - does not hold for frames born after the lineage consumed a separator: those model real continuation shapes, and cov4 found one. Snowflake primaryExpression's base alternatives include the self-referential PRIOR base_=primaryExpression DOT derefName=columnNameComponent For 'CONNECT BY PRIOR B.ID = B.PARENT_ID' the wildcard cascade re-descends into primaryExpression at that alternative after consuming PRIOR, hypothesizing a frame that owes the alternative's DOT after the completed base operand. The real stack of this input contains exactly that frame: discharging its obligation let the iterate/exit substitution resolve ITERATE after 'PRIOR B', so the inner primary swallowed '.ID' and the outer alternative's DOT then failed at '=' (mismatched input '=' expecting {'::', '[', '.', ':'}). Adaptive exits the inner loop. The new CONSUMED_TAINT bit (32, max-merge-safe) marks any token consumption by a boundary-popped lineage, loop-disciplined or foreign. Only separator-free (truly juxtaposed) births remain phantom. In practice the discharged obligations were all born past a separator, so the escape wins of 37c57a4 revert (Snowflake d=687 1e->31e, Trino d=318 0->32e, Bigquery d=370 0->32e, Redshift d=409 1e->41e) - the premise behind them was unsound; what remains is the pre-37c57a405 behavior for those decisions, with the discharge machinery kept for genuinely juxtaposed births. Verification: Snowflake differential corpus 2895 files 0 divergences (cov4 and the minimized 'SELECT * FROM t AS B CONNECT BY PRIOR B.ID = B.PARENT_ID' included); Rust unit tests; tool-testsuite 694/694; runtime-testsuite rust.** 707/707. Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
…bligations 37c57a4 discharged the exit lineage's pending-token obligations of phantom-born frames outright. The premise is unsound in both directions, and 399ed90's separator test did not save it: the caller shape that makes the exit uniquely viable (Snowflake's self-referential base alternative 'PRIOR base_=primaryExpression DOT derefName', hit by 'CONNECT BY PRIOR B.ID = ...') may never appear in the conflict's context set at all - the phantom cascade's re-descent does not have to reach it for a real stack to sit on it. No static analysis of the conflict can decide this; only the runtime stack can. The discharge is now a stack guard. Iterate/exit states vetoed only by phantom-born obligations (conflict substitutions and budget frontier alike) accept as GUARDED iterate; the guard's roots are every pending position of the precedence rule's BASE alternatives - the operator alternatives' own pending positions are owed by the iterate reading too and are excluded (partitioned by local control-flow reachability from the precpred heads, cut at the LoopEndState: the transformed rule wraps from there back into the base block). At runtime the guard chase defers to adaptivePredict exactly when an invoking frame on the real stack returns into such a position - 'PRIOR B' (base_ returning into the alternative's pending DOT) defers, while 'f(B.ID)' (an argument returning into a different rule's separators) and standalone 'B.ID' (enclosing loop invocation, not pending) take the static iterate. The base_ operand frames of the loop's own dereference alternative are operator positions, so chained dereferences stay static as well. Real-born obligations still veto outright (the Send/list witness is preserved), and both useless-guard eliminations apply (empty danger reverts to a plain iterate accept; a root reachable from the loop exit reverts to escape). Escape wins of 37c57a4 are restored, now guarded: Snowflake d=687 31e -> 1e (30 obligation-guarded), Trino d=318 32e -> 0 (32), Bigquery d=370 32e -> 0 (32), Redshift d=409 41e -> 1e (40), Databricks 29. Verification: Snowflake differential corpus 2895 files 0 divergences (cov4 defers correctly, DFA-TRACE shows guard-defer on the PRIOR stack and guard-take on the standalone dereference in the same query); Rust unit tests; tool-testsuite 694/694; runtime-testsuite rust.** 707/707. Signed-off-by: Bo Lin <bo.lin@dbtlabs.com>
sadboy
force-pushed
the
linb/v2.0
branch
2 times, most recently
from
July 31, 2026 20:23
e58f8ad to
972a68e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.