Extend XQTS runner: QT4/FTTS/Update suites, assertion fixes, batch runner, Saxon 12#49
Open
joewiz wants to merge 33 commits intoeXist-db:feature/qt4-xquery-updatefrom
Open
Extend XQTS runner: QT4/FTTS/Update suites, assertion fixes, batch runner, Saxon 12#49joewiz wants to merge 33 commits intoeXist-db:feature/qt4-xquery-updatefrom
joewiz wants to merge 33 commits intoeXist-db:feature/qt4-xquery-updatefrom
Conversation
Add qt4cg/qt4tests as a downloadable test suite (--xqts-version QT4). Parse and execute multi-step XQuery Update test cases with mutable in-memory documents. Add XP40/XQ40 spec values and XQUpdate feature. Handle revalidation and put dependency types. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add W3C XQFTTS as a downloadable test suite (--xqts-version FTTS). Handle Fragment and Inspect comparison types. Support stop-word and thesaurus URI maps with thread-safe registration. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add exist-expath dependency and register the ExpathFileModule (http://expath.org/ns/file) in the embedded server's conf.xml. This resolves all 190 expath-file test failures caused by XPST0017 "Call to undeclared function: file:exists". Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Allows running sbt with -Dmaven.repo.local=/path/to/repo to use a session-local Maven repository, avoiding conflicts between concurrent sessions that install different exist-core snapshots. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Move jetty-jakarta-servlet-api exclusion to global excludeDependencies so it covers transitive paths through exist-expath (fixes dedup error) - Disable prepended shell script in CI builds (corrupts ZIP offsets, causing "An unexpected error" from the Java launcher) - Use explicit java -jar in CI test step Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Parse the <sandpit> element from test environments, copy the sandpit source directory to a per-test-case temp directory before execution, set the static base URI to the temp directory so relative file paths resolve correctly, and clean up after execution. This enables the EXPath File test set (190 tests) and upd-fn-put test set (17 tests) which require a writable working directory. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add whitespace-only text node filter in XMLUnit diff to avoid spurious mismatches from insignificant whitespace differences - Capture update expression return values for copy-modify-return tests that have no separate verification query Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Change getBroker() to authenticate("admin", "") to get an admin-level
broker instead of a guest broker. Required for modules that need
filesystem or privileged access (e.g., EXPath File module operations).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The XQTS assert expressions like ?columns and ?get(1,1) use unary lookup which requires a context item. Previously, the result was only available as $result variable but not as the context item, causing XPDY0002 errors for any assertion using the ? lookup operator. Only set context for single-item results (e.g., maps from parse-csv) to avoid per-item evaluation for multi-item sequences like csv-to-arrays. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…dard namespaces
The runner was extracting only the local part of XPathException error
codes, losing the namespace URI. QT4 test catalogs use EQName notation
(e.g., Q{http://expath.org/ns/file}is-dir) for extension module error
codes. Now error codes from non-standard namespaces (not xqt-errors or
exist-xqt-errors) are formatted as Q{ns}local for proper matching.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…tion When a query returns an error and the expected result is an AllOf assertion containing an Error assertion, match the error code against the Error inside the AllOf. Previously, only direct Error and AnyOf assertions were matched; AllOf was not handled, causing false failures for tests like EXPath File read-binary bounds checking where the catalog wraps the expected error in <all-of>. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…rors When an XPath assertion (assert, assert-eq, assert-deep-eq, assert-permutation, assert-serialization) raises a query error during evaluation (e.g., XPTY0004 type mismatch), this indicates the assertion failed — not that the runner itself errored. Previously these were reported as ErrorResults, inflating the error count and masking the real nature of the failure. Now they are reported as FailureResults with the error details in the message. Fixes fn-parse-json-717 and fn-parse-json-731 which errored due to type mismatches in assertion XPath evaluation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…tServer Add sequenceToStringRaw() for serialization-matches assertions where the exact serialized output must be preserved without newline replacement. Refactor sequenceToString into a shared implementation with a sanitize flag. Add serializationProperties field to Result to capture query context serialization options (e.g., declare option output:method "json") for use in assertion evaluation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The sandpit implementation (fab30c1) sets the static base URI for test query execution, but assertion evaluation ran in a separate XQuery context that defaulted to the JVM working directory. This caused 21 EXPath File tests to fail because assertion expressions like Q{http://expath.org/ns/file}read-text("test.txt") resolved relative paths against the wrong directory. Thread the test case's static base URI through processAssertion → all assertion methods → executeQueryWith$Result → connection.executeQuery, so assertion expressions see the same base URI as the test query. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fix isTestSetCompleted() to compare completed test cases against started tests rather than the full catalog (which includes test cases filtered by spec/feature requirements). Add fallback path for when ParsedTestSet is stuck in the Pekko mailbox. Prevent duplicate serialization. Reduce stall timeout from 600s to 60s with hung test reporting. Add 30-second shutdown deadline for actor system termination. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add run-batched.sh that splits test sets into batches, runs each in a fresh JVM, and aggregates results. Includes per-test-set timing report, resume mode, and configurable batch size/heap. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add --parallel N flag to run-batched.sh that distributes batches across N concurrent streams. Each stream runs in its own JVM with an isolated eXist-db home directory (via -Dexist.home) to avoid BrokerPool data directory lock conflicts. Batches are distributed round-robin across streams. Results accumulate in the same output directory (JUnit XML files have unique names per test set, so no conflicts). QT4 full run: 5m37s with --parallel 3 (was ~9m sequential). FTTS verified: identical results between sequential and parallel modes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…atch The set +e / set -e toggle inside run_batch() caused backgrounded subshells to exit after the first batch failure. When set -e is re-enabled inside a function running in a backgrounded subshell, any subsequent command failure terminates the entire stream. Fix: replace set +e / set -e with || true pattern to capture the timeout exit code without toggling errexit. This allows each stream to continue processing all its batches regardless of individual batch outcomes. Before: --parallel 3 produced 150 results (1 batch per stream). After: --parallel 3 produces 630 results (all 13 batches). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…pressions
The W3C XQTS 3.1 test suite uses the deprecated map{key:=value} syntax
in some assertion expressions and test queries. Since eXist-db no longer
supports := in maps, these fail with XPST0003.
Uses a heuristic: replace := preceded by a non-whitespace char (map
entries like "key":=value) but not variable bindings ($var := value
which have a space before :=).
…query expressions" This reverts commit a590bcb.
When a batch approaches its 300s timeout, automatically capture a thread dump of the Java process via jstack. The dump is saved to $OUTPUT_DIR/jstack-batch-N.txt, showing exactly which threads are stuck and what locks they're contending on. Tested with op-to (known OOM hanger): thread dump reveals Pekko dispatcher threads BLOCKED on java.lang.Shutdown monitor — OOM'd threads all trying to call System.exit() simultaneously, deadlocking on the JVM shutdown lock. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
OOM during test execution causes multiple Pekko dispatcher threads to call System.exit() simultaneously, deadlocking on java.lang.Shutdown's monitor. ExitOnOutOfMemoryError makes the JVM call _exit() immediately on OOM — no shutdown hooks, no deadlock. The batch runner's timeout + jstack handles diagnostics; the JVM just needs to die cleanly. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Support PARSER=rd or PARSER=antlr2 environment variable in run-batched.sh. Sets -Dexist.parser on the JVM command line. Defaults to antlr2 if not specified. Displays parser name in the run header for clear identification of results. Usage: PARSER=rd ./run-batched.sh --xqts-version QT4 --output-dir results/rd PARSER=antlr2 ./run-batched.sh --xqts-version QT4 --output-dir results/antlr2 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…tibility The runner doesn't use Jetty directly — it only needs exist-core for XQuery evaluation. But Jetty comes in as a transitive dependency, and Ivy can't resolve Jetty 12 Maven POM constructs (property references in dependencyManagement), producing broken pseudo-versions. Exclude all org.eclipse.jetty group IDs (core, toolchain, websocket, ee10) so the runner builds against both Jetty 11 (develop) and Jetty 12 (next) branches of eXist-db. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…lding Two bugs blocking serialization tests: 1. Not assertion parsing (9 tests): stepOutAssertions only handled Assertions (AllOf/AnyOf) on the stack, not Not. When </not> triggered stepOutAssertions with a Not(Some(...)) on top, it threw "Unable to associate non-assertions object". Also consolidated the duplicate ALL_OF end-element handler — ALL_OF, ANY_OF, and NOT are now handled in a single case. 2. serializationMatches query building (4 tests): The method embedded the expected regex in a backtick string constructor ``[...]``, but patterns containing <? (e.g., <?xml) trigger an eXist parser bug where <? is interpreted as a processing instruction start inside the string constructor (XPST0003). Fixed by passing the regex and flags as external variables. Note: The eXist parser bug with <? inside backtick string constructors should be investigated separately — ``[<?xml]`` should be valid XQuery. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bug 1: Not(None) inside AllOf/AnyOf — When <not> is a child of <all-of> or <any-of>, addAssertion() appended Not() to the Assertions list, then appended the inner assertion as a sibling instead of nesting it inside the Not. Fix: check if the last element of an Assertions container is Not(None) and fill it with the new assertion. Bug 2: Serialization options lost after context.reset() — The runner called the 3-arg XQuery.execute() which passes null for outputProperties. eXist's execute() calls context.checkOptions(null) (no-op) then context.reset(), clearing declared serialization options before the runner could read them. Fix: pass a Properties object to execute() so eXist extracts options BEFORE resetting the context. This restores declare option output:method "html" etc. for serialization tests. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add --timeout flag (default 180s, was hardcoded 300s) for faster recovery when BrokerPool shutdown hangs - Kill lingering Java processes after each batch via pkill -9 matching the batch's unique exist.home directory - Adjust jstack capture timing relative to the configurable timeout The FLWOR fix increased rd parser test execution by ~4,483 tests, overwhelming the batch runner with BrokerPool shutdown hangs on 13/13 batches. Shorter timeout + process cleanup allows the runner to recover and continue. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
180s was too short for rd parser batches — the rd parser legitimately takes longer on some test sets (FLWOR parsing complexity), causing 32/32 batches to timeout during test execution. Keep the configurable --timeout flag and process cleanup, just set the default higher. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Exit code 1 from the XQTS runner means "some test failures found" — normal expected behavior. Only count timeout (124/137) and crashes (exit > 1, not 255) as batch failures. Exit 255 is a runner error (non-fatal, produces partial results). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Update the runner's Saxon dependency to match eXist-db's Saxon 12 upgrade (eXist-db/exist#6143). The runner's own Saxon usage (AnyURIValue, TransformerFactoryImpl) is compatible with both versions — no source changes needed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2 tasks
joewiz
changed the title
This was referenced Apr 13, 2026
Resolve conflicts in favor of saxon-12-runner branch: - build.sbt: Saxon-HE 12.5 (not 9.9), broader Jetty exclusions for both Jetty 11 and 12 compatibility - ExistServer: pass outputProperties to execute() so serialization options are captured before context.reset() - TestCaseRunnerActor: use map-based serialization with all properties (not just method/indent); bind regex/flags variables in serializationMatches assertion - XQTSRunnerActor: 60s stall timeout (not 600s), hung test tracking, forceSerializeAndShutdown helper Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extends the XQTS runner to support three additional W3C/community test suites beyond the existing XQuery 3.1 suite, fixes assertion evaluation bugs that could mask real test failures, adds performance improvements that cut full QT4 run time from 37+ minutes to under 6 minutes, and upgrades Saxon-HE from 9.9 to 12.5 to match eXist-db's Saxon 12 upgrade.
Consolidates what was previously split across PR #45 and PR #49.
--xqts-version QT4)--xqts-version FTTS)expath-filetest setPerformance Improvements
isTestSetCompleted()compared against full catalog instead of started tests; Pekko dispatcher blocked by BrokerPool preventedParsedTestSetdeliverycontext.system.terminate()halt(0)--parallel 3exist.homedirectoriesCommits
1. Core test suite support (3 commits)
2. Build & infrastructure (2 commits)
3. Sandpit support (1 commit)
4. Assertion reliability fixes (9 commits)
5. Completion detection & shutdown (2 commits)
6. Batch runner (5 commits)
--parallel N)7. Additional fixes (4 commits)
8. Saxon upgrade (1 commit)
When to merge Saxon upgrade
The Saxon upgrade commit is the last commit in this branch. It can be cherry-picked or this PR can be merged after eXist-db/exist#6212 (Saxon 12 upgrade) lands on develop. The rest of the PR is independent of the Saxon version.
Usage
Test Plan
sbt compilepassessbt assemblybuilds without deduplication errorsnextintegration branch: 630 test sets, 35581/41611 (85.5%)[This PR was co-authored with Claude Code. -Joe]
🤖 Generated with Claude Code