Skip to content

fix(nevermore-cli): tag only the script template as the batch test entry - #782

Open
Quenty wants to merge 1 commit into
mainfrom
users/jamesonnen/batch-tags-only-script-template
Open

fix(nevermore-cli): tag only the script template as the batch test entry#782
Quenty wants to merge 1 commit into
mainfrom
users/jamesonnen/batch-tags-only-script-template

Conversation

@Quenty

@Quenty Quenty commented Jul 30, 2026

Copy link
Copy Markdown
Owner

batch test can run the wrong script and report it as a pass. A game in my org
has had zero test coverage in CI since the moment it added a second server
script — the batch ran that script instead of the test bootstrap, and CI's only
complaint was 0ms and no jest report in output.

Why

combine-test-places.luau tags every Script under ServerScriptService:

for _, descendant in ServerScriptService:GetDescendants() do
	if descendant:IsA("Script") then
		descendant:AddTag("_BatchTest_" .. packages[1].slug)
	end
end

and batch-test-runner.luau loadstrings CollectionService:GetTagged(...)[1].
With one script in the project those are the same script. With two, the runner
takes whichever the tag lists first — and since a script that returns without
erroring looks exactly like a passing test bootstrap, the marker says PASS and
the only symptom is an empty section.

Meanwhile scriptPath — the target's scriptTemplate, the thing the
single-package path actually executes — is passed all the way down to the lune
script and then ignored. The header comment says it becomes a Script under
ReplicatedStorage._BatchScripts; the code never did that.

The change

  • combine-test-places.luau injects the scriptTemplate file as a Script under
    ReplicatedStorage._BatchScripts and tags only that, for every package
    rather than only the first. It exits with a clear message if the file is
    missing, instead of silently producing a place with nothing to run. A Script
    under ReplicatedStorage is inert, so it is a source carrier and nothing else.
  • batch-test-runner.luau refuses a tag that resolves to more than one instance
    and names them, rather than picking one. That is the state that used to be
    silent.

This also makes batch agree with the single-package path about what "the test
script" is, rather than agreeing by coincidence of project shape.

Verification

Against a real game whose src/scripts/Server holds two scripts, using a local
build of the CLI:

result
published 4.42.0 FAILED (0ms) — 7 log messages, none of them jest
this branch Passed (645/707) in 1m21s

The 645/707 matches what that repo's CI reported the day before it added the
second script, so the fix restores exactly the suite that went missing.

npm test in tools/nevermore-cli: 121 passed. No unit test added — the
regression lives in a lune script that needs a real .rbxl to exercise, and the
two guards this adds (a hard exit at combine time, a refusal at run time) are
what would have caught it.

https://claude.ai/code/session_01Ei9RRAgYwL3MGtMkr6vyKE

combine-test-places tagged every Script under ServerScriptService with
`_BatchTest_<slug>`, and batch-test-runner executes `GetTagged(...)[1]`. With one
script in a project those are the same script; with two, the runner loadstrings
whichever the tag listed first and the package's real test bootstrap never runs.
Nothing catches it downstream either -- a script that returns without erroring
reports as a pass, so the batch reports 0ms and no jest output for a suite that
was never asked to run.

The `scriptPath` each triple already carries -- the target's `scriptTemplate`,
and what the single-package path executes -- is now injected as a Script under
ReplicatedStorage._BatchScripts and is the only thing tagged, which is what the
file's own header said it did. The runner refuses an ambiguous tag rather than
picking from it, so a project that reintroduces one gets told.

Verified against a game with two server scripts: FAILED (0ms, no jest report)
before, 645/707 passed after.

Claude-Session: https://claude.ai/code/session_01Ei9RRAgYwL3MGtMkr6vyKE
@Quenty
Quenty deployed to integration July 30, 2026 07:58 — with GitHub Actions Active
@github-actions

github-actions Bot commented Jul 30, 2026

Copy link
Copy Markdown

Deploy Results

ℹ️ No changed packages with deploy targets were discovered for this PR. · View logs

Test Results

ℹ️ No changed packages with test targets were discovered for this PR. · View logs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant