fix(rollout): route verifier timeout cleanup by service

[zhiheng-yang]

Problem

[verifier].service controls which Compose service runs test.sh.
The main verifier path respects that setting, but the timeout recovery path did not.

For a verifier configured on scorer, the flow was effectively:

• run test.sh on scorer;
• inspect test-stdout.txt on the default service;
• run the orphan pkill on the default service.

Those services do not share the same processes or necessarily the same files, so the recovery code could inspect the wrong output and leave the timed-out verifier running.

I reproduced the timeout path with verifier.service = "scorer".
The two recovery calls were routed as follows:

• main: [None, None]
• this patch: ['scorer', 'scorer']

Change

Pass the configured verifier service to both the silent-timeout probe and the orphan cleanup helper.
The timeout and retry policy are unchanged.

• src/benchflow/rollout/_setup.py keeps recovery on the verifier service.
• tests/test_verifier_wedge_retry.py covers both main and scorer, plus the timeout-with-output path.

Tests

uv run pytest -q tests/test_verifier_wedge_retry.py

Result: 4 passed.

[devin-ai-integration]

Devin Review found 1 potential issue.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

[devin-ai-integration]

🟡 Timed-out verifier retries can overlap

When a legacy or custom verifier silently times out, _kill_orphan_verifier misses its process. The original can run beside the retry and corrupt its score.

Prompt for agents

The silent-timeout retry in src/benchflow/rollout/_setup.py calls _kill_orphan_verifier before starting the second attempt, but that helper only runs pkill against /verifier/test.sh. Verifier.verify supports legacy /tests/test.sh, verifier.md script strategies with arbitrary commands, and reward-kit runners, so those timed-out processes survive and can race the retry's output files. Make timeout cleanup target the actual command or process group started by Verifier, while preserving service routing and avoiding unrelated processes. Add coverage using a real legacy or custom verifier command rather than only a mocked verify coroutine.

---

Was this helpful? React with 👍 or 👎 to provide feedback.