Skip to content

fix(rollout): clear stale LLM trajectory on rollout reuse - #1094

Open
zhiheng-yang wants to merge 1 commit into
benchflow-ai:mainfrom
zhiheng-yang:fix/reused-rollout-llm-artifact
Open

fix(rollout): clear stale LLM trajectory on rollout reuse#1094
zhiheng-yang wants to merge 1 commit into
benchflow-ai:mainfrom
zhiheng-yang:fix/reused-rollout-llm-artifact

Conversation

@zhiheng-yang

Copy link
Copy Markdown

Problem

_init_rollout() can be called with explicit job and rollout names, so the target directory may already exist.
In that case, an old trajectory/llm_trajectory.jsonl is left in place.

Usually live capture replaces the file after the first completed model call.
If the new run fails before then, nothing overwrites it.
Training export and artifact inspection can then read the previous run's trajectory as if it came from the current attempt.

I reproduced this by initializing the same rollout path twice and placing a trajectory in between.
After the second initialization:

  • main: stale_exists=True
  • this patch: stale_exists=False

Change

Remove this one attempt-owned file when the rollout directory is initialized.
The rest of the directory is left alone, and the separate continue-run flow is unchanged.

  • src/benchflow/rollout/_setup.py removes the stale trajectory.
  • tests/test_sdk_internals.py covers reuse of the same rollout path.

Tests

uv run pytest -q tests/test_sdk_internals.py

Result: 51 passed.

@zhiheng-yang
zhiheng-yang marked this pull request as ready for review September 3, 2026 23:06

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Devin Review: 1 flag

Not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

Devin Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant