fix(ci): enable ft-long fault-tolerance soak tests

[guapisolo]

Summary

Enable all three ft-long H200 soak tests and stop GSM8K once its accuracy target passes.

Symptom & Reproduction

• Symptom: The run-ci-ft-long path could not complete its three intended H200 soak tests: registrations were disabled, then enabled runs exceeded stale time budgets.
• Reproduction: PR run 29293029757 reproduces the runtime timeouts after enabling the three previously disabled registrations.

PYTHONPATH=. python tests/ci/run_suite.py --hw cuda --suite stage-c-8-gpu-h200 --list-only --labels run-ci-ft-long

Root Cause

1. register_cuda_ci stored a disabled reason for each ft-long registration.
2. filter_tests skipped every matching registration while disabled remained set.
3. est_time values predated the enabled H200 runs and produced 3000s and 11250s timeouts.
4. MetricChecker.on_eval recorded target success without returning it to the training loop.

Fix

Remove the temporary registration gates and calibrate all three est_time values from observed H200 runtimes. Add the opt-in --ci-metric-checker-stop-on-success path so MetricChecker.on_eval reports cumulative success through RolloutManager.eval to train.py and train_async.py; the realistic GSM8K soak enables it while retaining the 0.55 threshold and 250-rollout upper bound.

Verification

• Metric success test: test_on_eval_reports_cumulative_success covers cumulative success across evaluations.
• Metric failure test: test_dispose_rejects_runs_without_a_successful_eval covers rejection when no evaluation reaches the target.
• Rollout manager test: TestEval.test_reports_stop_after_metric_success covers the stop signal returned after a successful evaluation.
• Argument test: test_ci_metric_checker_stop_on_success_flag_is_parsed covers the new opt-in CLI flag.
• H200 soak run: 29355800844 passes all three enabled ft-long tests.
• Early-stop signal: The GSM8K soak stops at rollout 139 after eval/gsm8k=0.561789 exceeds 0.55.

Review Focus

• Scrutinize the stop signal across MetricChecker.on_eval, RolloutManager.eval, train.py, and train_async.py.
• Scrutinize the opt-in boundary from miles_validate_args to scenario_realistic_gsm8k.py.
• Scrutinize the three est_time calibrations relative to observed H200 elapsed times under the runner timeout multiplier.

[gemini-code-assist]

Code Review

This pull request re-enables several fault-tolerance (FT) soak tests by removing the disabled parameter from the register_cuda_ci registration calls in test_trainer_ft_random_dp2_cp2_real_rollout.py, test_trainer_ft_random_dp2_cp2_tp2_ep2.py, and test_trainer_ft_realistic_gsm8k.py. I have no additional feedback to provide as there are no review comments.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.