Skip to content

Fix flaky Weber_BMC2015 gradient check #3226

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
FFroehlich wants to merge 2 commits into
base: main
Choose a base branch
from
+23 −2
Open

Fix flaky Weber_BMC2015 gradient check #3226

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7fb286f
Fix flaky Weber_BMC2015 gradient check
FFroehlich Aug 7, 2026
f919dda
Merge branch 'main' into fix/weber-gradient-check-flaky
FFroehlich Aug 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 23 additions & 2 deletions tests/benchmark_models/test_petab_benchmark.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -173,6 +173,15 @@ class GradientCheckSettings:
1e-5,
]
)
# Finite difference schemes to use. All of them have to agree within
# (atol|rtol)_consistency for a step size to be considered.
Comment on lines +176 to +177
method_ids: list[MethodId] = field(
default_factory=lambda: [
MethodId.CENTRAL,
MethodId.FORWARD,
MethodId.BACKWARD,
]
)
rng_seed: int = 0
ss_computation_mode: SteadyStateComputationMode = (
SteadyStateComputationMode.integrationOnly
Expand Down Expand Up @@ -237,6 +246,18 @@ class GradientCheckSettings:
atol_check=1e-6,
rtol_check=1e-2,
rng_seed=2,
# No single step size works for all directions with the default
# forward/backward/central agreement requirement: the noise parameters
# (e.g. std_yPKDpN25) are so strongly curved that forward and backward
# differences only agree with the central one for steps <= 5e-5, while at
# those steps the finite differences of parameters with a small nominal
# value (e.g. a32 ~ 1e-4) are dominated by integrator noise. Since
# `Consistency` averages over all step sizes it accepts, the noisy small
# steps then bias the result by ~1%, right at `rtol_check` -- which is what
# made this test flip-flop between runs. Use central differences only, over
# the step size window in which every direction is converged.
method_ids=[MethodId.CENTRAL],
step_sizes=[5e-3, 2e-3, 1e-3, 5e-4],
)
settings["Zheng_PNAS2012"] = GradientCheckSettings(
rng_seed=1,
Expand Down Expand Up @@ -484,7 +505,7 @@ def test_benchmark_gradient(
point=point,
sizes=cur_settings.step_sizes,
direction_ids=parameter_ids,
method_ids=[MethodId.CENTRAL, MethodId.FORWARD, MethodId.BACKWARD],
method_ids=cur_settings.method_ids,
success_checker=Consistency(
rtol=cur_settings.rtol_consistency,
atol=cur_settings.atol_consistency,
Expand Down Expand Up @@ -810,7 +831,7 @@ def test_nominal_parameters_llh_v2(problem_id):
point=point,
sizes=cur_settings.step_sizes,
direction_ids=parameter_ids,
method_ids=[MethodId.CENTRAL, MethodId.FORWARD, MethodId.BACKWARD],
method_ids=cur_settings.method_ids,
success_checker=Consistency(
rtol=cur_settings.rtol_consistency,
atol=cur_settings.atol_consistency,
Expand Down
Loading