Skip to content

feat: migrate sensitivity_report.py to trade_study.sensitivity_from_table() - #119

Merged
jc-macdonald merged 1 commit into
mainfrom
feat/sensitivity-report-use-sensitivity-from-table
Aug 21, 2026
Merged

feat: migrate sensitivity_report.py to trade_study.sensitivity_from_table()#119
jc-macdonald merged 1 commit into
mainfrom
feat/sensitivity-report-use-sensitivity-from-table

Conversation

@jc-macdonald

Copy link
Copy Markdown
Collaborator

Summary

Replaces the hand-rolled marginal Spearman correlation in sensitivity_report.py with trade_study.sensitivity_from_table() (jcm-sci/trade-study#113, built for exactly this) for the 9 continuous VBPCA hyperparameters, per p-bucket.

Why this matters

Unlike Spearman, Sobol-via-surrogate correctly detects non-monotonic (e.g. U-shaped) effects -- exactly the kind defaults.py's own docstring already documents for hp_va: "its tercile means do show a real, nonlinear U-shape a monotonic Spearman correlation understates." That gap was known but this script kept using the method that produces it.

sensitivity_from_table only screens continuous factors (matching screen()'s own contract), so the 4 discrete VBPCA hyperparameters (niter_broadprior/maxiters/patience/rmsstop_window) keep the original marginal-Spearman treatment -- now explicitly labeled "spearman_marginal" in the output rather than mixed in unlabeled alongside the Sobol results.

Verification

Ran against the real, current surrogate_train.json (not synthetic data):

- smallp: top rank_mae factors = xprobe_fraction (sobol_via_surrogate=0.46), niter_broadprior (spearman_marginal=0.17), hp_va (sobol_via_surrogate=0.16)
- trans: top rank_mae factors = xprobe_fraction (sobol_via_surrogate=0.77), hp_v (sobol_via_surrogate=0.06), niter_broadprior (spearman_marginal=0.05)
- large: top rank_mae factors = xprobe_fraction (sobol_via_surrogate=0.75), niter_broadprior (spearman_marginal=0.22), maxiters (spearman_marginal=0.08)

xprobe_fraction dominates in all three buckets via this completely different method -- corroborating the same finding the old marginal-Spearman analysis already reported in defaults.py's docstring ("xprobe_fraction is the strongest, most significant, and most consistent per-bucket predictor of rank_mae"), not hp_va despite it being called the "dominant lever" there. Figure regenerated cleanly (figure_sensitivity_rank_mae.png etc.) with sane-looking bars.

Test plan

  • just ci passes
  • Ran end-to-end against real local surrogate_train.json, inspected the regenerated figures and JSON output for sanity

🤖 Generated with Claude Code

Replaces the hand-rolled marginal Spearman correlation with
trade_study.sensitivity_from_table() (jcm-sci/trade-study#113) for the
9 continuous VBPCA hyperparameters, per p-bucket. Unlike Spearman, this
correctly detects non-monotonic (e.g. U-shaped) effects -- exactly the
kind defaults.py's own docstring already documents for hp_va (a real
nonlinear U-shape the marginal-correlation method this migration
retires understates). surrogate_cv_r2 is reported alongside each
bucket's indices so a poorly-fit surrogate's numbers aren't mistaken
for a reliable finding.

Sobol/Morris both require continuous factors (sensitivity_from_table
silently drops anything else, matching screen()'s contract), so the 4
discrete VBPCA hyperparameters (niter_broadprior/maxiters/patience/
rmsstop_window) keep the original marginal-Spearman treatment,
explicitly labeled "spearman_marginal" in the output rather than mixed
in unlabeled with the Sobol results.

Verified against the real, current surrogate_train.json: xprobe_fraction
dominates rank_mae sensitivity in all three buckets (S1=0.46-0.77),
corroborating -- via a completely different method -- the same
marginal-Spearman finding defaults.py's docstring already documents
("xprobe_fraction is the strongest, most significant, and most
consistent per-bucket predictor"), not hp_va despite it being called
the "dominant lever" there.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@jc-macdonald
jc-macdonald merged commit 86fb063 into main Aug 21, 2026
7 checks passed
@jc-macdonald
jc-macdonald deleted the feat/sensitivity-report-use-sensitivity-from-table branch August 21, 2026 12:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant