Skip to content

feat(wandb): WandB plot quality overhaul — deep HPO and XAI insights#36

Merged
Palamabron merged 5 commits into
mainfrom
feature/xai-wandb-report
Jun 14, 2026
Merged

feat(wandb): WandB plot quality overhaul — deep HPO and XAI insights#36
Palamabron merged 5 commits into
mainfrom
feature/xai-wandb-report

Conversation

@Palamabron

@Palamabron Palamabron commented Jun 14, 2026

Copy link
Copy Markdown
Owner

Summary

  • Fix broken charts: prediction histogram reduced from 50k+ rows to 50 bins; per-era line charts now use numeric era_index x-axis (fixes alphabetical string sort)
  • Add missing visualizations: drawdown curve, per-era correlation distribution histogram, bar charts for feature exposure, worst stability, and ensemble correlation matrix, line_series chart for feature importance over eras
  • Fix WandB string metric bug: feature_importance_model_type moved from wandb.log() (silently coerced to NaN) to wandb.run.summary
  • Expand HPO summary table: 18 → 30 columns — adds model_1/2/3_type (split for WandB parallel coordinates), XGBoost/LightGBM hyperparams, feature selection, noise injection, augmentation flags; enables answering "which hyperparams drive performance?"
  • Add convergence chart: log_hpo_best_so_far() logs trial and running-best corr_sharpe per step to a search-convergence run, showing whether the HPO search is improving over time
  • Remove duplicate metric: metric/corr_sharpe deduplicated in log_hpo_trial_metrics

Jakub Szulc and others added 2 commits June 14, 2026 13:54
- shap_report: extend to LightGBM (gain), CatBoost, sklearn trees (RF/ET);
  compute_universal_feature_importance dispatches across all model types,
  normalizes and averages; log_universal_feature_importance replaces the
  XGBoost-only log_xgboost_shap_importance (removed)
- wandb_diagnostics: wire feature_report.py per-era stability report to WandB
  (_log_feature_report: top by mean, top/worst by stability); add
  _log_era_stratified_importance from actual trained pipeline models;
  log_experiment_diagnostics gains log_feature_report and log_era_importance flags
- wandb_utils: add log_importance_artifact (CSV Artifact for feature importance)
- hpo_pipeline: add _run_best_trial_diagnostics — after HPO completes, retrain
  best config on 80% split, log full XAI diagnostics + Artifact to dedicated
  'best-trial-diagnostics' WandB run in the same group
- tests/test_xai.py: 11 tests covering XGBoost, LightGBM, CatBoost, RF, Ridge
  (empty), mixed pipelines, top_n, sort order, key filtering

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix prediction histogram: use np.histogram(bins=50) instead of per-row
  Table (50k+ rows → 50 rows)
- Fix per-era line charts: add era_index numeric x-axis to prevent
  alphabetical string sort; add drawdown curve from peak cumulative corr
- Add per-era correlation distribution histogram (shows negative era shape)
- Add bar charts to: feature_exposure_top, feature_worst_stability,
  ensemble correlation matrix (A→B pair format)
- Replace wide era×feature heatmap Table with wandb.plot.line_series
  (one line per feature over eras — shows temporal stability visually)
- Fix feature_importance_model_type: move from wandb.log() (coerced to
  NaN) to wandb.run.summary
- Expand HPO summary table from 18 → 30 columns: adds model_1/2/3_type
  (split for parallel coordinates), XGBoost/LightGBM hyperparams,
  feature selection, noise injection, augmentation flags
- Add log_hpo_best_so_far(): logs trial and running-best corr_sharpe per
  step to a 'search-convergence' run; wired in _run_local after each trial
- Remove duplicate metric/corr_sharpe from log_hpo_trial_metrics

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Palamabron

Copy link
Copy Markdown
Owner Author

Code review

Found 1 issue:

  1. log_hpo_best_so_far calls wandb.init(reinit=True) + wandb.finish() on every trial, creating N independent 1-point WandB runs instead of one multi-step convergence run. The docstring says "Opens or reuses a 'search-convergence' run" but reinit=True always finishes the previous run and starts a fresh one — there is no resume. The convergence chart will never render as a single curve; WandB will show N separate single-datapoint runs in the group instead.

To fix, persist the run ID across calls (e.g. via wandb.init(id=run_id, resume="allow") with the ID stored in a file or passed as a parameter) or collect all points and log them in a single wandb.init block at the end of the HPO search.

https://github.com/Palamabron/AlphaPulse/blob/60f8dd473570c84c6a95f49760e00bd8b9ea85a6/src/alphapulse/logging_/wandb_utils.py#L299-L313

🤖 Generated with Claude Code

- If this code review was useful, please react with 👍. Otherwise, react with 👎.

Jakub Szulc and others added 3 commits June 14, 2026 14:24
…er-trial runs

log_hpo_best_so_far created a new WandB run per trial (reinit=True + finish()),
producing N independent 1-point runs rather than a single multi-step convergence
curve. Replace with log_hpo_convergence() which opens one run after all trials
complete and logs all steps sequentially — the same pattern as log_hpo_summary_table.
Remove the per-trial call from _run_local and add a post-loop batch call.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…overhaul

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously init_wandb_run received only {config_path, seed, gpu}.
Now _build_wandb_config extracts the full ExperimentV1 schema:

- model_types, model_{i}_type + hyperparams (flat + nested params)
- is_multihead flag; model_{i}_input_group for MultiHeadPipeline heads
- model_{i}_local_preprocessor_{j}_type + params per head
- preprocessor_types, preprocessor_{i}_type + params (global)
- feature_group_{name}_n_features for each defined group
- ensemble_method, n_rounds, neutralization_proportion, primary_metric

Also adds architecture diagram (docs/assets/architecture.drawio.png)
and updates README with diagram, layer table, directory structure,
preprocessor list (Autoencoder, Compression, EraStable), and roadmap.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Palamabron

Copy link
Copy Markdown
Owner Author

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

Reviewed the 10 changed files (1,453 additions, 92 deletions):

  • shap_report.py_collect_by_type correctly unifies MultiHeadPipeline and flat Pipeline traversal; XGB/LGBM/CatBoost/sklearn importance extractors are guarded with total > 0 checks; _aggregate_importance correctly averages across model types.
  • wandb_diagnostics.py — Era-index fix for chronological ordering is correct; drawdown computation (np.maximum.accumulate) is correct; prediction histogram binning avoids logging 50k rows; _log_era_stratified_importance properly bounds runtime with max_eras=30.
  • wandb_utils.pylog_hpo_summary_table column and data counts both increase by 12 (consistent); or-fallback for model_1_* keys is safe since learning rates are always positive; top_level_keys deduplication in log_hpo_trial_metrics correctly removes the metric/corr_sharpe duplicate.
  • hpo_pipeline.py_run_best_trial_diagnostics correctly uses feature_cols to select X_feat; load_train_only_frame is imported at the module level (line 26); 80/20 era split logic is sound.
  • run_experiment.py_build_wandb_config flat-key expansion is correct; is_multihead flag, per-model routing fields, and feature-group sizes are all properly extracted.
  • tests/test_xai.py — Good coverage across XGBoost, LightGBM, CatBoost, RandomForest, Ridge (empty), and mixed pipelines.

🤖 Generated with Claude Code

- If this code review was useful, please react with 👍. Otherwise, react with 👎.

@Palamabron Palamabron merged commit 9cb8f0f into main Jun 14, 2026
2 checks passed
@Palamabron Palamabron deleted the feature/xai-wandb-report branch June 14, 2026 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant