diff --git a/.env-example b/.env-example index 7533942..fbc8ef4 100644 --- a/.env-example +++ b/.env-example @@ -3,6 +3,3 @@ NUMERAI_PRIVATE_API_KEY=... # Optional: for wandb logging (or use `wandb login`) WANDB_API_KEY=... - -# Optional: for TabPFN3ReasoningModel / TabPFN3Reasoning API (pip install 'alphapulse[foundation-api]') -TABPFN_API_KEY=... diff --git a/CHANGELOG.md b/CHANGELOG.md index 46b2887..ccc82a6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,21 +4,23 @@ All notable changes to AlphaPulse are documented here. --- -## [Unreleased] — WandB XAI & Plot Quality Overhaul +## [Unreleased] -- **Universal feature importance:** `compute_universal_feature_importance` extracts and normalizes importance from any supported model type (XGBoost pred_contribs, LightGBM gain, CatBoost PredictionValuesChange, sklearn `feature_importances_`), averages across all models present, and logs a ranked bar chart to WandB. -- **Era-stratified importance:** `_log_era_stratified_importance` slices validation data by era, computes importance per slice, and logs a `line_series` chart showing each feature's importance trajectory over eras — directly reveals temporal stability. -- **Per-era stability report wired:** `compute_feature_report` (LightGBM proxy) now surfaces in WandB via `_log_feature_report`; logs top features by mean importance, top by era stability, and worst by era stability — each with bar charts. -- **Best-trial diagnostics run:** After HPO, the best config is retrained on an 80/20 era split and all expensive diagnostics (`log_era_importance=True`, top-50 importance artifact) are logged to a dedicated `best-trial-diagnostics` WandB run. -- **Prediction histogram fixed:** `_log_prediction_diagnostics` now uses `np.histogram(bins=50)` (50 rows) instead of logging every prediction row (50k+ rows). -- **Per-era line charts fixed:** `era_index` (0, 1, 2…) used as x-axis — fixes alphabetical string sort that scrambled chronological order. -- **Drawdown curve added:** Per-era drawdown from peak cumulative correlation logged alongside the cumulative correlation line chart. -- **Correlation distribution histogram:** Distribution of per-era Spearman correlations logged as a bar chart — directly answers "how many negative eras does this model have?" -- **Missing bar charts added:** Feature exposure top-15, ensemble model-pair correlation (A→B format), worst stability by era — all now have companion bar charts. -- **HPO summary table expanded:** 18 → 30 columns; adds `model_1/2/3_type` (split, for WandB parallel coordinates), XGBoost/LightGBM hyperparams, feature selection, noise injection, augmentation flags. -- **Convergence chart:** `log_hpo_convergence` logs all trial scores and running-best `corr_sharpe` in a single WandB run after the HPO search completes, rendering as a proper convergence curve. -- **String metric bug fixed:** `feature_importance_model_type` moved from `wandb.log()` (coerced to NaN) to `wandb.run.summary`. -- **Duplicate metric removed:** `metric/corr_sharpe` deduplicated in `log_hpo_trial_metrics`. +## [0.6.0] — MMC Scoring & W&B Chart Diagnostics + +- **MMC on validation split:** `load_mmc_validation_frame` aligns `validation.parquet` with `meta_model.parquet`; HPO merges `mmc`, `mmc_sharpe`, and `payout_score` after train-era holdout evaluation so W&B `metric/mmc` is no longer null. +- **W&B diagnostics as charts:** Raw `diagnostics/` tables replaced with matplotlib horizontal bar charts, correlation heatmaps, and line charts; NaN metrics are skipped in trial logging. +- **Live W&B training logs:** `wandb_logging.py` bridges loguru to the W&B Logs panel and logs per-round XGBoost metrics during training. +- **MultiTarget diagnostics:** `pipeline/model_access.py` provides `iter_trained_models`, `model_prediction_map`, and `multitarget_blend_weights` for SHAP and ensemble diagnostics on multi-target pipelines. +- **Feature catalog & HPO routing:** `features/catalog.py`, `hpo/feature_routing.py`, and `hpo/target_strategy.py` resolve feature sets and multi-target training from `features.json`. +- **HPO export module:** `hpo/export.py` centralises pipeline fitting for Numerai pickle export from flat HPO configs. +- **Universal feature importance:** `compute_universal_feature_importance` extracts importance from XGBoost, LightGBM, CatBoost, and sklearn tree models; logs ranked bar charts to W&B. +- **Era-stratified importance:** `_log_era_stratified_importance` logs per-era importance `line_series` for temporal stability analysis. +- **Per-era stability report:** `compute_feature_report` (LightGBM proxy) surfaces top/mean/worst stability features as bar charts in W&B. +- **Best-trial diagnostics run:** After HPO, the best config is retrained and logged to a dedicated `best-trial-diagnostics` WandB run with full XAI artifacts. +- **Per-era line charts fixed:** `era_index` used as x-axis for chronological ordering; drawdown and correlation distribution charts added. +- **HPO summary expanded:** 30-column trial table plus scatter charts for corr Sharpe, MMC Sharpe, and runtime; `log_hpo_convergence` renders running-best curve in one W&B run. +- **W&B metric fixes:** `feature_importance_model_type` stored in `wandb.run.summary`; duplicate `metric/corr_sharpe` removed; finite-check on NaN metrics. ## [0.5.0] — Production Hardening diff --git a/README.md b/README.md index 7917503..abffda4 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# AlphaPulse v0.5.0 +# AlphaPulse v0.6.0 AlphaPulse is a config-driven framework for building, training, and deploying ML pipelines for the [Numerai](https://numer.ai) stock-market prediction tournament. It covers the full workflow: dataset download, experiment definition, backtesting, hyperparameter optimization (HPO), and automated weekly submission. @@ -13,7 +13,7 @@ The framework is organized into five layers: | **Data** | `NumeraiDataLoader`, parquet files, `features.json` | Downloads and loads Numerai dataset splits (train/validation/live) | | **Configuration** | `ExperimentV1` YAML schema, HPO search space, `TrialDB`, AutoResearch agent | Defines what to train — via static YAML, automated HPO, or Claude-agent-driven research | | **Core Pipeline** | Preprocessors, Models, `Pipeline` / `MultiHeadPipeline`, Ensemble, `FeatureNeutralizer` | Fits and combines models; handles feature routing, ensembling, and prediction neutralization | -| **Evaluation** | `Backtester`, `PurgedEraCV`, SHAP report, W&B diagnostics | Computes era-aware metrics (CORR, Sharpe, MMC) and XAI reports | +| **Evaluation** | `Backtester`, `PurgedEraCV`, SHAP report, W&B diagnostics (charts) | Era-aware metrics (CORR, Sharpe, MMC on validation split) and matplotlib XAI plots | | **Export & Submission** | `predict.pkl`, live inference, submission validation, Numerai upload | Produces tournament-ready predictions and submits them | > The diagram is editable — open `docs/assets/architecture.drawio` in [draw.io](https://app.diagrams.net) to modify it. @@ -194,9 +194,22 @@ uv run python scripts/hpo_pipeline.py \ --train-subsample 0.125 \ --num-trials 30 \ --output-dir artifacts/hpo_x8 \ - --local + --local \ + --wandb-project alphapulse-hpo ``` +**Useful flags:** +- `--wandb-project ` — log every trial to Weights & Biases (project name is timestamped and saved for `--resume`) +- `--resume` — skip trials already recorded in `trials.db` +- `--trial-timeout ` — kill stuck trials (default: 1800) +- `--max-hours ` — stop after a time budget +- `--objective corr_sharpe|mmc_sharpe|payout_score` — optimization target + +W&B trial runs include scalar metrics (`corr_sharpe`, `mmc_sharpe`, `metric/mmc`) and per-trial +`diagnostics/` charts (per-era correlation, feature exposure, SHAP importance). After the search, +a `best-trial-diagnostics` run and `search-convergence` / `hpo-summary` runs are logged to the +same W&B group. + The best resulting configuration will be saved to `artifacts/hpo_x8/best_config.json`. ### 4\. Run AutoResearch (Agent-Driven Research Loop) @@ -519,11 +532,12 @@ make eda-lint ├── src/alphapulse/ # Core framework source code │ ├── autoresearch/ # Agent-driven research loop (loop, agent, mutations, state) │ ├── evaluation/ # Backtesting, metrics, SHAP report, W&B diagnostics, submission validation -│ ├── experiments/ # YAML schema (ExperimentV1), runner -│ ├── hpo/ # HPO objective, search space, builder, registry, TrialDB (SQLite) -│ ├── logging_/ # Leaderboard and W&B helpers +│ ├── experiments/ # YAML schema (ExperimentV1), runner, data loaders (incl. MMC validation frame) +│ ├── features/ # Feature/target catalog loaded from features.json +│ ├── hpo/ # HPO objective, search space, builder, registry, TrialDB (SQLite), export +│ ├── logging_/ # Leaderboard, W&B helpers, live loguru → W&B Logs bridge │ ├── models/ # All model implementations + factory -│ ├── pipeline/ # Pipeline, MultiHeadPipeline, MultiTargetPipeline, ensemble, neutralizer, stacker +│ ├── pipeline/ # Pipeline, MultiHeadPipeline, MultiTargetPipeline, model_access, ensemble, neutralizer │ ├── preprocessors/ # All preprocessor implementations + factory (incl. autoencoder, compression, era-stable selector) │ ├── utils/ # Global seed utility │ └── validation/ # PurgedEraCV @@ -549,6 +563,14 @@ Commit messages: prefer conventional commits (e.g. `feat: ...`, `fix: ...`, `doc See [CHANGELOG.md](CHANGELOG.md) for completed releases. +**Completed — v0.6.0 (MMC + W&B Diagnostics):** +- **MMC on validation split:** HPO scores `mmc`, `mmc_sharpe`, and `payout_score` on `validation.parquet` rows aligned with `meta_model.parquet` (train holdout ids do not overlap meta-model ids). +- **W&B diagnostics as charts:** `diagnostics/` logs matplotlib bar/heatmap/line charts instead of raw tables; horizontal bar charts for feature importance and exposure; ensemble correlation heatmap. +- **Live W&B training logs:** loguru lines and per-round XGBoost metrics stream to W&B during HPO trials. +- **MultiTarget diagnostics:** `pipeline/model_access.py` unifies model iteration and prediction collection for SHAP and ensemble diagnostics across `Pipeline` and `MultiTargetPipeline`. +- **Feature catalog & routing:** `features/catalog.py` and HPO feature routing resolve `features.json` sets and YAML groups into per-model column lists. +- **HPO summary charts:** scatter plots for trial corr Sharpe, MMC Sharpe, and runtime in the `hpo-summary` W&B run. + **Completed — v0.5.0 (Production Hardening + XAI):** - **HPO fault tolerance:** Each local trial runs in an isolated subprocess; crashes mark the trial failed and the sweep continues. A SQLite-backed `TrialDB` persists trial state. `--resume` skips already-completed trials. - **Provenance artifact:** On every export, a hermetically sealed bundle is written: resolved config, `uv export` dependency snapshot, and git commit hash. diff --git a/eda/pages/hpo_analysis.py b/eda/pages/hpo_analysis.py index 4298db7..21c078e 100644 --- a/eda/pages/hpo_analysis.py +++ b/eda/pages/hpo_analysis.py @@ -403,9 +403,14 @@ def load_trials(path: str, _min_sharpe: float) -> pd.DataFrame: max_value=min(100, len(df)), value=20, ) +rank_col = ( + "payout_score" + if "payout_score" in df.columns and df["payout_score"].notna().any() + else "sharpe" +) show_cols = [ "trial", - "sharpe", + rank_col, "mean_era_corr", "std_era_corr", "max_drawdown", @@ -416,12 +421,12 @@ def load_trials(path: str, _min_sharpe: float) -> pd.DataFrame: "use_neutralization", "elapsed_seconds", ] -if "payout_score" in df.columns: - show_cols.insert(2, "payout_score") +if rank_col == "payout_score": + show_cols.insert(2, "corr_sharpe" if "corr_sharpe" in df.columns else "sharpe") -leaderboard = df.nlargest(top_n, "sharpe")[show_cols] +leaderboard = df.nlargest(top_n, rank_col)[show_cols] st.dataframe( - leaderboard.style.background_gradient(subset=["sharpe"], cmap="RdYlGn"), + leaderboard.style.background_gradient(subset=[rank_col], cmap="RdYlGn"), use_container_width=True, height=420, ) diff --git a/experiments/mmc_asymmetric_lgbm_tabicl_v1.yaml b/experiments/mmc_asymmetric_lgbm_tabicl_v1.yaml new file mode 100644 index 0000000..e6abdf8 --- /dev/null +++ b/experiments/mmc_asymmetric_lgbm_tabicl_v1.yaml @@ -0,0 +1,43 @@ +version: "1" +data: + data_dir: data/v5.2 + train_subsample: 0.125 + target_col: target + seed: 42 +features: + columns: null + groups: {} +preprocessing: + - type: StandardScaler + params: {} +models: + - type: LightGBM + params: + params: + num_leaves: 31 + learning_rate: 0.01 + objective: regression + metric: rmse + - type: TabICL + params: + n_estimators: 4 + max_train_rows: 8000 + compression: autoencoder + compression_components: 128 + compression_epochs: 10 + kv_cache: false + batch_size: 8 +ensemble_method: weighted +ensemble_params: + optimize_weights: true + objective: payout_score + min_weight: 0.05 + max_weight: 0.90 +neutralization: + proportion: 0.5 +train: + n_rounds: 2000 + early_stopping_rounds: 100 +evaluation: + primary_metric: payout_score + walk_forward: false diff --git a/experiments/mmc_catboost_baseline_v1.yaml b/experiments/mmc_catboost_baseline_v1.yaml new file mode 100644 index 0000000..18219ff --- /dev/null +++ b/experiments/mmc_catboost_baseline_v1.yaml @@ -0,0 +1,36 @@ +version: "1" +data: + data_dir: data/v5.2 + train_subsample: 0.125 + target_col: target + seed: 42 +features: + columns: null + groups: {} +preprocessing: + - type: RobustScaler + params: {} +models: + - type: CatBoost + params: + params: + depth: 6 + learning_rate: 0.03 + l2_leaf_reg: 5.0 + min_data_in_leaf: 200 + colsample_bylevel: 0.3 + loss_function: RMSE + verbose: 0 + iterations: 400 + early_stopping_rounds: 50 +ensemble_method: single +neutralization: + proportion: 0.35 +meta_neutralization: + proportion: 0.0 +train: + n_rounds: 400 + early_stopping_rounds: 50 +evaluation: + primary_metric: payout_score + walk_forward: false diff --git a/experiments/mmc_lgbm_catboost_ensemble_v1.yaml b/experiments/mmc_lgbm_catboost_ensemble_v1.yaml new file mode 100644 index 0000000..f750b60 --- /dev/null +++ b/experiments/mmc_lgbm_catboost_ensemble_v1.yaml @@ -0,0 +1,53 @@ +version: "1" +data: + data_dir: data/v5.2 + train_subsample: 0.125 + target_col: target + seed: 42 +features: + columns: null + groups: {} +preprocessing: + - type: StandardScaler + params: {} +models: + - type: LightGBM + params: + params: + num_leaves: 31 + learning_rate: 0.01 + min_child_samples: 200 + reg_alpha: 0.5 + reg_lambda: 5.0 + colsample_bytree: 0.3 + subsample: 0.7 + objective: regression + metric: rmse + - type: CatBoost + params: + params: + depth: 6 + learning_rate: 0.03 + l2_leaf_reg: 5.0 + min_data_in_leaf: 200 + colsample_bylevel: 0.3 + loss_function: RMSE + verbose: 0 + iterations: 400 + early_stopping_rounds: 50 +ensemble_method: weighted +ensemble_params: + optimize_weights: true + objective: payout_score + min_weight: 0.05 + max_weight: 0.90 +neutralization: + proportion: 0.5 +meta_neutralization: + proportion: 0.55 +train: + n_rounds: 400 + early_stopping_rounds: 50 +evaluation: + primary_metric: payout_score + walk_forward: false diff --git a/experiments/mmc_lgbm_meta_neutral_v1.yaml b/experiments/mmc_lgbm_meta_neutral_v1.yaml new file mode 100644 index 0000000..19d04da --- /dev/null +++ b/experiments/mmc_lgbm_meta_neutral_v1.yaml @@ -0,0 +1,36 @@ +version: "1" +data: + data_dir: data/v5.2 + train_subsample: 0.125 + target_col: target + seed: 42 +features: + columns: null + groups: {} +preprocessing: + - type: StandardScaler + params: {} +models: + - type: LightGBM + params: + params: + num_leaves: 31 + learning_rate: 0.01 + min_child_samples: 200 + reg_alpha: 0.5 + reg_lambda: 5.0 + colsample_bytree: 0.3 + subsample: 0.7 + objective: regression + metric: rmse +ensemble_method: single +neutralization: + proportion: 0.5 +meta_neutralization: + proportion: 0.6 +train: + n_rounds: 400 + early_stopping_rounds: 50 +evaluation: + primary_metric: payout_score + walk_forward: false diff --git a/experiments/mmc_multitarget_lgbm_v1.yaml b/experiments/mmc_multitarget_lgbm_v1.yaml new file mode 100644 index 0000000..d3eea6a --- /dev/null +++ b/experiments/mmc_multitarget_lgbm_v1.yaml @@ -0,0 +1,35 @@ +version: "1" +data: + data_dir: data/v5.2 + train_subsample: 0.125 + target_col: target + auxiliary_targets: + - target_jerome_v4_20 + - target_ralph_v4_20 + - target_tyler_v4_20 + target_blend_method: equal + seed: 42 +features: + columns: null + groups: {} +preprocessing: + - type: StandardScaler + params: {} +models: + - type: LightGBM + params: + params: + num_leaves: 31 + learning_rate: 0.01 + objective: regression + metric: rmse +ensemble_method: single +ensemble_params: {} +neutralization: + proportion: 0.5 +train: + n_rounds: 2000 + early_stopping_rounds: 100 +evaluation: + primary_metric: corr_sharpe + walk_forward: false diff --git a/pyproject.toml b/pyproject.toml index ee84cf2..172e623 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "alphapulse" -version = "0.5.0" +version = "0.6.0" description = "Config-driven framework for building, training, and deploying Numerai competition pipelines." readme = "README.md" requires-python = ">=3.12" @@ -62,8 +62,10 @@ foundation = [ "tabpfn>=7.0", "tabicl>=2.0", ] -foundation-api = [ - "tabpfn-client>=0.3.0", +packboost = [ + "packboost @ git+https://github.com/Pranshu-Bahadur/PackBoost.git", + "torch", + "ninja", ] eda = [ "streamlit>=1.30", @@ -133,7 +135,9 @@ ignore = [ [tool.ruff.lint.per-file-ignores] "tests/**" = ["S101", "S108", "T20", "E501"] -"scripts/**" = ["T20", "S607"] +"scripts/**" = ["T20", "S607", "E501"] +"src/alphapulse/logging_/cli.py" = ["PLW0603"] +"src/alphapulse/logging_/wandb_logging.py" = ["PLW0603"] "src/alphapulse/autoresearch/agent.py" = ["E501", "S603", "S607"] [tool.ruff.format] @@ -200,10 +204,10 @@ module = [ "torch.*", "tabpfn", "tabpfn.*", - "tabpfn_client", - "tabpfn_client.*", "tabicl", "tabicl.*", + "packboost", + "packboost.*", ] ignore_missing_imports = true diff --git a/scripts/export_numerai_pickle.py b/scripts/export_numerai_pickle.py index b6e1bbc..8c0dbb1 100644 --- a/scripts/export_numerai_pickle.py +++ b/scripts/export_numerai_pickle.py @@ -20,24 +20,10 @@ from loguru import logger from alphapulse.evaluation.export_validation import smoke_test_predict_fn -from alphapulse.experiments.data import load_train_only_frame -from alphapulse.experiments.split import internal_val_split -from alphapulse.hpo.builder import TREE_MODEL_NAMES, build_pipeline_or_multi -from alphapulse.hpo.search_space import get_train_kwargs_from_flat, resolve_flat_config +from alphapulse.hpo.export import build_hpo_pipeline_from_flat from alphapulse.utils import set_global_seed -def _needs_era_from_flat_config(flat: dict) -> bool: - if bool(flat.get("use_packboost", False)): - return True - num_models = int(flat.get("num_models", 1)) - for i in range(1, min(num_models, 3) + 1): - model_type = flat.get(f"model_{i}_type", "") - if model_type == "Packboost" or model_type in TREE_MODEL_NAMES: - return True - return False - - def _artifact_stem(flat_config: dict[str, Any], target_col: str) -> str: """Build canonical artifact stem: TIMESTAMP_ARCH_TARGET_HASH.""" ts = datetime.datetime.now(datetime.UTC).strftime("%Y%m%dT%H%M%S") @@ -97,68 +83,44 @@ def main( if not isinstance(flat_config, dict): raise ValueError(f"Expected a JSON object in {best_config_path}") - need_era = _needs_era_from_flat_config(flat_config) - feature_set = flat_config.get("feature_set") - logger.info( - "Loading train data (subsample={}, feature_set={!r})...", + "Building pipeline from HPO config " + "(subsample={}, routing={}, target_mode={})...", train_subsample, - feature_set, + flat_config.get("use_feature_routing", False), + flat_config.get("target_mode", "single"), ) - X_train, y_train, feature_cols = load_train_only_frame( - data_dir=data_dir, + fit_result = build_hpo_pipeline_from_flat( + flat_config, + data_dir, train_subsample=train_subsample, - target_col=target_col, seed=seed, - feature_columns=None, - need_era=need_era, - feature_set=feature_set, + target_col_fallback=target_col, + allow_target_resample=False, ) - gc.collect() - - mem_mb = X_train.memory_usage(deep=True).sum() / 1e6 - logger.info("Train shape: {} ({:.1f} MB)", X_train.shape, mem_mb) + pipeline = fit_result.pipeline + feature_cols = fit_result.feature_columns + pipeline_config = fit_result.pipeline_cfg + flat_config = fit_result.flat + primary_target = fit_result.primary_target - era_train = X_train["era"] if "era" in X_train.columns else None - stacking_needs_val = ( - int(flat_config.get("num_models", 1)) > 1 - and flat_config.get("ensemble_method") == "stacking" - ) - X_train_fit, y_train_fit, X_val_internal, y_val_internal = internal_val_split( - X_train, - y_train, - era_train=era_train, - force_internal=stacking_needs_val, - ) - del X_train, y_train gc.collect() - - pipeline_config = resolve_flat_config(flat_config) - pipeline = build_pipeline_or_multi(pipeline_config, feature_columns=feature_cols) - train_kwargs = get_train_kwargs_from_flat(flat_config) - - logger.info("Fitting pipeline...") - pipeline.fit( - X_train_fit, - y_train_fit, - X_val=X_val_internal, - y_val=y_val_internal, - **train_kwargs, + logger.info( + "Pipeline fitted: {} features, primary_target={!r}", + len(feature_cols), + primary_target, ) - del X_train_fit, y_train_fit, X_val_internal, y_val_internal - gc.collect() - output_dir.mkdir(parents=True, exist_ok=True) - stem = _artifact_stem(flat_config, target_col) + stem = _artifact_stem(flat_config, primary_target) (output_dir / "resolved_pipeline_config.json").write_text( json.dumps(pipeline_config, indent=2), encoding="utf-8", ) - prov = _provenance(flat_config, pipeline_config, target_col) + prov = _provenance(flat_config, pipeline_config, primary_target) prov_path = output_dir / f"{stem}_provenance.json" prov_path.write_text(json.dumps(prov, indent=2), encoding="utf-8") logger.info("Provenance bundle saved to: {}", prov_path) diff --git a/scripts/hpo_pipeline.py b/scripts/hpo_pipeline.py index 31e256b..2dc7809 100644 --- a/scripts/hpo_pipeline.py +++ b/scripts/hpo_pipeline.py @@ -1,7 +1,7 @@ """HPO pipeline: search over preprocessing, models, and ensembles. Supports two modes: - --local Random search (no extra dependencies). + --local Optuna TPE Bayesian search (default sampler) with subprocess isolation. (default) Ray Tune distributed search (requires ``pip install 'alphapulse[hpo]'``). Each trial is scored via era holdout (fast mode, default) or walk-forward backtesting @@ -12,27 +12,52 @@ Pass --resume to skip already-completed trials recorded in the trial database. """ +import gc import json import multiprocessing +import os +import random import time import uuid from pathlib import Path -from typing import Literal +from typing import Any, Literal +import numpy as np +import pandas as pd import tyro from dotenv import load_dotenv from loguru import logger -from alphapulse.experiments.data import load_train_only_frame +from alphapulse.experiments.data import load_train_only_frame, load_train_targets_frame +from alphapulse.features.catalog import load_feature_catalog, load_target_catalog +from alphapulse.hpo.feature_routing import resolve_feature_routing from alphapulse.hpo.objective import TrialResult, run_trial -from alphapulse.hpo.search_space import sample_random_config +from alphapulse.hpo.optuna_search import ( + DEFAULT_N_STARTUP_TRIALS, + SamplerName, + create_hpo_study, + suggest_flat_config, + tell_trial_result, +) +from alphapulse.hpo.target_strategy import ( + apply_target_strategy_to_flat, + strategy_from_flat, + validate_target_strategy_early, +) from alphapulse.hpo.trial_db import TrialDB from alphapulse.logging_.leaderboard import ( + BestCriteria, entry_from_hpo_result, print_leaderboard, save_leaderboard, + selection_score_from_metrics, ) from alphapulse.utils import set_global_seed +from alphapulse.utils.gpu_cleanup import ( + cleanup_after_trial_subprocess, + cleanup_stale_gpu_processes, + release_cuda_memory, +) _MP_CTX = multiprocessing.get_context("spawn") _WANDB_GROUP_FILE = "wandb_group.txt" @@ -55,6 +80,8 @@ def _trial_result_from_db_row(row: dict) -> TrialResult: metrics = row["metrics"] or {} corr_sharpe = float(metrics.get("corr_sharpe", float("-inf"))) flat_config = row["flat_config"] + mmc_sharpe = metrics.get("mmc_sharpe") + payout_score = metrics.get("payout_score") return TrialResult( trial_number=int(row["trial_number"]), sharpe=corr_sharpe, @@ -64,6 +91,12 @@ def _trial_result_from_db_row(row: dict) -> TrialResult: params=flat_config, error=row["error"], corr_sharpe=corr_sharpe, + mmc_sharpe=float(mmc_sharpe) + if mmc_sharpe is not None and np.isfinite(mmc_sharpe) + else None, + payout_score=float(payout_score) + if payout_score is not None and np.isfinite(payout_score) + else None, ) @@ -86,9 +119,72 @@ def _trial_worker( ) -> None: """Run a single trial inside a subprocess and push the result to the queue.""" wandb_initialized = False + X_train = None + y_train = None + targets_df = None + era_train = None + metrics: dict = {} try: load_dotenv() worker_config = dict(flat_config) + worker_config["_data_dir"] = data_dir + worker_config["_train_subsample"] = train_subsample + worker_config.setdefault("primary_target", target_col) + if "hpo_objective" not in worker_config: + worker_config.setdefault("hpo_objective", "payout_score") + + feature_catalog = load_feature_catalog(data_dir) + target_catalog = load_target_catalog(data_dir) + strategy = strategy_from_flat(worker_config) + routing = resolve_feature_routing(worker_config, feature_catalog) + feature_columns = routing.feature_columns or None + active_groups = list(worker_config.get("active_groups") or []) + worker_config["active_groups"] = active_groups + worker_config["active_groups_str"] = "+".join(active_groups) + worker_config["active_groups_count"] = len(active_groups) + worker_config["routed_feature_count"] = len(feature_columns or []) + + if strategy.target_mode == "multi_blend" and strategy.auxiliary_targets: + X_train, y_train, targets_df, feature_cols = load_train_targets_frame( + Path(data_dir), + train_subsample=train_subsample, + primary_target=strategy.primary_target, + auxiliary_targets=strategy.auxiliary_targets, + seed=seed, + feature_columns=feature_columns, + need_era=True, + ) + else: + X_train, y_train, feature_cols = load_train_only_frame( + Path(data_dir), + train_subsample=train_subsample, + target_col=strategy.primary_target, + seed=seed, + feature_columns=feature_columns, + need_era=True, + ) + targets_df = None + + era_train = X_train["era"] + targets_for_validation = ( + targets_df + if targets_df is not None + else pd.DataFrame({strategy.primary_target: y_train}) + ) + validation = validate_target_strategy_early( + targets_for_validation, + strategy, + catalog=target_catalog, + rng=random.Random(seed), + ) + if not validation.ok: + raise ValueError(validation.reason or "target strategy validation failed") + worker_config = apply_target_strategy_to_flat( + worker_config, validation.strategy + ) + if validation.strategy.target_mode == "single": + targets_df = None + wandb_active = ( wandb_project is not None and wandb_group is not None @@ -100,38 +196,33 @@ def _trial_worker( import wandb from alphapulse.hpo.objective import TrialResult + from alphapulse.logging_.wandb_logging import attach_wandb_loguru from alphapulse.logging_.wandb_utils import log_hpo_trial_metrics - num = flat_config.get("num_models", 1) + num = worker_config.get("num_models", 1) model_types = "+".join( - str(flat_config.get(f"model_{i}_type", "?")) for i in range(1, num + 1) + str(worker_config.get(f"model_{i}_type", "?")) + for i in range(1, num + 1) ) - preprocessors = flat_config.get("scaler_type", "StandardScaler") - if flat_config.get("use_packboost"): + preprocessors = worker_config.get("scaler_type", "StandardScaler") + if worker_config.get("use_packboost"): preprocessors += "+Packboost" wandb.init( project=wandb_project, group=wandb_group, name=f"trial_{trial_number:03d}", config={ - **flat_config, + **worker_config, "model_types": model_types, "preprocessors": preprocessors, }, - reinit=True, + reinit="finish_previous", + settings=wandb.Settings(console="wrap"), ) + attach_wandb_loguru() wandb_initialized = True t0 = time.perf_counter() - X_train, y_train, feature_cols = load_train_only_frame( - Path(data_dir), - train_subsample=train_subsample, - target_col=target_col, - seed=seed, - feature_columns=None, - need_era=True, - ) - era_train = X_train["era"] metrics = run_trial( worker_config, X_train=X_train, @@ -139,19 +230,29 @@ def _trial_worker( era_train=era_train, feature_cols=feature_cols, seed=seed, + targets_df=targets_df, + catalog=feature_catalog, ) elapsed = time.perf_counter() - t0 if wandb_initialized: assert trial_number is not None corr_sharpe = float(metrics.get("corr_sharpe", float("-inf"))) + mmc_sharpe = metrics.get("mmc_sharpe") + payout_score = metrics.get("payout_score") trial_result = TrialResult( trial_number=trial_number, sharpe=corr_sharpe, metrics=metrics, - model_type=str(flat_config.get("model_1_type", "XGBoost")), + model_type=str(worker_config.get("model_1_type", "XGBoost")), elapsed_seconds=elapsed, - params=flat_config, + params=worker_config, corr_sharpe=corr_sharpe, + mmc_sharpe=float(mmc_sharpe) + if mmc_sharpe is not None and np.isfinite(mmc_sharpe) + else None, + payout_score=float(payout_score) + if payout_score is not None and np.isfinite(payout_score) + else None, ) log_hpo_trial_metrics( trial_result, @@ -160,30 +261,146 @@ def _trial_worker( preprocessors=preprocessors, ) - result_queue.put({"ok": True, "metrics": metrics, "elapsed_seconds": elapsed}) + result_queue.put( + { + "ok": True, + "metrics": metrics, + "elapsed_seconds": elapsed, + "flat_config": worker_config, + } + ) except Exception as exc: result_queue.put({"ok": False, "error": str(exc)}) finally: + if X_train is not None: + del X_train + if y_train is not None: + del y_train + if targets_df is not None: + del targets_df + if era_train is not None: + del era_train + if metrics: + del metrics + release_cuda_memory() if wandb_initialized: import wandb - wandb.finish(quiet=True) + from alphapulse.logging_.wandb_logging import detach_wandb_loguru + detach_wandb_loguru() + wandb.finish() + gc.collect() -def _best_from_db(db: TrialDB, objective: str) -> tuple[float, dict]: + +def _best_from_db( + db: TrialDB, + objective: str, + *, + criteria: BestCriteria = "objective", +) -> tuple[float, dict]: best_score = float("-inf") best_config: dict = {} for row in db.load_all_trials(): if row["status"] != "completed" or not row["metrics"]: continue metrics = row["metrics"] - score = float(metrics.get(objective, metrics.get("corr_sharpe", float("-inf")))) + score = selection_score_from_metrics( + metrics, objective=objective, criteria=criteria + ) if score > best_score: best_score = score best_config = row["flat_config"] return best_score, best_config +_WORKER_RUNTIME_KEYS = frozenset( + { + "_data_dir", + "_train_subsample", + "log_wandb_diagnostics", + "wandb_log_shap", + } +) + + +def _persistable_flat_config(flat: dict[str, Any]) -> dict[str, Any]: + return {k: v for k, v in flat.items() if k not in _WORKER_RUNTIME_KEYS} + + +def _resolve_best_criteria(objective: str, best_criteria: BestCriteria) -> BestCriteria: + if best_criteria != "objective": + return best_criteria + if objective == "payout_score": + return "robust_payout" + return "objective" + + +def _warn_resume_eval_mode_mismatch( + *, + resume: bool, + fast: bool, + completed_rows: list[dict], +) -> None: + if not resume or not completed_rows: + return + prior_fast_flags = { + bool(row["flat_config"].get("hpo_fast")) + for row in completed_rows + if row["status"] == "completed" and row.get("flat_config") + } + if len(prior_fast_flags) != 1: + return + prior_fast = prior_fast_flags.pop() + if prior_fast == fast: + return + logger.warning( + "Resume eval-mode mismatch: completed trials used fast={} but this run uses " + "fast={}. New trials will not be comparable with earlier leaderboard scores.", + prior_fast, + fast, + ) + + +def _load_diagnostics_train_data( + best_config: dict, + data_dir: Path, + train_subsample: float, + target_col: str, + seed: int, +) -> tuple[ + pd.DataFrame, pd.Series, pd.DataFrame | None, list[str], dict[str, list[str]] +]: + feature_catalog = load_feature_catalog(data_dir) + routing = resolve_feature_routing(best_config, feature_catalog) + feature_columns = routing.feature_columns or None + strategy = strategy_from_flat(best_config) + primary = strategy.primary_target or target_col + + if strategy.target_mode == "multi_blend" and strategy.auxiliary_targets: + X_train, y_train, targets_df, feature_cols = load_train_targets_frame( + data_dir, + train_subsample=train_subsample, + primary_target=primary, + auxiliary_targets=strategy.auxiliary_targets, + seed=seed, + feature_columns=feature_columns, + need_era=True, + ) + else: + X_train, y_train, feature_cols = load_train_only_frame( + data_dir, + train_subsample=train_subsample, + target_col=primary, + seed=seed, + feature_columns=feature_columns, + need_era=True, + ) + targets_df = None + + return X_train, y_train, targets_df, feature_cols, routing.feature_groups + + def _run_best_trial_diagnostics( *, best_config: dict, @@ -226,15 +443,17 @@ def _run_best_trial_diagnostics( from alphapulse.logging_.wandb_utils import log_importance_artifact try: - X_train, y_train, _ = load_train_only_frame( - data_dir, - train_subsample=train_subsample, - target_col=target_col, - seed=seed, - feature_columns=None, - need_era=True, + X_train, y_train, targets_df, feature_cols, feature_groups = ( + _load_diagnostics_train_data( + best_config, + data_dir, + train_subsample, + target_col, + seed, + ) ) era_train = X_train["era"] + primary_target = strategy_from_flat(best_config).primary_target eras_sorted = sorted(era_train.unique(), key=str) n_holdout = max(5, len(eras_sorted) // 5) @@ -248,6 +467,7 @@ def _run_best_trial_diagnostics( pipeline_cfg = apply_gpu_pipeline_config(pipeline_cfg) train_kwargs = get_train_kwargs_from_flat(best_config) + targets_train = targets_df.loc[train_mask] if targets_df is not None else None pipeline = _fit_pipeline( pipeline_cfg, feature_cols, @@ -256,6 +476,8 @@ def _run_best_trial_diagnostics( train_kwargs, flat_config=best_config, seed=seed, + feature_groups=feature_groups or None, + targets_df=targets_train, ) ho_mask = era_train.isin(holdout_set) @@ -267,6 +489,17 @@ def _run_best_trial_diagnostics( metrics = Backtester(pipeline, feature_columns=feature_cols).evaluate( X_ho, y_ho, era_ho ) + from alphapulse.hpo.objective import _merge_validation_mmc_metrics + + metrics, _ = _merge_validation_mmc_metrics( + metrics, + pipeline=pipeline, + data_dir=data_dir, + feature_cols=feature_cols, + target_col=primary_target, + train_subsample=train_subsample, + seed=seed, + ) wandb.init( project=wandb_project, @@ -274,8 +507,12 @@ def _run_best_trial_diagnostics( name="best-trial-diagnostics", job_type="diagnostics", config=best_config, - reinit=True, + reinit="finish_previous", + settings=wandb.Settings(console="wrap"), ) + from alphapulse.logging_.wandb_logging import attach_wandb_loguru + + attach_wandb_loguru() log_experiment_diagnostics( pipeline=pipeline, @@ -295,11 +532,17 @@ def _run_best_trial_diagnostics( if importance: log_importance_artifact(importance, name="best-trial-feature-importance") - wandb.finish(quiet=True) + from alphapulse.logging_.wandb_logging import detach_wandb_loguru + + detach_wandb_loguru() + wandb.finish() except Exception as exc: logger.warning("Best-trial diagnostics failed: {}", exc) try: - wandb.finish(quiet=True) + from alphapulse.logging_.wandb_logging import detach_wandb_loguru + + detach_wandb_loguru() + wandb.finish() except Exception as finish_exc: logger.debug("wandb.finish cleanup error: {}", finish_exc) @@ -312,15 +555,20 @@ def _run_local( seed: int, num_trials: int, output_dir: Path, - objective: str = "corr_sharpe", + objective: str = "payout_score", wandb_project: str | None = None, resume: bool = False, - trial_timeout: int = 1800, + trial_timeout: int = 3600, gpu: bool = False, fast: bool = True, + max_models: int = 2, wandb_diagnostics: bool = True, + max_hours: float | None = None, + sampler: SamplerName = "tpe", + n_startup_trials: int = DEFAULT_N_STARTUP_TRIALS, + best_criteria: BestCriteria = "objective", ) -> None: - """Local random-search HPO with subprocess isolation and SQLite trial DB.""" + """Local HPO with subprocess isolation, Optuna TPE, and SQLite trial DB.""" X_train, y_train, feature_cols = load_train_only_frame( data_dir, @@ -337,6 +585,18 @@ def _run_local( ) output_dir.mkdir(parents=True, exist_ok=True) + if gpu: + preflight = cleanup_stale_gpu_processes(parent_pid=os.getpid()) + if preflight["killed_pids"]: + logger.warning( + "GPU preflight killed stale PIDs: {}", + preflight["killed_pids"], + ) + if preflight["remaining_gpu_pids"]: + logger.warning( + "GPU preflight: non-parent processes still on GPU: {}", + preflight["remaining_gpu_pids"], + ) db_path = output_dir / "trials.db" wandb_group = _load_or_create_wandb_group(output_dir, wandb_project) if wandb_project: @@ -347,7 +607,7 @@ def _run_local( wandb_diagnostics, ) - trial_kwargs = { + trial_kwargs: dict[str, str | float] = { "data_dir": str(data_dir), "train_subsample": train_subsample, "target_col": target_col, @@ -356,15 +616,51 @@ def _run_local( results: list[TrialResult] = [] best_score = float("-inf") best_config: dict = {} + sweep_t0 = time.perf_counter() + max_seconds = max_hours * 3600.0 if max_hours is not None else None + if max_seconds is not None: + logger.info( + "Time budget: {:.1f}h ({} trials max cap)", + max_hours, + num_trials, + ) + + study = create_hpo_study( + output_dir, + seed=seed, + sampler=sampler, + resume=resume, + n_startup_trials=n_startup_trials, + ) + logger.info( + "Optuna sampler: {} (storage={}, n_startup_trials={})", + sampler, + output_dir / "optuna.db", + n_startup_trials if sampler == "tpe" else "n/a", + ) with TrialDB(db_path) as db: already_done = db.completed_trials() if resume else set() + resolved_best_criteria = _resolve_best_criteria(objective, best_criteria) + if resolved_best_criteria == "robust_payout": + logger.info( + "Best config selection uses robust payout " + "(validation payout penalized by weak holdout CORR)" + ) + _warn_resume_eval_mode_mismatch( + resume=resume, + fast=fast, + completed_rows=db.load_all_trials() if resume else [], + ) if resume: - best_score, best_config = _best_from_db(db, objective) + best_score, best_config = _best_from_db( + db, objective, criteria=resolved_best_criteria + ) if best_config: logger.info( - "Resuming: global best {}={:.4f} from {} completed trial(s)", + "Resuming: global best {} ({})={:.4f} from {} completed trial(s)", objective, + resolved_best_criteria, best_score, len(already_done), ) @@ -374,25 +670,49 @@ def _run_local( ) for i in range(num_trials): + if max_seconds is not None: + elapsed_sweep = time.perf_counter() - sweep_t0 + if elapsed_sweep >= max_seconds: + logger.info( + "Time budget reached ({:.1f}h elapsed), stopping after {} " + "trial(s)", + elapsed_sweep / 3600.0, + len(results), + ) + break if i in already_done: logger.info( "Trial {}/{}: skipped (already completed)", i + 1, num_trials ) continue - flat_config = sample_random_config(seed=seed + i, fast=fast) + optuna_trial = study.ask() + flat_config = suggest_flat_config( + optuna_trial, fast=fast, max_models=max_models, data_dir=data_dir + ) if gpu: flat_config["use_gpu"] = True if fast: flat_config["hpo_fast"] = True + flat_config["hpo_objective"] = objective db.insert_trial(i, flat_config) + time_left_suffix = "" + if max_seconds is not None: + remaining_min = ( + (max_seconds or 0) - (time.perf_counter() - sweep_t0) + ) / 60 + time_left_suffix = f", {remaining_min:.0f}m left" + logger.info( - "Trial {}/{} starting (fast={}, models={})", + "Trial {}/{} starting (fast={}, models={}, groups={}, features={}{})", i + 1, num_trials, fast, flat_config.get("model_1_type", "?"), + "+".join(flat_config.get("active_groups", [])) or "default", + flat_config.get("routed_feature_count", "n/a"), + time_left_suffix, ) t0 = time.perf_counter() @@ -460,17 +780,27 @@ def _run_local( if payload.get("ok"): metrics = payload["metrics"] corr_sharpe = metrics.get("corr_sharpe", float("-inf")) - trial_score = float(metrics.get(objective, corr_sharpe)) + trial_score = selection_score_from_metrics( + metrics, + objective=objective, + criteria=resolved_best_criteria, + ) worker_elapsed = payload.get("elapsed_seconds") + worker_flat = payload.get("flat_config") + completed_flat = ( + _persistable_flat_config(worker_flat) + if isinstance(worker_flat, dict) + else flat_config + ) result = TrialResult( trial_number=i, sharpe=corr_sharpe, metrics=metrics, - model_type=flat_config.get("model_1_type", "XGBoost"), + model_type=completed_flat.get("model_1_type", "XGBoost"), elapsed_seconds=float(worker_elapsed) if worker_elapsed is not None else elapsed, - params=flat_config, + params=completed_flat, corr_sharpe=corr_sharpe, ) db.update_trial( @@ -478,7 +808,9 @@ def _run_local( status="completed", metrics=metrics, elapsed_seconds=elapsed, + flat_config=completed_flat, ) + flat_config = completed_flat else: error_msg = payload.get("error", "unknown") trial_score = float("-inf") @@ -495,12 +827,40 @@ def _run_local( i, status="failed", error=error_msg, elapsed_seconds=elapsed ) + if gpu: + trial_failed_hard = result.error is not None + cleanup = cleanup_after_trial_subprocess( + p.pid, + parent_pid=os.getpid(), + kill_worker_tree=trial_failed_hard, + ) + if cleanup["killed_pids"]: + logger.warning( + "GPU cleanup killed PIDs: {}", + cleanup["killed_pids"], + ) + remaining = cleanup["remaining_gpu_pids"] + if remaining: + logger.warning( + "GPU cleanup: other processes still on GPU: {}", + remaining, + ) + results.append(result) + metrics = result.metrics or {} logger.info( - "Trial {}/{}: corr_sharpe={:.4f} ({:.1f}s){}", + "Trial {}/{}: val_sharpe={:.4f}, holdout_sharpe={:.4f}, " + "payout={:.4f} ({:.1f}s){}", i + 1, num_trials, - result.sharpe, + float(metrics.get("val_corr_sharpe", float("nan"))), + float( + metrics.get( + "holdout_corr_sharpe", + metrics.get("corr_sharpe", float("nan")), + ) + ), + float(metrics.get("payout_score", float("nan"))), result.elapsed_seconds, f" [ERROR: {result.error}]" if result.error else "", ) @@ -530,13 +890,27 @@ def _run_local( best_score = trial_score best_config = flat_config - best_score, best_config = _best_from_db(db, objective) + tell_trial_result( + study, + optuna_trial, + trial_score, + failed=bool(result.error), + ) + + best_score, best_config = _best_from_db( + db, objective, criteria=resolved_best_criteria + ) results = _all_results_from_db(db) best_path = output_dir / "best_config.json" with open(best_path, "w", encoding="utf-8") as f: json.dump(best_config, f, indent=2) - logger.info("Best {} score: {:.4f}", objective, best_score) + logger.info( + "Best {} score ({}) : {:.4f}", + objective, + resolved_best_criteria, + best_score, + ) logger.info("Best config saved to: {}", best_path) all_results_path = output_dir / "all_trials.json" @@ -568,7 +942,12 @@ def _run_local( ) log_hpo_summary_table(results, project=wandb_project, group=wandb_group) - log_hpo_convergence(results, project=wandb_project, group=wandb_group) + log_hpo_convergence( + results, + project=wandb_project, + group=wandb_group, + objective=objective, + ) logger.info("WandB summary table logged to project={}", wandb_project) if wandb_project and wandb_group and best_config: @@ -593,7 +972,7 @@ def _run_ray( seed: int, num_trials: int, output_dir: Path, - objective: str = "corr_sharpe", + objective: str = "payout_score", wandb_project: str | None = None, ) -> None: """Ray Tune distributed HPO.""" @@ -698,31 +1077,52 @@ def main( local: bool = False, resume: bool = False, objective: Literal[ - "corr_sharpe", "mean_per_era_correlation", "max_drawdown" - ] = "corr_sharpe", + "corr_sharpe", "mean_per_era_correlation", "max_drawdown", "payout_score" + ] = "payout_score", wandb_project: str | None = None, - trial_timeout: int = 1800, + trial_timeout: int = 3600, gpu: bool = False, fast: bool = True, + max_models: int = 2, wandb_diagnostics: bool = True, + max_hours: float | None = None, + sampler: SamplerName = "tpe", + n_startup_trials: int = DEFAULT_N_STARTUP_TRIALS, + best_criteria: BestCriteria = "objective", ) -> None: """Run HPO search over preprocessing, models, and ensemble strategies. - Use --local for random search without Ray, or omit for Ray Tune. - Use --objective to choose the optimization target (default: corr_sharpe). + Use --local for Optuna-guided search without Ray, or omit for Ray Tune. + Use --objective to choose the optimization target (default: payout_score). + Use --sampler tpe for Bayesian optimization (default) or --sampler random. + Use --n-startup-trials N to set TPE random exploration trials before + Bayesian optimization (default: 25). Pass --wandb-project to log every trial to Weights & Biases. + The project name is suffixed with a launch timestamp + (e.g. alphapulse-hpo-20260614-232943) and saved in the output dir for --resume. With WandB enabled, diagnostics (per-era charts, feature exposure, SHAP for XGBoost) are logged under the ``diagnostics/`` prefix in each trial run. Pass --no-wandb-diagnostics to log metrics only. Pass --resume to continue an interrupted sweep (requires --local). Pass --trial-timeout N to cap each subprocess trial at N seconds (default: 1800). - Pass --gpu to enable CUDA for XGBoost, LightGBM, and CatBoost. + Pass --max-hours N to stop after N wall-clock hours (local mode only). + Pass --gpu to enable CUDA for XGBoost, LightGBM, CatBoost, and PackBoost. Fast mode (default) uses era holdout and a tighter search space so trials finish within ~30 minutes on full data. Pass --no-fast for full walk-forward evaluation (slower). + Pass --max-models N to cap ensemble size per trial (default: 2 in fast mode, 3 + in walk-forward mode). Use --max-models 3 in fast mode for 3-model ensembles. + When --objective payout_score, best_config.json defaults to robust payout + selection (validation payout penalized by weak holdout CORR). Override with + --best-criteria objective to keep raw validation payout. """ load_dotenv() set_global_seed(seed) + if wandb_project: + from alphapulse.logging_.wandb_utils import resolve_wandb_project + + wandb_project = resolve_wandb_project(wandb_project, output_dir=output_dir) + logger.info("WandB project: {}", wandb_project) if local: _run_local( data_dir=data_dir, @@ -737,7 +1137,12 @@ def main( trial_timeout=trial_timeout, gpu=gpu, fast=fast, + max_models=max_models if fast else max(max_models, 3), wandb_diagnostics=wandb_diagnostics, + max_hours=max_hours, + sampler=sampler, + n_startup_trials=n_startup_trials, + best_criteria=best_criteria, ) else: _run_ray( diff --git a/scripts/run_experiment.py b/scripts/run_experiment.py index caf28e9..d523b11 100644 --- a/scripts/run_experiment.py +++ b/scripts/run_experiment.py @@ -99,14 +99,18 @@ def main( from alphapulse.experiments.schema import ExperimentV1 if wandb_project: - from alphapulse.logging_.wandb_utils import init_wandb_run + from alphapulse.logging_.wandb_utils import ( + init_wandb_run, + resolve_wandb_project, + ) exp_dict = load_experiment_dict(config) exp_parsed = ExperimentV1.model_validate(exp_dict) wandb_cfg = _build_wandb_config( exp_parsed, config_path=str(config), seed=seed, gpu=gpu ) - init_wandb_run(project=wandb_project, name=config.stem, config=wandb_cfg) + resolved_project = resolve_wandb_project(wandb_project, output_dir=artifact_dir) + init_wandb_run(project=resolved_project, name=config.stem, config=wandb_cfg) result = run_experiment_from_path( config, diff --git a/scripts/smoke_foundation_packboost.py b/scripts/smoke_foundation_packboost.py new file mode 100644 index 0000000..9c2f4e0 --- /dev/null +++ b/scripts/smoke_foundation_packboost.py @@ -0,0 +1,175 @@ +"""Smoke-test PackBoost CUDA and all locally available foundation models.""" + +from __future__ import annotations + +import sys +import time +import traceback +from collections.abc import Callable +from dataclasses import dataclass + +import numpy as np +import pandas as pd + +N_ROWS = 120 +N_FEATURES = 24 +N_ERAS = 12 + + +@dataclass +class SmokeResult: + name: str + ok: bool + seconds: float + detail: str = "" + + +def _toy_era_data() -> tuple[pd.DataFrame, pd.Series]: + rng = np.random.default_rng(42) + cols = [f"f_{i}" for i in range(N_FEATURES)] + x = pd.DataFrame(rng.integers(0, 5, size=(N_ROWS, N_FEATURES)), columns=cols) + x["era"] = np.repeat([f"era_{i:04d}" for i in range(N_ERAS)], N_ROWS // N_ERAS) + y = pd.Series(rng.standard_normal(N_ROWS), dtype=np.float32) + return x, y + + +def _toy_float_data() -> tuple[pd.DataFrame, pd.Series]: + rng = np.random.default_rng(43) + cols = [f"f_{i}" for i in range(N_FEATURES)] + x = pd.DataFrame(rng.standard_normal((N_ROWS, N_FEATURES)), columns=cols) + y = pd.Series(rng.standard_normal(N_ROWS), dtype=np.float32) + return x, y + + +def _run(name: str, fn: Callable[[], str]) -> SmokeResult: + t0 = time.perf_counter() + try: + detail = fn() + return SmokeResult( + name=name, ok=True, seconds=time.perf_counter() - t0, detail=detail + ) + except Exception as exc: + return SmokeResult( + name=name, + ok=False, + seconds=time.perf_counter() - t0, + detail=f"{type(exc).__name__}: {exc}\n{traceback.format_exc()}", + ) + + +def smoke_packboost() -> str: + from alphapulse.models.packboost_model import PackboostModel + + x, y = _toy_era_data() + model = PackboostModel( + device="cuda", + n_rounds_base=8, + n_rounds_boost=4, + n_worst_eras=2, + nfolds=4, + max_depth=4, + nfeatsets=2, + ) + metrics = model.train(x, y) + preds = model.predict(x) + if not np.isfinite(preds).all(): + raise RuntimeError("non-finite PackBoost predictions") + return f"n_boost_eras={metrics.get('n_boost_eras', 0):.0f}, pred_mean={preds.mean():.6f}" + + +def smoke_tabpfn() -> str: + from alphapulse.models.foundation_models import TabPFNModel + + x, y = _toy_float_data() + model = TabPFNModel( + max_train_rows=80, + max_features=16, + compression="pca", + compression_components=8, + n_estimators=2, + ignore_pretraining_limits=True, + ) + model.train(x, y) + preds = model.predict(x) + if not np.isfinite(preds).all(): + raise RuntimeError("non-finite TabPFN predictions") + return f"pred_mean={preds.mean():.6f}" + + +def smoke_tabpfn3() -> str: + from alphapulse.models.foundation_models import TabPFN3Model + + x, y = _toy_float_data() + model = TabPFN3Model( + max_train_rows=80, + max_features=16, + compression="pca", + compression_components=8, + n_estimators=2, + ) + model.train(x, y) + preds = model.predict(x) + if not np.isfinite(preds).all(): + raise RuntimeError("non-finite TabPFN3 predictions") + return f"pred_mean={preds.mean():.6f}" + + +def smoke_tabicl() -> str: + from alphapulse.models.foundation_models import TabICLModel + + x, y = _toy_float_data() + model = TabICLModel( + max_train_rows=80, + max_features=16, + compression="pca", + compression_components=8, + n_estimators=2, + ) + model.train(x, y) + preds = model.predict(x) + if not np.isfinite(preds).all(): + raise RuntimeError("non-finite TabICL predictions") + return f"pred_mean={preds.mean():.6f}" + + +def main() -> int: + from alphapulse.hpo.search_space import available_foundation_models + from alphapulse.models.packboost_backend import packboost_cuda_available + + tests: list[tuple[str, Callable[[], str]]] = [] + if packboost_cuda_available(): + tests.append(("Packboost(CUDA)", smoke_packboost)) + else: + print("SKIP Packboost: CUDA/PackBoost unavailable") + + for model_name in available_foundation_models(): + if model_name == "TabPFN": + tests.append(("TabPFN", smoke_tabpfn)) + elif model_name == "TabPFN3": + tests.append(("TabPFN3", smoke_tabpfn3)) + elif model_name == "TabICL": + tests.append(("TabICL", smoke_tabicl)) + + results: list[SmokeResult] = [] + print(f"Running {len(tests)} smoke test(s)...") + for name, fn in tests: + print(f"\n--- {name} ---") + result = _run(name, fn) + results.append(result) + status = "PASS" if result.ok else "FAIL" + print( + f"{status} ({result.seconds:.1f}s) {result.detail.splitlines()[0] if result.detail else ''}" + ) + if not result.ok: + print(result.detail) + + passed = sum(1 for r in results if r.ok) + print(f"\n=== Summary: {passed}/{len(results)} passed ===") + for r in results: + mark = "OK" if r.ok else "FAIL" + print(f" [{mark}] {r.name} ({r.seconds:.1f}s)") + return 0 if passed == len(results) else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/update_wandb_hpo_report.py b/scripts/update_wandb_hpo_report.py new file mode 100644 index 0000000..8930671 --- /dev/null +++ b/scripts/update_wandb_hpo_report.py @@ -0,0 +1,183 @@ +"""Update the published AlphaPulse HPO W&B report in place.""" + +from __future__ import annotations + +import json +import sqlite3 +from pathlib import Path + +import wandb_workspaces.reports.v2 as wr + +REPORT_URL = ( + "https://wandb.ai/dsc-pjatk-warsaw/alphapulse-hpo-20260616-232741/reports/" + "AlphaPulse-HPO-Summary--VmlldzoxNzI1NDg1Ng==" +) +PROJECT = "alphapulse-hpo-20260616-232741" +TRIALS_DB = Path("artifacts/hpo_9h_v11/trials.db") + +METRICS_SECTION_TITLE = "Metric definitions (holdout vs validation)" +METRICS_SECTION_BODY = """\ +AlphaPulse HPO logs **two evaluation splits**. Do not compare columns across splits. + +| W&B / leaderboard name | Split | Meaning | +|---|---|---| +| `holdout/HoldoutSharpe` | train holdout | corr_sharpe on the last train eras | +| `holdout/HoldoutMeanCorr` | train holdout | mean per-era Spearman on holdout | +| `validation/ValidationSharpe` | validation | corr_sharpe on `validation.parquet` | +| `validation/ValidationMmcSharpe` | validation | MMC Sharpe with `meta_model.parquet` | +| `validation/ValidationMeanCorr` | validation | mean per-era Spearman on validation | +| `validation/PayoutScore` | validation | **HPO objective** | + +**Payout formula (Numerai-style):** + +`PayoutScore = 0.75 * ValidationSharpe + 2.25 * ValidationMmcSharpe` + +Leaderboard ranks by `PayoutScore` (validation). `HoldoutSharpe` is a separate generalization check on train. + +**Diagnostics charts** are prefixed by split: `diagnostics/holdout/...` vs `diagnostics/validation/...`. +""" + + +def _trial_stats(db_path: Path) -> tuple[int, int, int, dict[str, float | int | str]]: + conn = sqlite3.connect(db_path) + counts = dict( + conn.execute("SELECT status, COUNT(*) FROM trials GROUP BY status").fetchall() + ) + completed = int(counts.get("completed", 0)) + failed = int(counts.get("failed", 0)) + total = completed + failed + int(counts.get("running", 0)) + + best_payout: tuple | None = None + best_holdout: tuple | None = None + for tn, metrics_raw, fc_raw in conn.execute( + "SELECT trial_number, metrics, flat_config FROM trials WHERE status='completed'" + ): + metrics = json.loads(metrics_raw or "{}") + fc = json.loads(fc_raw or "{}") + payout = metrics.get("payout_score") + holdout = metrics.get("holdout_corr_sharpe", metrics.get("corr_sharpe")) + val_sh = metrics.get("val_corr_sharpe") + mmc = metrics.get("mmc_sharpe") + n = int(fc.get("num_models", 1)) + models = "+".join(str(fc.get(f"model_{i}_type", "?")) for i in range(1, n + 1)) + row = (int(tn), holdout, val_sh, payout, mmc, models) + if payout is not None and (best_payout is None or payout > best_payout[3]): + best_payout = row + if holdout is not None and (best_holdout is None or holdout > best_holdout[1]): + best_holdout = row + + if best_payout is None or best_holdout is None: + raise RuntimeError("No completed trials with metrics in trials.db") + + return ( + total, + completed, + failed, + { + "best_payout_trial": best_payout[0], + "best_payout": float(best_payout[3]), + "best_payout_val_sh": float(best_payout[2] or 0), + "best_payout_holdout_sh": float(best_payout[1] or 0), + "best_payout_mmc": float(best_payout[4] or 0), + "best_payout_models": best_payout[5], + "best_holdout_trial": best_holdout[0], + "best_holdout_sh": float(best_holdout[1] or 0), + "best_holdout_val_sh": float(best_holdout[2] or 0), + "best_holdout_models": best_holdout[5], + }, + ) + + +def _snapshot_markdown(total: int, completed: int, failed: int) -> str: + return ( + f"Generated from project `dsc-pjatk-warsaw/{PROJECT}`.\n\n" + f"This report summarizes the hyperparameter and ensemble search: " + f"**{total} trials started**, **{completed} completed**, **{failed} failed** " + f"at last snapshot." + ) + + +def _executive_summary_fixed(stats: dict[str, float | int | str], failed: int) -> str: + return ( + f"**Main finding:** `trial_{int(stats['best_holdout_trial']):03d}` " + f"(`{stats['best_holdout_models']}`) has the strongest holdout signal: " + f"`holdout/HoldoutSharpe = {float(stats['best_holdout_sh']):.3f}` and " + f"`validation/ValidationSharpe = {float(stats['best_holdout_val_sh']):.3f}`.\n\n" + f"**Payout leader:** `trial_{int(stats['best_payout_trial']):03d}` " + f"(`{stats['best_payout_models']}`) reaches " + f"`validation/PayoutScore = {float(stats['best_payout']):.3f}` " + f"with `validation/ValidationMmcSharpe = {float(stats['best_payout_mmc']):.3f}`, " + f"but `holdout/HoldoutSharpe = {float(stats['best_payout_holdout_sh']):.3f}`.\n\n" + f"**Operational caveat:** prioritize stability — failed trials often reflect " + f"TabPFN timeouts or GPU OOM; check `failed` count in the snapshot above." + ) + + +def _top_runs_table(stats: dict[str, float | int | str]) -> str: + hp_t = int(stats["best_holdout_trial"]) + pp_t = int(stats["best_payout_trial"]) + return f"""\ +| Rank lens | Best run | Model setup | Key metrics | Interpretation | +|---|---:|---|---|---| +| ValidationSharpe | `trial_{hp_t:03d}` | `{stats["best_holdout_models"]}` | ValidationSharpe `{float(stats["best_holdout_val_sh"]):.3f}`; HoldoutSharpe `{float(stats["best_holdout_sh"]):.3f}` | Best holdout generalization among completed runs. | +| HoldoutSharpe | `trial_{hp_t:03d}` | `{stats["best_holdout_models"]}` | HoldoutSharpe `{float(stats["best_holdout_sh"]):.3f}` | Same run — strongest train-era holdout Sharpe. | +| PayoutScore | `trial_{pp_t:03d}` | `{stats["best_payout_models"]}` | PayoutScore `{float(stats["best_payout"]):.3f}`; ValidationSharpe `{float(stats["best_payout_val_sh"]):.3f}`; HoldoutSharpe `{float(stats["best_payout_holdout_sh"]):.3f}` | Best validation payout objective; weak holdout Sharpe — tradeoff, not a single clear winner. | +""" + + +def _has_metrics_section(blocks: list) -> bool: + return any( + getattr(b, "text", None) == METRICS_SECTION_TITLE + for b in blocks + if type(b).__name__ == "H2" + ) + + +def main() -> None: + total, completed, failed, stats = _trial_stats(TRIALS_DB) + report = wr.Report.from_url(REPORT_URL) + + report.blocks[1].text = _snapshot_markdown(total, completed, failed) + report.blocks[3].text = _executive_summary_fixed(stats, failed) + report.blocks[5].text = ( + "The project is an HPO and ensemble search over tabular/predictive models. " + "Families include `TabPFN`, `TabICL`, `LightGBM`, `XGBoost`, `CatBoost`, and " + "`Packboost`, with ensemble modes `single`, `weighted`, and `stacking`.\n\n" + "Logged metrics use explicit holdout vs validation names (see next section). " + "The HPO objective is `validation/PayoutScore`." + ) + report.blocks[9].text = _top_runs_table(stats) + report.blocks[12].text = ( + "Diagnostics are split by evaluation lane:\n" + "- `diagnostics/holdout/...` — per-era correlation, drawdown, feature exposure, " + "SHAP/importance on train holdout\n" + "- `diagnostics/validation/...` — payout/MMC scalars and validation per-era charts\n\n" + "Artifacts include `run_table` and `wandb-history` collections per trial." + ) + + if not _has_metrics_section(report.blocks): + report.blocks.insert(6, wr.H2(text=METRICS_SECTION_TITLE)) + report.blocks.insert(7, wr.MarkdownBlock(text=METRICS_SECTION_BODY)) + + report.save(draft=False) + + verify = wr.Report.from_url(REPORT_URL) + texts = [] + for block in verify.blocks: + if hasattr(block, "text") and block.text: + texts.append(block.text) + joined = "\n".join(texts) + for needle in ( + METRICS_SECTION_TITLE, + "PayoutScore = 0.75 * ValidationSharpe + 2.25 * ValidationMmcSharpe", + "holdout/HoldoutSharpe", + f"trial_{int(stats['best_payout_trial']):03d}", + ): + if needle not in joined: + raise RuntimeError(f"Verification failed: missing {needle!r}") + print("Report updated and verified.") + print(REPORT_URL) + + +if __name__ == "__main__": + main() diff --git a/src/alphapulse/__about__.py b/src/alphapulse/__about__.py index fcfaa0e..e0c96d5 100644 --- a/src/alphapulse/__about__.py +++ b/src/alphapulse/__about__.py @@ -1,4 +1,4 @@ # SPDX-FileCopyrightText: 2025-present jakub.szulc # # SPDX-License-Identifier: MIT -__version__ = "0.5.0" +__version__ = "0.6.0" diff --git a/src/alphapulse/autoresearch/mutations.py b/src/alphapulse/autoresearch/mutations.py index 3e00636..6bf170e 100644 --- a/src/alphapulse/autoresearch/mutations.py +++ b/src/alphapulse/autoresearch/mutations.py @@ -15,7 +15,6 @@ "ExtraTrees", "TabPFN", "TabPFN3", - "TabPFN3Reasoning", "TabICL", ] VALID_PREPROCESSORS = [ diff --git a/src/alphapulse/evaluation/backtester.py b/src/alphapulse/evaluation/backtester.py index f920c7d..f3c58fb 100644 --- a/src/alphapulse/evaluation/backtester.py +++ b/src/alphapulse/evaluation/backtester.py @@ -17,10 +17,22 @@ def predict_with_optional_eras( predictor: PredictorProtocol, X: pd.DataFrame, era: pd.Series, + meta_model_preds: np.ndarray | None = None, ) -> np.ndarray: + meta_kw = ( + {"meta_model": meta_model_preds} + if meta_model_preds is not None + and float(getattr(predictor, "meta_neutralize_proportion", 0.0)) > 0.0 + else {} + ) if float(getattr(predictor, "neutralize_proportion", 0.0)) > 0.0: return np.asarray( - predictor.predict(X, eras=era), # type: ignore[call-arg] + predictor.predict(X, eras=era, **meta_kw), # type: ignore[call-arg] + dtype=np.float64, + ) + if meta_kw: + return np.asarray( + predictor.predict(X, **meta_kw), dtype=np.float64, ) return np.asarray(predictor.predict(X), dtype=np.float64) @@ -81,7 +93,9 @@ def evaluate( ``mmc_sharpe``, ``payout_score``, ``fnc_sharpe``. """ X_use = X[self.feature_columns] if self.feature_columns is not None else X - preds = predict_with_optional_eras(self.predictor, X_use, era) + preds = predict_with_optional_eras( + self.predictor, X_use, era, meta_model_preds=meta_model_preds + ) if self.neutralizer is not None: preds = self.neutralizer.neutralize(preds, X_use, era) @@ -100,7 +114,7 @@ def evaluate( mmc_weight=mmc_weight, ) - if meta_arr is not None: + if meta_arr is not None and np.isfinite(meta_arr).sum() >= 2: metrics["mmc"] = mmc_score(y, preds, meta_arr, era) if compute_fnc: diff --git a/src/alphapulse/evaluation/metrics.py b/src/alphapulse/evaluation/metrics.py index 3816099..be8e609 100644 --- a/src/alphapulse/evaluation/metrics.py +++ b/src/alphapulse/evaluation/metrics.py @@ -435,15 +435,16 @@ def calculate_metrics( } if meta_model_preds is not None: meta_arr = np.asarray(meta_model_preds, dtype=np.float64) - ms = era_sharpe_of_mmc(y_true, y_pred, meta_arr, eras) - result["mmc_sharpe"] = ms - ps = payout_score( - y_true, - y_pred, - meta_arr, - eras, - corr_weight=corr_weight, - mmc_weight=mmc_weight, - ) - result["payout_score"] = ps + if np.isfinite(meta_arr).sum() >= 2: + ms = era_sharpe_of_mmc(y_true, y_pred, meta_arr, eras) + result["mmc_sharpe"] = ms + ps = payout_score( + y_true, + y_pred, + meta_arr, + eras, + corr_weight=corr_weight, + mmc_weight=mmc_weight, + ) + result["payout_score"] = ps return result diff --git a/src/alphapulse/evaluation/shap_report.py b/src/alphapulse/evaluation/shap_report.py index 04de279..2bdd044 100644 --- a/src/alphapulse/evaluation/shap_report.py +++ b/src/alphapulse/evaluation/shap_report.py @@ -7,8 +7,7 @@ from ..models.base import _numeric from ..models.era_ensemble_model import EraEnsembleModel from ..models.xgboost_model import XGBoostModel -from ..pipeline.multihead import MultiHeadPipeline -from ..pipeline.pipeline import Pipeline +from ..pipeline.model_access import PipelineLike, iter_trained_models SHAP_SAMPLE_ROWS = 2000 @@ -25,16 +24,11 @@ def _wandb_active() -> bool: def _collect_by_type( - pipeline: Pipeline | MultiHeadPipeline, + pipeline: PipelineLike, model_class: type, ) -> _ModelList: results: _ModelList = [] - sources = ( - [h.model for h in pipeline.heads] - if isinstance(pipeline, MultiHeadPipeline) - else pipeline.models - ) - for m in sources: + for m in iter_trained_models(pipeline): if isinstance(m, model_class): results.append((m.name, m)) elif isinstance(m, EraEnsembleModel): @@ -44,23 +38,23 @@ def _collect_by_type( return results -def _collect_xgboost_models(pipeline: Pipeline | MultiHeadPipeline) -> _ModelList: +def _collect_xgboost_models(pipeline: PipelineLike) -> _ModelList: return _collect_by_type(pipeline, XGBoostModel) -def _collect_lgbm_models(pipeline: Pipeline | MultiHeadPipeline) -> _ModelList: +def _collect_lgbm_models(pipeline: PipelineLike) -> _ModelList: from ..models.lightgbm_model import LightGBMModel return _collect_by_type(pipeline, LightGBMModel) -def _collect_catboost_models(pipeline: Pipeline | MultiHeadPipeline) -> _ModelList: +def _collect_catboost_models(pipeline: PipelineLike) -> _ModelList: from ..models.catboost_model import CatBoostModel return _collect_by_type(pipeline, CatBoostModel) -def _collect_sklearn_tree_models(pipeline: Pipeline | MultiHeadPipeline) -> _ModelList: +def _collect_sklearn_tree_models(pipeline: PipelineLike) -> _ModelList: from ..models.sklearn_models import ExtraTreesModel, RandomForestModel results: _ModelList = [] @@ -141,7 +135,7 @@ def _aggregate_importance( def compute_universal_feature_importance( - pipeline: Pipeline | MultiHeadPipeline, + pipeline: PipelineLike, X: pd.DataFrame, *, feature_cols: list[str], @@ -218,11 +212,12 @@ def compute_universal_feature_importance( def log_universal_feature_importance( - pipeline: Pipeline | MultiHeadPipeline, + pipeline: PipelineLike, X: pd.DataFrame, *, feature_cols: list[str], top_n: int = 20, + diagnostics_prefix: str = "diagnostics/holdout", ) -> dict[str, float]: """Log universal feature importance for all supported model types to WandB. @@ -246,20 +241,20 @@ def log_universal_feature_importance( if not importance: return {} - table = wandb.Table(columns=["feature", "mean_abs_contribution"]) - for feature, score in importance.items(): - table.add_data(feature, score) - wandb.log( - { - "diagnostics/feature_importance_top": table, - "diagnostics/feature_importance_bar": wandb.plot.bar( - table, - "feature", - "mean_abs_contribution", - title=f"Top feature importance ({model_type})", - ), - } + from .wandb_diagnostics import _log_horizontal_bar_chart + + features = list(importance.keys()) + scores = [float(importance[f]) for f in features] + _log_horizontal_bar_chart( + wandb, + labels=features, + values=scores, + key=f"{diagnostics_prefix}/feature_importance_bar", + title=f"Top feature importance ({model_type})", + xlabel="Mean |contribution|", ) if wandb.run is not None: - wandb.run.summary["diagnostics/feature_importance_model_type"] = model_type + wandb.run.summary[f"{diagnostics_prefix}/feature_importance_model_type"] = ( + model_type + ) return importance diff --git a/src/alphapulse/evaluation/wandb_diagnostics.py b/src/alphapulse/evaluation/wandb_diagnostics.py index 329506e..11de613 100644 --- a/src/alphapulse/evaluation/wandb_diagnostics.py +++ b/src/alphapulse/evaluation/wandb_diagnostics.py @@ -6,15 +6,24 @@ import pandas as pd from ..evaluation.metrics import per_era_correlation, rank_normalize -from ..pipeline.multihead import MultiHeadPipeline +from ..pipeline.model_access import ( + PipelineLike, + model_prediction_map, + multitarget_blend_weights, +) +from ..pipeline.multi_target import MultiTargetPipeline from ..pipeline.pipeline import Pipeline -from ..pipeline.row_utils import protected_metadata_frame -MAX_SCATTER_POINTS = 5000 MAX_HEXBIN_POINTS = 10_000 FEATURE_EXPOSURE_TOP_N = 15 MAX_FNC_FEATURES = 200 _ERA_IMPORTANCE_MIN_ROWS = 10 +_PRED_PLOT_BINS = 20 +_PRED_PLOT_DPI = 150 + + +def _diag_key(split: str, name: str) -> str: + return f"diagnostics/{split}/{name}" def _wandb_active() -> bool: @@ -26,23 +35,135 @@ def _wandb_active() -> bool: return False +def _log_wandb_figure(wandb: Any, key: str, fig: Any) -> None: + import io + + from PIL import Image + + buf = io.BytesIO() + fig.savefig(buf, format="png", dpi=_PRED_PLOT_DPI, bbox_inches="tight") + buf.seek(0) + wandb.log({key: wandb.Image(Image.open(buf))}) + + +def _new_figure(*, figsize: tuple[float, float]) -> Any: + import matplotlib as mpl + from matplotlib.backends.backend_agg import FigureCanvasAgg + from matplotlib.figure import Figure + + mpl.use("Agg", force=True) + fig = Figure(figsize=figsize, dpi=_PRED_PLOT_DPI) + FigureCanvasAgg(fig) + return fig + + +def _log_horizontal_bar_chart( + wandb: Any, + *, + labels: list[str], + values: list[float], + key: str, + title: str, + xlabel: str, +) -> None: + if not labels: + return + + n = len(labels) + fig_h = max(3.5, min(14.0, n * 0.32)) + fig = _new_figure(figsize=(9, fig_h)) + ax = fig.add_subplot(111) + y = np.arange(n) + ax.barh(y, values, color="steelblue", edgecolor="white", height=0.75) + ax.set_yticks(y, labels=labels, fontsize=8) + ax.invert_yaxis() + ax.set_xlabel(xlabel) + ax.set_title(title) + fig.tight_layout() + _log_wandb_figure(wandb, key, fig) + + +def _log_correlation_heatmap( + wandb: Any, + names: list[str], + corr: dict[str, dict[str, float]], + key: str, + *, + title: str, +) -> None: + if len(names) < 2: + return + + mat = np.array([[float(corr[a][b]) for b in names] for a in names]) + size = max(5.0, min(12.0, len(names) * 0.65)) + fig = _new_figure(figsize=(size, size)) + ax = fig.add_subplot(111) + im = ax.imshow(mat, cmap="RdBu_r", vmin=-1.0, vmax=1.0, aspect="auto") + ax.set_xticks(range(len(names))) + ax.set_yticks(range(len(names))) + ax.set_xticklabels(names, rotation=45, ha="right", fontsize=8) + ax.set_yticklabels(names, fontsize=8) + for i in range(len(names)): + for j in range(len(names)): + ax.text( + j, + i, + f"{mat[i, j]:.2f}", + ha="center", + va="center", + fontsize=7, + color="white" if abs(mat[i, j]) > 0.5 else "black", + ) + fig.colorbar(im, ax=ax, fraction=0.046, pad=0.04) + ax.set_title(title) + fig.tight_layout() + _log_wandb_figure(wandb, key, fig) + + +def _log_mmc_metrics_from_dict( + wandb: Any, metrics: dict[str, float], *, split: str = "validation" +) -> None: + logged: dict[str, float] = {} + scalar_names = { + "mmc": "ValidationMmc", + "mmc_sharpe": "ValidationMmcSharpe", + "payout_score": "PayoutScore", + "corr_sharpe": "ValidationSharpe", + "mean_per_era_correlation": "ValidationMeanCorr", + } + for key, wandb_name in scalar_names.items(): + value = metrics.get(key) + if key == "corr_sharpe" and metrics.get("val_corr_sharpe") is not None: + value = metrics.get("val_corr_sharpe") + if ( + key == "mean_per_era_correlation" + and metrics.get("val_mean_per_era_correlation") is not None + ): + value = metrics.get("val_mean_per_era_correlation") + if value is not None and np.isfinite(value): + logged[_diag_key(split, wandb_name)] = float(value) + if logged: + wandb.log(logged) + + +def _subsample_finite_pairs( + y: np.ndarray, p: np.ndarray, *, max_points: int, seed: int = 0 +) -> tuple[np.ndarray, np.ndarray]: + mask = np.isfinite(y) & np.isfinite(p) + y = y[mask] + p = p[mask] + if len(y) <= max_points: + return y, p + idx = np.random.default_rng(seed).choice(len(y), size=max_points, replace=False) + return y[idx], p[idx] + + def _collect_model_predictions( - pipeline: Pipeline | MultiHeadPipeline, + pipeline: PipelineLike, X_val: pd.DataFrame, feature_cols: list[str], ) -> dict[str, np.ndarray]: - if isinstance(pipeline, MultiHeadPipeline): - preds = pipeline.predict(X_val[feature_cols] if feature_cols else X_val) - return {"ensemble": preds} - - X_feat = X_val[feature_cols] if feature_cols else X_val - era_meta = protected_metadata_frame(X_feat) - X_t = pipeline._preprocess(X_feat, era_meta) - X_numeric = X_t.select_dtypes(include=[np.number]) - - if len(pipeline.models) == 1: - return {pipeline.models[0].name: pipeline.models[0].predict(X_numeric)} - return {m.name: m.predict(X_numeric) for m in pipeline.models} + return model_prediction_map(pipeline, X_val, feature_cols) def _feature_exposure_summary( @@ -89,7 +210,7 @@ def _feature_exposure_summary( def log_experiment_diagnostics( *, - pipeline: Pipeline | MultiHeadPipeline, + pipeline: PipelineLike, X_val: pd.DataFrame, y_val: pd.Series, era_val: pd.Series, @@ -100,22 +221,24 @@ def log_experiment_diagnostics( log_feature_report: bool = True, log_era_importance: bool = False, compute_fnc: bool | None = None, + split: str = "holdout", ) -> None: """Log comprehensive XAI and backtest diagnostics to the active WandB run. Args: pipeline: Trained pipeline. - X_val: Validation features (may include era column). - y_val: Validation targets. + X_val: Evaluation features (may include era column). + y_val: Evaluation targets. era_val: Era labels aligned with X_val. feature_cols: Feature column names (must not include "era"). - metrics: Backtest metrics dict. + metrics: Backtest metrics dict for this *split* (holdout or validation). meta_model_preds: Optional meta-model predictions for MMC logging. log_shap: If True, log universal feature importance (all model types). log_feature_report: If True, log per-era stability report via LightGBM proxy. log_era_importance: If True, log era-stratified importance from pipeline models (expensive — recommended only for best-trial diagnostics). compute_fnc: Whether to log FNC. Auto-detected from feature count when None. + split: Data split label used in W&B keys (`holdout` or `validation`). """ if not _wandb_active(): return @@ -127,46 +250,58 @@ def log_experiment_diagnostics( eras_for_predict = era_val if neutralize > 0 else None if isinstance(pipeline, Pipeline): preds = pipeline.predict(X_use, eras=eras_for_predict) + elif isinstance(pipeline, MultiTargetPipeline): + preds = pipeline.predict(X_use) else: preds = pipeline.predict(X_val[feature_cols] if feature_cols else X_val) - _log_per_era_correlation(y_val, preds, era_val) - _log_prediction_diagnostics(y_val, preds) - _log_feature_exposure(preds, X_use, era_val) + _log_per_era_correlation(y_val, preds, era_val, split=split) + _log_prediction_diagnostics(y_val, preds, split=split) + _log_feature_exposure(preds, X_use, era_val, split=split) if isinstance(pipeline, Pipeline) and len(pipeline.models) > 1: - _log_ensemble_diagnostics(pipeline, X_val, feature_cols, y_val, era_val) - - if meta_model_preds is not None: - wandb.log( - { - "diagnostics/mmc_sharpe": metrics.get("mmc_sharpe"), - "diagnostics/payout_score": metrics.get("payout_score"), - } + _log_ensemble_diagnostics( + pipeline, X_val, feature_cols, y_val, era_val, split=split + ) + elif isinstance(pipeline, MultiTargetPipeline) and len(pipeline._models) > 1: + _log_ensemble_diagnostics( + pipeline, X_val, feature_cols, y_val, era_val, split=split ) + if split == "validation" and ( + meta_model_preds is not None + or any(k in metrics for k in ("mmc", "mmc_sharpe", "payout_score")) + ): + _log_mmc_metrics_from_dict(wandb, metrics, split=split) + use_fnc = compute_fnc if use_fnc is None: use_fnc = len(feature_cols) <= MAX_FNC_FEATURES if use_fnc and "fnc_sharpe" in metrics: - wandb.log({"diagnostics/fnc_sharpe": metrics["fnc_sharpe"]}) + wandb.log({_diag_key(split, "fnc_sharpe"): metrics["fnc_sharpe"]}) if log_shap: from ..evaluation.shap_report import log_universal_feature_importance log_universal_feature_importance( - pipeline, X_use, feature_cols=feature_cols, top_n=20 + pipeline, + X_use, + feature_cols=feature_cols, + top_n=20, + diagnostics_prefix=f"diagnostics/{split}", ) if log_feature_report: - _log_feature_report(X_use, y_val, era_val, feature_cols) + _log_feature_report(X_use, y_val, era_val, feature_cols, split=split) if log_era_importance: - _log_era_stratified_importance(pipeline, X_use, feature_cols, era_val) + _log_era_stratified_importance( + pipeline, X_use, feature_cols, era_val, split=split + ) def _log_per_era_correlation( - y_val: pd.Series, preds: np.ndarray, era_val: pd.Series + y_val: pd.Series, preds: np.ndarray, era_val: pd.Series, *, split: str ) -> None: import wandb @@ -197,23 +332,26 @@ def _log_per_era_correlation( float(drawdown.loc[era]), ) + split_label = "train holdout" if split == "holdout" else "validation" wandb.log( { - "diagnostics/per_era_correlation_table": table, - "diagnostics/per_era_correlation": wandb.plot.line( - table, "era_index", "correlation", title="Per-era Spearman correlation" + _diag_key(split, "per_era_correlation"): wandb.plot.line( + table, + "era_index", + "correlation", + title=f"Per-era Spearman correlation ({split_label})", ), - "diagnostics/cumulative_correlation": wandb.plot.line( + _diag_key(split, "cumulative_correlation"): wandb.plot.line( table, "era_index", "cumulative_correlation", - title="Cumulative per-era correlation", + title=f"Cumulative per-era correlation ({split_label})", ), - "diagnostics/drawdown_curve": wandb.plot.line( + _diag_key(split, "drawdown_curve"): wandb.plot.line( table, "era_index", "drawdown", - title="Drawdown from peak cumulative correlation", + title=f"Drawdown from peak cumulative correlation ({split_label})", ), } ) @@ -228,119 +366,155 @@ def _log_per_era_correlation( dist_table.add_data(float(m), int(c)) wandb.log( { - "diagnostics/corr_distribution": wandb.plot.bar( + _diag_key(split, "corr_distribution"): wandb.plot.bar( dist_table, "bin_center", "count", - title="Distribution of per-era correlations", + title=f"Distribution of per-era correlations ({split_label})", ) } ) -def _log_prediction_diagnostics(y_val: pd.Series, preds: np.ndarray) -> None: +def _log_prediction_diagnostics( + y_val: pd.Series, preds: np.ndarray, *, split: str +) -> None: import wandb + split_label = "train holdout" if split == "holdout" else "validation" ranked = rank_normalize(preds) finite_ranked = ranked[np.isfinite(ranked)] - counts, edges = np.histogram(finite_ranked, bins=50) - midpoints = 0.5 * (edges[:-1] + edges[1:]) - hist_table = wandb.Table(columns=["bin_center", "count"]) - for mid, cnt in zip(midpoints, counts, strict=False): - hist_table.add_data(float(mid), int(cnt)) - wandb.log( - { - "diagnostics/prediction_histogram": wandb.plot.bar( - hist_table, - "bin_center", - "count", - title="Rank-normalized prediction distribution (50 bins)", - ) - } - ) - - n = min(len(y_val), MAX_SCATTER_POINTS) - if n < len(y_val): - idx = np.random.default_rng(0).choice(len(y_val), size=n, replace=False) - y_sample = y_val.iloc[idx] - p_sample = preds[idx] - else: - y_sample = y_val - p_sample = preds - - scatter = wandb.Table(columns=["target", "prediction"]) - for yt, pp in zip(y_sample, p_sample, strict=False): - if np.isfinite(yt) and np.isfinite(pp): - scatter.add_data(float(yt), float(pp)) - wandb.log( - { - "diagnostics/pred_vs_target_scatter": wandb.plot.scatter( - scatter, - "target", - "prediction", - title="Predictions vs target (sampled)", - ) - } + if len(finite_ranked): + fig = _new_figure(figsize=(8, 4)) + ax = fig.add_subplot(111) + ax.hist( + finite_ranked, + bins=_PRED_PLOT_BINS, + range=(0.0, 1.0), + color="steelblue", + edgecolor="white", + alpha=0.9, + ) + uniform_ref = len(finite_ranked) / _PRED_PLOT_BINS + ax.axhline( + uniform_ref, + color="tomato", + linestyle="--", + linewidth=1.2, + label="uniform reference", + ) + ax.set_xlim(0.0, 1.0) + ax.set_xlabel("Rank-normalized prediction") + ax.set_ylabel("Count") + ax.set_title(f"Prediction distribution ({split_label})") + ax.legend(loc="upper right", fontsize=9) + fig.tight_layout() + _log_wandb_figure(wandb, _diag_key(split, "prediction_histogram"), fig) + + y_arr = y_val.to_numpy(dtype=np.float64) + p_arr = np.asarray(preds, dtype=np.float64) + y_plot, p_plot = _subsample_finite_pairs( + y_arr, p_arr, max_points=MAX_HEXBIN_POINTS, seed=0 ) - - residuals = y_val.to_numpy(dtype=np.float64) - preds + if len(y_plot): + y_lo, y_hi = np.percentile(y_plot, [1, 99]) + p_lo, p_hi = np.percentile(p_plot, [1, 99]) + if y_lo == y_hi: + y_lo, y_hi = y_lo - 1e-6, y_hi + 1e-6 + if p_lo == p_hi: + p_lo, p_hi = p_lo - 1e-6, p_hi + 1e-6 + + fig = _new_figure(figsize=(12, 4.5)) + axes = [fig.add_subplot(1, 2, i + 1) for i in range(2)] + hb = axes[0].hexbin( + y_plot, + p_plot, + gridsize=35, + cmap="Blues", + mincnt=1, + extent=(y_lo, y_hi, p_lo, p_hi), + ) + fig.colorbar(hb, ax=axes[0], label="count") + axes[0].set_xlabel("Target (1–99 pct)") + axes[0].set_ylabel("Raw prediction (1–99 pct)") + axes[0].set_title(f"Pred vs target — hexbin (n={len(y_plot):,})") + + step = max(1, len(y_plot) // 2500) + axes[1].scatter( + y_plot[::step], + p_plot[::step], + alpha=0.2, + s=6, + c="steelblue", + edgecolors="none", + rasterized=True, + ) + axes[1].set_xlim(y_lo, y_hi) + axes[1].set_ylim(p_lo, p_hi) + axes[1].set_xlabel("Target (1–99 pct)") + axes[1].set_ylabel("Raw prediction (1–99 pct)") + axes[1].set_title("Pred vs target — subsampled scatter") + fig.tight_layout() + _log_wandb_figure(wandb, _diag_key(split, "pred_vs_target_scatter"), fig) + + residuals = y_arr - p_arr finite = residuals[np.isfinite(residuals)] if len(finite): wandb.log( { - "diagnostics/residual_mean": float(np.mean(finite)), - "diagnostics/residual_std": float(np.std(finite, ddof=0)), - "diagnostics/residual_mae": float(np.mean(np.abs(finite))), + _diag_key(split, "residual_mean"): float(np.mean(finite)), + _diag_key(split, "residual_std"): float(np.std(finite, ddof=0)), + _diag_key(split, "residual_mae"): float(np.mean(np.abs(finite))), } ) def _log_feature_exposure( - preds: np.ndarray, features: pd.DataFrame, eras: pd.Series + preds: np.ndarray, features: pd.DataFrame, eras: pd.Series, *, split: str ) -> None: import wandb + split_label = "train holdout" if split == "holdout" else "validation" summary = _feature_exposure_summary(preds, features, eras) wandb.log( { - "diagnostics/feature_exposure_max": summary["max_mean_abs_corr"], - "diagnostics/feature_exposure_mean": summary["mean_abs_corr"], + _diag_key(split, "feature_exposure_max"): summary["max_mean_abs_corr"], + _diag_key(split, "feature_exposure_mean"): summary["mean_abs_corr"], } ) if summary["top"]: - table = wandb.Table(columns=["feature", "mean_abs_corr"]) - for row in summary["top"]: - table.add_data(row["feature"], row["mean_abs_corr"]) - wandb.log( - { - "diagnostics/feature_exposure_top": table, - "diagnostics/feature_exposure_bar": wandb.plot.bar( - table, - "feature", - "mean_abs_corr", - title="Feature exposure (top 15 by mean |corr| with predictions)", - ), - } + _log_horizontal_bar_chart( + wandb, + labels=[row["feature"] for row in summary["top"]], + values=[row["mean_abs_corr"] for row in summary["top"]], + key=_diag_key(split, "feature_exposure_bar"), + title=f"Feature exposure ({split_label}, top 15)", + xlabel="Mean |corr| with predictions", ) def _log_ensemble_diagnostics( - pipeline: Pipeline | MultiHeadPipeline, + pipeline: PipelineLike, X_val: pd.DataFrame, feature_cols: list[str], y_val: pd.Series, era_val: pd.Series, + *, + split: str, ) -> None: import wandb from ..evaluation.ensemble_diagnostics import compute_ensemble_diagnostics + split_label = "train holdout" if split == "holdout" else "validation" oof = _collect_model_predictions(pipeline, X_val, feature_cols) weights = None if isinstance(pipeline, Pipeline) and pipeline.ensemble_method == "weighted": w = pipeline._ensemble.params.get("weights") if w is not None: weights = np.asarray(w, dtype=np.float64) + elif isinstance(pipeline, MultiTargetPipeline): + weights = multitarget_blend_weights(pipeline) diag = compute_ensemble_diagnostics( oof, @@ -350,33 +524,37 @@ def _log_ensemble_diagnostics( ) wandb.log( { - "diagnostics/effective_model_count": diag["effective_model_count"], - "diagnostics/mean_pairwise_correlation": diag["mean_pairwise_correlation"], + _diag_key(split, "effective_model_count"): diag["effective_model_count"], + _diag_key(split, "mean_pairwise_correlation"): diag[ + "mean_pairwise_correlation" + ], } ) names = diag["model_names"] corr = diag["correlation_matrix"] - table = wandb.Table(columns=["model_a", "model_b", "correlation"]) - for i, a in enumerate(names): - for j, b in enumerate(names): - if j >= i: - table.add_data(a, b, corr[a][b]) - - pair_table = wandb.Table(columns=["pair", "correlation"]) - for i, a in enumerate(names): - for j, b in enumerate(names): - if j > i: - pair_table.add_data(f"{a}→{b}", corr[a][b]) - - logged: dict[str, Any] = {"diagnostics/ensemble_correlation_matrix": table} + _log_correlation_heatmap( + wandb, + names, + corr, + _diag_key(split, "ensemble_correlation_heatmap"), + title=f"Model prediction correlations ({split_label})", + ) if len(names) > 1: - logged["diagnostics/ensemble_correlation_bar"] = wandb.plot.bar( - pair_table, - "pair", - "correlation", - title="Model pair correlations (lower = more diverse ensemble)", + pairs: list[str] = [] + pair_corrs: list[float] = [] + for i, a in enumerate(names): + for j, b in enumerate(names): + if j > i: + pairs.append(f"{a} → {b}") + pair_corrs.append(float(corr[a][b])) + _log_horizontal_bar_chart( + wandb, + labels=pairs, + values=pair_corrs, + key=_diag_key(split, "ensemble_correlation_bar"), + title=f"Model pair correlations ({split_label})", + xlabel="Spearman correlation", ) - wandb.log(logged) def _log_feature_report( @@ -385,12 +563,13 @@ def _log_feature_report( era_val: pd.Series, feature_cols: list[str], *, + split: str, top_n: int = 20, ) -> None: """Log per-era feature stability report (LightGBM proxy) to WandB. - Calls compute_feature_report and logs three tables: top features by mean - importance, top features by era stability, and worst features by stability. + Calls compute_feature_report and logs horizontal bar charts for mean + importance, most stable features, and least stable features. Silently skips if lightgbm is not installed. """ if not _wandb_active(): @@ -409,67 +588,47 @@ def _log_feature_report( except Exception: return - wandb.log({"diagnostics/feature_n_eras_used": report["n_eras_used"]}) + split_label = "train holdout" if split == "holdout" else "validation" + wandb.log({_diag_key(split, "feature_n_eras_used"): report["n_eras_used"]}) if report["top_by_mean"]: - table_mean = wandb.Table(columns=["feature", "mean_importance"]) - for row in report["top_by_mean"]: - table_mean.add_data(row["feature"], row["mean_importance"]) - wandb.log( - { - "diagnostics/feature_top_by_mean": table_mean, - "diagnostics/feature_importance_mean_bar": wandb.plot.bar( - table_mean, - "feature", - "mean_importance", - title="Top features by mean importance (LightGBM proxy, per era)", - ), - } + _log_horizontal_bar_chart( + wandb, + labels=[row["feature"] for row in report["top_by_mean"]], + values=[row["mean_importance"] for row in report["top_by_mean"]], + key=_diag_key(split, "feature_importance_mean_bar"), + title=f"Top features by mean importance ({split_label})", + xlabel="Mean importance", ) if report["top_by_stability"]: - table_stab = wandb.Table(columns=["feature", "stability", "mean_importance"]) - for row in report["top_by_stability"]: - table_stab.add_data( - row["feature"], row["stability"], row["mean_importance"] - ) - wandb.log( - { - "diagnostics/feature_top_by_stability": table_stab, - "diagnostics/feature_stability_bar": wandb.plot.bar( - table_stab, - "feature", - "stability", - title="Most stable features across eras (mean/std ratio)", - ), - } + _log_horizontal_bar_chart( + wandb, + labels=[row["feature"] for row in report["top_by_stability"]], + values=[row["stability"] for row in report["top_by_stability"]], + key=_diag_key(split, "feature_stability_bar"), + title=f"Most stable features ({split_label})", + xlabel="Stability", ) if report["bottom_by_stability"]: - table_worst = wandb.Table(columns=["feature", "stability", "mean_importance"]) - for row in report["bottom_by_stability"]: - table_worst.add_data( - row["feature"], row["stability"], row["mean_importance"] - ) - wandb.log( - { - "diagnostics/feature_worst_stability": table_worst, - "diagnostics/feature_worst_stability_bar": wandb.plot.bar( - table_worst, - "feature", - "stability", - title="Least stable features across eras (worst to prune)", - ), - } + _log_horizontal_bar_chart( + wandb, + labels=[row["feature"] for row in report["bottom_by_stability"]], + values=[row["stability"] for row in report["bottom_by_stability"]], + key=_diag_key(split, "feature_worst_stability_bar"), + title=f"Least stable features ({split_label})", + xlabel="Stability", ) def _log_era_stratified_importance( - pipeline: Pipeline | MultiHeadPipeline, + pipeline: PipelineLike, X_val: pd.DataFrame, feature_cols: list[str], era_val: pd.Series, *, + split: str, top_n: int = 20, max_eras: int = 30, ) -> None: @@ -528,34 +687,25 @@ def _log_era_stratified_importance( std_imp = imp_matrix.std(axis=0, ddof=0) stability = mean_imp / (std_imp + 1e-10) - stab_table = wandb.Table( - columns=["feature", "mean_importance", "std_importance", "stability"] - ) - for feat, mean_v, std_v, stab_v in zip( - all_features, mean_imp, std_imp, stability, strict=False - ): - stab_table.add_data(feat, float(mean_v), float(std_v), float(stab_v)) - wandb.log( - { - "diagnostics/era_importance_stability": stab_table, - "diagnostics/era_importance_stability_bar": wandb.plot.bar( - stab_table, - "feature", - "stability", - title="Era-stratified importance stability (mean/std)", - ), - } + split_label = "train holdout" if split == "holdout" else "validation" + _log_horizontal_bar_chart( + wandb, + labels=all_features, + values=[float(v) for v in stability], + key=_diag_key(split, "era_importance_stability_bar"), + title=f"Era-stratified importance stability ({split_label})", + xlabel="Stability", ) xs = list(range(len(era_labels))) ys = [[float(imp.get(f, 0.0)) for imp in era_imps] for f in all_features] wandb.log( { - "diagnostics/era_importance_over_time": wandb.plot.line_series( + _diag_key(split, "era_importance_over_time"): wandb.plot.line_series( xs=xs, ys=ys, keys=all_features, - title="Feature importance across eras (each line = one feature)", + title=f"Feature importance across eras ({split_label})", xname="era_index", ), } diff --git a/src/alphapulse/experiments/data.py b/src/alphapulse/experiments/data.py index 24434ec..bc61d55 100644 --- a/src/alphapulse/experiments/data.py +++ b/src/alphapulse/experiments/data.py @@ -1,9 +1,20 @@ import json from pathlib import Path +import numpy as np import pandas as pd -META_MODEL_COLUMN_CANDIDATES = ("meta_model", "prediction", "meta") +META_MODEL_COLUMN = "numerai_meta_model" + + +def meta_model_from_benchmarks( + live_benchmark_models: pd.DataFrame, + index: pd.Index, +) -> np.ndarray | None: + if META_MODEL_COLUMN not in live_benchmark_models.columns: + return None + aligned = live_benchmark_models[META_MODEL_COLUMN].reindex(index) + return np.asarray(aligned.to_numpy(dtype=np.float64), dtype=np.float64) def load_meta_model_series( @@ -21,24 +32,66 @@ def load_meta_model_series( return None meta_df = pd.read_parquet(path) - value_col = next( - (c for c in META_MODEL_COLUMN_CANDIDATES if c in meta_df.columns), - None, - ) - if value_col is None: - numeric_cols = meta_df.select_dtypes(include=["number"]).columns - if len(numeric_cols) == 0: - return None - value_col = str(numeric_cols[0]) + if META_MODEL_COLUMN not in meta_df.columns: + return None if "id" in meta_df.columns: - aligned = meta_df.set_index("id")[value_col] + aligned = meta_df.set_index("id")[META_MODEL_COLUMN] return aligned.reindex(index) - if meta_df.index.equals(index): - return meta_df[value_col] - if len(meta_df) == len(index): - return pd.Series(meta_df[value_col].to_numpy(), index=index) - return meta_df[value_col].reindex(index) + if meta_df.index.name == "id": + return meta_df[META_MODEL_COLUMN].reindex(index) + return None + + +def load_mmc_validation_frame( + data_dir: Path, + *, + feature_cols: list[str], + target_col: str, + train_subsample: float = 1.0, + seed: int = 42, +) -> tuple[pd.DataFrame, pd.Series, pd.Series, np.ndarray] | None: + """Load validation rows with aligned Numerai meta-model predictions for MMC scoring. + + ``meta_model.parquet`` covers validation ids only (not train), so MMC during HPO + must be evaluated on this split rather than on train-era holdout. + """ + meta_path = data_dir / "meta_model.parquet" + val_path = data_dir / "validation.parquet" + if not meta_path.exists() or not val_path.exists(): + return None + + meta_df = pd.read_parquet(meta_path) + if META_MODEL_COLUMN not in meta_df.columns: + return None + + read_cols = list(dict.fromkeys([*feature_cols, target_col, "era"])) + val_df = pd.read_parquet(val_path, columns=read_cols) + common_idx = val_df.index.intersection(meta_df.index) + if common_idx.empty: + return None + + val_df = val_df.loc[common_idx] + meta_series = meta_df[META_MODEL_COLUMN].reindex(common_idx) + valid_meta = meta_series.notna() + if not valid_meta.any(): + return None + val_df = val_df.loc[valid_meta] + meta_series = meta_series.loc[valid_meta] + + if not 0.0 < train_subsample <= 1.0: + raise ValueError(f"train_subsample must be in (0, 1], got {train_subsample}") + if train_subsample < 1.0: + val_df = val_df.sample(frac=train_subsample, random_state=seed) + meta_series = meta_series.loc[val_df.index] + + X_val = val_df[feature_cols + (["era"] if "era" in val_df.columns else [])] + return ( + X_val, + val_df[target_col], + val_df["era"], + meta_series.to_numpy(dtype=np.float64), + ) def load_feature_names( @@ -105,6 +158,25 @@ def resolve_feature_columns( ] +def load_validation_frames( + data_dir: Path, + target_col: str, + feature_cols: list[str], + *, + need_era: bool, +) -> tuple[pd.DataFrame, pd.Series, pd.Series]: + val_path = Path(data_dir) / "validation.parquet" + if not val_path.exists(): + raise FileNotFoundError( + f"Expected {val_path}. Run scripts/download_dataset.py first." + ) + + read_cols = list(dict.fromkeys(feature_cols + [target_col, "era"])) + val_df = pd.read_parquet(val_path, columns=read_cols) + x_cols = feature_cols + (["era"] if need_era else []) + return val_df[x_cols], val_df[target_col], val_df["era"] + + def load_train_val_frames( data_dir: Path, train_subsample: float, @@ -114,49 +186,19 @@ def load_train_val_frames( need_era: bool, benchmark_columns: list[str] | None = None, ) -> tuple[pd.DataFrame, pd.Series, pd.DataFrame, pd.Series, pd.Series, list[str]]: - if not 0.0 < train_subsample <= 1.0: - raise ValueError(f"train_subsample must be in (0, 1], got {train_subsample}") - - train_path = data_dir / "train.parquet" - val_path = data_dir / "validation.parquet" - if not train_path.exists() or not val_path.exists(): - raise FileNotFoundError( - f"Expected {train_path} and {val_path}. " - "Run scripts/download_dataset.py first." - ) - - excluded = set(benchmark_columns or []) - feature_names = feature_columns or load_feature_names(data_dir) - extra_cols = [target_col] + (["era"] if need_era else []) - if feature_names: - read_cols = list(dict.fromkeys(feature_names + extra_cols + ["era"])) - train_df = pd.read_parquet(train_path, columns=read_cols) - val_df = pd.read_parquet(val_path, columns=read_cols) - cols = [c for c in feature_names if c in train_df.columns and c not in excluded] - else: - train_df = pd.read_parquet(train_path) - val_df = pd.read_parquet(val_path) - cols = resolve_feature_columns( - train_df, data_dir, feature_columns, benchmark_columns - ) - - if not cols: - raise ValueError("No feature columns resolved.") - cols = [c for c in cols if c not in {"era", "id"}] - if not cols: - raise ValueError("No feature columns resolved after excluding metadata.") - read_cols = cols + (["era"] if need_era else []) - train_df = train_df.sample(frac=train_subsample, random_state=seed) - if "era" in train_df.columns: - train_df = train_df.sort_values("era", kind="mergesort") - return ( - train_df[read_cols], - train_df[target_col], - val_df[read_cols], - val_df[target_col], - val_df["era"], - cols, + X_train, y_train, feature_cols = load_train_only_frame( + data_dir, + train_subsample=train_subsample, + target_col=target_col, + seed=seed, + feature_columns=feature_columns, + need_era=need_era, + benchmark_columns=benchmark_columns, + ) + X_val, y_val, era_val = load_validation_frames( + data_dir, target_col, feature_cols, need_era=need_era ) + return X_train, y_train, X_val, y_val, era_val, feature_cols def load_train_only_frame( @@ -168,6 +210,7 @@ def load_train_only_frame( need_era: bool, feature_set: str | None = None, benchmark_columns: list[str] | None = None, + extra_target_columns: list[str] | None = None, ) -> tuple[pd.DataFrame, pd.Series, list[str]]: """Load only train.parquet (no validation) with column pruning to reduce RAM.""" if not 0.0 < train_subsample <= 1.0: @@ -178,12 +221,18 @@ def load_train_only_frame( raise FileNotFoundError(f"Expected {train_path}") excluded = set(benchmark_columns or []) + extra_targets = list(dict.fromkeys(extra_target_columns or [])) feature_names = feature_columns or load_feature_names( data_dir, feature_set=feature_set ) if feature_names: read_cols = list( - dict.fromkeys(feature_names + [target_col] + (["era"] if need_era else [])) + dict.fromkeys( + feature_names + + [target_col] + + extra_targets + + (["era"] if need_era else []) + ) ) train_df = pd.read_parquet(train_path, columns=read_cols) cols = [c for c in feature_names if c in train_df.columns and c not in excluded] @@ -206,6 +255,53 @@ def load_train_only_frame( return train_df[read_cols], train_df[target_col], cols +def load_train_targets_frame( + data_dir: Path, + train_subsample: float, + primary_target: str, + auxiliary_targets: list[str] | None, + seed: int, + feature_columns: list[str] | None, + need_era: bool, + benchmark_columns: list[str] | None = None, +) -> tuple[pd.DataFrame, pd.Series, pd.DataFrame, list[str]]: + """Load train data with feature columns and targets for multi-target HPO.""" + aux = [c for c in dict.fromkeys(auxiliary_targets or []) if c != primary_target] + target_cols = list(dict.fromkeys([primary_target, *aux])) + + if not 0.0 < train_subsample <= 1.0: + raise ValueError(f"train_subsample must be in (0, 1], got {train_subsample}") + + train_path = data_dir / "train.parquet" + if not train_path.exists(): + raise FileNotFoundError(f"Expected {train_path}") + + excluded = set(benchmark_columns or []) + feature_names = feature_columns or load_feature_names(data_dir) + if not feature_names: + raise ValueError("feature_columns required for multi-target HPO load") + + read_cols = list( + dict.fromkeys(feature_names + target_cols + (["era"] if need_era else [])) + ) + train_df = pd.read_parquet(train_path, columns=read_cols) + feature_cols = [ + c for c in feature_names if c in train_df.columns and c not in excluded + ] + if not feature_cols: + raise ValueError("No feature columns resolved.") + + train_df = train_df.sample(frac=train_subsample, random_state=seed) + if "era" in train_df.columns: + train_df = train_df.sort_values("era", kind="mergesort") + + x_cols = feature_cols + (["era"] if need_era else []) + X_train = train_df[x_cols] + y_primary = train_df[primary_target] + targets_df = train_df[target_cols].copy() + return X_train, y_primary, targets_df, feature_cols + + def load_train_frame_with_era( data_dir: Path, train_subsample: float, diff --git a/src/alphapulse/experiments/pipeline_build.py b/src/alphapulse/experiments/pipeline_build.py new file mode 100644 index 0000000..4d91329 --- /dev/null +++ b/src/alphapulse/experiments/pipeline_build.py @@ -0,0 +1,37 @@ +from typing import Any + +from .schema import ExperimentV1 + + +def is_multi_target_experiment(exp: ExperimentV1) -> bool: + return bool(exp.data.auxiliary_targets) + + +def experiment_target_flat(exp: ExperimentV1) -> dict[str, Any]: + aux = [ + t + for t in dict.fromkeys(exp.data.auxiliary_targets or []) + if t != exp.data.target_col + ] + if not aux: + return { + "target_mode": "single", + "primary_target": exp.data.target_col, + "auxiliary_targets": [], + "target_blend_method": exp.data.target_blend_method, + } + n_subs = exp.models[0].n_subs if exp.models else 10 + return { + "target_mode": "multi_blend", + "primary_target": exp.data.target_col, + "auxiliary_targets": aux, + "target_blend_method": exp.data.target_blend_method, + "n_subs": n_subs, + } + + +def needs_internal_val_for_experiment(exp: ExperimentV1) -> bool: + from ..pipeline.ensemble import needs_internal_val_for_ensemble + + pipeline_cfg = exp.to_pipeline_config() + return needs_internal_val_for_ensemble(pipeline_cfg) diff --git a/src/alphapulse/experiments/runner.py b/src/alphapulse/experiments/runner.py index d21131b..c9c36c0 100644 --- a/src/alphapulse/experiments/runner.py +++ b/src/alphapulse/experiments/runner.py @@ -11,15 +11,27 @@ from ..evaluation import Backtester from ..evaluation.era_split import EraSplitEvaluator, evaluate_holdout_last_n_eras -from ..hpo.builder import TREE_MODEL_NAMES, build_pipeline_or_multi +from ..hpo.builder import ( + TREE_MODEL_NAMES, + build_multi_target_from_config, + build_pipeline_or_multi, +) +from ..pipeline.multi_target import MultiTargetPipeline from ..pipeline.multihead import MultiHeadPipeline from ..pipeline.pipeline import Pipeline from .data import ( load_meta_model_series, load_train_frame_with_era, - load_train_val_frames, + load_train_only_frame, + load_train_targets_frame, + load_validation_frames, ) from .hashing import config_hash +from .pipeline_build import ( + experiment_target_flat, + is_multi_target_experiment, + needs_internal_val_for_experiment, +) from .schema import ExperimentV1 from .split import internal_val_split @@ -59,6 +71,31 @@ def _need_era_column(exp: ExperimentV1) -> bool: return False +def _describe_models(exp: ExperimentV1) -> str: + parts: list[str] = [] + for m in exp.models: + era_tag = ( + f", era_ensemble n_subs={m.n_subs}" if m.type in TREE_MODEL_NAMES else "" + ) + parts.append(f"{m.type}{era_tag}") + return " + ".join(parts) + + +def _describe_preprocessors(exp: ExperimentV1) -> str: + names = [p.type for p in exp.preprocessing] + return " -> ".join(names) if names else "none" + + +def _describe_pipeline_models( + pipeline: Pipeline | MultiHeadPipeline | MultiTargetPipeline, +) -> str: + if isinstance(pipeline, MultiHeadPipeline): + return f"MultiHead({len(pipeline.heads)} heads)" + if isinstance(pipeline, MultiTargetPipeline): + return f"MultiTarget({len(pipeline.target_columns)} targets)" + return " + ".join(m.name for m in pipeline.models) + + def run_experiment( exp: ExperimentV1, *, @@ -78,6 +115,20 @@ def run_experiment( artifact paths, and any error string. """ t0 = time.perf_counter() + from ..logging_.cli import configure_cli_logging + + configure_cli_logging() + model_summary = _describe_models(exp) + logger.info( + "Experiment start: target={} train_subsample={} models=[{}] preprocessors=[{}] " + "ensemble={} n_rounds={}", + exp.data.target_col, + exp.data.train_subsample, + model_summary, + _describe_preprocessors(exp), + exp.ensemble_method, + exp.train.n_rounds, + ) pipeline_cfg = exp.to_pipeline_config() if use_gpu: from ..hpo.search_space import apply_gpu_pipeline_config @@ -95,18 +146,33 @@ def run_experiment( need_era = _need_era_column(exp) data_dir = Path(exp.data.data_dir) + logger.info("Loading train data from {} ...", data_dir) + multi_target = is_multi_target_experiment(exp) + targets_df = None try: - X_train, y_train, X_val, y_val, era_val, feature_cols = load_train_val_frames( - data_dir, - train_subsample=exp.data.train_subsample, - target_col=exp.data.target_col, - seed=exp.data.seed, - feature_columns=exp.features.columns, - need_era=need_era, - benchmark_columns=exp.data.benchmark_columns or None, - ) + if multi_target: + X_train, y_train, targets_df, feature_cols = load_train_targets_frame( + data_dir, + train_subsample=exp.data.train_subsample, + primary_target=exp.data.target_col, + auxiliary_targets=exp.data.auxiliary_targets, + seed=exp.data.seed, + feature_columns=exp.features.columns, + need_era=need_era, + benchmark_columns=exp.data.benchmark_columns or None, + ) + else: + X_train, y_train, feature_cols = load_train_only_frame( + data_dir, + train_subsample=exp.data.train_subsample, + target_col=exp.data.target_col, + seed=exp.data.seed, + feature_columns=exp.features.columns, + need_era=need_era, + benchmark_columns=exp.data.benchmark_columns or None, + ) except Exception as e: - logger.exception("Experiment data load failed") + logger.exception("Experiment train data load failed") return RunResult( error=str(e), config_hash=ch, @@ -114,7 +180,15 @@ def run_experiment( pipeline_config=pipeline_cfg, ) - stacking_needs_val = exp.ensemble_method == "stacking" and len(exp.models) > 1 + n_eras = int(X_train["era"].nunique()) if "era" in X_train.columns else 0 + logger.info( + "Train loaded: rows={} features={} eras={}", + len(X_train), + len(feature_cols), + n_eras, + ) + + stacking_needs_val = needs_internal_val_for_experiment(exp) era_train = X_train["era"] if "era" in X_train.columns else None X_train_fit, y_train_fit, X_val_internal, y_val_internal = internal_val_split( X_train, @@ -122,22 +196,96 @@ def run_experiment( era_train=era_train, force_internal=stacking_needs_val, ) + targets_train_fit = None + targets_val_internal = None + if multi_target and targets_df is not None: + targets_train_fit = targets_df.loc[X_train_fit.index] + if X_val_internal is not None: + targets_val_internal = targets_df.loc[X_val_internal.index] + if X_val_internal is not None: + logger.info( + "Internal val split: train_rows={} val_rows={}", + len(X_train_fit), + len(X_val_internal), + ) - pipeline: Pipeline | MultiHeadPipeline = build_pipeline_or_multi( - pipeline_cfg, feature_columns=feature_cols, feature_groups=exp.features.groups - ) + target_flat = experiment_target_flat(exp) + if multi_target: + pipeline: Pipeline | MultiHeadPipeline | MultiTargetPipeline = ( + build_multi_target_from_config( + pipeline_cfg, + target_flat, + feature_columns=feature_cols, + feature_groups=exp.features.groups, + ) + ) + else: + pipeline = build_pipeline_or_multi( + pipeline_cfg, + feature_columns=feature_cols, + feature_groups=exp.features.groups, + ) + logger.info("Pipeline built: {}", _describe_pipeline_models(pipeline)) train_kw: dict[str, Any] = { "n_rounds": exp.train.n_rounds, "early_stopping_rounds": exp.train.early_stopping_rounds, } + logger.info( + "Training started (n_rounds={}, early_stopping={}) ...", + exp.train.n_rounds, + exp.train.early_stopping_rounds, + ) try: - pipeline.fit( - X_train_fit, - y_train_fit, - X_val=X_val_internal, - y_val=y_val_internal, - **train_kw, + era_train_fit = ( + era_train.loc[X_train_fit.index] if era_train is not None else None ) + era_val_fit = ( + era_train.loc[X_val_internal.index] + if era_train is not None and X_val_internal is not None + else None + ) + if multi_target: + assert targets_train_fit is not None + train_metrics = pipeline.fit( + X_train_fit.drop(columns=["era"], errors="ignore"), + targets_train_fit, + X_val=X_val_internal.drop(columns=["era"], errors="ignore") + if X_val_internal is not None + else None, + targets_val=targets_val_internal, + era_train=era_train_fit, + era_val=era_val_fit, + **train_kw, + ) + else: + train_metrics = pipeline.fit( + X_train_fit, + y_train_fit, + X_val=X_val_internal, + y_val=y_val_internal, + era_val=era_val_fit, + **train_kw, + ) + if isinstance(pipeline, Pipeline): + from ..hpo.objective import ( + _needs_validation_ensemble_opt, + _optimize_ensemble_on_validation, + ) + + if _needs_validation_ensemble_opt(pipeline_cfg): + _optimize_ensemble_on_validation( + pipeline, + data_dir=data_dir, + feature_cols=feature_cols, + target_col=exp.data.target_col, + train_subsample=exp.data.train_subsample, + seed=exp.data.seed, + pipeline_cfg=pipeline_cfg, + ) + if pipeline.ensemble_weights is not None: + pipeline_cfg.setdefault("ensemble_params", {})["weights"] = ( + pipeline.ensemble_weights + ) except Exception as e: logger.exception("Pipeline fit failed") return RunResult( @@ -146,8 +294,33 @@ def run_experiment( duration_sec=time.perf_counter() - t0, pipeline_config=pipeline_cfg, ) + logger.info("Training finished: {}", train_metrics) + del X_train, y_train, X_train_fit, y_train_fit, X_val_internal, y_val_internal + if era_train is not None: + del era_train gc.collect() + + logger.info("Loading validation data ...") + try: + X_val, y_val, era_val = load_validation_frames( + data_dir, + exp.data.target_col, + feature_cols, + need_era=need_era, + ) + except Exception as e: + logger.exception("Experiment validation data load failed") + return RunResult( + error=str(e), + config_hash=ch, + duration_sec=time.perf_counter() - t0, + pipeline_config=pipeline_cfg, + ) + logger.info( + "Validation loaded: rows={} eras={}", len(X_val), int(era_val.nunique()) + ) + meta_path = exp.evaluation.meta_model_path meta_model_preds = None meta_series = load_meta_model_series( @@ -156,8 +329,29 @@ def run_experiment( if meta_series is not None: meta_model_preds = meta_series.reindex(X_val.index).to_numpy(dtype=np.float64) + if ( + isinstance(pipeline, Pipeline) + and pipeline._meta_neutralizer is not None + and meta_model_preds is not None + and np.isfinite(meta_model_preds).sum() >= 2 + ): + X_feat = X_val[feature_cols] + base_preds = pipeline.predict(X_feat, eras=era_val, meta_model=None) + optimized = pipeline._meta_neutralizer.optimize_proportion( + base_preds, + meta_model_preds, + y_val, + era_val, + objective="payout_score", + bounds=(0.5, 0.75), + corr_weight=exp.evaluation.corr_weight, + mmc_weight=exp.evaluation.mmc_weight, + ) + pipeline.meta_neutralize_proportion = optimized + compute_fnc = len(feature_cols) <= 200 backtester = Backtester(pipeline, feature_columns=feature_cols) + logger.info("Backtesting on validation set ...") metrics = backtester.evaluate( X_val, y_val, @@ -167,6 +361,12 @@ def run_experiment( corr_weight=exp.evaluation.corr_weight, mmc_weight=exp.evaluation.mmc_weight, ) + logger.info( + "Backtest done: corr_sharpe={:.4f} mean_corr={:.4f} pct_positive_eras={:.1%}", + metrics.get("corr_sharpe", float("nan")), + metrics.get("mean_per_era_correlation", float("nan")), + metrics.get("pct_positive_eras", float("nan")), + ) should_log_diag = log_wandb_diagnostics if should_log_diag is None: @@ -177,6 +377,7 @@ def run_experiment( except ImportError: should_log_diag = False if should_log_diag: + logger.info("Logging W&B diagnostics ...") from ..evaluation.wandb_diagnostics import log_experiment_diagnostics log_experiment_diagnostics( @@ -199,17 +400,73 @@ def run_experiment( metrics[f"holdout_{k}"] = v if ev.walk_forward: - X_wf, y_wf, era_wf, _ = load_train_frame_with_era( - data_dir, - train_subsample=exp.data.train_subsample, - target_col=exp.data.target_col, - seed=exp.data.seed, - feature_columns=exp.features.columns, - need_era=need_era, - benchmark_columns=exp.data.benchmark_columns or None, - ) + if multi_target: + X_wf, y_wf, targets_wf, _ = load_train_targets_frame( + data_dir, + train_subsample=exp.data.train_subsample, + primary_target=exp.data.target_col, + auxiliary_targets=exp.data.auxiliary_targets, + seed=exp.data.seed, + feature_columns=exp.features.columns, + need_era=need_era, + benchmark_columns=exp.data.benchmark_columns or None, + ) + era_wf = X_wf["era"] + else: + X_wf, y_wf, era_wf, _ = load_train_frame_with_era( + data_dir, + train_subsample=exp.data.train_subsample, + target_col=exp.data.target_col, + seed=exp.data.seed, + feature_columns=exp.features.columns, + need_era=need_era, + benchmark_columns=exp.data.benchmark_columns or None, + ) + targets_wf = None + + def train_fn( + X_tr: Any, y_tr: Any + ) -> Pipeline | MultiHeadPipeline | MultiTargetPipeline: + if multi_target: + assert targets_wf is not None + p_mt: Pipeline | MultiHeadPipeline | MultiTargetPipeline = ( + build_multi_target_from_config( + pipeline_cfg, + target_flat, + feature_columns=feature_cols, + feature_groups=exp.features.groups, + ) + ) + era_col = X_tr["era"] if "era" in X_tr.columns else None + X_fit, _, X_val_inner, _ = internal_val_split( + X_tr, y_tr, era_train=era_col, force_internal=stacking_needs_val + ) + targets_split = targets_wf.loc[X_tr.index] + targets_train = targets_split.loc[X_fit.index] + targets_val = ( + targets_split.loc[X_val_inner.index] + if X_val_inner is not None + else None + ) + era_train_wf = era_col.loc[X_fit.index] if era_col is not None else None + era_val_wf = ( + era_col.loc[X_val_inner.index] + if era_col is not None and X_val_inner is not None + else None + ) + p_mt.fit( + X_fit.drop(columns=["era"], errors="ignore"), + targets_train, + X_val=X_val_inner.drop(columns=["era"], errors="ignore") + if X_val_inner is not None + else None, + targets_val=targets_val, + era_train=era_train_wf, + era_val=era_val_wf, + **train_kw, + ) + return p_mt - def train_fn(X_tr: Any, y_tr: Any) -> Pipeline | MultiHeadPipeline: p = build_pipeline_or_multi( pipeline_cfg, feature_columns=feature_cols, @@ -219,7 +476,19 @@ def train_fn(X_tr: Any, y_tr: Any) -> Pipeline | MultiHeadPipeline: X_fit, y_fit, X_val_inner, y_val_inner = internal_val_split( X_tr, y_tr, era_train=era_col, force_internal=stacking_needs_val ) - p.fit(X_fit, y_fit, X_val=X_val_inner, y_val=y_val_inner, **train_kw) + era_val_wf = ( + era_col.loc[X_val_inner.index] + if era_col is not None and X_val_inner is not None + else None + ) + p.fit( + X_fit, + y_fit, + X_val=X_val_inner, + y_val=y_val_inner, + era_val=era_val_wf, + **train_kw, + ) return p wf_metrics = EraSplitEvaluator( @@ -245,10 +514,12 @@ def train_fn(X_tr: Any, y_tr: Any) -> Pipeline | MultiHeadPipeline: f.write(ch) paths["config_hash"] = str(hash_path) + duration = time.perf_counter() - t0 + logger.info("Experiment complete in {:.1f}s", duration) return RunResult( metrics=metrics, config_hash=ch, - duration_sec=time.perf_counter() - t0, + duration_sec=duration, paths=paths, pipeline_config=pipeline_cfg, ) diff --git a/src/alphapulse/experiments/schema.py b/src/alphapulse/experiments/schema.py index 081d45c..ea82f09 100644 --- a/src/alphapulse/experiments/schema.py +++ b/src/alphapulse/experiments/schema.py @@ -42,8 +42,19 @@ class DataConfig(BaseModel): target_col: str = "target" seed: int = 42 auxiliary_targets: list[str] | None = None + target_blend_method: Literal["equal", "sharpe"] = "equal" benchmark_columns: list[str] = Field(default_factory=list) + @model_validator(mode="after") + def validate_auxiliary_targets(self) -> Self: + aux = self.auxiliary_targets or [] + if self.target_col in aux: + raise ValueError( + f"auxiliary_targets must not include primary target_col " + f"{self.target_col!r}" + ) + return self + class FeatureConfig(BaseModel): columns: list[str] | None = None @@ -88,10 +99,14 @@ class NeutralizationConfig(BaseModel): features: list[str] | None = None -_METRIC_ALIASES: dict[str, str] = { - "sharpe": "corr_sharpe", - "correlation": "mean_per_era_correlation", -} +class MetaNeutralizationConfig(BaseModel): + """Meta-model neutralization applied after feature neutralization. + + Removes linear exposure of predictions to the Numerai meta model, improving + MMC by reducing overlap with the stake-weighted meta model. + """ + + proportion: float = Field(0.0, ge=0.0, le=1.0) class EvaluationConfig(BaseModel): @@ -102,15 +117,6 @@ class EvaluationConfig(BaseModel): "mmc_sharpe", ] = "corr_sharpe" - @model_validator(mode="before") - @classmethod - def _map_metric_aliases(cls, data: Any) -> Any: - if isinstance(data, dict) and "primary_metric" in data: - data["primary_metric"] = _METRIC_ALIASES.get( - data["primary_metric"], data["primary_metric"] - ) - return data - era_holdout_last_n: int | None = None walk_forward: bool = False walk_forward_min_train_eras: int = Field(default=1, ge=1) @@ -136,6 +142,9 @@ class ExperimentV1(BaseModel): ensemble_method: Literal["single", "weighted", "stacking"] = "single" ensemble_params: dict[str, Any] = Field(default_factory=dict) neutralization: NeutralizationConfig = Field(default_factory=NeutralizationConfig) + meta_neutralization: MetaNeutralizationConfig = Field( + default_factory=MetaNeutralizationConfig + ) train: TrainConfig = Field(default_factory=TrainConfig) evaluation: EvaluationConfig = Field(default_factory=EvaluationConfig) @@ -174,4 +183,5 @@ def to_pipeline_config(self) -> dict[str, Any]: "feature_groups": dict(self.features.groups), "neutralize_proportion": self.neutralization.proportion, "neutralize_features": self.neutralization.features, + "meta_neutralize_proportion": self.meta_neutralization.proportion, } diff --git a/src/alphapulse/features/__init__.py b/src/alphapulse/features/__init__.py new file mode 100644 index 0000000..b5c0c6a --- /dev/null +++ b/src/alphapulse/features/__init__.py @@ -0,0 +1,13 @@ +from .catalog import ( + FeatureCatalog, + TargetCatalog, + load_feature_catalog, + load_target_catalog, +) + +__all__ = [ + "FeatureCatalog", + "TargetCatalog", + "load_feature_catalog", + "load_target_catalog", +] diff --git a/src/alphapulse/features/catalog.py b/src/alphapulse/features/catalog.py new file mode 100644 index 0000000..1e1d75a --- /dev/null +++ b/src/alphapulse/features/catalog.py @@ -0,0 +1,116 @@ +import json +import re +from dataclasses import dataclass +from functools import lru_cache +from pathlib import Path + +import pandas as pd + +LEGACY_EXCLUDED = frozenset( + {"v2_equivalent_features", "v3_equivalent_features", "fncv3_features"} +) +SIZE_GROUPS = frozenset({"small", "medium", "all"}) +STAT_GROUPS = frozenset( + { + "intelligence", + "charisma", + "strength", + "dexterity", + "constitution", + "wisdom", + "agility", + "serenity", + "sunshine", + "rain", + "midnight", + "faith", + } +) +_HORIZON_RE = re.compile(r"_(\d+)$") + + +@dataclass(frozen=True) +class FeatureCatalog: + feature_sets: dict[str, list[str]] + searchable_names: list[str] + + def columns(self, name: str) -> list[str]: + if name not in self.feature_sets: + raise KeyError(f"Unknown feature group: {name!r}") + return list(self.feature_sets[name]) + + def union(self, names: list[str]) -> list[str]: + seen: set[str] = set() + out: list[str] = [] + for name in names: + for col in self.columns(name): + if col not in seen: + seen.add(col) + out.append(col) + return out + + def category(self, name: str) -> str: + if name in SIZE_GROUPS: + return "size" + if name in STAT_GROUPS: + return "stat" + return "other" + + +@dataclass(frozen=True) +class TargetCatalog: + targets: list[str] + + def parse_horizon(self, target: str) -> int | None: + if target == "target": + return 20 + match = _HORIZON_RE.search(target) + if match: + return int(match.group(1)) + return None + + def valid_targets(self, df: pd.DataFrame, cols: list[str]) -> list[str]: + return [c for c in cols if c in df.columns and c in self.targets] + + +def _load_features_json(data_dir: Path) -> dict: + path = data_dir / "features.json" + if not path.exists(): + raise FileNotFoundError(f"Expected {path}") + with open(path, encoding="utf-8") as f: + data = json.load(f) + if not isinstance(data, dict): + raise ValueError(f"Invalid features.json at {path}") + return data + + +@lru_cache(maxsize=8) +def load_feature_catalog(data_dir: str | Path) -> FeatureCatalog: + data = _load_features_json(Path(data_dir)) + raw_sets = data.get("feature_sets", {}) + if not isinstance(raw_sets, dict): + raise ValueError("features.json missing feature_sets dict") + + feature_sets: dict[str, list[str]] = {} + for name, cols in raw_sets.items(): + if name in LEGACY_EXCLUDED: + continue + if not isinstance(name, str) or not isinstance(cols, list): + continue + if all(isinstance(c, str) for c in cols): + feature_sets[name] = cols + + searchable = sorted( + name for name in feature_sets if name in SIZE_GROUPS or name in STAT_GROUPS + ) + return FeatureCatalog(feature_sets=feature_sets, searchable_names=searchable) + + +@lru_cache(maxsize=8) +def load_target_catalog(data_dir: str | Path) -> TargetCatalog: + data = _load_features_json(Path(data_dir)) + raw_targets = data.get("targets", []) + if not isinstance(raw_targets, list): + raise ValueError("features.json missing targets list") + targets = [t for t in raw_targets if isinstance(t, str)] + return TargetCatalog(targets=targets) diff --git a/src/alphapulse/hpo/__init__.py b/src/alphapulse/hpo/__init__.py index 700d908..b411b7c 100644 --- a/src/alphapulse/hpo/__init__.py +++ b/src/alphapulse/hpo/__init__.py @@ -8,6 +8,13 @@ needs_multi_head_pipeline, ) from .objective import TrialResult, ray_trainable, run_trial +from .optuna_search import ( + DEFAULT_N_STARTUP_TRIALS, + SamplerName, + create_hpo_study, + suggest_flat_config, + tell_trial_result, +) from .registry import MODEL_REGISTRY, PREPROCESSOR_REGISTRY from .search_space import ( get_full_param_space, @@ -33,4 +40,9 @@ "resolve_flat_config", "run_trial", "sample_random_config", + "SamplerName", + "DEFAULT_N_STARTUP_TRIALS", + "create_hpo_study", + "suggest_flat_config", + "tell_trial_result", ] diff --git a/src/alphapulse/hpo/builder.py b/src/alphapulse/hpo/builder.py index 5308d9e..e357ab7 100644 --- a/src/alphapulse/hpo/builder.py +++ b/src/alphapulse/hpo/builder.py @@ -1,8 +1,13 @@ from collections.abc import Callable +from pathlib import Path from typing import Any, Literal +import numpy as np +import pandas as pd + from ..models.base import BaseModel from ..models.era_ensemble_model import EraEnsembleModel +from ..pipeline.multi_target import MultiTargetPipeline from ..pipeline.multihead import HeadSpec, MultiHeadPipeline from ..pipeline.pipeline import Pipeline from ..preprocessors.base import BasePreprocessor @@ -15,6 +20,35 @@ ) +class _PipelineModelAdapter(BaseModel): + def __init__(self, pipeline: Pipeline | MultiHeadPipeline) -> None: + super().__init__(name=f"Adapter_{type(pipeline).__name__}") + self._pipeline = pipeline + + def train( + self, + X_train: pd.DataFrame, + y_train: pd.Series, + X_val: pd.DataFrame | None = None, + y_val: pd.Series | None = None, + **kwargs: Any, + ) -> dict[str, float]: + metrics = self._pipeline.fit( + X_train, y_train, X_val=X_val, y_val=y_val, **kwargs + ) + self.is_trained = True + return metrics + + def predict(self, X: pd.DataFrame) -> np.ndarray: + return self._pipeline.predict(X) + + def save(self, path: Path) -> None: + self._pipeline.save_pipeline(path) + + def load(self, path: Path) -> "_PipelineModelAdapter": + raise NotImplementedError("Adapter load is not supported") + + def _merge_params( defaults: dict[str, Any], override: dict[str, Any] | None ) -> dict[str, Any]: @@ -122,7 +156,7 @@ def build_models(config: list[dict[str, Any]]) -> list[BaseModel]: def model_spec_needs_head_split(item: dict[str, Any]) -> bool: - if item.get("input_columns") or item.get("input_group"): + if item.get("input_columns") or item.get("input_group") or item.get("input_groups"): return True return len(item.get("preprocessors") or []) > 0 @@ -153,6 +187,7 @@ def build_multi_head_pipeline( model=models[i], input_columns=m.get("input_columns"), input_group=m.get("input_group"), + input_groups=m.get("input_groups"), local_preprocessors=local_pres, feature_groups=fg, ) @@ -201,6 +236,7 @@ def build_pipeline( ensemble_params=ensemble_params, neutralize_proportion=config.get("neutralize_proportion", 0.0), neutralize_features=config.get("neutralize_features"), + meta_neutralize_proportion=config.get("meta_neutralize_proportion", 0.0), ) @@ -212,7 +248,7 @@ def build_pipeline_or_multi( """Build either a ``Pipeline`` or ``MultiHeadPipeline`` from config. Automatically selects ``MultiHeadPipeline`` when any model specifies - ``input_columns``, ``input_group``, or local preprocessors. + ``input_columns``, ``input_group``, ``input_groups``, or local preprocessors. Args: config: Nested pipeline configuration dict. @@ -228,3 +264,48 @@ def build_pipeline_or_multi( config, feature_columns=feature_columns, feature_groups=feature_groups ) return build_pipeline(config, feature_columns=feature_columns) + + +def build_multi_target_from_config( + config: dict[str, Any], + flat: dict[str, Any], + feature_columns: list[str] | None = None, + feature_groups: dict[str, list[str]] | None = None, +) -> MultiTargetPipeline: + preprocessors = build_preprocessors(config.get("preprocessors", [])) + strategy_targets = [str(flat.get("primary_target", "target"))] + aux = flat.get("auxiliary_targets") or [] + if isinstance(aux, list): + strategy_targets.extend(str(a) for a in aux) + target_columns = list(dict.fromkeys(strategy_targets)) + blend_method = str(flat.get("target_blend_method", "equal")) + if blend_method not in ("equal", "sharpe"): + blend_method = "equal" + + def model_factory() -> BaseModel: + if needs_multi_head_pipeline(config) or len(config.get("models", [])) > 1: + pipeline = build_multi_head_pipeline( + config, + feature_columns=feature_columns, + feature_groups=feature_groups, + ) + return _PipelineModelAdapter(pipeline) + models_cfg = config.get("models", []) + if not models_cfg: + raise ValueError("Config must have at least one model for multi-target HPO") + spec = models_cfg[0] + return instantiate_model( + str(spec.get("type", "XGBoost")), + spec.get("params"), + index=0, + n_subs=int(spec.get("n_subs", flat.get("n_subs", 10))), + use_era_ensemble=bool(spec.get("use_era_ensemble", True)), + ) + + return MultiTargetPipeline( + preprocessors=preprocessors, + model_factory=model_factory, + target_columns=target_columns, + primary_target=str(flat.get("primary_target", "target")), + blend_method=blend_method, + ) diff --git a/src/alphapulse/hpo/export.py b/src/alphapulse/hpo/export.py new file mode 100644 index 0000000..028bea5 --- /dev/null +++ b/src/alphapulse/hpo/export.py @@ -0,0 +1,246 @@ +import random +from dataclasses import dataclass +from pathlib import Path +from typing import Any, cast + +import pandas as pd + +from ..experiments.data import load_train_only_frame, load_train_targets_frame +from ..features.catalog import FeatureCatalog, load_feature_catalog, load_target_catalog +from ..pipeline.model_access import PipelineLike +from .builder import TREE_MODEL_NAMES +from .feature_routing import FeatureRoutingResult, resolve_feature_routing +from .objective import _fit_pipeline, _resolve_pipeline_cfg +from .search_space import apply_gpu_pipeline_config, get_train_kwargs_from_flat +from .target_strategy import ( + TargetStrategy, + apply_target_strategy_to_flat, + strategy_from_flat, + validate_target_strategy_early, +) + + +@dataclass(frozen=True) +class HpoBuildContext: + flat: dict[str, Any] + strategy: TargetStrategy + routing: FeatureRoutingResult + pipeline_cfg: dict[str, Any] + feature_groups: dict[str, list[str]] | None + feature_columns: list[str] + + +@dataclass(frozen=True) +class HpoFitResult: + pipeline: PipelineLike + feature_columns: list[str] + pipeline_cfg: dict[str, Any] + flat: dict[str, Any] + primary_target: str + + +def needs_era_from_flat_config(flat: dict[str, Any]) -> bool: + if bool(flat.get("use_packboost", False)): + return True + num_models = int(flat.get("num_models", 1)) + for i in range(1, min(num_models, 3) + 1): + model_type = flat.get(f"model_{i}_type", "") + if model_type == "Packboost" or model_type in TREE_MODEL_NAMES: + return True + return False + + +def prepare_hpo_flat( + flat: dict[str, Any], + data_dir: Path, + *, + target_col_fallback: str = "target", +) -> dict[str, Any]: + out = dict(flat) + out["_data_dir"] = str(data_dir) + out.setdefault("primary_target", target_col_fallback) + return out + + +def resolve_hpo_build_context( + flat: dict[str, Any], + *, + catalog: FeatureCatalog | None = None, +) -> HpoBuildContext: + data_dir = flat.get("_data_dir") + if catalog is None: + if not data_dir: + raise ValueError("_data_dir required in flat config for HPO build") + catalog = load_feature_catalog(data_dir) + + strategy = strategy_from_flat(flat) + routing = resolve_feature_routing(flat, catalog) + pipeline_cfg, feature_groups, routed_columns = _resolve_pipeline_cfg(flat, catalog) + if flat.get("use_gpu"): + pipeline_cfg = apply_gpu_pipeline_config(pipeline_cfg) + + if routed_columns: + feature_columns = list(routed_columns) + elif routing.feature_columns: + feature_columns = list(routing.feature_columns) + else: + feature_columns = ( + catalog.columns("medium") if "medium" in catalog.feature_sets else [] + ) + + return HpoBuildContext( + flat=flat, + strategy=strategy, + routing=routing, + pipeline_cfg=pipeline_cfg, + feature_groups=feature_groups, + feature_columns=feature_columns, + ) + + +def load_hpo_training_frames( + context: HpoBuildContext, + data_dir: Path, + *, + train_subsample: float, + seed: int, +) -> tuple[pd.DataFrame, pd.Series, pd.DataFrame | None, list[str]]: + need_era = needs_era_from_flat_config(context.flat) + feature_columns = context.feature_columns or None + strategy = context.strategy + + if strategy.target_mode == "multi_blend" and strategy.auxiliary_targets: + X_train, y_train, targets_df, feature_cols = load_train_targets_frame( + data_dir, + train_subsample=train_subsample, + primary_target=strategy.primary_target, + auxiliary_targets=strategy.auxiliary_targets, + seed=seed, + feature_columns=feature_columns, + need_era=need_era, + ) + return X_train, y_train, targets_df, feature_cols + + X_train, y_train, feature_cols = load_train_only_frame( + data_dir, + train_subsample=train_subsample, + target_col=strategy.primary_target, + seed=seed, + feature_columns=feature_columns, + need_era=need_era, + ) + return X_train, y_train, None, feature_cols + + +def validate_and_apply_target_strategy( + flat: dict[str, Any], + targets_df: pd.DataFrame, + data_dir: Path, + *, + allow_resample: bool = False, + seed: int | None = None, +) -> dict[str, Any]: + strategy = strategy_from_flat(flat) + catalog = load_target_catalog(data_dir) if allow_resample else None + rng = random.Random(seed) if allow_resample and seed is not None else None + validation = validate_target_strategy_early( + targets_df, + strategy, + catalog=catalog, + rng=rng, + ) + if not validation.ok: + raise ValueError(validation.reason or "target strategy validation failed") + return apply_target_strategy_to_flat(flat, validation.strategy) + + +def fit_hpo_pipeline( + context: HpoBuildContext, + X_train: pd.DataFrame, + y_train: pd.Series, + *, + targets_df: pd.DataFrame | None = None, + seed: int | None = None, +) -> PipelineLike: + train_kwargs = get_train_kwargs_from_flat(context.flat) + return cast( + PipelineLike, + _fit_pipeline( + context.pipeline_cfg, + context.feature_columns, + X_train, + y_train, + train_kwargs, + flat_config=context.flat, + seed=seed, + feature_groups=context.feature_groups, + targets_df=targets_df, + ), + ) + + +def build_hpo_pipeline_from_flat( + flat: dict[str, Any], + data_dir: Path, + *, + train_subsample: float, + seed: int, + target_col_fallback: str = "target", + allow_target_resample: bool = False, +) -> HpoFitResult: + prepared = prepare_hpo_flat(flat, data_dir, target_col_fallback=target_col_fallback) + context = resolve_hpo_build_context(prepared) + X_train, y_train, targets_df, feature_cols = load_hpo_training_frames( + context, + data_dir, + train_subsample=train_subsample, + seed=seed, + ) + context = HpoBuildContext( + flat=prepared, + strategy=context.strategy, + routing=context.routing, + pipeline_cfg=context.pipeline_cfg, + feature_groups=context.feature_groups, + feature_columns=feature_cols, + ) + + targets_for_validation = ( + targets_df + if targets_df is not None + else pd.DataFrame({context.strategy.primary_target: y_train}) + ) + validated_flat = validate_and_apply_target_strategy( + prepared, + targets_for_validation, + data_dir, + allow_resample=allow_target_resample, + seed=seed, + ) + context = resolve_hpo_build_context(validated_flat) + context = HpoBuildContext( + flat=validated_flat, + strategy=context.strategy, + routing=context.routing, + pipeline_cfg=context.pipeline_cfg, + feature_groups=context.feature_groups, + feature_columns=feature_cols, + ) + if context.strategy.target_mode == "single": + targets_df = None + + pipeline = fit_hpo_pipeline( + context, + X_train, + y_train, + targets_df=targets_df, + seed=seed, + ) + primary = str(validated_flat.get("primary_target", target_col_fallback)) + return HpoFitResult( + pipeline=pipeline, + feature_columns=feature_cols, + pipeline_cfg=context.pipeline_cfg, + flat=validated_flat, + primary_target=primary, + ) diff --git a/src/alphapulse/hpo/feature_routing.py b/src/alphapulse/hpo/feature_routing.py new file mode 100644 index 0000000..df4bf34 --- /dev/null +++ b/src/alphapulse/hpo/feature_routing.py @@ -0,0 +1,356 @@ +import random +from dataclasses import dataclass, field +from typing import TYPE_CHECKING, Any, Literal + +from ..features.catalog import SIZE_GROUPS, STAT_GROUPS, FeatureCatalog + +if TYPE_CHECKING: + import optuna + +BuildPath = Literal["default", "simple", "grouped", "multihead"] + +FAST_MAX_ACTIVE_GROUPS = 4 +SLOW_MAX_ACTIVE_GROUPS = 6 +MAX_ROUTED_FEATURES = 1000 + +LANE_PREPROCESSORS_FAST = ("StandardScaler", "RobustScaler", "VarianceFeatureSelector") +LANE_PREPROCESSORS_SLOW = LANE_PREPROCESSORS_FAST + ("EraStableFeatureSelector",) + + +@dataclass +class FeatureRoutingResult: + build_path: BuildPath + feature_groups: dict[str, list[str]] + feature_columns: list[str] + pipeline_config_patch: dict[str, Any] = field(default_factory=dict) + + +def _lane_steps_to_preprocessors(steps: list[str]) -> list[dict[str, Any]]: + out: list[dict[str, Any]] = [] + for step in steps: + if step in ("StandardScaler", "RobustScaler"): + out.append({"type": step, "params": {}}) + elif step == "VarianceFeatureSelector": + out.append({"type": "VarianceSelector", "params": {"keep_fraction": 0.75}}) + elif step == "EraStableFeatureSelector": + out.append( + { + "type": "EraStableSelector", + "params": {"keep_fraction": 0.5, "n_estimators": 50}, + } + ) + return out + + +def _biased_group_choice( + rng: random.Random, catalog: FeatureCatalog, n: int +) -> list[str]: + size_pool = [g for g in catalog.searchable_names if g in SIZE_GROUPS] + stat_pool = [g for g in catalog.searchable_names if g in STAT_GROUPS] + chosen: list[str] = [] + if size_pool and rng.random() < 0.85: + weights = [3 if g == "medium" else (1 if g == "all" else 2) for g in size_pool] + chosen.append(rng.choices(size_pool, weights=weights, k=1)[0]) + if stat_pool and len(chosen) < n: + n_stat = min(n - len(chosen), rng.randint(0, n - len(chosen) + 1)) + if n_stat > 0: + chosen.extend(rng.sample(stat_pool, min(n_stat, len(stat_pool)))) + while len(chosen) < n: + pool = [g for g in catalog.searchable_names if g not in chosen] + if not pool: + break + chosen.append(rng.choice(pool)) + return chosen[:n] + + +def _union_size(catalog: FeatureCatalog, groups: list[str]) -> int: + return len(catalog.union(groups)) + + +def _fit_groups_under_limit( + catalog: FeatureCatalog, candidate_groups: list[str] +) -> list[str]: + groups: list[str] = [] + for group in candidate_groups: + next_groups = groups + [group] + if _union_size(catalog, next_groups) <= MAX_ROUTED_FEATURES: + groups = next_groups + if groups: + return groups + + searchable = sorted( + catalog.searchable_names, + key=lambda name: len(catalog.columns(name)), + ) + for group in searchable: + if _union_size(catalog, [group]) <= MAX_ROUTED_FEATURES: + return [group] + + raise ValueError( + f"No feature group fits the routing limit of {MAX_ROUTED_FEATURES} features" + ) + + +def sample_feature_routing( + rng: random.Random, + catalog: FeatureCatalog, + num_models: int, + *, + fast: bool = False, +) -> dict[str, Any]: + max_groups = FAST_MAX_ACTIVE_GROUPS if fast else SLOW_MAX_ACTIVE_GROUPS + n_groups = rng.randint(1, min(max_groups, len(catalog.searchable_names))) + sampled_groups = _biased_group_choice(rng, catalog, n_groups) + active_groups = _fit_groups_under_limit(catalog, sampled_groups) + routed_feature_columns = catalog.union(active_groups) + + lane_pool = LANE_PREPROCESSORS_FAST if fast else LANE_PREPROCESSORS_SLOW + n_lanes = rng.randint(1, min(2, num_models)) + lane_steps: dict[int, list[str]] = {} + for lane_id in range(n_lanes): + if rng.random() < 0.5: + lane_steps[lane_id] = [] + else: + step = rng.choice(lane_pool) + lane_steps[lane_id] = ( + [step] if step not in ("StandardScaler", "RobustScaler") else [] + ) + + model_groups: dict[int, list[str]] = {i: [] for i in range(1, num_models + 1)} + for group in active_groups: + model_idx = rng.randint(1, num_models) + model_groups[model_idx].append(group) + + for model_idx in range(1, num_models + 1): + if not model_groups[model_idx]: + model_groups[model_idx] = [rng.choice(active_groups)] + + flat: dict[str, Any] = { + "use_feature_routing": True, + "active_groups": active_groups, + "active_groups_count": len(active_groups), + "routed_feature_count": len(routed_feature_columns), + } + for model_idx in range(1, num_models + 1): + flat[f"model_{model_idx}_groups"] = model_groups[model_idx] + flat[f"model_{model_idx}_lane"] = rng.randint(0, n_lanes - 1) + for lane_id, steps in lane_steps.items(): + flat[f"lane_{lane_id}_steps"] = steps + return flat + + +def suggest_feature_routing( + trial: "optuna.Trial", + catalog: FeatureCatalog, + num_models: int, + *, + fast: bool = False, +) -> dict[str, Any]: + size_pool = [g for g in catalog.searchable_names if g in SIZE_GROUPS and g != "all"] + candidate: list[str] = [] + if size_pool: + size_group = trial.suggest_categorical("routing_size_group", size_pool) + candidate.append(size_group) + + stat_pool = [g for g in catalog.searchable_names if g in STAT_GROUPS] + for group in stat_pool: + if trial.suggest_categorical(f"routing_use_{group}", [False, True]): + candidate.append(group) + + if not candidate: + candidate = _fit_groups_under_limit(catalog, []) + active_groups = _fit_groups_under_limit(catalog, candidate) + routed_feature_columns = catalog.union(active_groups) + + lane_pool = LANE_PREPROCESSORS_FAST if fast else LANE_PREPROCESSORS_SLOW + n_lanes = trial.suggest_int("routing_n_lanes", 1, min(2, num_models)) + max_groups = FAST_MAX_ACTIVE_GROUPS if fast else SLOW_MAX_ACTIVE_GROUPS + lane_steps: dict[int, list[str]] = {} + for lane_id in range(n_lanes): + use_lane_pp = trial.suggest_categorical( + f"routing_lane_{lane_id}_use_pp", [False, True] + ) + if use_lane_pp: + step = trial.suggest_categorical( + f"routing_lane_{lane_id}_step", list(lane_pool) + ) + lane_steps[lane_id] = ( + [step] if step not in ("StandardScaler", "RobustScaler") else [] + ) + else: + lane_steps[lane_id] = [] + + model_groups: dict[int, list[str]] = {i: [] for i in range(1, num_models + 1)} + for group in active_groups: + model_idx = trial.suggest_int(f"routing_assign_{group}_model", 1, num_models) + model_groups[model_idx].append(group) + + flat: dict[str, Any] = { + "use_feature_routing": True, + "active_groups": active_groups, + "active_groups_count": len(active_groups), + "routed_feature_count": len(routed_feature_columns), + } + for model_idx in range(1, num_models + 1): + lane = trial.suggest_int(f"routing_model_{model_idx}_lane", 0, n_lanes - 1) + fallback_idx = trial.suggest_int( + f"routing_model_{model_idx}_fallback_idx", 0, max_groups - 1 + ) + if not model_groups[model_idx]: + model_groups[model_idx] = [active_groups[fallback_idx % len(active_groups)]] + flat[f"model_{model_idx}_groups"] = model_groups[model_idx] + flat[f"model_{model_idx}_lane"] = lane + for lane_id, steps in lane_steps.items(): + flat[f"lane_{lane_id}_steps"] = steps + return flat + + +def validate_routing( + flat: dict[str, Any], + catalog: FeatureCatalog, + num_models: int, +) -> None: + if not flat.get("use_feature_routing"): + return + active = flat.get("active_groups") or [] + if not active: + raise ValueError("use_feature_routing requires non-empty active_groups") + for group in active: + if group not in catalog.feature_sets: + raise ValueError(f"Unknown active group: {group}") + for model_idx in range(1, num_models + 1): + groups = flat.get(f"model_{model_idx}_groups") or [] + if not groups: + raise ValueError(f"model_{model_idx} has no assigned groups") + for group in groups: + if group not in active: + raise ValueError(f"model_{model_idx} references inactive group {group}") + + +def resolve_feature_routing( + flat: dict[str, Any], + catalog: FeatureCatalog, +) -> FeatureRoutingResult: + if not flat.get("use_feature_routing"): + cols = catalog.columns("medium") if "medium" in catalog.feature_sets else [] + return FeatureRoutingResult( + build_path="default", + feature_groups={}, + feature_columns=cols, + pipeline_config_patch={}, + ) + + num_models = int(flat.get("num_models", 1)) + validate_routing(flat, catalog, num_models) + + active_groups: list[str] = list(flat.get("active_groups") or []) + feature_groups = {g: catalog.columns(g) for g in active_groups} + feature_columns = catalog.union(active_groups) + if len(feature_columns) > MAX_ROUTED_FEATURES: + raise ValueError( + "Feature routing exceeds max feature limit: " + f"{len(feature_columns)} > {MAX_ROUTED_FEATURES}" + ) + + model_group_map: dict[int, list[str]] = {} + model_lane_map: dict[int, int] = {} + for model_idx in range(1, num_models + 1): + model_group_map[model_idx] = list(flat.get(f"model_{model_idx}_groups") or []) + model_lane_map[model_idx] = int(flat.get(f"model_{model_idx}_lane", 0)) + + unique_lanes = {model_lane_map[i] for i in range(1, num_models + 1)} + single_lane_steps: list[str] = [] + if num_models == 1: + lane_id = model_lane_map[1] + single_lane_steps = list(flat.get(f"lane_{lane_id}_steps") or []) + single_has_lane_preprocessors = bool( + _lane_steps_to_preprocessors(single_lane_steps) + ) + + if num_models > 1: + build_path: BuildPath = "multihead" + elif len(unique_lanes) > 1 or single_has_lane_preprocessors: + build_path = "grouped" + else: + build_path = "simple" + + patch: dict[str, Any] = {"feature_groups": feature_groups} + + if build_path == "simple": + union_cols = catalog.union(model_group_map.get(1, active_groups)) + patch["models"] = [ + { + "input_columns": union_cols, + } + ] + elif build_path == "grouped": + groups_for_model = model_group_map[1] + lane_groups: dict[str, list[str]] = {} + lane_pipelines: dict[str, list[dict[str, Any]]] = {} + lane_id = model_lane_map[1] + steps = flat.get(f"lane_{lane_id}_steps") or [] + local_steps = _lane_steps_to_preprocessors(list(steps)) + for group in groups_for_model: + lane_groups[group] = catalog.columns(group) + lane_pipelines[group] = list(local_steps) + patch["preprocessors"] = [ + { + "type": "Grouped", + "params": { + "groups": lane_groups, + "pipelines": lane_pipelines, + }, + } + ] + patch["models"] = [{"input_columns": feature_columns}] + else: + models_patch: list[dict[str, Any]] = [] + for model_idx in range(1, num_models + 1): + lane_id = model_lane_map[model_idx] + steps = flat.get(f"lane_{lane_id}_steps") or [] + local_pp = _lane_steps_to_preprocessors(list(steps)) + models_patch.append( + { + "input_groups": model_group_map[model_idx], + "preprocessors": local_pp, + } + ) + patch["models"] = models_patch + + return FeatureRoutingResult( + build_path=build_path, + feature_groups=feature_groups, + feature_columns=feature_columns, + pipeline_config_patch=patch, + ) + + +def merge_routing_into_pipeline_config( + pipeline_cfg: dict[str, Any], + routing: FeatureRoutingResult, +) -> dict[str, Any]: + if routing.build_path == "default": + return pipeline_cfg + + cfg = dict(pipeline_cfg) + patch = routing.pipeline_config_patch + + if "preprocessors" in patch: + if routing.build_path == "grouped": + cfg["preprocessors"] = list(cfg.get("preprocessors", [])) + list( + patch["preprocessors"] + ) + else: + cfg["preprocessors"] = patch["preprocessors"] + + if "feature_groups" in patch: + cfg["feature_groups"] = patch["feature_groups"] + + models = [dict(m) for m in cfg.get("models", [])] + patch_models = patch.get("models") or [] + for i, pm in enumerate(patch_models): + if i >= len(models): + break + models[i] = {**models[i], **pm} + cfg["models"] = models + return cfg diff --git a/src/alphapulse/hpo/objective.py b/src/alphapulse/hpo/objective.py index 70e66f1..ec9772c 100644 --- a/src/alphapulse/hpo/objective.py +++ b/src/alphapulse/hpo/objective.py @@ -1,5 +1,7 @@ +import gc from dataclasses import dataclass, field -from typing import Any +from pathlib import Path +from typing import Any, Literal, cast import numpy as np import pandas as pd @@ -15,9 +17,25 @@ EraSplitEvaluator, ) from ..experiments.split import internal_val_split +from ..features.catalog import FeatureCatalog, load_feature_catalog from ..models.diffusion_augmenter import SyntheticDataAugmenter -from .builder import build_pipeline_or_multi +from ..pipeline.ensemble import needs_internal_val_for_ensemble +from ..pipeline.ensemble_optimizer import ( + DEFAULT_MAX_WEIGHT, + DEFAULT_MIN_WEIGHT, + EnsembleOptimizer, +) +from ..pipeline.pipeline import Pipeline +from .builder import ( + build_multi_target_from_config, + build_pipeline_or_multi, +) +from .feature_routing import ( + merge_routing_into_pipeline_config, + resolve_feature_routing, +) from .search_space import get_train_kwargs_from_flat, resolve_flat_config +from .target_strategy import strategy_from_flat @dataclass(frozen=True) @@ -38,6 +56,28 @@ def ray_trainable(config: dict[str, Any], **kwargs: Any) -> dict[str, float]: return run_trial(config, **kwargs) +def _resolve_pipeline_cfg( + config: dict[str, Any], + catalog: FeatureCatalog | None, +) -> tuple[dict[str, Any], dict[str, list[str]] | None, list[str] | None]: + pipeline_cfg = resolve_flat_config(config) + if not config.get("use_feature_routing"): + return pipeline_cfg, None, None + + cat = catalog + if cat is None: + data_dir = config.get("_data_dir") + if not data_dir: + raise ValueError("_data_dir required when use_feature_routing is enabled") + cat = load_feature_catalog(data_dir) + + routing = resolve_feature_routing(config, cat) + pipeline_cfg = merge_routing_into_pipeline_config(pipeline_cfg, routing) + feature_groups = routing.feature_groups or None + feature_columns = routing.feature_columns or None + return pipeline_cfg, feature_groups, feature_columns + + def _apply_synthetic_augmentation( X_tr: pd.DataFrame, y_tr: pd.Series, @@ -74,6 +114,90 @@ def _apply_synthetic_augmentation( return X_out, y_out +def _is_multi_target(flat_config: dict[str, Any] | None) -> bool: + if not flat_config: + return False + return flat_config.get("target_mode") == "multi_blend" and bool( + flat_config.get("auxiliary_targets") + ) + + +def _needs_validation_ensemble_opt(pipeline_cfg: dict[str, Any]) -> bool: + if pipeline_cfg.get("ensemble_method") != "weighted": + return False + params = pipeline_cfg.get("ensemble_params") or {} + if not params.get("optimize_weights"): + return False + if params.get("weights") is not None: + return False + return len(pipeline_cfg.get("models") or []) > 1 + + +def _optimize_ensemble_on_validation( + pipeline: Pipeline, + *, + data_dir: Path, + feature_cols: list[str], + target_col: str, + train_subsample: float, + seed: int | None, + pipeline_cfg: dict[str, Any], + flat_config: dict[str, Any] | None = None, +) -> np.ndarray | None: + from ..experiments.data import load_mmc_validation_frame + + frame = load_mmc_validation_frame( + data_dir, + feature_cols=feature_cols, + target_col=target_col, + train_subsample=train_subsample, + seed=seed or 42, + ) + if frame is None: + return None + + X_val, y_val, era_val, meta_preds = frame + X_feat = X_val[feature_cols] + pred_matrix = pipeline.predict_model_matrix(X_feat) + ensemble_params = pipeline_cfg.get("ensemble_params") or {} + objective = str( + ensemble_params.get( + "objective", + flat_config.get("hpo_objective", "payout_score") + if flat_config + else "payout_score", + ) + ) + if objective not in ("corr_sharpe", "payout_score"): + objective = "payout_score" + objective_lit = cast(Literal["corr_sharpe", "payout_score"], objective) + + optimizer = EnsembleOptimizer( + objective=objective_lit, + corr_weight=float(ensemble_params.get("corr_weight", 0.75)), + mmc_weight=float(ensemble_params.get("mmc_weight", 2.25)), + min_weight=float(ensemble_params.get("min_weight", DEFAULT_MIN_WEIGHT)), + max_weight=float(ensemble_params.get("max_weight", DEFAULT_MAX_WEIGHT)), + seed=int(ensemble_params.get("seed", seed or 42)), + ) + min_w = ensemble_params.get("min_weights") + max_w = ensemble_params.get("max_weights") + optimizer.fit( + pred_matrix, + y_val.to_numpy(dtype=np.float64), + era_val, + meta_model_preds=meta_preds, + min_weights=list(min_w) if min_w is not None else None, + max_weights=list(max_w) if max_w is not None else None, + ) + if optimizer.weights_ is None: + return None + pipeline.set_ensemble_weights(optimizer.weights_) + if flat_config is not None: + flat_config["ensemble_weights"] = [float(w) for w in optimizer.weights_] + return optimizer.weights_ + + def _fit_pipeline( pipeline_cfg: dict[str, Any], feature_cols: list[str], @@ -82,28 +206,182 @@ def _fit_pipeline( train_kwargs: dict[str, Any], flat_config: dict[str, Any] | None = None, seed: int | None = None, + feature_groups: dict[str, list[str]] | None = None, + targets_df: pd.DataFrame | None = None, ) -> Any: X_fit_src = X_tr y_fit_src = y_tr - if flat_config and flat_config.get("use_augmentation"): + targets_fit = targets_df + if ( + flat_config + and flat_config.get("use_augmentation") + and not _is_multi_target(flat_config) + ): X_fit_src, y_fit_src = _apply_synthetic_augmentation( X_tr, y_tr, flat_config, feature_cols, seed ) - pipeline = build_pipeline_or_multi( - pipeline_cfg, feature_columns=feature_cols, feature_groups=None - ) + if targets_fit is not None: + targets_fit = targets_fit.loc[X_tr.index].reset_index(drop=True) + aug_targets = targets_fit.copy() + targets_fit = pd.concat( + [aug_targets, aug_targets.iloc[:0]], ignore_index=True + ) + era_col = X_fit_src["era"] if "era" in X_fit_src.columns else None - stacking_needs_val = ( - pipeline_cfg.get("ensemble_method") == "stacking" - and len(pipeline_cfg.get("models", [])) > 1 + force_internal = needs_internal_val_for_ensemble(pipeline_cfg) + + if _is_multi_target(flat_config): + assert flat_config is not None + pipeline: Any = build_multi_target_from_config( + pipeline_cfg, + flat_config, + feature_columns=feature_cols, + feature_groups=feature_groups, + ) + X_fit, _, X_val_inner, _ = internal_val_split( + X_fit_src, + y_fit_src, + era_train=era_col, + force_internal=force_internal, + ) + targets_split = ( + targets_fit.loc[X_fit_src.index] if targets_fit is not None else None + ) + targets_train = ( + targets_split.loc[X_fit.index] if targets_split is not None else None + ) + targets_val = ( + targets_split.loc[X_val_inner.index] + if targets_split is not None and X_val_inner is not None + else None + ) + era_train_fit = era_col.loc[X_fit.index] if era_col is not None else None + era_val_fit = ( + era_col.loc[X_val_inner.index] + if era_col is not None and X_val_inner is not None + else None + ) + pipeline.fit( + X_fit.drop(columns=["era"], errors="ignore"), + targets_train, + X_val=X_val_inner.drop(columns=["era"], errors="ignore") + if X_val_inner is not None + else None, + targets_val=targets_val, + era_train=era_train_fit, + era_val=era_val_fit, + **train_kwargs, + ) + return pipeline + + pipeline = build_pipeline_or_multi( + pipeline_cfg, + feature_columns=feature_cols, + feature_groups=feature_groups, ) X_fit, y_fit, X_val_inner, y_val_inner = internal_val_split( - X_fit_src, y_fit_src, era_train=era_col, force_internal=stacking_needs_val + X_fit_src, y_fit_src, era_train=era_col, force_internal=force_internal + ) + era_val_fit = ( + era_col.loc[X_val_inner.index] + if era_col is not None and X_val_inner is not None + else None ) - pipeline.fit(X_fit, y_fit, X_val=X_val_inner, y_val=y_val_inner, **train_kwargs) + pipeline.fit( + X_fit, + y_fit, + X_val=X_val_inner, + y_val=y_val_inner, + era_val=era_val_fit, + **train_kwargs, + ) + if flat_config is not None and hasattr(pipeline, "ensemble_weights"): + weights = pipeline.ensemble_weights + if weights is not None: + flat_config["ensemble_weights"] = weights return pipeline +_HOLDOUT_METRIC_KEYS = ( + "corr_sharpe", + "mean_per_era_correlation", + "std_per_era_correlation", + "max_drawdown", + "pct_positive_eras", + "n_valid_eras", +) + + +def _merge_validation_mmc_metrics( + metrics: dict[str, float], + *, + pipeline: Any, + data_dir: Path, + feature_cols: list[str], + target_col: str, + train_subsample: float, + seed: int | None, +) -> tuple[ + dict[str, float], + tuple[pd.DataFrame, pd.Series, pd.Series, np.ndarray] | None, +]: + from ..experiments.data import load_mmc_validation_frame + + frame = load_mmc_validation_frame( + data_dir, + feature_cols=feature_cols, + target_col=target_col, + train_subsample=train_subsample, + seed=seed or 42, + ) + if frame is None: + return metrics, None + + X_val, y_val, era_val, meta_preds = frame + mmc_metrics = Backtester(pipeline, feature_columns=feature_cols).evaluate( + X_val, + y_val, + era_val, + meta_model_preds=meta_preds, + ) + merged = dict(metrics) + for key in _HOLDOUT_METRIC_KEYS: + if key in merged: + merged[f"holdout_{key}"] = merged[key] + for key in ("mmc", "mmc_sharpe", "payout_score"): + value = mmc_metrics.get(key) + if value is not None and np.isfinite(value): + merged[key] = float(value) + for key in _HOLDOUT_METRIC_KEYS: + value = mmc_metrics.get(key) + if value is not None and np.isfinite(value): + merged[f"val_{key}"] = float(value) + return merged, frame + + +def _holdout_metrics_for_diagnostics(metrics: dict[str, float]) -> dict[str, float]: + scoped: dict[str, float] = {} + for key in _HOLDOUT_METRIC_KEYS: + holdout_key = f"holdout_{key}" + if holdout_key in metrics: + scoped[key] = float(metrics[holdout_key]) + elif key in metrics: + scoped[key] = float(metrics[key]) + return scoped + + +def _validation_metrics_for_diagnostics(metrics: dict[str, float]) -> dict[str, float]: + scoped: dict[str, float] = {} + for key in _HOLDOUT_METRIC_KEYS: + val_key = f"val_{key}" + if val_key in metrics: + scoped[key] = float(metrics[val_key]) + for key in ("mmc", "mmc_sharpe", "payout_score"): + if key in metrics: + scoped[key] = float(metrics[key]) + return scoped + + def _evaluate_holdout( *, X_train: pd.DataFrame, @@ -115,6 +393,8 @@ def _evaluate_holdout( holdout_eras: int = HPO_FAST_HOLDOUT_ERAS, config: dict[str, Any] | None = None, seed: int | None = None, + feature_groups: dict[str, list[str]] | None = None, + targets_df: pd.DataFrame | None = None, ) -> dict[str, float]: eras_sorted = sorted(era_train.unique(), key=str) min_train = WF_MIN_TRAIN_ERAS @@ -135,16 +415,52 @@ def _evaluate_holdout( train_kwargs, flat_config=config, seed=seed, + feature_groups=feature_groups, + targets_df=targets_df.loc[train_mask] if targets_df is not None else None, ) + if ( + config + and config.get("_data_dir") + and isinstance(pipeline, Pipeline) + and _needs_validation_ensemble_opt(pipeline_cfg) + ): + data_dir = Path(str(config["_data_dir"])) + target_col = str(config.get("primary_target", "target")) + train_subsample = float(config.get("_train_subsample", 1.0)) + _optimize_ensemble_on_validation( + pipeline, + data_dir=data_dir, + feature_cols=feature_cols, + target_col=target_col, + train_subsample=train_subsample, + seed=seed, + pipeline_cfg=pipeline_cfg, + flat_config=config, + ) ho_mask = era_train.isin(holdout_set) X_ho = X_train.loc[ho_mask] y_ho = y_train.loc[ho_mask] era_ho = era_train.loc[ho_mask] + metrics = Backtester(pipeline, feature_columns=feature_cols).evaluate( X_ho, y_ho, era_ho, ) + mmc_frame: tuple[pd.DataFrame, pd.Series, pd.Series, np.ndarray] | None = None + if config and config.get("_data_dir"): + data_dir = Path(str(config["_data_dir"])) + target_col = str(config.get("primary_target", "target")) + train_subsample = float(config.get("_train_subsample", 1.0)) + metrics, mmc_frame = _merge_validation_mmc_metrics( + metrics, + pipeline=pipeline, + data_dir=data_dir, + feature_cols=feature_cols, + target_col=target_col, + train_subsample=train_subsample, + seed=seed, + ) if config and config.get("log_wandb_diagnostics"): from ..evaluation.wandb_diagnostics import log_experiment_diagnostics @@ -154,10 +470,29 @@ def _evaluate_holdout( y_val=y_ho, era_val=era_ho, feature_cols=feature_cols, - metrics=metrics, + metrics=_holdout_metrics_for_diagnostics(metrics), log_shap=bool(config.get("wandb_log_shap", True)), compute_fnc=False, + split="holdout", ) + if mmc_frame is not None: + X_val, y_val, era_val, meta_preds = mmc_frame + log_experiment_diagnostics( + pipeline=pipeline, + X_val=X_val, + y_val=y_val, + era_val=era_val, + feature_cols=feature_cols, + metrics=_validation_metrics_for_diagnostics(metrics), + meta_model_preds=meta_preds, + log_shap=False, + log_feature_report=False, + log_era_importance=False, + compute_fnc=False, + split="validation", + ) + del pipeline + gc.collect() return metrics @@ -170,55 +505,53 @@ def run_trial( feature_cols: list[str], seed: int | None = None, fast_eval: bool | None = None, + targets_df: pd.DataFrame | None = None, + catalog: FeatureCatalog | None = None, ) -> dict[str, float]: - """Train a single HPO trial and return backtest metrics. - - When ``fast_eval`` is True (default for ``hpo_fast`` configs), scores a - single holdout on the last ``HPO_FAST_HOLDOUT_ERAS`` eras instead of full - walk-forward CV. This keeps trials under the 30-minute budget on full data. - - Args: - config: Flat parameter dictionary (as produced by - ``sample_random_config`` or Ray Tune). - X_train: Training feature DataFrame (may include an "era" column). - y_train: Training target Series. - era_train: Era labels aligned to X_train. - feature_cols: Feature column names (must not include "era"). - seed: Optional integer seed for reproducibility. - fast_eval: Override fast holdout mode. When None, reads ``hpo_fast`` - from *config* (defaults to False). - - Returns: - Dictionary of backtest metrics (keys include ``corr_sharpe``, - ``mean_per_era_correlation``, ``max_drawdown``, ``pct_positive_eras``, - ``n_valid_eras``). - """ + """Train a single HPO trial and return backtest metrics.""" if seed is not None: rng = np.random.default_rng(seed) np.random.seed(rng.integers(0, 2**31)) use_fast = fast_eval if fast_eval is not None else bool(config.get("hpo_fast")) - pipeline_cfg = resolve_flat_config(config) + pipeline_cfg, feature_groups, routed_columns = _resolve_pipeline_cfg( + config, catalog + ) + if routed_columns: + feature_cols = [c for c in routed_columns if c in X_train.columns] + if config.get("use_gpu"): from .search_space import apply_gpu_pipeline_config pipeline_cfg = apply_gpu_pipeline_config(pipeline_cfg) train_kwargs = get_train_kwargs_from_flat(config) + strategy = strategy_from_flat(config) + y_eval = ( + targets_df[strategy.primary_target] + if targets_df is not None and strategy.primary_target in targets_df.columns + else y_train + ) + if use_fast: return _evaluate_holdout( X_train=X_train, - y_train=y_train, + y_train=y_eval, era_train=era_train, feature_cols=feature_cols, pipeline_cfg=pipeline_cfg, train_kwargs=train_kwargs, config=config, seed=seed, + feature_groups=feature_groups, + targets_df=targets_df, ) def train_fn(X_tr: pd.DataFrame, y_tr: pd.Series) -> Any: + local_targets = None + if targets_df is not None: + local_targets = targets_df.loc[X_tr.index] return _fit_pipeline( pipeline_cfg, feature_cols, @@ -227,6 +560,8 @@ def train_fn(X_tr: pd.DataFrame, y_tr: pd.Series) -> Any: train_kwargs, flat_config=config, seed=seed, + feature_groups=feature_groups, + targets_df=local_targets, ) diagnostics_state: dict[str, Any] = {} @@ -246,7 +581,7 @@ def last_fold_callback( min_train_eras=WF_MIN_TRAIN_ERAS, ).evaluate_walk_forward( X_train, - y_train, + y_eval, era_train, train_fn, last_fold_callback=last_fold_callback @@ -266,6 +601,7 @@ def last_fold_callback( log_shap=bool(config.get("wandb_log_shap", True)), compute_fnc=False, ) + gc.collect() return metrics @@ -277,20 +613,36 @@ def run_trial_fast_walk_forward( era_train: pd.Series, feature_cols: list[str], seed: int | None = None, + targets_df: pd.DataFrame | None = None, + catalog: FeatureCatalog | None = None, ) -> dict[str, float]: """Walk-forward with reduced folds and capped train window for HPO.""" if seed is not None: rng = np.random.default_rng(seed) np.random.seed(rng.integers(0, 2**31)) - pipeline_cfg = resolve_flat_config(config) + pipeline_cfg, feature_groups, routed_columns = _resolve_pipeline_cfg( + config, catalog + ) + if routed_columns: + feature_cols = [c for c in routed_columns if c in X_train.columns] if config.get("use_gpu"): from .search_space import apply_gpu_pipeline_config pipeline_cfg = apply_gpu_pipeline_config(pipeline_cfg) train_kwargs = get_train_kwargs_from_flat(config) + strategy = strategy_from_flat(config) + y_eval = ( + targets_df[strategy.primary_target] + if targets_df is not None and strategy.primary_target in targets_df.columns + else y_train + ) + def train_fn(X_tr: pd.DataFrame, y_tr: pd.Series) -> Any: + local_targets = None + if targets_df is not None: + local_targets = targets_df.loc[X_tr.index] return _fit_pipeline( pipeline_cfg, feature_cols, @@ -299,12 +651,16 @@ def train_fn(X_tr: pd.DataFrame, y_tr: pd.Series) -> Any: train_kwargs, flat_config=config, seed=seed, + feature_groups=feature_groups, + targets_df=local_targets, ) - return EraSplitEvaluator( + metrics = EraSplitEvaluator( feature_columns=feature_cols, n_splits=HPO_FAST_WF_N_SPLITS, n_purge=WF_N_PURGE, min_train_eras=WF_MIN_TRAIN_ERAS, max_train_eras=HPO_FAST_MAX_TRAIN_ERAS, - ).evaluate_walk_forward(X_train, y_train, era_train, train_fn) + ).evaluate_walk_forward(X_train, y_eval, era_train, train_fn) + gc.collect() + return metrics diff --git a/src/alphapulse/hpo/optuna_search.py b/src/alphapulse/hpo/optuna_search.py new file mode 100644 index 0000000..09083f4 --- /dev/null +++ b/src/alphapulse/hpo/optuna_search.py @@ -0,0 +1,377 @@ +from pathlib import Path +from typing import Any, Literal + +import optuna +from optuna.samplers import RandomSampler, TPESampler +from optuna.trial import TrialState + +from ..features.catalog import load_feature_catalog, load_target_catalog +from .feature_routing import suggest_feature_routing +from .search_space import ( + BOOSTING_MODELS, + FOUNDATION_MODELS, + NEUTRALIZATION_PROPORTION_RANGE, + _finalize_neutralization_sampling, + _sampled_model_types, + available_foundation_models, + uses_neutralization_for_models, +) +from .target_strategy import apply_target_strategy_to_flat, suggest_target_strategy + +SamplerName = Literal["tpe", "random"] +DEFAULT_N_STARTUP_TRIALS = 25 + + +def create_hpo_study( + output_dir: Path, + *, + seed: int, + sampler: SamplerName = "tpe", + resume: bool = False, + n_startup_trials: int = DEFAULT_N_STARTUP_TRIALS, +) -> optuna.Study: + storage_url = f"sqlite:///{(output_dir / 'optuna.db').resolve()}" + optuna_sampler: TPESampler | RandomSampler + if sampler == "tpe": + optuna_sampler = TPESampler( + seed=seed, + multivariate=True, + warn_independent_sampling=False, + n_startup_trials=n_startup_trials, + ) + else: + optuna_sampler = RandomSampler(seed=seed) + return optuna.create_study( + direction="maximize", + sampler=optuna_sampler, + storage=storage_url, + study_name="alphapulse_hpo", + load_if_exists=resume, + ) + + +def tell_trial_result( + study: optuna.Study, + optuna_trial: optuna.trial.Trial, + score: float, + *, + failed: bool = False, +) -> None: + if failed or score != score or score in (float("inf"), float("-inf")): + study.tell(optuna_trial, state=TrialState.FAIL) + return + study.tell(optuna_trial, score) + + +def _model_pool(*, fast: bool) -> list[str]: + pool = list(BOOSTING_MODELS) + pool.extend(available_foundation_models(hpo_fast=fast)) + return list(dict.fromkeys(pool)) + + +def _suggest_model_type(trial: optuna.Trial, param: str, *, fast: bool) -> str: + return trial.suggest_categorical(param, _model_pool(fast=fast)) + + +def _active_model_types(cfg: dict[str, Any]) -> set[str]: + return set(_sampled_model_types(cfg)) + + +def _suggest_xgb_params(trial: optuna.Trial, *, fast: bool) -> dict[str, Any]: + params: dict[str, Any] = { + "xgb_max_depth": trial.suggest_categorical("xgb_max_depth", [3, 5, 7]), + "xgb_learning_rate": trial.suggest_float( + "xgb_learning_rate", 1e-3, 0.1, log=True + ), + } + if fast: + params["xgb_n_rounds"] = trial.suggest_categorical( + "xgb_n_rounds", [150, 250, 400] + ) + params["xgb_early_stopping"] = trial.suggest_categorical( + "xgb_early_stopping", [20, 30, 50] + ) + else: + params["xgb_n_rounds"] = trial.suggest_categorical( + "xgb_n_rounds", [300, 500, 800] + ) + params["xgb_early_stopping"] = trial.suggest_categorical( + "xgb_early_stopping", [30, 50, 100] + ) + return params + + +def _suggest_lgbm_params(trial: optuna.Trial, *, fast: bool) -> dict[str, Any]: + params: dict[str, Any] = { + "lgbm_num_leaves": trial.suggest_categorical( + "lgbm_num_leaves", [16, 31, 63] if fast else [16, 31, 63, 127] + ), + "lgbm_learning_rate": trial.suggest_float( + "lgbm_learning_rate", 5e-3, 0.05, log=True + ), + "lgbm_min_child_samples": trial.suggest_categorical( + "lgbm_min_child_samples", [100, 200, 500] + ), + "lgbm_reg_alpha": trial.suggest_float("lgbm_reg_alpha", 0.1, 2.0), + "lgbm_reg_lambda": trial.suggest_float("lgbm_reg_lambda", 1.0, 10.0), + "lgbm_colsample_bytree": trial.suggest_float("lgbm_colsample_bytree", 0.2, 0.5), + "lgbm_subsample": trial.suggest_float("lgbm_subsample", 0.5, 0.8), + } + if fast: + params["lgbm_n_rounds"] = trial.suggest_categorical( + "lgbm_n_rounds", [200, 400, 600] + ) + params["lgbm_early_stopping"] = trial.suggest_categorical( + "lgbm_early_stopping", [30, 50] + ) + else: + params["lgbm_n_rounds"] = trial.suggest_categorical( + "lgbm_n_rounds", [300, 500, 800, 1500] + ) + params["lgbm_early_stopping"] = trial.suggest_categorical( + "lgbm_early_stopping", [50, 100] + ) + return params + + +def _suggest_catboost_params(trial: optuna.Trial) -> dict[str, Any]: + return { + "catboost_depth": trial.suggest_categorical("catboost_depth", [4, 5, 6]), + "catboost_learning_rate": trial.suggest_float( + "catboost_learning_rate", 0.01, 0.05, log=True + ), + "catboost_l2_leaf_reg": trial.suggest_float("catboost_l2_leaf_reg", 3.0, 15.0), + "catboost_min_data_in_leaf": trial.suggest_categorical( + "catboost_min_data_in_leaf", [100, 200, 500] + ), + "catboost_colsample_bylevel": trial.suggest_float( + "catboost_colsample_bylevel", 0.2, 0.4 + ), + } + + +def _suggest_packboost_model_params( + trial: optuna.Trial, *, fast: bool +) -> dict[str, Any]: + params: dict[str, Any] = { + "packboost_model_n_worst_eras": trial.suggest_categorical( + "packboost_model_n_worst_eras", [3, 5, 7] + ), + "packboost_model_boost_weight": trial.suggest_float( + "packboost_model_boost_weight", 0.2, 0.5 + ), + "packboost_model_n_rounds_boost": trial.suggest_categorical( + "packboost_model_n_rounds_boost", [100, 200] + ), + } + if fast: + params["packboost_model_n_rounds_base"] = trial.suggest_categorical( + "packboost_model_n_rounds_base", [200, 300] + ) + else: + params["packboost_model_n_rounds_base"] = trial.suggest_categorical( + "packboost_model_n_rounds_base", [300, 500] + ) + return params + + +def _suggest_packboost_preprocessor_params( + trial: optuna.Trial, *, fast: bool +) -> dict[str, Any]: + params: dict[str, Any] = { + "packboost_n_worst_eras": trial.suggest_categorical( + "packboost_n_worst_eras", [3, 5, 7] + ), + "packboost_boost_weight": trial.suggest_float( + "packboost_boost_weight", 0.1, 0.5 + ), + "packboost_n_rounds_boost": trial.suggest_categorical( + "packboost_n_rounds_boost", [100, 150, 200] + ), + } + if fast: + params["packboost_n_rounds_base"] = trial.suggest_categorical( + "packboost_n_rounds_base", [150, 250, 350] + ) + else: + params["packboost_n_rounds_base"] = trial.suggest_categorical( + "packboost_n_rounds_base", [200, 300, 500] + ) + return params + + +def _suggest_foundation_params(trial: optuna.Trial, *, fast: bool) -> dict[str, Any]: + if fast: + return { + "foundation_max_train_rows": trial.suggest_categorical( + "foundation_max_train_rows", [2_000, 3_000, 5_000] + ), + "foundation_compression": trial.suggest_categorical( + "foundation_compression", ["pca", "svd"] + ), + "foundation_n_components": trial.suggest_categorical( + "foundation_n_components", [64, 128, 256] + ), + "foundation_n_estimators": trial.suggest_categorical( + "foundation_n_estimators", [2, 4] + ), + "foundation_compression_epochs": trial.suggest_categorical( + "foundation_compression_epochs", [5, 10] + ), + } + return { + "foundation_max_train_rows": trial.suggest_categorical( + "foundation_max_train_rows", [5_000, 10_000, 20_000] + ), + "foundation_compression": trial.suggest_categorical( + "foundation_compression", ["pca", "svd"] + ), + "foundation_n_components": trial.suggest_categorical( + "foundation_n_components", [128, 256, 512] + ), + "foundation_n_estimators": trial.suggest_categorical( + "foundation_n_estimators", [2, 4, 8] + ), + "foundation_compression_epochs": trial.suggest_categorical( + "foundation_compression_epochs", [5, 10, 20] + ), + } + + +def _suggest_model_hyperparams( + trial: optuna.Trial, + active_types: set[str], + *, + fast: bool, +) -> dict[str, Any]: + params: dict[str, Any] = {} + if "XGBoost" in active_types: + params.update(_suggest_xgb_params(trial, fast=fast)) + if "LightGBM" in active_types: + params.update(_suggest_lgbm_params(trial, fast=fast)) + if "CatBoost" in active_types: + params.update(_suggest_catboost_params(trial)) + if "Packboost" in active_types: + params.update(_suggest_packboost_model_params(trial, fast=fast)) + if active_types.intersection(FOUNDATION_MODELS): + params.update(_suggest_foundation_params(trial, fast=fast)) + return params + + +def _suggest_core_params( + trial: optuna.Trial, *, fast: bool, max_models: int = 3 +) -> dict[str, Any]: + cfg: dict[str, Any] = { + "scaler_type": trial.suggest_categorical( + "scaler_type", ["StandardScaler", "RobustScaler"] + ), + "use_gpu": False, + "augmenter_backend": "auto", + } + + model_cap = max(1, min(int(max_models), 3)) + if fast: + cfg["hpo_fast"] = True + cfg["use_packboost"] = False + cfg["num_models"] = trial.suggest_int("num_models", 1, model_cap) + else: + cfg["use_packboost"] = trial.suggest_categorical("use_packboost", [False, True]) + cfg["num_models"] = trial.suggest_int("num_models", 1, model_cap) + + num_models = int(cfg["num_models"]) + if num_models > 1: + cfg["ensemble_method"] = trial.suggest_categorical( + "ensemble_method", ["single", "weighted", "stacking"] + ) + else: + cfg["ensemble_method"] = "single" + + cfg["model_1_type"] = _suggest_model_type(trial, "model_1_type", fast=fast) + if num_models >= 2: + cfg["model_2_type"] = _suggest_model_type(trial, "model_2_type", fast=fast) + if num_models >= 3: + cfg["model_3_type"] = _suggest_model_type(trial, "model_3_type", fast=fast) + + if cfg.get("use_packboost"): + cfg.update(_suggest_packboost_preprocessor_params(trial, fast=fast)) + + active_types = _active_model_types(cfg) + cfg.update(_suggest_model_hyperparams(trial, active_types, fast=fast)) + + tree_models = {"XGBoost", "LightGBM", "CatBoost", "RandomForest", "ExtraTrees"} + if active_types.intersection(tree_models): + if fast: + cfg["n_subs"] = trial.suggest_categorical("n_subs", [3, 5]) + else: + cfg["n_subs"] = trial.suggest_categorical("n_subs", [5, 8, 10, 15]) + + if num_models > 1 and cfg.get("ensemble_method") == "stacking": + cfg["stacking_meta_learner"] = trial.suggest_categorical( + "stacking_meta_learner", ["ridge", "xgboost"] + ) + + cfg["use_augmentation"] = trial.suggest_categorical( + "use_augmentation", [False, True] + ) + if cfg["use_augmentation"]: + cfg["augmenter_top_fraction"] = trial.suggest_float( + "augmenter_top_fraction", 0.05, 0.20 + ) + cfg["augmenter_n_synthetic"] = trial.suggest_categorical( + "augmenter_n_synthetic", [200, 500, 1000] + ) + + if uses_neutralization_for_models(list(active_types)): + cfg["use_neutralization"] = True + cfg["neutralization_proportion"] = trial.suggest_float( + "neutralization_proportion", *NEUTRALIZATION_PROPORTION_RANGE + ) + else: + cfg["use_neutralization"] = False + + cfg["use_meta_neutralization"] = trial.suggest_categorical( + "use_meta_neutralization", [False, True] + ) + if cfg["use_meta_neutralization"]: + cfg["meta_neutralization_proportion"] = trial.suggest_float( + "meta_neutralization_proportion", 0.5, 0.75 + ) + + return cfg + + +def suggest_flat_config( + trial: optuna.Trial, + *, + fast: bool = False, + max_models: int | None = None, + data_dir: str | Path | None = None, +) -> dict[str, Any]: + model_cap = ( + 2 if max_models is None and fast else (3 if max_models is None else max_models) + ) + cfg = _finalize_neutralization_sampling( + _suggest_core_params(trial, fast=fast, max_models=model_cap) + ) + + if data_dir is not None: + target_catalog = load_target_catalog(data_dir) + strategy = suggest_target_strategy(trial, target_catalog, fast=fast) + cfg = apply_target_strategy_to_flat(cfg, strategy) + + feature_catalog = load_feature_catalog(data_dir) + routing_fragment = suggest_feature_routing( + trial, + feature_catalog, + int(cfg.get("num_models", 1)), + fast=fast, + ) + cfg.update(routing_fragment) + else: + cfg.setdefault("target_mode", "single") + cfg.setdefault("primary_target", "target") + cfg.setdefault("auxiliary_targets", []) + cfg.setdefault("target_blend_method", "equal") + cfg.setdefault("use_feature_routing", False) + + return cfg diff --git a/src/alphapulse/hpo/registry.py b/src/alphapulse/hpo/registry.py index 31e7db9..466e0d7 100644 --- a/src/alphapulse/hpo/registry.py +++ b/src/alphapulse/hpo/registry.py @@ -5,7 +5,6 @@ from ..models.foundation_models import ( TabICLModel, TabPFN3Model, - TabPFN3ReasoningModel, TabPFNModel, ) from ..models.lightgbm_model import LightGBMModel @@ -128,15 +127,6 @@ "random_state": 42, }, ), - "TabPFN3Reasoning": ( - TabPFN3ReasoningModel, - { - "thinking_mode": True, - "thinking_effort": "medium", - "thinking_timeout_s": 300, - "thinking_metric": "rmse", - }, - ), "TabICL": ( TabICLModel, { diff --git a/src/alphapulse/hpo/search_space.py b/src/alphapulse/hpo/search_space.py index c8ad6f1..f8654d8 100644 --- a/src/alphapulse/hpo/search_space.py +++ b/src/alphapulse/hpo/search_space.py @@ -1,4 +1,5 @@ import random as _random_mod +from pathlib import Path from typing import Any try: @@ -7,16 +8,23 @@ tune = None from ..evaluation.era_split import HPO_FAST_N_SUBS_CAP +from ..features.catalog import load_feature_catalog, load_target_catalog +from .feature_routing import sample_feature_routing +from .target_strategy import apply_target_strategy_to_flat, sample_target_strategy BOOSTING_MODELS = ["XGBoost", "LightGBM", "Packboost", "CatBoost"] -FOUNDATION_MODELS = ["TabPFN", "TabICL", "TabPFN3", "TabPFN3Reasoning"] +FOUNDATION_MODELS = ["TabPFN", "TabICL", "TabPFN3"] FOUNDATION_SAMPLE_PROB = 0.05 AUGMENTER_SAMPLE_PROB = 0.05 HPO_FAST_FOUNDATION_SAMPLE_PROB = 0.03 -HPO_SLOW_FOUNDATION_TYPES = ("TabPFN3", "TabPFN3Reasoning") +HPO_SLOW_FOUNDATION_TYPES = ("TabPFN3",) MIN_NEUTRALIZATION_PROPORTION = 0.15 DEFAULT_NEUTRALIZATION_PROPORTION = 0.35 NEUTRALIZATION_PROPORTION_RANGE = (MIN_NEUTRALIZATION_PROPORTION, 0.8) +MIN_META_NEUTRALIZATION_PROPORTION = 0.5 +DEFAULT_META_NEUTRALIZATION_PROPORTION = 0.6 +META_NEUTRALIZATION_PROPORTION_RANGE = (MIN_META_NEUTRALIZATION_PROPORTION, 0.75) +FOUNDATION_ENSEMBLE_MAX_WEIGHT = 0.35 def uses_neutralization_for_models(model_types: list[str]) -> bool: @@ -61,6 +69,34 @@ def resolve_neutralize_proportion( return max(MIN_NEUTRALIZATION_PROPORTION, min(1.0, proportion)) +def resolve_meta_neutralize_proportion(flat: dict[str, Any]) -> float: + if not flat.get("use_meta_neutralization", False): + return 0.0 + proportion = float( + flat.get( + "meta_neutralization_proportion", + DEFAULT_META_NEUTRALIZATION_PROPORTION, + ) + ) + return max( + MIN_META_NEUTRALIZATION_PROPORTION, + min(1.0, proportion), + ) + + +def _ensemble_weight_bounds( + model_types: list[str], flat: dict[str, Any] +) -> tuple[list[float], list[float]]: + default_min = float(flat.get("ensemble_min_weight", 0.05)) + default_max = float(flat.get("ensemble_max_weight", 0.90)) + min_weights = [default_min] * len(model_types) + max_weights = [ + FOUNDATION_ENSEMBLE_MAX_WEIGHT if t in FOUNDATION_MODELS else default_max + for t in model_types + ] + return min_weights, max_weights + + def _torch_available() -> bool: try: import torch # noqa: F401 @@ -74,7 +110,7 @@ def resolve_foundation_compression( compression: str | None, *, hpo_fast: bool = False ) -> str | None: if compression is None and hpo_fast: - compression = "autoencoder" + compression = "pca" if compression == "autoencoder" and not _torch_available(): return "pca" return compression @@ -94,15 +130,6 @@ def available_foundation_models(*, hpo_fast: bool = False) -> list[str]: available.append("TabICL") except ImportError: pass - import os - - if os.environ.get("TABPFN_API_KEY"): - try: - import tabpfn_client # noqa: F401 - - available.append("TabPFN3Reasoning") - except ImportError: - pass if hpo_fast: return [m for m in available if m not in HPO_SLOW_FOUNDATION_TYPES] return available @@ -137,6 +164,8 @@ def apply_gpu_model_params(model_type: str, params: dict[str, Any]) -> dict[str, else: out["task_type"] = "GPU" out.pop("colsample_bylevel", None) + elif model_type == "Packboost": + out["device"] = "cuda" return out @@ -163,6 +192,18 @@ def apply_gpu_pipeline_config(config: dict[str, Any]) -> dict[str, Any]: } ) cfg["models"] = models + + preprocessors = [] + for item in cfg.get("preprocessors", []): + pp_type = item.get("type", "") + params = dict(item.get("params") or {}) + preprocessors.append( + { + **item, + "params": apply_gpu_model_params(pp_type, params), + } + ) + cfg["preprocessors"] = preprocessors return cfg @@ -187,8 +228,41 @@ def _loguniform(low: float, high: float, rng: _random_mod.Random) -> float: return float(low * (high / low) ** rng.random()) +def _enrich_hpo_sampling( + cfg: dict[str, Any], + rng: _random_mod.Random, + *, + fast: bool, + data_dir: str | Path | None = None, +) -> dict[str, Any]: + if data_dir is not None: + target_catalog = load_target_catalog(data_dir) + strategy = sample_target_strategy(rng, target_catalog, fast=fast) + cfg = apply_target_strategy_to_flat(cfg, strategy) + + feature_catalog = load_feature_catalog(data_dir) + routing_fragment = sample_feature_routing( + rng, + feature_catalog, + int(cfg.get("num_models", 1)), + fast=fast, + ) + cfg.update(routing_fragment) + else: + cfg.setdefault("target_mode", "single") + cfg.setdefault("primary_target", "target") + cfg.setdefault("auxiliary_targets", []) + cfg.setdefault("target_blend_method", "equal") + cfg.setdefault("use_feature_routing", False) + return cfg + + def sample_random_config( - seed: int | None = None, *, phase: str = "phase_b", fast: bool = False + seed: int | None = None, + *, + phase: str = "phase_b", + fast: bool = False, + data_dir: str | Path | None = None, ) -> dict[str, Any]: """Sample a random flat HPO configuration for local search. @@ -227,6 +301,15 @@ def sample_random_config( "lgbm_n_rounds": rng.choice([300, 500, 800]), "lgbm_min_child_samples": rng.choice([100, 200]), "lgbm_early_stopping": rng.choice([50, 100]), + "lgbm_reg_alpha": rng.uniform(0.1, 2.0), + "lgbm_reg_lambda": rng.uniform(1.0, 10.0), + "lgbm_colsample_bytree": rng.uniform(0.2, 0.5), + "lgbm_subsample": rng.uniform(0.5, 0.8), + "catboost_depth": rng.choice([4, 5, 6]), + "catboost_learning_rate": _loguniform(0.01, 0.05, rng), + "catboost_l2_leaf_reg": rng.uniform(3.0, 15.0), + "catboost_min_data_in_leaf": rng.choice([100, 200, 500]), + "catboost_colsample_bylevel": rng.uniform(0.2, 0.4), "packboost_model_n_worst_eras": 3, "packboost_model_boost_weight": rng.uniform(0.2, 0.3), "packboost_model_n_rounds_base": 300, @@ -235,13 +318,19 @@ def sample_random_config( "stacking_meta_learner": "ridge", "use_neutralization": True, "neutralization_proportion": rng.uniform(*NEUTRALIZATION_PROPORTION_RANGE), + "use_meta_neutralization": rng.random() < 0.35, + "meta_neutralization_proportion": rng.uniform( + *META_NEUTRALIZATION_PROPORTION_RANGE + ), } if fast: cfg["hpo_fast"] = True cfg["n_subs"] = rng.choice([3, 5]) cfg["xgb_n_rounds"] = rng.choice([150, 250, 350]) cfg["lgbm_n_rounds"] = rng.choice([200, 400, 600]) - return _finalize_neutralization_sampling(cfg) + return _enrich_hpo_sampling( + _finalize_neutralization_sampling(cfg), rng, fast=fast, data_dir=data_dir + ) cfg = { "scaler_type": rng.choice(["StandardScaler", "RobustScaler"]), @@ -259,11 +348,20 @@ def sample_random_config( "xgb_learning_rate": _loguniform(1e-3, 0.1, rng), "xgb_n_rounds": rng.choice([300, 500, 800]), "xgb_early_stopping": rng.choice([30, 50, 100]), - "lgbm_num_leaves": rng.choice([16, 31, 63, 127]), + "lgbm_num_leaves": rng.choice([16, 31, 63]), "lgbm_learning_rate": _loguniform(5e-3, 0.05, rng), "lgbm_n_rounds": rng.choice([300, 500, 800, 1500]), "lgbm_min_child_samples": rng.choice([100, 200, 500]), "lgbm_early_stopping": rng.choice([50, 100]), + "lgbm_reg_alpha": rng.uniform(0.1, 2.0), + "lgbm_reg_lambda": rng.uniform(1.0, 10.0), + "lgbm_colsample_bytree": rng.uniform(0.2, 0.5), + "lgbm_subsample": rng.uniform(0.5, 0.8), + "catboost_depth": rng.choice([4, 5, 6, 7]), + "catboost_learning_rate": _loguniform(0.01, 0.05, rng), + "catboost_l2_leaf_reg": rng.uniform(3.0, 15.0), + "catboost_min_data_in_leaf": rng.choice([100, 200, 500]), + "catboost_colsample_bylevel": rng.uniform(0.2, 0.4), "packboost_model_n_worst_eras": rng.choice([3, 5, 7]), "packboost_model_boost_weight": rng.uniform(0.2, 0.5), "packboost_model_n_rounds_base": rng.choice([300, 500]), @@ -288,14 +386,16 @@ def sample_random_config( cfg["lgbm_early_stopping"] = rng.choice([30, 50]) cfg["packboost_n_rounds_base"] = rng.choice([150, 250, 350]) cfg["packboost_model_n_rounds_base"] = rng.choice([200, 300]) - cfg["foundation_max_train_rows"] = rng.choice([3_000, 5_000, 8_000]) - cfg["foundation_compression"] = rng.choice(["autoencoder", "pca", "svd"]) + cfg["foundation_max_train_rows"] = rng.choice([2_000, 3_000, 5_000]) + cfg["foundation_compression"] = rng.choice(["pca", "svd"]) cfg["foundation_n_components"] = rng.choice([64, 128, 256]) cfg["foundation_n_estimators"] = rng.choice([2, 4]) cfg["foundation_compression_epochs"] = rng.choice([5, 10]) cfg["use_packboost"] = False cfg["use_augmentation"] = rng.random() < 0.03 - return _finalize_neutralization_sampling(cfg) + return _enrich_hpo_sampling( + _finalize_neutralization_sampling(cfg), rng, fast=fast, data_dir=data_dir + ) def get_full_param_space() -> dict[str, Any]: @@ -365,17 +465,15 @@ def resolve_flat_config(flat: dict[str, Any]) -> dict[str, Any]: {"type": flat.get("scaler_type", "StandardScaler"), "params": {}} ] if flat.get("use_packboost"): - preprocessors.append( - { - "type": "Packboost", - "params": { - "n_worst_eras": flat.get("packboost_n_worst_eras", 5), - "boost_weight": flat.get("packboost_boost_weight", 0.3), - "n_rounds_base": flat.get("packboost_n_rounds_base", 300), - "n_rounds_boost": flat.get("packboost_n_rounds_boost", 100), - }, - } - ) + pb_params: dict[str, Any] = { + "n_worst_eras": flat.get("packboost_n_worst_eras", 5), + "boost_weight": flat.get("packboost_boost_weight", 0.3), + "n_rounds_base": flat.get("packboost_n_rounds_base", 300), + "n_rounds_boost": flat.get("packboost_n_rounds_boost", 100), + } + if flat.get("use_gpu"): + pb_params["device"] = "cuda" + preprocessors.append({"type": "Packboost", "params": pb_params}) def model_params(t: str, index: int) -> dict[str, Any]: if t == "XGBoost": @@ -394,6 +492,11 @@ def model_params(t: str, index: int) -> dict[str, Any]: "num_leaves": flat.get("lgbm_num_leaves", 31), "learning_rate": flat.get("lgbm_learning_rate", 0.01), "min_child_samples": flat.get("lgbm_min_child_samples", 200), + "reg_alpha": flat.get("lgbm_reg_alpha", 0.1), + "reg_lambda": flat.get("lgbm_reg_lambda", 1.0), + "colsample_bytree": flat.get("lgbm_colsample_bytree", 0.3), + "subsample": flat.get("lgbm_subsample", 0.7), + "subsample_freq": 1, "objective": "regression", "metric": "rmse", "verbosity": -1, @@ -451,12 +554,17 @@ def model_params(t: str, index: int) -> dict[str, Any]: if t == "Ridge": return {"alpha": flat.get("ridge_alpha", 100.0)} if t == "Packboost": - return { + params: dict[str, Any] = { "n_worst_eras": flat.get("packboost_model_n_worst_eras", 5), "boost_weight": flat.get("packboost_model_boost_weight", 0.3), "n_rounds_base": flat.get("packboost_model_n_rounds_base", 500), "n_rounds_boost": flat.get("packboost_model_n_rounds_boost", 200), + "max_depth": flat.get("packboost_max_depth", 7), + "nfolds": flat.get("packboost_nfolds", 8), } + if flat.get("use_gpu"): + params["device"] = "cuda" + return params if t in FOUNDATION_MODELS: key_map = { "foundation_max_train_rows": "max_train_rows", @@ -494,12 +602,31 @@ def model_params(t: str, index: int) -> dict[str, Any]: ensemble_method = "single" ensemble_params: dict[str, Any] = {} if ensemble_method == "weighted" and num_models > 1: - ensemble_params["weights"] = [1.0 / num_models] * num_models + saved_weights = flat.get("ensemble_weights") + if saved_weights and len(saved_weights) == num_models: + ensemble_params["weights"] = list(saved_weights) + elif flat.get("ensemble_params", {}).get("weights") is not None: + ensemble_params["weights"] = flat["ensemble_params"]["weights"] + else: + ensemble_params["optimize_weights"] = True + ensemble_params["objective"] = flat.get( + "ensemble_objective", + flat.get("hpo_objective", "corr_sharpe"), + ) + ensemble_params["min_weight"] = float(flat.get("ensemble_min_weight", 0.05)) + ensemble_params["max_weight"] = float(flat.get("ensemble_max_weight", 0.90)) + min_w, max_w = _ensemble_weight_bounds(types, flat) + if any(t in FOUNDATION_MODELS for t in types): + ensemble_params["min_weights"] = min_w + ensemble_params["max_weights"] = max_w + ensemble_params["corr_weight"] = float(flat.get("corr_weight", 0.75)) + ensemble_params["mmc_weight"] = float(flat.get("mmc_weight", 2.25)) if ensemble_method == "stacking" and num_models > 1: ensemble_params["meta_learner"] = flat.get("stacking_meta_learner", "ridge") ensemble_params["meta_params"] = {} neutralize_proportion = resolve_neutralize_proportion(flat, types) + meta_neutralize_proportion = resolve_meta_neutralize_proportion(flat) return { "preprocessors": preprocessors, @@ -507,6 +634,7 @@ def model_params(t: str, index: int) -> dict[str, Any]: "ensemble_method": ensemble_method, "ensemble_params": ensemble_params, "neutralize_proportion": neutralize_proportion, + "meta_neutralize_proportion": meta_neutralize_proportion, } diff --git a/src/alphapulse/hpo/target_strategy.py b/src/alphapulse/hpo/target_strategy.py new file mode 100644 index 0000000..2aee9e7 --- /dev/null +++ b/src/alphapulse/hpo/target_strategy.py @@ -0,0 +1,245 @@ +import random +from dataclasses import dataclass, field +from pathlib import Path +from typing import TYPE_CHECKING, Any, Literal, cast + +import pandas as pd + +from ..features.catalog import TargetCatalog, load_target_catalog +from ..pipeline.multi_target import _MIN_TRAIN_ROWS + +if TYPE_CHECKING: + import optuna + +TargetMode = Literal["single", "multi_blend"] +MAX_AUXILIARY_RESAMPLE_ATTEMPTS = 3 +MAX_NAN_FRACTION = 0.5 + +PRIMARY_TARGET = "target" + + +@dataclass +class TargetStrategy: + target_mode: TargetMode + primary_target: str + auxiliary_targets: list[str] = field(default_factory=list) + target_blend_method: Literal["equal", "sharpe"] = "equal" + + +@dataclass(frozen=True) +class TargetValidationResult: + strategy: TargetStrategy + ok: bool + reason: str | None = None + + +def _sample_auxiliary( + rng: random.Random, + catalog: TargetCatalog, + primary: str, + *, + max_aux: int, +) -> list[str]: + pool = [t for t in catalog.targets if t != primary] + if not pool or max_aux <= 0: + return [] + n = rng.randint(1, min(max_aux, len(pool))) + return rng.sample(pool, n) + + +def sample_target_strategy( + rng: random.Random, + catalog: TargetCatalog, + *, + fast: bool = False, +) -> TargetStrategy: + max_aux = 1 if fast else rng.randint(1, 3) + multi_prob = 0.30 if fast else 0.35 + primary = PRIMARY_TARGET + if primary not in catalog.targets: + raise ValueError(f"Primary tournament target {primary!r} missing from catalog") + if rng.random() < multi_prob: + aux = _sample_auxiliary(rng, catalog, primary, max_aux=max_aux) + return TargetStrategy( + target_mode="multi_blend", + primary_target=primary, + auxiliary_targets=aux, + target_blend_method=rng.choice(["equal", "sharpe"]), + ) + return TargetStrategy( + target_mode="single", + primary_target=primary, + auxiliary_targets=[], + target_blend_method="equal", + ) + + +def suggest_target_strategy( + trial: "optuna.Trial", + catalog: TargetCatalog, + *, + fast: bool = False, +) -> TargetStrategy: + primary = PRIMARY_TARGET + if primary not in catalog.targets: + raise ValueError(f"Primary tournament target {primary!r} missing from catalog") + mode = trial.suggest_categorical("target_mode", ["single", "multi_blend"]) + pool = [t for t in catalog.targets if t != primary] + max_slots = 1 if fast else 3 + aux: list[str] = [] + if pool: + for i in range(max_slots): + choice = trial.suggest_categorical(f"auxiliary_target_{i}", ["none", *pool]) + if choice != "none" and choice not in aux: + aux.append(choice) + if mode == "single" or not aux: + return TargetStrategy( + target_mode="single", + primary_target=primary, + auxiliary_targets=[], + target_blend_method="equal", + ) + blend_method = cast( + Literal["equal", "sharpe"], + trial.suggest_categorical("target_blend_method", ["equal", "sharpe"]), + ) + return TargetStrategy( + target_mode="multi_blend", + primary_target=primary, + auxiliary_targets=aux, + target_blend_method=blend_method, + ) + + +def _target_stats(series: pd.Series) -> tuple[int, float]: + valid = int(series.notna().sum()) + total = len(series) + nan_fraction = 1.0 - (valid / total) if total else 1.0 + return valid, nan_fraction + + +def _auxiliary_is_valid(series: pd.Series) -> bool: + valid, nan_fraction = _target_stats(series) + return valid >= _MIN_TRAIN_ROWS and nan_fraction <= MAX_NAN_FRACTION + + +def validate_target_strategy_early( + targets_df: pd.DataFrame, + strategy: TargetStrategy, + *, + catalog: TargetCatalog | None = None, + rng: random.Random | None = None, +) -> TargetValidationResult: + if strategy.primary_target not in targets_df.columns: + return TargetValidationResult( + strategy=strategy, + ok=False, + reason=f"primary target {strategy.primary_target!r} missing from data", + ) + + primary_valid, _ = _target_stats(targets_df[strategy.primary_target]) + if primary_valid < _MIN_TRAIN_ROWS: + return TargetValidationResult( + strategy=strategy, + ok=False, + reason=f"primary target has only {primary_valid} valid rows", + ) + + if strategy.target_mode == "single" or not strategy.auxiliary_targets: + return TargetValidationResult(strategy=strategy, ok=True) + + invalid_aux = [ + col + for col in strategy.auxiliary_targets + if col not in targets_df.columns or not _auxiliary_is_valid(targets_df[col]) + ] + if not invalid_aux: + return TargetValidationResult(strategy=strategy, ok=True) + + if catalog is None or rng is None: + downgraded = TargetStrategy( + target_mode="single", + primary_target=strategy.primary_target, + auxiliary_targets=[], + target_blend_method=strategy.target_blend_method, + ) + return TargetValidationResult( + strategy=downgraded, + ok=True, + reason="downgraded multi_blend to single (no resample context)", + ) + + current = strategy + for _ in range(MAX_AUXILIARY_RESAMPLE_ATTEMPTS): + pool = [ + t + for t in catalog.targets + if t != current.primary_target + and t not in current.auxiliary_targets + and t in targets_df.columns + ] + if not pool: + break + replacement = rng.choice(pool) + new_aux = [ + replacement if col in invalid_aux else col + for col in current.auxiliary_targets + ] + current = TargetStrategy( + target_mode="multi_blend", + primary_target=current.primary_target, + auxiliary_targets=new_aux, + target_blend_method=current.target_blend_method, + ) + invalid_aux = [ + col + for col in current.auxiliary_targets + if col not in targets_df.columns or not _auxiliary_is_valid(targets_df[col]) + ] + if not invalid_aux: + return TargetValidationResult(strategy=current, ok=True) + + downgraded = TargetStrategy( + target_mode="single", + primary_target=current.primary_target, + auxiliary_targets=[], + target_blend_method=current.target_blend_method, + ) + return TargetValidationResult( + strategy=downgraded, + ok=True, + reason="downgraded multi_blend to single after auxiliary resample attempts", + ) + + +def apply_target_strategy_to_flat( + flat: dict[str, Any], strategy: TargetStrategy +) -> dict[str, Any]: + out = dict(flat) + out["target_mode"] = strategy.target_mode + out["primary_target"] = strategy.primary_target + out["auxiliary_targets"] = list(strategy.auxiliary_targets) + out["target_blend_method"] = strategy.target_blend_method + return out + + +def strategy_from_flat(flat: dict[str, Any]) -> TargetStrategy: + mode = flat.get("target_mode", "single") + if mode not in ("single", "multi_blend"): + mode = "single" + aux = flat.get("auxiliary_targets") or [] + if not isinstance(aux, list): + aux = [] + blend = flat.get("target_blend_method", "equal") + if blend not in ("equal", "sharpe"): + blend = "equal" + return TargetStrategy( + target_mode=mode, + primary_target=str(flat.get("primary_target", "target")), + auxiliary_targets=[str(a) for a in aux], + target_blend_method=blend, + ) + + +def load_target_catalog_for_data(data_dir: str | Path) -> TargetCatalog: + return load_target_catalog(data_dir) diff --git a/src/alphapulse/hpo/trial_db.py b/src/alphapulse/hpo/trial_db.py index e747460..74c07e2 100644 --- a/src/alphapulse/hpo/trial_db.py +++ b/src/alphapulse/hpo/trial_db.py @@ -64,18 +64,33 @@ def update_trial( metrics: dict[str, Any] | None = None, error: str | None = None, elapsed_seconds: float | None = None, + flat_config: dict[str, Any] | None = None, ) -> None: - self._conn.execute( - "UPDATE trials SET status=?, metrics=?, error=?, elapsed_seconds=? " - "WHERE trial_number=?", - ( - status, - json.dumps(metrics) if metrics is not None else None, - error, - elapsed_seconds, - trial_number, - ), - ) + if flat_config is not None: + self._conn.execute( + "UPDATE trials SET status=?, metrics=?, error=?, elapsed_seconds=?, " + "flat_config=? WHERE trial_number=?", + ( + status, + json.dumps(metrics) if metrics is not None else None, + error, + elapsed_seconds, + json.dumps(flat_config), + trial_number, + ), + ) + else: + self._conn.execute( + "UPDATE trials SET status=?, metrics=?, error=?, elapsed_seconds=? " + "WHERE trial_number=?", + ( + status, + json.dumps(metrics) if metrics is not None else None, + error, + elapsed_seconds, + trial_number, + ), + ) self._conn.commit() def completed_trials(self) -> set[int]: diff --git a/src/alphapulse/logging_/__init__.py b/src/alphapulse/logging_/__init__.py index d724c92..ecabaa8 100644 --- a/src/alphapulse/logging_/__init__.py +++ b/src/alphapulse/logging_/__init__.py @@ -1,3 +1,4 @@ +from .cli import configure_cli_logging from .leaderboard import ( TrialLeaderboardEntry, entry_from_hpo_result, @@ -15,9 +16,11 @@ log_hpo_trial, log_metrics, log_research_step, + resolve_wandb_project, ) __all__ = [ + "configure_cli_logging", "TrialLeaderboardEntry", "entry_from_hpo_result", "entry_from_trial_record", @@ -30,6 +33,7 @@ "log_hpo_trial", "log_metrics", "log_research_step", + "resolve_wandb_project", "print_leaderboard", "save_leaderboard", ] diff --git a/src/alphapulse/logging_/cli.py b/src/alphapulse/logging_/cli.py new file mode 100644 index 0000000..df0259e --- /dev/null +++ b/src/alphapulse/logging_/cli.py @@ -0,0 +1,20 @@ +import sys + +from loguru import logger + +_CONFIGURED = False + + +def configure_cli_logging(*, level: str = "INFO") -> None: + """Send structured progress logs to stderr for CLI scripts.""" + global _CONFIGURED + if _CONFIGURED: + return + logger.remove() + logger.add( + sys.stderr, + level=level, + format="{time:HH:mm:ss} | {level:<8} | {message}", + enqueue=True, + ) + _CONFIGURED = True diff --git a/src/alphapulse/logging_/leaderboard.py b/src/alphapulse/logging_/leaderboard.py index ee09029..cb65f59 100644 --- a/src/alphapulse/logging_/leaderboard.py +++ b/src/alphapulse/logging_/leaderboard.py @@ -1,14 +1,44 @@ from __future__ import annotations import json +from collections.abc import Callable from dataclasses import asdict, dataclass from pathlib import Path -from typing import TYPE_CHECKING, Any +from typing import TYPE_CHECKING, Any, Literal + +import numpy as np if TYPE_CHECKING: from ..autoresearch.state import TrialRecord from ..hpo.objective import TrialResult +BestCriteria = Literal["objective", "robust_payout"] +NEGATIVE_HOLDOUT_PAYOUT_FACTOR = 0.25 +ROBUST_CONSISTENCY_FLOOR = 0.5 + + +def compute_robust_payout_score( + payout_score: float | None, + val_corr_sharpe: float | None, + holdout_corr_sharpe: float | None, +) -> float | None: + """Down-rank high validation payout when holdout CORR does not confirm it.""" + payout = _finite_optional(payout_score) + if payout is None: + return None + holdout = _finite_optional(holdout_corr_sharpe) + if holdout is None: + return payout + if holdout <= 0.0: + return payout * NEGATIVE_HOLDOUT_PAYOUT_FACTOR + val = _finite_optional(val_corr_sharpe) + if val is None or val <= 0.0: + return payout * min(1.0, holdout / 0.2) + consistency = min(1.0, holdout / val) + return payout * ( + ROBUST_CONSISTENCY_FLOOR + (1.0 - ROBUST_CONSISTENCY_FLOOR) * consistency + ) + @dataclass class TrialLeaderboardEntry: @@ -20,6 +50,12 @@ class TrialLeaderboardEntry: model_types: str elapsed_seconds: float error: str | None = None + payout_score: float | None = None + mmc_sharpe: float | None = None + val_corr_sharpe: float | None = None + val_mean_per_era_correlation: float | None = None + holdout_corr_sharpe: float | None = None + robust_payout_score: float | None = None def _model_types_from_flat(flat: dict[str, Any]) -> str: @@ -32,56 +68,186 @@ def _model_types_from_flat(flat: dict[str, Any]) -> str: return "+".join(str(t) for t in types) +def _finite_optional(raw: Any) -> float | None: + if raw is None: + return None + try: + value = float(raw) + except (TypeError, ValueError): + return None + return value if np.isfinite(value) else None + + +def _payout_from_result( + payout_score: float | None, + metrics: dict[str, Any], +) -> float | None: + if payout_score is not None: + return _finite_optional(payout_score) + return _finite_optional(metrics.get("payout_score")) + + +def _mmc_from_result( + mmc_sharpe: float | None, + metrics: dict[str, Any], +) -> float | None: + if mmc_sharpe is not None: + return _finite_optional(mmc_sharpe) + return _finite_optional(metrics.get("mmc_sharpe")) + + +def _rank_score(entry: TrialLeaderboardEntry) -> float: + if entry.payout_score is not None: + return entry.payout_score + return entry.sharpe + + +def _robust_rank_score(entry: TrialLeaderboardEntry) -> float: + if entry.robust_payout_score is not None: + return entry.robust_payout_score + return _rank_score(entry) + + +def _uses_payout_score(entries: list[TrialLeaderboardEntry]) -> bool: + return any(e.payout_score is not None for e in entries) + + +def _uses_robust_payout(entries: list[TrialLeaderboardEntry]) -> bool: + return any( + e.robust_payout_score is not None and e.holdout_corr_sharpe is not None + for e in entries + ) + + +def selection_score_from_metrics( + metrics: dict[str, Any], + *, + objective: str, + criteria: BestCriteria = "objective", +) -> float: + objective_score = float( + metrics.get(objective, metrics.get("corr_sharpe", float("-inf"))) + ) + if criteria != "robust_payout" or objective != "payout_score": + return objective_score + robust = compute_robust_payout_score( + _finite_optional(metrics.get("payout_score")), + _finite_optional(metrics.get("val_corr_sharpe")), + _finite_optional(metrics.get("holdout_corr_sharpe")), + ) + if robust is not None: + return robust + return objective_score + + +def _fmt_metric(value: float | None, width: int = 7, precision: int = 4) -> str: + if value is None: + return "N/A".rjust(width) + if value == float("-inf"): + return "-inf".rjust(width) + if value == float("inf"): + return " inf".rjust(width) + if not np.isfinite(value): + return "N/A".rjust(width) + return f"{value:{width}.{precision}f}" + + def entry_from_hpo_result(result: TrialResult) -> TrialLeaderboardEntry: metrics = result.metrics + holdout_sharpe = _finite_optional(metrics.get("holdout_corr_sharpe")) + if holdout_sharpe is None: + holdout_sharpe = _finite_optional(result.sharpe) + holdout_corr = _finite_optional(metrics.get("holdout_mean_per_era_correlation")) + if holdout_corr is None: + holdout_corr = float(metrics.get("mean_per_era_correlation", 0.0)) + holdout_std = _finite_optional(metrics.get("holdout_std_per_era_correlation")) + if holdout_std is None: + holdout_std = _finite_optional(metrics.get("std_per_era_correlation")) + holdout_dd = _finite_optional(metrics.get("holdout_max_drawdown")) + if holdout_dd is None: + holdout_dd = _finite_optional(metrics.get("max_drawdown")) + payout = _payout_from_result(result.payout_score, metrics) + val_corr_sharpe = _finite_optional(metrics.get("val_corr_sharpe")) return TrialLeaderboardEntry( trial_number=result.trial_number, - sharpe=result.sharpe, - mean_per_era_correlation=float(metrics.get("mean_per_era_correlation", 0.0)), - std_per_era_correlation=metrics.get("std_per_era_correlation"), - max_drawdown=metrics.get("max_drawdown"), + sharpe=holdout_sharpe if holdout_sharpe is not None else result.sharpe, + mean_per_era_correlation=holdout_corr, + std_per_era_correlation=holdout_std, + max_drawdown=holdout_dd, model_types=_model_types_from_flat(result.params), elapsed_seconds=result.elapsed_seconds, error=result.error, + payout_score=payout, + mmc_sharpe=_mmc_from_result(result.mmc_sharpe, metrics), + val_corr_sharpe=val_corr_sharpe, + val_mean_per_era_correlation=_finite_optional( + metrics.get("val_mean_per_era_correlation") + ), + holdout_corr_sharpe=holdout_sharpe, + robust_payout_score=compute_robust_payout_score( + payout, val_corr_sharpe, holdout_sharpe + ), ) def entry_from_trial_record(record: TrialRecord) -> TrialLeaderboardEntry: + metrics = record.metrics + holdout_sharpe = _finite_optional(metrics.get("holdout_corr_sharpe")) + if holdout_sharpe is None: + holdout_sharpe = _finite_optional(record.sharpe) + holdout_corr = _finite_optional(metrics.get("holdout_mean_per_era_correlation")) + if holdout_corr is None: + holdout_corr = float(metrics.get("mean_per_era_correlation", 0.0)) + holdout_std = _finite_optional(metrics.get("holdout_std_per_era_correlation")) + if holdout_std is None: + holdout_std = _finite_optional(metrics.get("std_per_era_correlation")) + holdout_dd = _finite_optional(metrics.get("holdout_max_drawdown")) + if holdout_dd is None: + holdout_dd = _finite_optional(metrics.get("max_drawdown")) + payout = _payout_from_result(record.payout_score, metrics) + val_corr_sharpe = _finite_optional(metrics.get("val_corr_sharpe")) return TrialLeaderboardEntry( trial_number=record.trial_number, - sharpe=record.sharpe, - mean_per_era_correlation=float( - record.metrics.get("mean_per_era_correlation", 0.0) - ), - std_per_era_correlation=record.metrics.get("std_per_era_correlation"), - max_drawdown=record.metrics.get("max_drawdown"), + sharpe=holdout_sharpe if holdout_sharpe is not None else record.sharpe, + mean_per_era_correlation=holdout_corr, + std_per_era_correlation=holdout_std, + max_drawdown=holdout_dd, model_types="+".join(record.model_types), elapsed_seconds=record.elapsed_seconds, error=record.error, + payout_score=payout, + mmc_sharpe=_mmc_from_result(record.mmc_sharpe, metrics), + val_corr_sharpe=val_corr_sharpe, + val_mean_per_era_correlation=_finite_optional( + metrics.get("val_mean_per_era_correlation") + ), + holdout_corr_sharpe=holdout_sharpe, + robust_payout_score=compute_robust_payout_score( + payout, val_corr_sharpe, holdout_sharpe + ), ) -def format_leaderboard( +def _format_payout_table( entries: list[TrialLeaderboardEntry], *, - top_n: int = 10, - current_trial: int | None = None, -) -> str: - sorted_entries = sorted(entries, key=lambda e: e.sharpe, reverse=True)[:top_n] - lines = [ - f"--- LEADERBOARD (top {top_n} by sharpe) ---", - " Rank | Trial | Sharpe | Corr | StdCorr | MaxDD | " - "Models | Time", - ] + top_n: int, + current_trial: int | None, + title: str, + sort_key: Callable[[TrialLeaderboardEntry], float], + score_label: str, + score_getter: Callable[[TrialLeaderboardEntry], float | None], +) -> list[str]: + header = ( + f"--- {title} ---\n" + " Payout = 0.75*ValidationSharpe + 2.25*ValidationMmcSharpe\n" + f" Rank | Trial | {score_label:>11} | ValidationSharpe | " + "ValidationMmcSharpe | ValidationMeanCorr | HoldoutSharpe | " + "HoldoutMeanCorr | Models | Time" + ) + lines = [header] + sorted_entries = sorted(entries, key=sort_key, reverse=True)[:top_n] for rank, entry in enumerate(sorted_entries, start=1): - std_corr = ( - f"{entry.std_per_era_correlation:8.4f}" - if entry.std_per_era_correlation is not None - else " N/A" - ) - max_dd = ( - f"{entry.max_drawdown:6.3f}" if entry.max_drawdown is not None else " N/A" - ) marker = ( " *" if current_trial is not None and entry.trial_number == current_trial @@ -89,10 +255,84 @@ def format_leaderboard( ) lines.append( f" {rank:4d} | {entry.trial_number:5d} | " - f"{entry.sharpe:7.4f} | {entry.mean_per_era_correlation:7.4f} | " - f"{std_corr} | {max_dd} | " + f"{_fmt_metric(score_getter(entry))} | " + f"{_fmt_metric(entry.val_corr_sharpe)} | " + f"{_fmt_metric(entry.mmc_sharpe)} | " + f"{_fmt_metric(entry.val_mean_per_era_correlation)} | " + f"{_fmt_metric(entry.holdout_corr_sharpe)} | " + f"{_fmt_metric(entry.mean_per_era_correlation)} | " f"{entry.model_types[:19]:<19} | {entry.elapsed_seconds:4.0f}s{marker}" ) + return lines + + +def format_leaderboard( + entries: list[TrialLeaderboardEntry], + *, + top_n: int = 10, + current_trial: int | None = None, +) -> str: + by_payout = _uses_payout_score(entries) + lines: list[str] = [] + if by_payout: + lines.extend( + _format_payout_table( + entries, + top_n=top_n, + current_trial=current_trial, + title=f"LEADERBOARD (top {top_n} by payout on validation)", + sort_key=_rank_score, + score_label="Payout", + score_getter=lambda entry: entry.payout_score, + ) + ) + if _uses_robust_payout(entries): + lines.append("") + lines.extend( + _format_payout_table( + entries, + top_n=top_n, + current_trial=current_trial, + title=( + f"LEADERBOARD (top {top_n} by robust payout: " + "payout penalized when holdout CORR is weak vs validation)" + ), + sort_key=_robust_rank_score, + score_label="RobustPayout", + score_getter=lambda entry: entry.robust_payout_score, + ) + ) + else: + header = ( + f"--- LEADERBOARD (top {top_n} by holdout corr_sharpe) ---\n" + " Rank | Trial | HoldoutSharpe | HoldoutMeanCorr | HoldoutStdCorr | " + "HoldoutMaxDrawdown | Models | Time" + ) + lines = [header] + sorted_entries = sorted(entries, key=_rank_score, reverse=True)[:top_n] + for rank, entry in enumerate(sorted_entries, start=1): + std_corr = ( + f"{entry.std_per_era_correlation:8.4f}" + if entry.std_per_era_correlation is not None + else " N/A" + ) + max_dd = ( + f"{entry.max_drawdown:6.3f}" + if entry.max_drawdown is not None + else " N/A" + ) + marker = ( + " *" + if current_trial is not None and entry.trial_number == current_trial + else "" + ) + lines.append( + f" {rank:4d} | {entry.trial_number:5d} | " + f"{_fmt_metric(entry.holdout_corr_sharpe)} | " + f"{_fmt_metric(entry.mean_per_era_correlation)} | " + f"{std_corr} | {max_dd} | " + f"{entry.model_types[:19]:<19} | {entry.elapsed_seconds:4.0f}s{marker}" + ) if current_trial is not None: lines.append("* = current trial") return "\n".join(lines) @@ -111,6 +351,6 @@ def print_leaderboard( def save_leaderboard(path: Path, entries: list[TrialLeaderboardEntry]) -> None: path = Path(path) path.parent.mkdir(parents=True, exist_ok=True) - sorted_entries = sorted(entries, key=lambda e: e.sharpe, reverse=True) + sorted_entries = sorted(entries, key=_rank_score, reverse=True) payload = [asdict(e) for e in sorted_entries] path.write_text(json.dumps(payload, indent=2), encoding="utf-8") diff --git a/src/alphapulse/logging_/wandb_logging.py b/src/alphapulse/logging_/wandb_logging.py new file mode 100644 index 0000000..eec966e --- /dev/null +++ b/src/alphapulse/logging_/wandb_logging.py @@ -0,0 +1,72 @@ +from __future__ import annotations + +from typing import Any + +_WANDB_LOGURU_SINK_ID: int | None = None + + +def wandb_run_active() -> bool: + try: + import wandb + + return wandb.run is not None + except ImportError: + return False + + +def attach_wandb_loguru(*, level: str = "INFO") -> None: + """Send loguru lines to wandb-wrapped stderr for the W&B Logs panel.""" + global _WANDB_LOGURU_SINK_ID + import sys + + from loguru import logger + + if not wandb_run_active(): + return + logger.remove() + _WANDB_LOGURU_SINK_ID = logger.add( + sys.stderr, + level=level, + format="{time:YYYY-MM-DD HH:mm:ss.SSS} | {level:<8} | {message}", + enqueue=False, + ) + + +def detach_wandb_loguru() -> None: + global _WANDB_LOGURU_SINK_ID + from loguru import logger + + if _WANDB_LOGURU_SINK_ID is not None: + logger.remove(_WANDB_LOGURU_SINK_ID) + _WANDB_LOGURU_SINK_ID = None + + +def log_boosting_round_metrics( + *, + model_name: str, + round_num: int, + metrics: dict[str, float], +) -> None: + if not metrics or not wandb_run_active(): + return + import wandb + + logged: dict[str, Any] = {"train/round": round_num} + for key, value in metrics.items(): + logged[f"train/{model_name}/{key}"] = value + wandb.log(logged) + + +def parse_xgb_evals_log( + evals_log: dict[str, dict[str, list[float] | list[tuple[float, float]]]], +) -> dict[str, float]: + parsed: dict[str, float] = {} + for dataset, metric_map in (evals_log or {}).items(): + for metric_name, values in metric_map.items(): + if not values: + continue + last = values[-1] + parsed[f"{dataset}_{metric_name}"] = float( + last[0] if isinstance(last, tuple) else last + ) + return parsed diff --git a/src/alphapulse/logging_/wandb_utils.py b/src/alphapulse/logging_/wandb_utils.py index 9956ac6..391737d 100644 --- a/src/alphapulse/logging_/wandb_utils.py +++ b/src/alphapulse/logging_/wandb_utils.py @@ -1,9 +1,33 @@ from typing import TYPE_CHECKING, Any if TYPE_CHECKING: + from pathlib import Path + from ..hpo.objective import TrialResult +def resolve_wandb_project( + base: str, + *, + output_dir: "Path | None" = None, + stamp_file: str = "wandb_project.txt", +) -> str: + """Return a timestamped W&B project name, persisted for resume in *output_dir*.""" + from datetime import datetime + from pathlib import Path + + if output_dir is not None: + path = Path(output_dir) / stamp_file + if path.exists(): + return path.read_text(encoding="utf-8").strip() + + stamped = f"{base}-{datetime.now().strftime('%Y%m%d-%H%M%S')}" + if output_dir is not None: + Path(output_dir).mkdir(parents=True, exist_ok=True) + path.write_text(stamped, encoding="utf-8") + return stamped + + def init_wandb( project: str, config: dict[str, Any] | None = None, @@ -34,13 +58,18 @@ def init_wandb_run( """Start a long-lived WandB run (e.g. for the full AutoResearch loop).""" import wandb - wandb.init(project=project, name=name, config=config or {}, reinit=True) + wandb.init( + project=project, name=name, config=config or {}, reinit="finish_previous" + ) def finish_wandb_run() -> None: import wandb - wandb.finish(quiet=True) + from .wandb_logging import detach_wandb_loguru + + detach_wandb_loguru() + wandb.finish() def log_research_step( @@ -90,14 +119,15 @@ def log_hpo_summary_table( columns = [ "trial", - "sharpe", - "corr_sharpe", - "mmc_sharpe", - "payout_score", - "mean_era_corr", - "std_era_corr", - "max_drawdown", - "pct_positive_eras", + "HoldoutSharpe", + "ValidationSharpe", + "ValidationMmcSharpe", + "PayoutScore", + "HoldoutMeanCorr", + "ValidationMeanCorr", + "HoldoutStdCorr", + "HoldoutMaxDrawdown", + "HoldoutPctPositiveEras", "model_types", "model_1_type", "model_2_type", @@ -118,6 +148,9 @@ def log_hpo_summary_table( "feature_selection_type", "use_feature_selection", "use_augmentation", + "active_groups", + "active_groups_count", + "routed_feature_count", "elapsed_seconds", ] table = wandb.Table(columns=columns) @@ -130,16 +163,25 @@ def log_hpo_summary_table( ) table.add_data( r.trial_number, - r.sharpe, r.corr_sharpe if r.corr_sharpe not in (float("-inf"), float("inf")) else None, + r.metrics.get("val_corr_sharpe"), r.mmc_sharpe, r.payout_score, - r.metrics.get("mean_per_era_correlation"), - r.metrics.get("std_per_era_correlation"), - r.metrics.get("max_drawdown"), - r.metrics.get("pct_positive_eras"), + r.metrics.get("holdout_mean_per_era_correlation") + if r.metrics.get("holdout_mean_per_era_correlation") is not None + else r.metrics.get("mean_per_era_correlation"), + r.metrics.get("val_mean_per_era_correlation"), + r.metrics.get("holdout_std_per_era_correlation") + if r.metrics.get("holdout_std_per_era_correlation") is not None + else r.metrics.get("std_per_era_correlation"), + r.metrics.get("holdout_max_drawdown") + if r.metrics.get("holdout_max_drawdown") is not None + else r.metrics.get("max_drawdown"), + r.metrics.get("holdout_pct_positive_eras") + if r.metrics.get("holdout_pct_positive_eras") is not None + else r.metrics.get("pct_positive_eras"), model_types, r.params.get("model_1_type"), r.params.get("model_2_type"), @@ -151,17 +193,18 @@ def log_hpo_summary_table( r.params.get("ensemble_method"), r.params.get("use_neutralization"), r.params.get("neutralization_proportion"), - r.params.get("model_1_max_depth") or r.params.get("xgb_max_depth"), - r.params.get("model_1_learning_rate") or r.params.get("xgb_learning_rate"), - r.params.get("model_1_num_leaves") or r.params.get("lgbm_num_leaves"), - r.params.get("model_1_learning_rate_lgbm") - or r.params.get("lgbm_learning_rate"), - r.params.get("model_1_min_child_samples") - or r.params.get("lgbm_min_child_samples"), + r.params.get("xgb_max_depth"), + r.params.get("xgb_learning_rate"), + r.params.get("lgbm_num_leaves"), + r.params.get("lgbm_learning_rate"), + r.params.get("lgbm_min_child_samples"), r.params.get("use_noise_injection"), r.params.get("feature_selection_type"), r.params.get("use_feature_selection"), r.params.get("use_augmentation"), + "+".join(r.params.get("active_groups", [])), + r.params.get("active_groups_count"), + r.params.get("routed_feature_count"), r.elapsed_seconds, ) @@ -170,10 +213,96 @@ def log_hpo_summary_table( group=group, name="hpo-summary", job_type="summary", - reinit=True, + reinit="finish_previous", ) - wandb.log({"trials_summary": table}) - wandb.finish(quiet=True) + summary_charts: dict[str, Any] = {} + if any(r.error is None for r in results): + summary_charts["hpo/trial_PayoutScore"] = wandb.plot.scatter( + table, + "trial", + "PayoutScore", + title="Trial validation PayoutScore", + ) + summary_charts["hpo/trial_HoldoutSharpe"] = wandb.plot.scatter( + table, + "trial", + "HoldoutSharpe", + title="Trial holdout HoldoutSharpe", + ) + summary_charts["hpo/trial_ValidationSharpe"] = wandb.plot.scatter( + table, + "trial", + "ValidationSharpe", + title="Trial validation ValidationSharpe", + ) + summary_charts["hpo/trial_ValidationMmcSharpe"] = wandb.plot.scatter( + table, + "trial", + "ValidationMmcSharpe", + title="Trial validation ValidationMmcSharpe", + ) + summary_charts["hpo/trial_elapsed"] = wandb.plot.scatter( + table, + "trial", + "elapsed_seconds", + title="Trial runtime (seconds)", + ) + wandb.log({"trials_summary": table, **summary_charts}) + wandb.finish() + + +def _finite_metric(value: Any) -> float | None: + import numpy as np + + if value is None: + return None + try: + numeric = float(value) + except (TypeError, ValueError): + return None + return numeric if np.isfinite(numeric) else None + + +def _log_split_metrics( + logged: dict[str, Any], + metrics: dict[str, Any], + result: "TrialResult", +) -> None: + holdout_corr = _finite_metric(metrics.get("holdout_corr_sharpe")) + if holdout_corr is None: + holdout_corr = _finite_metric(result.corr_sharpe) + if holdout_corr is not None: + logged["holdout/HoldoutSharpe"] = holdout_corr + holdout_mean = _finite_metric(metrics.get("holdout_mean_per_era_correlation")) + if holdout_mean is None: + holdout_mean = _finite_metric(metrics.get("mean_per_era_correlation")) + if holdout_mean is not None: + logged["holdout/HoldoutMeanCorr"] = holdout_mean + holdout_dd = _finite_metric(metrics.get("holdout_max_drawdown")) + if holdout_dd is None: + holdout_dd = _finite_metric(metrics.get("max_drawdown")) + if holdout_dd is not None: + logged["holdout/HoldoutMaxDrawdown"] = holdout_dd + + val_corr = _finite_metric(metrics.get("val_corr_sharpe")) + if val_corr is not None: + logged["validation/ValidationSharpe"] = val_corr + val_mean = _finite_metric(metrics.get("val_mean_per_era_correlation")) + if val_mean is not None: + logged["validation/ValidationMeanCorr"] = val_mean + mmc = _finite_metric(result.mmc_sharpe) + if mmc is None: + mmc = _finite_metric(metrics.get("mmc_sharpe")) + if mmc is not None: + logged["validation/ValidationMmcSharpe"] = mmc + payout = _finite_metric(result.payout_score) + if payout is None: + payout = _finite_metric(metrics.get("payout_score")) + if payout is not None: + logged["validation/PayoutScore"] = payout + mmc_mean = _finite_metric(metrics.get("mmc")) + if mmc_mean is not None: + logged["validation/ValidationMmc"] = mmc_mean def log_hpo_trial_metrics( @@ -183,10 +312,10 @@ def log_hpo_trial_metrics( model_types: str | None = None, preprocessors: str | None = None, ) -> None: + import numpy as np import wandb logged: dict[str, Any] = { - "sharpe": result.sharpe, "objective": objective, "elapsed_seconds": result.elapsed_seconds, } @@ -194,16 +323,34 @@ def log_hpo_trial_metrics( logged["model_types"] = model_types if preprocessors is not None: logged["preprocessors"] = preprocessors - if result.corr_sharpe not in (float("-inf"), float("inf")): - logged["corr_sharpe"] = result.corr_sharpe - if result.mmc_sharpe is not None: - logged["mmc_sharpe"] = result.mmc_sharpe - if result.payout_score is not None: - logged["payout_score"] = result.payout_score - top_level_keys = {"sharpe", "corr_sharpe", "mmc_sharpe", "payout_score"} - for k, v in (result.metrics or {}).items(): - if k not in top_level_keys: - logged[f"metric/{k}"] = v + active_groups = result.params.get("active_groups", []) + if isinstance(active_groups, list): + logged["active_groups"] = "+".join(str(g) for g in active_groups) + logged["active_groups_count"] = len(active_groups) + routed_feature_count = result.params.get("routed_feature_count") + if routed_feature_count is not None: + logged["routed_feature_count"] = routed_feature_count + metrics = result.metrics or {} + _log_split_metrics(logged, metrics, result) + skip_metric_keys = { + "corr_sharpe", + "mean_per_era_correlation", + "std_per_era_correlation", + "max_drawdown", + "pct_positive_eras", + "n_valid_eras", + "mmc", + "mmc_sharpe", + "payout_score", + } + for key, value in metrics.items(): + if key in skip_metric_keys or key.startswith(("holdout_", "val_")): + continue + if value is None: + continue + if isinstance(value, float) and not np.isfinite(value): + continue + logged[f"metric/{key}"] = value wandb.log(logged) @@ -271,13 +418,13 @@ def log_hpo_trial( group=group, name=f"trial_{result.trial_number:03d}", config=config_for_wandb, - reinit=True, + reinit="finish_previous", ) log_hpo_trial_metrics( result, objective, model_types=model_types, preprocessors=preprocessors ) - wandb.finish(quiet=True) + wandb.finish() def log_hpo_convergence( @@ -285,17 +432,10 @@ def log_hpo_convergence( *, project: str, group: str, + objective: str = "payout_score", ) -> None: - """Log per-trial corr_sharpe and running best in a single WandB convergence run. - - All trials are logged as ordered steps within one run so that WandB renders - a proper convergence curve (trial scores + running maximum line). - - Args: - results: All TrialResult objects from the HPO search, in trial order. - project: WandB project name. - group: WandB group name (same as HPO run group). - """ + """Log per-trial holdout/validation metrics and running best in one WandB run.""" + import numpy as np import wandb wandb.init( @@ -303,24 +443,40 @@ def log_hpo_convergence( group=group, name="search-convergence", job_type="convergence", - reinit=True, + reinit="finish_previous", ) best_so_far = float("-inf") for r in results: if r.error: continue - trial_corr = ( - r.corr_sharpe - if r.corr_sharpe not in (float("-inf"), float("inf")) - else r.sharpe - ) - if trial_corr > best_so_far: - best_so_far = trial_corr - wandb.log( - { - "trial_corr_sharpe": trial_corr, - "best_corr_sharpe_so_far": best_so_far, - }, - step=r.trial_number, - ) - wandb.finish(quiet=True) + holdout_corr = r.corr_sharpe + if holdout_corr in (float("-inf"), float("inf")): + holdout_corr = None + payout = r.payout_score + if payout is not None and not np.isfinite(payout): + payout = None + mmc = r.mmc_sharpe + if mmc is not None and not np.isfinite(mmc): + mmc = None + val_corr = r.metrics.get("val_corr_sharpe") if r.metrics else None + if val_corr is not None and not np.isfinite(float(val_corr)): + val_corr = None + + trial_objective = payout if objective == "payout_score" else holdout_corr + if trial_objective is not None and trial_objective > best_so_far: + best_so_far = trial_objective + + logged: dict[str, Any] = {} + if holdout_corr is not None: + logged["holdout/HoldoutSharpe"] = holdout_corr + if val_corr is not None: + logged["validation/ValidationSharpe"] = float(val_corr) + if mmc is not None: + logged["validation/ValidationMmcSharpe"] = mmc + if payout is not None: + logged["validation/PayoutScore"] = payout + if trial_objective is not None: + logged[f"best_{objective}_so_far"] = best_so_far + if logged: + wandb.log(logged, step=r.trial_number) + wandb.finish() diff --git a/src/alphapulse/models/__init__.py b/src/alphapulse/models/__init__.py index ebf1294..afa587f 100644 --- a/src/alphapulse/models/__init__.py +++ b/src/alphapulse/models/__init__.py @@ -6,7 +6,6 @@ from .foundation_models import ( TabICLModel, TabPFN3Model, - TabPFN3ReasoningModel, TabPFNModel, ) from .lightgbm_model import LightGBMModel @@ -27,7 +26,6 @@ "SyntheticDataAugmenter", "TabICLModel", "TabPFN3Model", - "TabPFN3ReasoningModel", "TabPFNModel", "XGBoostModel", "suggest_augmentation", diff --git a/src/alphapulse/models/era_ensemble_model.py b/src/alphapulse/models/era_ensemble_model.py index 4c14d15..cf93c6c 100644 --- a/src/alphapulse/models/era_ensemble_model.py +++ b/src/alphapulse/models/era_ensemble_model.py @@ -5,6 +5,7 @@ import numpy as np import pandas as pd +from loguru import logger from sklearn.linear_model import Ridge from .base import BaseModel @@ -55,6 +56,10 @@ def train( UserWarning, stacklevel=2, ) + logger.info( + "{}: no era column, training single base model", + self.name, + ) model = self.base_model_factory() metrics = model.train(X_train, y_train, X_val=X_val, y_val=y_val, **kwargs) self._sub_models = [model] @@ -69,6 +74,12 @@ def train( unique_eras = np.sort(era_train.unique()) n_parts = min(self.n_subs, len(unique_eras)) era_partitions = np.array_split(unique_eras, n_parts) + logger.info( + "{}: training {} era partitions ({} unique eras)", + self.name, + n_parts, + len(unique_eras), + ) sub_preds: list[np.ndarray] = [] self._sub_models = [] @@ -79,6 +90,14 @@ def train( mask = era_train.isin(era_group) X_sub = X_train[mask] y_sub = y_train[mask] + logger.info( + "{} sub {}/{}: rows={} eras={}", + self.name, + i + 1, + n_parts, + len(X_sub), + len(era_group), + ) X_sub_val: pd.DataFrame | None = None y_sub_val: pd.Series | None = None @@ -101,6 +120,11 @@ def train( sub_preds.append(model.predict(X_train)) + logger.info( + "{}: fitting Ridge meta-learner on {} sub-models", + self.name, + len(self._sub_models), + ) X_meta = np.column_stack(sub_preds) self._meta_model = Ridge(alpha=100.0).fit(X_meta, y_train) self.is_trained = True diff --git a/src/alphapulse/models/factory.py b/src/alphapulse/models/factory.py index 2eb3729..e39a731 100644 --- a/src/alphapulse/models/factory.py +++ b/src/alphapulse/models/factory.py @@ -263,16 +263,7 @@ def suggest_fixed( name=p.get("name"), ) - type_aliases = { - "xgboost": "XGBoost", - "lightgbm": "LightGBM", - "catboost": "CatBoost", - "random_forest": "RandomForest", - "extra_trees": "ExtraTrees", - "ridge": "Ridge", - } - registry_name = type_aliases.get(model_type, model_type) - return instantiate_model(registry_name, p, index=0, n_subs=n_subs) + return instantiate_model(model_type, p, index=0, n_subs=n_subs) def suggest_augmentation( diff --git a/src/alphapulse/models/foundation_models.py b/src/alphapulse/models/foundation_models.py index 88e47eb..2a22ea7 100644 --- a/src/alphapulse/models/foundation_models.py +++ b/src/alphapulse/models/foundation_models.py @@ -1,4 +1,3 @@ -import os from abc import abstractmethod from pathlib import Path from typing import Any @@ -22,8 +21,6 @@ TABPFN_MAX_FEATURES = 500 TABPFN3_MAX_TRAIN_ROWS = 100_000 TABPFN3_MAX_FEATURES = 2_000 -TABPFN3_REASONING_MAX_TRAIN_ROWS = 10_000 -TABPFN3_REASONING_MAX_FEATURES = 500 TABICL_MAX_TRAIN_ROWS = 60_000 TABICL_MAX_FEATURES = 500 TABPFN_PREDICT_CHUNK_ROWS = 256 @@ -113,7 +110,8 @@ def train( **kwargs: Any, ) -> dict[str, float]: regressor = self._make_regressor() - feat, y = self._prepare_train(X_train, y_train) + eras = X_train["era"] if "era" in X_train.columns else None + feat, y = self._prepare_train(X_train, y_train, eras=eras) regressor.fit(feat, y) self.model = regressor self.is_trained = True @@ -150,27 +148,66 @@ def load(self, path: Path) -> "FoundationModel": return self def _prepare_train( - self, X_train: pd.DataFrame, y_train: pd.Series + self, + X_train: pd.DataFrame, + y_train: pd.Series, + eras: pd.Series | None = None, ) -> tuple[pd.DataFrame, pd.Series]: feat = _numeric(X_train) if feat.shape[1] == 0: raise ValueError(f"{self.name}: no numeric feature columns found.") - feat, y = self._subsample(feat, y_train) + feat, y = self._subsample(feat, y_train, eras=eras) self._medians = feat.median() feat = feat.fillna(self._medians) feat = self._fit_compressor(feat, y) return feat, y - def _subsample( + def _subsample_random( self, feat: pd.DataFrame, y: pd.Series ) -> tuple[pd.DataFrame, pd.Series]: - if len(feat) <= self.max_train_rows: - return feat, y rng = np.random.default_rng(self.seed) idx = rng.choice(len(feat), size=self.max_train_rows, replace=False) idx.sort() return feat.iloc[idx], y.iloc[idx] + def _subsample_era_stratified( + self, feat: pd.DataFrame, y: pd.Series, eras: pd.Series + ) -> tuple[pd.DataFrame, pd.Series]: + era_vals = np.asarray(eras) + unique_eras = np.unique(era_vals) + n_eras = len(unique_eras) + if n_eras == 0: + return self._subsample_random(feat, y) + per_era = max(1, self.max_train_rows // n_eras) + rng = np.random.default_rng(self.seed) + selected_idx: list[int] = [] + for era in unique_eras: + era_idx = np.flatnonzero(era_vals == era) + if len(era_idx) <= per_era: + selected_idx.extend(era_idx.tolist()) + else: + chosen = rng.choice(era_idx, size=per_era, replace=False) + selected_idx.extend(sorted(chosen.tolist())) + if len(selected_idx) > self.max_train_rows: + selected_idx = sorted( + rng.choice( + selected_idx, size=self.max_train_rows, replace=False + ).tolist() + ) + return feat.iloc[selected_idx], y.iloc[selected_idx] + + def _subsample( + self, + feat: pd.DataFrame, + y: pd.Series, + eras: pd.Series | None = None, + ) -> tuple[pd.DataFrame, pd.Series]: + if len(feat) <= self.max_train_rows: + return feat, y + if eras is not None and len(eras) == len(feat): + return self._subsample_era_stratified(feat, y, eras) + return self._subsample_random(feat, y) + def _fit_compressor(self, feat: pd.DataFrame, y: pd.Series) -> pd.DataFrame: if self.compression is None or feat.shape[1] <= self.max_features: self._compressor = None @@ -297,64 +334,6 @@ def _make_regressor(self) -> Any: return TabPFNRegressor(**init_kwargs) -class TabPFN3ReasoningModel(FoundationModel): - """TabPFN v3 regression via Prior Labs API with reasoning mode. - - Requires: pip install 'alphapulse[foundation-api]' and TABPFN_API_KEY. - Note: fits are API-metered, so the default row budget is conservative. - """ - - def __init__( - self, - thinking_mode: bool = True, - thinking_effort: str = "medium", - thinking_timeout_s: int = 300, - thinking_metric: str = "rmse", - max_train_rows: int = TABPFN3_REASONING_MAX_TRAIN_ROWS, - max_features: int = TABPFN3_REASONING_MAX_FEATURES, - compression: str | None = DEFAULT_COMPRESSION, - compression_components: int | None = None, - compression_epochs: int = 20, - compression_device: str | None = None, - seed: int = DEFAULT_SEED, - name: str | None = "TabPFN3Reasoning", - ) -> None: - super().__init__( - max_train_rows=max_train_rows, - max_features=max_features, - compression=compression, - compression_components=compression_components, - compression_epochs=compression_epochs, - compression_device=compression_device, - seed=seed, - name=name, - ) - self.thinking_mode = thinking_mode - self.thinking_effort = thinking_effort - self.thinking_timeout_s = thinking_timeout_s - self.thinking_metric = thinking_metric - - def _make_regressor(self) -> Any: - try: - from tabpfn_client import TabPFNRegressor - except ImportError as exc: - raise ImportError( - "TabPFN3ReasoningModel requires tabpfn-client. " - "Install with: pip install 'alphapulse[foundation-api]'" - ) from exc - - if not os.environ.get("TABPFN_API_KEY"): - raise ValueError( - "TabPFN3ReasoningModel requires TABPFN_API_KEY environment variable." - ) - return TabPFNRegressor( - thinking_mode=self.thinking_mode, - thinking_effort=self.thinking_effort, - thinking_timeout_s=self.thinking_timeout_s, - thinking_metric=self.thinking_metric, - ) - - class TabICLModel(FoundationModel): """TabICL v2 regression via in-context learning. diff --git a/src/alphapulse/models/packboost_backend.py b/src/alphapulse/models/packboost_backend.py new file mode 100644 index 0000000..c591804 --- /dev/null +++ b/src/alphapulse/models/packboost_backend.py @@ -0,0 +1,132 @@ +from __future__ import annotations + +from typing import Any + +import numpy as np +import pandas as pd + +from .packboost_encoding import ( + bin_features_for_packboost, + default_nfeatsets, + encode_era_ids, + q30_predictions_to_float, + sort_rows_by_era, +) + + +def packboost_cuda_available() -> bool: + try: + require_packboost_cuda(device="cuda") + return True + except (ImportError, RuntimeError): + return False + + +def require_packboost_cuda(*, device: str = "cuda") -> None: + if device != "cuda": + raise ValueError(f"PackboostModel only supports device='cuda', got {device!r}.") + try: + import torch + except ImportError as exc: + raise ImportError( + "PackBoost requires PyTorch. Install with: uv sync --extra packboost" + ) from exc + if not torch.cuda.is_available(): + raise RuntimeError( + "PackBoost CUDA is required but no CUDA device is available." + ) + try: + from packboost.core import PackBoost # noqa: F401 + except ImportError as exc: + raise ImportError( + "PackBoost is not installed. Install with: uv sync --extra packboost" + ) from exc + except (OSError, RuntimeError) as exc: + raise RuntimeError( + "PackBoost CUDA kernels failed to load. " + "Ensure ninja and a matching CUDA toolkit are installed." + ) from exc + + +def resolve_packboost_device(device: str) -> str: + require_packboost_cuda(device=device) + return "cuda" + + +class PackBoostTrainer: + def __init__( + self, + *, + device: str = "auto", + max_depth: int = 7, + nfolds: int = 8, + lr: float = 0.07, + l2: float = 100_000.0, + nfeatsets: int = 32, + seed: int = 42, + ) -> None: + self.device = resolve_packboost_device(device) + self.max_depth = int(max_depth) + self.nfolds = int(nfolds) + self.lr = float(lr) + self.l2 = float(l2) + self.nfeatsets = int(nfeatsets) + self.seed = int(seed) + self._model: Any = None + + def fit( + self, + features: pd.DataFrame, + target: pd.Series, + *, + era: pd.Series | None = None, + val_features: pd.DataFrame | None = None, + val_target: pd.Series | None = None, + rounds: int, + ) -> None: + from packboost.core import PackBoost + + fit_features = features + fit_target = target + fit_era = era + if era is not None: + fit_features, fit_target, fit_era = sort_rows_by_era(features, target, era) + + x_train = bin_features_for_packboost(fit_features) + y_train = fit_target.to_numpy(dtype=np.float32) + n_features = x_train.shape[1] + nfeatsets = default_nfeatsets(n_features, self.nfeatsets) + + era_ids: np.ndarray | None = None + if fit_era is not None: + era_ids = encode_era_ids(fit_era) + + x_val: np.ndarray | None = None + y_val: np.ndarray | None = None + if val_features is not None and val_target is not None: + x_val = bin_features_for_packboost(val_features) + y_val = val_target.to_numpy(dtype=np.float32) + + model = PackBoost(device=self.device) + model.fit( + x_train, + y_train, + Xv=x_val, + Yv=y_val, + nfolds=self.nfolds, + rounds=int(rounds), + max_depth=self.max_depth, + lr=self.lr, + L2=self.l2, + nfeatsets=nfeatsets, + seed=self.seed, + era_ids=era_ids, + ) + self._model = model + + def predict(self, features: pd.DataFrame) -> np.ndarray: + if self._model is None: + raise ValueError("PackBoostTrainer is not fitted.") + x_test = bin_features_for_packboost(features) + pred_q30 = self._model.predict(x_test) + return q30_predictions_to_float(pred_q30) diff --git a/src/alphapulse/models/packboost_encoding.py b/src/alphapulse/models/packboost_encoding.py new file mode 100644 index 0000000..59961fa --- /dev/null +++ b/src/alphapulse/models/packboost_encoding.py @@ -0,0 +1,66 @@ +from __future__ import annotations + +import numpy as np +import pandas as pd + +Q30_SCALE = 1 << 30 + + +def bin_features_for_packboost(features: pd.DataFrame) -> np.ndarray: + arr = features.to_numpy(copy=False, dtype=np.float64) + arr = np.nan_to_num(arr, nan=0.0) + rounded = np.rint(arr) + if ( + arr.size > 0 + and arr.min() >= 0.0 + and arr.max() <= 4.0 + and np.allclose(arr, rounded, atol=1e-6) + ): + return np.asarray(np.clip(rounded, 0, 4), dtype=np.int8) + + n_rows = arr.shape[0] + if n_rows == 0: + return np.empty((0, arr.shape[1]), dtype=np.int8) + + out = np.empty(arr.shape, dtype=np.int8) + for col_idx in range(arr.shape[1]): + col = arr[:, col_idx] + order = np.argsort(col, kind="mergesort") + ranks = np.empty(n_rows, dtype=np.float64) + ranks[order] = np.arange(n_rows, dtype=np.float64) + if n_rows == 1: + out[:, col_idx] = 2 + else: + out[:, col_idx] = np.floor(ranks / (n_rows - 1) * 4.0).astype(np.int8) + return out + + +def encode_era_ids(era: pd.Series) -> np.ndarray: + ordered = sorted(era.unique(), key=str) + mapping = {label: idx for idx, label in enumerate(ordered)} + return np.asarray(era.map(mapping).astype(np.int32).to_numpy()) + + +def sort_rows_by_era( + features: pd.DataFrame, + target: pd.Series, + era: pd.Series, +) -> tuple[pd.DataFrame, pd.Series, pd.Series]: + era_codes = encode_era_ids(era) + order = np.argsort(era_codes, kind="stable") + idx = features.index[order] + return ( + features.loc[idx], + target.loc[idx], + era.loc[idx], + ) + + +def default_nfeatsets(n_features: int, requested: int = 32) -> int: + bitplanes = 4 * n_features + max_sets = max(1, bitplanes // 32) + return min(requested, max_sets) + + +def q30_predictions_to_float(predictions_q30: np.ndarray) -> np.ndarray: + return predictions_q30.astype(np.float64) / float(Q30_SCALE) diff --git a/src/alphapulse/models/packboost_model.py b/src/alphapulse/models/packboost_model.py index 66326ef..b82c865 100644 --- a/src/alphapulse/models/packboost_model.py +++ b/src/alphapulse/models/packboost_model.py @@ -3,11 +3,13 @@ import numpy as np import pandas as pd -import xgboost as xgb from ..evaluation.metrics import per_era_correlation from .base import BaseModel, _numeric -from .xgboost_model import _make_ray_callbacks +from .packboost_backend import PackBoostTrainer, require_packboost_cuda + +_MIN_TRAIN_ROWS = 10 +_MIN_VAL_ROWS = 10 class PackboostModel(BaseModel): @@ -22,23 +24,18 @@ def __init__( early_stopping_rounds_base: int = 50, n_rounds_boost: int = 200, early_stopping_rounds_boost: int = 30, + device: str = "cuda", + max_depth: int = 7, + nfolds: int = 8, + lr: float = 0.07, + l2: float = 100_000.0, + nfeatsets: int = 32, + seed: int = 42, name: str = "Packboost", ) -> None: super().__init__(name) - self.base_params = base_params or { - "max_depth": 5, - "learning_rate": 0.01, - "tree_method": "hist", - "objective": "reg:squarederror", - "eval_metric": "rmse", - } - self.boost_params = boost_params or { - "max_depth": 3, - "learning_rate": 0.05, - "tree_method": "hist", - "objective": "reg:squarederror", - "eval_metric": "rmse", - } + self.base_params = base_params or {} + self.boost_params = boost_params or {"max_depth": 3} self.era_column = era_column self.n_worst_eras = int(n_worst_eras) self.boost_weight = float(boost_weight) @@ -46,9 +43,16 @@ def __init__( self.early_stopping_rounds_base = int(early_stopping_rounds_base) self.n_rounds_boost = int(n_rounds_boost) self.early_stopping_rounds_boost = int(early_stopping_rounds_boost) - - self._base_model: xgb.Booster | None = None - self._era_models: dict[Any, xgb.Booster] = {} + self.device = device + self.max_depth = int(max_depth) + self.nfolds = int(nfolds) + self.lr = float(lr) + self.l2 = float(l2) + self.nfeatsets = int(nfeatsets) + self.seed = int(seed) + + self._base_trainer: PackBoostTrainer | None = None + self._era_trainers: dict[Any, PackBoostTrainer] = {} self._feature_columns: list[str] | None = None self._worst_eras: list[Any] = [] @@ -57,6 +61,17 @@ def _feature_matrix(self, X: pd.DataFrame) -> pd.DataFrame: return X[self._feature_columns] return _numeric(X) + def _make_trainer(self, *, max_depth: int | None = None) -> PackBoostTrainer: + return PackBoostTrainer( + device=self.device, + max_depth=max_depth if max_depth is not None else self.max_depth, + nfolds=self.nfolds, + lr=self.lr, + l2=self.l2, + nfeatsets=self.nfeatsets, + seed=self.seed, + ) + def train( self, X_train: pd.DataFrame, @@ -67,6 +82,8 @@ def train( early_stopping_rounds: int | None = None, **kwargs: Any, ) -> dict[str, float]: + require_packboost_cuda(device=self.device) + era_train = kwargs.get("era_train") if era_train is None and self.era_column in X_train.columns: era_train = X_train[self.era_column] @@ -78,146 +95,100 @@ def train( feat = self._feature_matrix(X_train) self._feature_columns = list(feat.columns) - dtrain = xgb.DMatrix(feat, label=y_train) - eval_set: list[tuple[xgb.DMatrix, str]] = [] - evals_result: dict[str, Any] = {} n_rounds_use = n_rounds if n_rounds is not None else self.n_rounds_base - early_use = ( - early_stopping_rounds - if early_stopping_rounds is not None - else self.early_stopping_rounds_base - ) - feat_val: pd.DataFrame | None = None era_val_series: pd.Series | None = None if X_val is not None and y_val is not None: feat_val = self._feature_matrix(X_val) - dval = xgb.DMatrix(feat_val, label=y_val) - eval_set = [(dtrain, "train"), (dval, "eval")] if self.era_column in X_val.columns: era_val_series = X_val[self.era_column] - callbacks: list[Any] = _make_ray_callbacks() - - self._base_model = xgb.train( - self.base_params, - dtrain, - num_boost_round=n_rounds_use, - evals=eval_set, - evals_result=evals_result, - early_stopping_rounds=early_use if eval_set else None, - verbose_eval=False, - callbacks=callbacks, + self._base_trainer = self._make_trainer() + self._base_trainer.fit( + feat, + y_train, + era=era_train, + val_features=feat_val, + val_target=y_val, + rounds=n_rounds_use, ) - - base_pred = self._base_model.predict(xgb.DMatrix(feat)) + base_pred = self._base_trainer.predict(feat) per_era_corr = per_era_correlation(y_train, base_pred, era_train) valid_corr = per_era_corr.dropna() if len(valid_corr) == 0: self._worst_eras = [] self.is_trained = True - return { - "base_final_rmse": float( - evals_result.get("train", {}).get("rmse", [0])[-1] - ) - } + return {"n_boost_eras": 0.0} ascending_order = per_era_corr.sort_values(ascending=True) self._worst_eras = ascending_order.index[: self.n_worst_eras].tolist() - MIN_ERA_ROWS = 10 - self._era_models = {} + self._era_trainers = {} + boost_depth = int(self.boost_params.get("max_depth", 3)) for era_id in self._worst_eras: mask = era_train == era_id - if int(mask.sum()) < MIN_ERA_ROWS: + if int(mask.sum()) < _MIN_TRAIN_ROWS: continue - X_era = feat.loc[mask] + x_era = feat.loc[mask] y_era = y_train.loc[mask] - eval_set_era: list[tuple[xgb.DMatrix, str]] = [] + x_era_fit = x_era + y_era_fit = y_era + x_era_val_local: pd.DataFrame | None = None + y_era_val_local: pd.Series | None = None if feat_val is not None and era_val_series is not None: mask_val = era_val_series == era_id - if int(mask_val.sum()) >= MIN_ERA_ROWS: - X_era_val = feat_val.loc[mask_val] - y_era_val = y_val.loc[mask_val] if y_val is not None else None - if y_era_val is not None and len(X_era_val) == len(y_era_val): - d_era_fit = xgb.DMatrix(X_era, label=y_era) - d_era_val = xgb.DMatrix(X_era_val, label=y_era_val) - eval_set_era = [(d_era_fit, "train"), (d_era_val, "eval")] - - if not eval_set_era: - n = len(X_era) + if int(mask_val.sum()) >= _MIN_VAL_ROWS: + x_era_val_local = feat_val.loc[mask_val] + y_era_val_local = y_val.loc[mask_val] if y_val is not None else None + + if x_era_val_local is None: + n = len(x_era) if n <= 2: - d_era_fit = xgb.DMatrix(X_era, label=y_era) - booster = xgb.train( - self.boost_params, - d_era_fit, - num_boost_round=self.n_rounds_boost, - verbose_eval=False, - ) - self._era_models[era_id] = booster + trainer = self._make_trainer(max_depth=boost_depth) + trainer.fit(x_era, y_era, rounds=self.n_rounds_boost) + self._era_trainers[era_id] = trainer continue - - n_val_era = max(1, int(n * 0.1)) - n_val_era = min(n_val_era, n - 1) - X_era_fit = X_era.iloc[: n - n_val_era] + n_val_era = max(1, min(int(n * 0.1), n - 1)) + x_era_fit = x_era.iloc[: n - n_val_era] y_era_fit = y_era.iloc[: n - n_val_era] - X_era_eval = X_era.iloc[n - n_val_era :] - y_era_eval = y_era.iloc[n - n_val_era :] - - d_era_fit = xgb.DMatrix(X_era_fit, label=y_era_fit) - d_era_eval = xgb.DMatrix(X_era_eval, label=y_era_eval) - eval_set_era = [(d_era_fit, "train"), (d_era_eval, "eval")] - - booster = xgb.train( - self.boost_params, - eval_set_era[0][0], - num_boost_round=self.n_rounds_boost, - evals=eval_set_era, - early_stopping_rounds=self.early_stopping_rounds_boost - if eval_set_era - else None, - verbose_eval=False, + x_era_val_local = x_era.iloc[n - n_val_era :] + y_era_val_local = y_era.iloc[n - n_val_era :] + + trainer = self._make_trainer(max_depth=boost_depth) + trainer.fit( + x_era_fit, + y_era_fit, + val_features=x_era_val_local, + val_target=y_era_val_local, + rounds=self.n_rounds_boost, ) - self._era_models[era_id] = booster + self._era_trainers[era_id] = trainer self.is_trained = True self.model = { - "base": self._base_model, - "era_models": self._era_models, + "base_trainer": self._base_trainer, + "era_trainers": self._era_trainers, "feature_columns": self._feature_columns, "worst_eras": self._worst_eras, } - - metrics: dict[str, float] = {} - for k, v in evals_result.items(): - for metric_name, values in v.items(): - if values: - metrics[f"{k}_{metric_name}"] = float(values[-1]) - metrics["n_boost_eras"] = float(len(self._era_models)) - return metrics + return {"n_boost_eras": float(len(self._era_trainers))} def predict(self, X: pd.DataFrame) -> np.ndarray: - if not self.is_trained or self._base_model is None: + if not self.is_trained or self._base_trainer is None: raise ValueError("PackboostModel is not trained.") feat = self._feature_matrix(X) - base_pred = self._base_model.predict(xgb.DMatrix(feat)) - out = np.array(base_pred, dtype=np.float64) - - if self.era_column not in X.columns or not self._era_models: - return out - - era_values = X[self.era_column] - for era_id, booster in self._era_models.items(): - mask = (era_values == era_id).values - if not mask.any(): - continue - era_feat = feat.loc[mask] - boost_pred = booster.predict(xgb.DMatrix(era_feat)) - out[mask] += self.boost_weight * boost_pred - + out = self._base_trainer.predict(feat) + if self.era_column in X.columns and self._era_trainers: + era_values = X[self.era_column] + for era_id, trainer in self._era_trainers.items(): + mask = (era_values == era_id).values + if not mask.any(): + continue + boost_pred = trainer.predict(feat.loc[mask]) + out[mask] += self.boost_weight * boost_pred return out def save(self, path: Path) -> None: @@ -226,8 +197,8 @@ def save(self, path: Path) -> None: path = Path(path) path.parent.mkdir(parents=True, exist_ok=True) state = { - "base": self._base_model, - "era_models": self._era_models, + "base_trainer": self._base_trainer, + "era_trainers": self._era_trainers, "feature_columns": self._feature_columns, "worst_eras": self._worst_eras, "era_column": self.era_column, @@ -241,8 +212,13 @@ def load(self, path: Path) -> "PackboostModel": with open(path, "rb") as f: state = cloudpickle.load(f) - self._base_model = state["base"] - self._era_models = state["era_models"] + if "base" in state or "era_models" in state: + raise ValueError( + "Legacy XGBoost Packboost checkpoint is not supported. " + "Retrain with PackBoost CUDA." + ) + self._base_trainer = state["base_trainer"] + self._era_trainers = state["era_trainers"] self._feature_columns = state["feature_columns"] self._worst_eras = state["worst_eras"] self.era_column = state.get("era_column", "era") diff --git a/src/alphapulse/models/xgboost_model.py b/src/alphapulse/models/xgboost_model.py index 4a33633..407eafe 100644 --- a/src/alphapulse/models/xgboost_model.py +++ b/src/alphapulse/models/xgboost_model.py @@ -4,10 +4,46 @@ import numpy as np import pandas as pd import xgboost as xgb +from loguru import logger from .base import BaseModel, _numeric +def _make_progress_callbacks(model_name: str, log_every: int = 10) -> list[Any]: + ray_callbacks = _make_ray_callbacks() + if ray_callbacks: + return ray_callbacks + + from ..logging_.wandb_logging import ( + log_boosting_round_metrics, + parse_xgb_evals_log, + wandb_run_active, + ) + + class _LogProgressCallback(xgb.callback.TrainingCallback): + def after_iteration( + self, + model: Any, + epoch: int, + evals_log: dict[str, dict[str, list[float] | list[tuple[float, float]]]], + ) -> bool: + if epoch != 0 and (epoch + 1) % log_every != 0: + return False + parsed = parse_xgb_evals_log(evals_log) + if parsed: + parts = [f"{k}={v:.6f}" for k, v in parsed.items()] + logger.info("{} round {}: {}", model_name, epoch + 1, " ".join(parts)) + if wandb_run_active(): + log_boosting_round_metrics( + model_name=model_name, + round_num=epoch + 1, + metrics=parsed, + ) + return False + + return [_LogProgressCallback()] + + def _make_ray_callbacks() -> list[Any]: try: from ray.tune import session as ray_session @@ -87,7 +123,16 @@ def train( eval_set = [(dtrain, "train"), (dval, "eval")] evals_result: dict[str, Any] = {} - callbacks: list[Any] = _make_ray_callbacks() + callbacks: list[Any] = _make_progress_callbacks(self.name) + + logger.info( + "{}: starting XGBoost train rows={} features={} rounds={} early_stop={}", + self.name, + len(feat_train), + feat_train.shape[1], + n_rounds, + early_stopping_rounds if eval_set else None, + ) self.model = xgb.train( self.params, @@ -100,6 +145,8 @@ def train( callbacks=callbacks, ) self.is_trained = True + best_iter = getattr(self.model, "best_iteration", n_rounds - 1) + logger.info("{}: finished at iteration {}", self.name, best_iter + 1) metrics: dict[str, float] = {} for key, values in evals_result.items(): diff --git a/src/alphapulse/pipeline/ensemble.py b/src/alphapulse/pipeline/ensemble.py index 4f8f5bd..b2da286 100644 --- a/src/alphapulse/pipeline/ensemble.py +++ b/src/alphapulse/pipeline/ensemble.py @@ -4,6 +4,25 @@ import numpy as np import pandas as pd +from .ensemble_optimizer import ( + DEFAULT_MAX_WEIGHT, + DEFAULT_MIN_WEIGHT, + EnsembleOptimizer, +) + + +def needs_internal_val_for_ensemble(pipeline_cfg: dict[str, Any]) -> bool: + models = pipeline_cfg.get("models") or [] + if len(models) <= 1: + return False + method = pipeline_cfg.get("ensemble_method", "single") + if method == "stacking": + return True + if method == "weighted": + params = pipeline_cfg.get("ensemble_params") or {} + return bool(params.get("optimize_weights")) + return False + class EnsembleStrategy: def __init__( @@ -17,17 +36,101 @@ def __init__( self._meta_learner: Any = None self._meta_learner_type: str | None = None + @property + def weights(self) -> np.ndarray | None: + return None if self._weights is None else self._weights.copy() + + def set_weights(self, weights: np.ndarray | list[float]) -> None: + w = np.asarray(weights, dtype=np.float64) + if w.ndim != 1: + raise ValueError("weights must be a 1-D array") + total = float(w.sum()) + if total <= 0.0 or not np.isfinite(total): + raise ValueError( + f"weights must sum to a positive finite value, got {total}" + ) + self._weights = w / total + def fit( self, n_models: int, get_val_predictions: Callable[[], np.ndarray] | None = None, y_val: pd.Series | None = None, + eras_val: pd.Series | None = None, + meta_model_preds: np.ndarray | None = None, ) -> None: if n_models <= 1: return if self.method == "weighted": weights = self.params.get("weights") + optimize_weights = bool(self.params.get("optimize_weights")) + if optimize_weights and weights is not None: + raise ValueError( + "ensemble_params cannot set both optimize_weights and fixed weights" + ) + if ( + optimize_weights + and get_val_predictions is not None + and y_val is not None + ): + stack_X = get_val_predictions() + if stack_X.shape[1] != n_models: + raise ValueError( + f"validation predictions have {stack_X.shape[1]} columns " + f"but expected {n_models}" + ) + finite_mask = np.isfinite(stack_X).all(axis=1) & np.isfinite( + np.asarray(y_val, dtype=np.float64) + ) + if eras_val is not None: + finite_mask &= eras_val.notna().to_numpy() + if not finite_mask.all(): + stack_X = stack_X[finite_mask] + y_val_arr = np.asarray(y_val, dtype=np.float64)[finite_mask] + eras_fit = ( + eras_val.iloc[finite_mask] + if eras_val is not None + else pd.Series(np.zeros(len(y_val_arr))) + ) + meta_fit = ( + np.asarray(meta_model_preds, dtype=np.float64)[finite_mask] + if meta_model_preds is not None + else None + ) + else: + y_val_arr = np.asarray(y_val, dtype=np.float64) + eras_fit = ( + eras_val + if eras_val is not None + else pd.Series(np.zeros(len(y_val_arr))) + ) + meta_fit = meta_model_preds + + objective = self.params.get("objective", "corr_sharpe") + if objective not in ("corr_sharpe", "payout_score"): + objective = "corr_sharpe" + optimizer = EnsembleOptimizer( + objective=objective, + corr_weight=float(self.params.get("corr_weight", 0.75)), + mmc_weight=float(self.params.get("mmc_weight", 2.25)), + min_weight=float(self.params.get("min_weight", DEFAULT_MIN_WEIGHT)), + max_weight=float(self.params.get("max_weight", DEFAULT_MAX_WEIGHT)), + seed=int(self.params.get("seed", 42)), + ) + min_w = self.params.get("min_weights") + max_w = self.params.get("max_weights") + optimizer.fit( + stack_X, + y_val_arr, + eras_fit, + meta_model_preds=meta_fit, + min_weights=list(min_w) if min_w is not None else None, + max_weights=list(max_w) if max_w is not None else None, + ) + self._weights = optimizer.weights_ + return + if weights is not None: w = np.asarray(weights, dtype=np.float64) if len(w) != n_models: diff --git a/src/alphapulse/pipeline/ensemble_optimizer.py b/src/alphapulse/pipeline/ensemble_optimizer.py index 3a73651..698e72e 100644 --- a/src/alphapulse/pipeline/ensemble_optimizer.py +++ b/src/alphapulse/pipeline/ensemble_optimizer.py @@ -6,7 +6,123 @@ from ..evaluation.metrics import era_sharpe, payout_score -MAX_RESTARTS = 3 +DEFAULT_MIN_WEIGHT = 0.05 +DEFAULT_MAX_WEIGHT = 0.90 + + +def validate_weight_bounds_list( + min_weights: list[float], max_weights: list[float] +) -> None: + k = len(min_weights) + if k != len(max_weights): + raise ValueError("min_weights and max_weights must have the same length") + if k < 1: + raise ValueError("k must be >= 1") + for i, (lo, hi) in enumerate(zip(min_weights, max_weights, strict=True)): + if lo < 0.0 or hi > 1.0 or lo > hi: + raise ValueError(f"invalid weight bounds at index {i}: min={lo}, max={hi}") + if sum(min_weights) > 1.0 + 1e-9: + raise ValueError(f"infeasible: sum(min_weights) > 1 ({sum(min_weights)})") + if sum(max_weights) < 1.0 - 1e-9: + raise ValueError(f"infeasible: sum(max_weights) < 1 ({sum(max_weights)})") + + +def validate_weight_bounds(k: int, min_weight: float, max_weight: float) -> None: + if k < 1: + raise ValueError("k must be >= 1") + if min_weight < 0.0 or max_weight > 1.0 or min_weight > max_weight: + raise ValueError( + f"invalid weight bounds: min_weight={min_weight}, max_weight={max_weight}" + ) + if k * min_weight > 1.0 + 1e-9: + raise ValueError(f"infeasible: k * min_weight > 1 ({k} * {min_weight})") + if k * max_weight < 1.0 - 1e-9: + raise ValueError(f"infeasible: k * max_weight < 1 ({k} * {max_weight})") + + +def project_weights_to_bounds_list( + weights: np.ndarray, + min_weights: list[float], + max_weights: list[float], +) -> np.ndarray: + k = len(weights) + lo = [float(v) for v in min_weights] + hi = [float(v) for v in max_weights] + validate_weight_bounds_list(lo, hi) + w = np.clip(np.asarray(weights, dtype=np.float64), lo, hi) + for _ in range(64): + total = float(w.sum()) + if abs(total - 1.0) < 1e-10: + break + slack_hi = np.asarray(hi, dtype=np.float64) - w + slack_lo = w - np.asarray(lo, dtype=np.float64) + delta = 1.0 - total + if delta > 0: + room = slack_hi.sum() + if room > 1e-12: + w += delta * (slack_hi / room) + else: + room = slack_lo.sum() + if room > 1e-12: + w += delta * (slack_lo / room) + w = np.clip(w, lo, hi) + if abs(float(w.sum()) - 1.0) > 1e-6: + w = project_weights_to_bounds(np.ones(k) / k, lo[0], hi[0]) + w = project_weights_to_bounds_list(w, lo, hi) + return w + + +def feasible_weight_starts_list( + min_weights: list[float], + max_weights: list[float], + rng: np.random.RandomState, +) -> list[np.ndarray]: + lo = [float(v) for v in min_weights] + hi = [float(v) for v in max_weights] + validate_weight_bounds_list(lo, hi) + k = len(lo) + starts = [ + project_weights_to_bounds_list(np.ones(k) / k, lo, hi), + ] + for _ in range(2): + raw = rng.dirichlet(np.ones(k)) + starts.append(project_weights_to_bounds_list(raw, lo, hi)) + return starts + + +def project_weights_to_bounds( + weights: np.ndarray, + min_weight: float, + max_weight: float, +) -> np.ndarray: + k = len(weights) + validate_weight_bounds(k, min_weight, max_weight) + w = np.clip(np.asarray(weights, dtype=np.float64), min_weight, max_weight) + for _ in range(64): + total = float(w.sum()) + if abs(total - 1.0) < 1e-10: + break + delta = (1.0 - total) / k + w = np.clip(w + delta, min_weight, max_weight) + if abs(float(w.sum()) - 1.0) > 1e-6: + w = np.ones(k, dtype=np.float64) / k + w = np.clip(w, min_weight, max_weight) + w = project_weights_to_bounds(w, min_weight, max_weight) + return w + + +def feasible_weight_starts( + k: int, + min_weight: float, + max_weight: float, + rng: np.random.RandomState, +) -> list[np.ndarray]: + validate_weight_bounds(k, min_weight, max_weight) + starts = [project_weights_to_bounds(np.ones(k) / k, min_weight, max_weight)] + for _ in range(2): + raw = rng.dirichlet(np.ones(k)) + starts.append(project_weights_to_bounds(raw, min_weight, max_weight)) + return starts class EnsembleOptimizer: @@ -18,6 +134,8 @@ def __init__( objective: Literal["corr_sharpe", "payout_score"] = "corr_sharpe", corr_weight: float = 0.75, mmc_weight: float = 2.25, + min_weight: float = DEFAULT_MIN_WEIGHT, + max_weight: float = DEFAULT_MAX_WEIGHT, ) -> None: self.method = method self.max_iter = max_iter @@ -25,6 +143,8 @@ def __init__( self.objective = objective self.corr_weight = corr_weight self.mmc_weight = mmc_weight + self.min_weight = float(min_weight) + self.max_weight = float(max_weight) self.weights_: np.ndarray | None = None self.sharpe_: float = float("-inf") @@ -35,6 +155,8 @@ def fit( eras_oof: pd.Series, *, meta_model_preds: np.ndarray | None = None, + min_weights: list[float] | None = None, + max_weights: list[float] | None = None, ) -> Self: """Optimise ensemble weights on OOF predictions. @@ -60,6 +182,19 @@ def fit( self.sharpe_ = era_sharpe(y_series, oof_matrix[:, 0], eras_oof) return self + lo = ( + [float(v) for v in min_weights] + if min_weights is not None + else [self.min_weight] * k + ) + hi = ( + [float(v) for v in max_weights] + if max_weights is not None + else [self.max_weight] * k + ) + validate_weight_bounds_list(lo, hi) + bounds = list(zip(lo, hi, strict=True)) + def neg_objective(w: np.ndarray) -> float: blend = oof_matrix @ w if use_payout and meta_arr is not None: @@ -76,19 +211,12 @@ def neg_objective(w: np.ndarray) -> float: return -score if np.isfinite(score) else 1e6 constraints = {"type": "eq", "fun": lambda w: w.sum() - 1.0} - bounds = [(0.0, 1.0)] * k rng = np.random.RandomState(self.seed) best_result = None best_val = float("inf") - starts = [ - np.ones(k) / k, - rng.dirichlet(np.ones(k)), - rng.dirichlet(np.ones(k) * 0.1), - ] - - for w0 in starts: + for w0 in feasible_weight_starts_list(lo, hi, rng): res = minimize( neg_objective, w0, @@ -102,8 +230,11 @@ def neg_objective(w: np.ndarray) -> float: best_result = res assert best_result is not None - self.weights_ = np.clip(np.asarray(best_result.x, dtype=np.float64), 0.0, None) - self.weights_ /= self.weights_.sum() + self.weights_ = project_weights_to_bounds_list( + np.asarray(best_result.x, dtype=np.float64), + lo, + hi, + ) self.sharpe_ = -best_val return self @@ -114,23 +245,6 @@ def predict(self, pred_matrix: np.ndarray) -> np.ndarray: class GreedyEnsembleSelector: - """Greedily select the best subset of models from OOF predictions. - - Starting from an empty ensemble, iteratively adds the model whose OOF - predictions most improve the objective (equal-weight average) when included. - Continues until no improvement or ``max_models`` is reached. - - This is faster than exhaustive search and produces sparse, high-quality - ensembles. Typical usage: run after collecting OOF predictions from many - HPO trials to find the best k of n configs. - - Args: - max_models: Maximum number of models to include. - metric: Objective to maximise. ``"payout_score"`` requires meta model. - corr_weight: CORR weight in payout formula. Default 0.75. - mmc_weight: MMC weight in payout formula. Default 2.25. - """ - def __init__( self, max_models: int = 20, @@ -168,16 +282,6 @@ def fit( meta_model_preds: np.ndarray | None = None, model_names: list[str] | None = None, ) -> Self: - """Select the best subset of models from OOF predictions. - - Args: - oof_matrix: Shape (n_samples, n_models) OOF prediction matrix. - y_oof: True target values. - eras_oof: Era labels. - meta_model_preds: Optional Numerai meta model predictions. - Required when ``metric="payout_score"``. - model_names: Optional display names for each column (for inspection). - """ n_models = oof_matrix.shape[1] y_series = pd.Series(y_oof) meta_arr = ( @@ -204,7 +308,7 @@ def fit( best_i = i if best_i == -1 or best_step_score <= best_score: - break # no more improvement + break selected.append(best_i) n_sel = len(selected) diff --git a/src/alphapulse/pipeline/model_access.py b/src/alphapulse/pipeline/model_access.py new file mode 100644 index 0000000..27854ab --- /dev/null +++ b/src/alphapulse/pipeline/model_access.py @@ -0,0 +1,61 @@ +from __future__ import annotations + +import numpy as np +import pandas as pd + +from ..models.base import BaseModel +from .multi_target import MultiTargetPipeline +from .multihead import MultiHeadPipeline +from .pipeline import Pipeline +from .row_utils import protected_metadata_frame + +PipelineLike = Pipeline | MultiHeadPipeline | MultiTargetPipeline + + +def iter_trained_models(pipeline: PipelineLike) -> list[BaseModel]: + if isinstance(pipeline, MultiHeadPipeline): + return [h.model for h in pipeline.heads] + if isinstance(pipeline, MultiTargetPipeline): + return list(pipeline._models.values()) + return list(pipeline.models) + + +def _preprocess_multitarget( + pipeline: MultiTargetPipeline, X: pd.DataFrame +) -> pd.DataFrame: + X_t = X + for pp in pipeline.preprocessors: + X_t = pp.transform(X_t) + return X_t + + +def model_prediction_map( + pipeline: PipelineLike, + X_val: pd.DataFrame, + feature_cols: list[str], +) -> dict[str, np.ndarray]: + if isinstance(pipeline, MultiHeadPipeline): + X_in = X_val[feature_cols] if feature_cols else X_val + return {"ensemble": pipeline.predict(X_in)} + + if isinstance(pipeline, MultiTargetPipeline): + X_in = X_val[feature_cols] if feature_cols else X_val + X_t = _preprocess_multitarget(pipeline, X_in) + return { + target: model.predict(X_t) for target, model in pipeline._models.items() + } + + X_feat = X_val[feature_cols] if feature_cols else X_val + era_meta = protected_metadata_frame(X_feat) + X_t = pipeline._preprocess(X_feat, era_meta) + X_numeric = X_t.select_dtypes(include=[np.number]) + models = list(pipeline.models) + if len(models) == 1: + return {models[0].name: models[0].predict(X_numeric)} + return {m.name: m.predict(X_numeric) for m in models} + + +def multitarget_blend_weights(pipeline: MultiTargetPipeline) -> np.ndarray | None: + if pipeline._weights is None: + return None + return np.asarray(pipeline._weights, dtype=np.float64) diff --git a/src/alphapulse/pipeline/multi_target.py b/src/alphapulse/pipeline/multi_target.py index 94c5e04..47a66f6 100644 --- a/src/alphapulse/pipeline/multi_target.py +++ b/src/alphapulse/pipeline/multi_target.py @@ -1,5 +1,6 @@ from collections.abc import Callable -from typing import Any +from pathlib import Path +from typing import Any, Self import numpy as np import pandas as pd @@ -46,6 +47,7 @@ def fit( X_val: pd.DataFrame | None = None, targets_val: pd.DataFrame | None = None, era_train: pd.Series | None = None, + era_val: pd.Series | None = None, **model_train_kwargs: Any, ) -> dict[str, float]: self.feature_columns = list(X.columns) @@ -71,10 +73,10 @@ def fit( if era_train is not None: era_train = era_train.loc[X_fit.index] - era_val: pd.Series | None = None X_val_t: pd.DataFrame | None = None if X_val is not None: - era_val = X_val["era"] if "era" in X_val.columns else None + if era_val is None and "era" in X_val.columns: + era_val = X_val["era"] X_val_t = X_val for pp in self.preprocessors: X_val_t = pp.transform(X_val_t) @@ -113,12 +115,17 @@ def fit( y_val_masked = None model = self.model_factory() + train_kw = dict(model_train_kwargs) + if era_train is not None: + train_kw["era_train"] = era_train.loc[X_fit_masked.index] + if era_val is not None and X_val_masked is not None: + train_kw["era_val"] = era_val.loc[X_val_masked.index] metrics = model.train( X_fit_masked, y_masked, X_val=X_val_masked, y_val=y_val_masked, - **model_train_kwargs, + **train_kw, ) self._models[target_col] = model for k, v in metrics.items(): @@ -214,7 +221,7 @@ def predict( live_features: pd.DataFrame, live_benchmark_models: pd.DataFrame, ) -> pd.DataFrame: - X = live_features[feature_columns] + X = live_features.reindex(columns=feature_columns, fill_value=0.0) raw_preds = pipeline.predict(X) ranked = rank_normalize(raw_preds) @@ -229,3 +236,23 @@ def predict( return pd.DataFrame({"prediction": ranked}, index=live_features.index) return predict + + def save_pipeline(self, path: Path) -> None: + import cloudpickle + + path = Path(path) + path.parent.mkdir(parents=True, exist_ok=True) + with open(path, "wb") as f: + cloudpickle.dump(self, f) + + @classmethod + def load_pipeline(cls, path: Path) -> Self: + import cloudpickle + + with open(path, "rb") as f: + loaded = cloudpickle.load(f) + if not isinstance(loaded, cls): + raise TypeError( + f"Expected a {cls.__name__} object, got {type(loaded).__name__}" + ) + return loaded diff --git a/src/alphapulse/pipeline/multihead.py b/src/alphapulse/pipeline/multihead.py index b458590..7220277 100644 --- a/src/alphapulse/pipeline/multihead.py +++ b/src/alphapulse/pipeline/multihead.py @@ -27,10 +27,12 @@ def __init__( input_group: str | None, local_preprocessors: list[BasePreprocessor], feature_groups: dict[str, list[str]], + input_groups: list[str] | None = None, ) -> None: self.model = model self.input_columns = input_columns self.input_group = input_group + self.input_groups = input_groups self.local_preprocessors = local_preprocessors self.feature_groups = feature_groups @@ -44,6 +46,27 @@ def resolved_columns(self, X: pd.DataFrame) -> list[str]: + (" (and more)" if len(missing) > 10 else "") ) return list(self.input_columns) + if self.input_groups: + seen: set[str] = set() + out: list[str] = [] + for group in self.input_groups: + if group not in self.feature_groups: + available = sorted(self.feature_groups.keys()) + raise ValueError( + f"Model {self.model.name!r} references input_groups containing " + f"{group!r}, which is not defined in feature_groups. " + f"Available groups: {available}" + ) + for col in self.feature_groups[group]: + if col in X.columns and col not in seen: + seen.add(col) + out.append(col) + if not out: + raise ValueError( + f"Model {self.model.name!r} input_groups={self.input_groups!r} " + "resolved to zero columns present in data" + ) + return out if self.input_group is not None: if self.input_group not in self.feature_groups: available = sorted(self.feature_groups.keys()) @@ -107,7 +130,7 @@ def _head_matrix( f"Missing columns for head {head.model.name}: {missing[:10]}..." ) X_h = X_global[cols].copy() - era_meta = protected_metadata_frame(X_h) + era_meta = protected_metadata_frame(X_global) for pp in head.local_preprocessors: if isinstance(pp, TrainEvalPreprocessor): pp.train() if fit else pp.eval() @@ -115,7 +138,18 @@ def _head_matrix( pp.fit(X_h, y) X_h = pp.transform(X_h) X_h = reattach_protected_columns(X_h, era_meta) - return X_h + return reattach_protected_columns(X_h, era_meta) + + def _with_era_column( + self, + frame: pd.DataFrame, + era: pd.Series | None, + ) -> pd.DataFrame: + if era is None or "era" in frame.columns: + return frame + out = frame.copy() + out["era"] = era.loc[frame.index] + return out def fit( self, @@ -125,6 +159,12 @@ def fit( y_val: pd.Series | None = None, **model_train_kwargs: Any, ) -> dict[str, float]: + era_train = model_train_kwargs.pop("era_train", None) + era_val = model_train_kwargs.pop("era_val", None) + X = self._with_era_column(X, era_train) + if X_val is not None: + X_val = self._with_era_column(X_val, era_val) + if self.feature_columns is None: self.feature_columns = list(X.columns) diff --git a/src/alphapulse/pipeline/neutralizer.py b/src/alphapulse/pipeline/neutralizer.py index 711da59..1cc88d7 100644 --- a/src/alphapulse/pipeline/neutralizer.py +++ b/src/alphapulse/pipeline/neutralizer.py @@ -1,9 +1,11 @@ +from typing import Literal + import numpy as np import pandas as pd from scipy.optimize import minimize_scalar from ..constants import _PROTECTED_COLS -from ..evaluation.metrics import era_sharpe +from ..evaluation.metrics import era_sharpe, era_sharpe_of_mmc, payout_score def _numeric_features(features: pd.DataFrame | np.ndarray) -> np.ndarray: @@ -16,6 +18,100 @@ def _numeric_features(features: pd.DataFrame | np.ndarray) -> np.ndarray: return np.asarray(features, dtype=np.float64) +def neutralize_against_meta( + predictions: np.ndarray | pd.Series, + meta_model: np.ndarray | pd.Series, + eras: pd.Series | None = None, + proportion: float = 0.5, +) -> np.ndarray: + """Remove linear exposure of predictions to the Numerai meta model (per era).""" + if proportion == 0.0: + return np.asarray(predictions, dtype=np.float64) + pred_arr = np.asarray(predictions, dtype=np.float64) + meta_arr = np.asarray(meta_model, dtype=np.float64).reshape(-1) + if len(pred_arr) != len(meta_arr): + raise ValueError( + f"predictions length {len(pred_arr)} != meta_model length {len(meta_arr)}" + ) + + def _neutralize_slice(pred: np.ndarray, meta: np.ndarray) -> np.ndarray: + if len(pred) < 2: + return pred.copy() + meta_c = meta - meta.mean() + pred_c = pred - pred.mean() + denom = float(np.dot(meta_c, meta_c)) + if denom <= 1e-12: + return pred.copy() + beta = float(np.dot(meta_c, pred_c) / denom) + exposure = meta_c * beta + return np.asarray(pred - proportion * exposure, dtype=np.float64) + + if eras is None: + return _neutralize_slice(pred_arr, meta_arr) + + era_vals = np.asarray(eras) + out = pred_arr.copy() + for era in np.unique(era_vals): + mask = era_vals == era + if mask.sum() < 2: + continue + out[mask] = _neutralize_slice(pred_arr[mask], meta_arr[mask]) + return out + + +class MetaModelNeutralizer: + def __init__(self, proportion: float = 0.5) -> None: + if not 0.0 <= proportion <= 1.0: + raise ValueError(f"proportion must be in [0, 1], got {proportion}") + self.proportion = proportion + + def neutralize( + self, + predictions: np.ndarray | pd.Series, + meta_model: np.ndarray | pd.Series, + eras: pd.Series | None = None, + ) -> np.ndarray: + return neutralize_against_meta( + predictions, meta_model, eras=eras, proportion=self.proportion + ) + + def optimize_proportion( + self, + predictions: np.ndarray | pd.Series, + meta_model: np.ndarray | pd.Series, + y_true: pd.Series, + eras: pd.Series, + *, + objective: Literal["corr_sharpe", "mmc_sharpe", "payout_score"] = "mmc_sharpe", + bounds: tuple[float, float] = (0.0, 1.0), + corr_weight: float = 0.75, + mmc_weight: float = 2.25, + ) -> float: + pred_arr = np.asarray(predictions, dtype=np.float64) + meta_arr = np.asarray(meta_model, dtype=np.float64) + + def neg_score(p: float) -> float: + out = neutralize_against_meta(pred_arr, meta_arr, eras=eras, proportion=p) + if objective == "payout_score": + score = payout_score( + y_true, out, meta_arr, eras, corr_weight, mmc_weight + ) + elif objective == "mmc_sharpe": + score = era_sharpe_of_mmc(y_true, out, meta_arr, eras) + else: + score = era_sharpe(y_true, out, eras) + return -score if np.isfinite(score) else 1e6 + + result = minimize_scalar( + neg_score, + bounds=bounds, + method="bounded", + options={"maxiter": 200, "xatol": 1e-4}, + ) + self.proportion = float(result.x) + return self.proportion + + class FeatureNeutralizer: def __init__(self, proportion: float = 0.5) -> None: if not 0.0 <= proportion <= 1.0: @@ -74,23 +170,38 @@ def optimize_proportion( y_true: pd.Series, eras: pd.Series, bounds: tuple[float, float] = (0.0, 1.0), + *, + objective: Literal["corr_sharpe", "mmc_sharpe", "payout_score"] = "corr_sharpe", + meta_model: np.ndarray | pd.Series | None = None, + corr_weight: float = 0.75, + mmc_weight: float = 2.25, ) -> float: pred_arr = np.asarray(predictions, dtype=np.float64) feat_arr = _numeric_features(features) era_vals = np.asarray(eras) + meta_arr = ( + np.asarray(meta_model, dtype=np.float64) if meta_model is not None else None + ) - def neg_sharpe(p: float) -> float: + def neg_score(p: float) -> float: out = pred_arr.copy() for era in np.unique(era_vals): mask = era_vals == era if mask.sum() < 2: continue out[mask] = self._neutralize_array(pred_arr[mask], feat_arr[mask], p) - s = era_sharpe(y_true, out, eras) - return -s if np.isfinite(s) else 1e6 + if objective == "payout_score" and meta_arr is not None: + score = payout_score( + y_true, out, meta_arr, eras, corr_weight, mmc_weight + ) + elif objective == "mmc_sharpe" and meta_arr is not None: + score = era_sharpe_of_mmc(y_true, out, meta_arr, eras) + else: + score = era_sharpe(y_true, out, eras) + return -score if np.isfinite(score) else 1e6 result = minimize_scalar( - neg_sharpe, + neg_score, bounds=bounds, method="bounded", options={"maxiter": 200, "xatol": 1e-4}, diff --git a/src/alphapulse/pipeline/pipeline.py b/src/alphapulse/pipeline/pipeline.py index 9b576f0..69d0df2 100644 --- a/src/alphapulse/pipeline/pipeline.py +++ b/src/alphapulse/pipeline/pipeline.py @@ -4,13 +4,15 @@ import numpy as np import pandas as pd +from loguru import logger from ..evaluation.metrics import rank_normalize +from ..experiments.data import meta_model_from_benchmarks from ..models.base import BaseModel from ..preprocessors.base import BasePreprocessor, TrainEvalPreprocessor from ..preprocessors.era_stable import EraStableFeatureSelector from .ensemble import EnsembleStrategy -from .neutralizer import FeatureNeutralizer +from .neutralizer import FeatureNeutralizer, MetaModelNeutralizer from .row_utils import ( filter_invalid_rows, filter_nan_rows, @@ -55,6 +57,7 @@ def __init__( benchmark_blend_weight: float = 0.0, neutralize_proportion: float = 0.0, neutralize_features: list[str] | None = None, + meta_neutralize_proportion: float = 0.0, ) -> None: if model is not None and models is not None: raise ValueError("Provide either model or models, not both.") @@ -82,17 +85,32 @@ def __init__( if self.neutralize_proportion > 0.0 else None ) + self.meta_neutralize_proportion = float(meta_neutralize_proportion) + self._meta_neutralizer: MetaModelNeutralizer | None = ( + MetaModelNeutralizer(proportion=self.meta_neutralize_proportion) + if self.meta_neutralize_proportion > 0.0 + else None + ) @property def ensemble_method(self) -> str: return self._ensemble.method + @property + def ensemble_weights(self) -> list[float] | None: + weights = self._ensemble.weights + if weights is None: + return None + return [float(w) for w in weights] + def fit( self, X: pd.DataFrame, y: pd.Series, X_val: pd.DataFrame | None = None, y_val: pd.Series | None = None, + era_val: pd.Series | None = None, + ensemble_meta_preds: np.ndarray | None = None, **model_train_kwargs: Any, ) -> dict[str, float]: """Fit preprocessors and train all models. @@ -117,7 +135,11 @@ def fit( era_meta = protected_metadata_frame(X) X_fit = X - for pp in self.preprocessors: + if self.preprocessors: + logger.info("Fitting {} preprocessor(s) ...", len(self.preprocessors)) + for i, pp in enumerate(self.preprocessors, start=1): + pp_name = type(pp).__name__ + logger.info("Preprocessor {}/{}: {}", i, len(self.preprocessors), pp_name) if isinstance(pp, TrainEvalPreprocessor): pp.train() if isinstance(pp, EraStableFeatureSelector) and era_meta is not None: @@ -132,7 +154,10 @@ def fit( raise ValueError("No valid training rows after preprocessing") X_val_t: pd.DataFrame | None = None + era_val_t: pd.Series | None = None if X_val is not None: + if era_val is None and "era" in X_val.columns: + era_val = X_val["era"] X_val, y_val = filter_invalid_rows(X_val, y_val) if len(X_val) == 0: X_val_t = None @@ -150,17 +175,39 @@ def fit( if len(X_val_t) == 0: X_val_t = None y_val = None + era_val_t = None + else: + era_val_t = ( + era_val.loc[X_val_t.index] if era_val is not None else None + ) if len(self.models) == 1: - metrics = self.models[0].train( + model = self.models[0] + logger.info( + "Training model {}: rows={} features={}", + model.name, + len(X_fit), + X_fit.shape[1], + ) + metrics = model.train( X_fit, y, X_val=X_val_t, y_val=y_val, **model_train_kwargs ) + logger.info("Model {} finished", model.name) else: metrics = {} - for m in self.models: + for idx, m in enumerate(self.models, start=1): + logger.info( + "Training model {}/{} {}: rows={} features={}", + idx, + len(self.models), + m.name, + len(X_fit), + X_fit.shape[1], + ) m_metrics = m.train( X_fit, y, X_val=X_val_t, y_val=y_val, **model_train_kwargs ) + logger.info("Model {} finished", m.name) for k, v in m_metrics.items(): metrics[f"{m.name}_{k}"] = v @@ -171,6 +218,8 @@ def get_val_preds() -> np.ndarray: n_models=len(self.models), get_val_predictions=get_val_preds if X_val_t is not None else None, y_val=y_val, + eras_val=era_val_t, + meta_model_preds=ensemble_meta_preds, ) return metrics @@ -190,10 +239,65 @@ def _infer(self, X_t: pd.DataFrame) -> np.ndarray: preds = np.column_stack([m.predict(X_t) for m in self.models]) return self._ensemble.combine(preds) + def set_ensemble_weights(self, weights: np.ndarray | list[float]) -> None: + self._ensemble.set_weights(weights) + + def predict_model_matrix(self, X: pd.DataFrame) -> np.ndarray: + """Return per-model predictions after preprocessing (n_rows, n_models).""" + for pp in self.preprocessors: + if isinstance(pp, TrainEvalPreprocessor): + pp.eval() + + input_invalid_mask = invalid_row_mask(X) + if input_invalid_mask.all(): + return np.zeros((len(X), len(self.models)), dtype=np.float64) + + if input_invalid_mask.any(): + valid_mask = ~input_invalid_mask + X_valid = X[valid_mask] + else: + valid_mask = None + X_valid = X + + era_meta = protected_metadata_frame(X_valid) + X_t = self._preprocess(X_valid, era_meta) + + post_nan_mask = ~X_t.isna().any(axis=1) if X_t.isna().any().any() else None + if post_nan_mask is not None: + X_t = X_t[post_nan_mask] + + if len(X_t) == 0: + return np.zeros((len(X), len(self.models)), dtype=np.float64) + + valid_matrix = np.column_stack([m.predict(X_t) for m in self.models]) + + if valid_mask is None and post_nan_mask is None: + return valid_matrix + + full = np.zeros((len(X), len(self.models)), dtype=np.float64) + if valid_mask is not None and post_nan_mask is not None: + combined = valid_mask.to_numpy(dtype=bool).copy() + valid_positions = np.flatnonzero(combined) + post_arr = np.asarray(post_nan_mask, dtype=bool) + combined[valid_positions[~post_arr]] = False + full[combined] = valid_matrix + elif valid_mask is not None: + full[valid_mask.to_numpy(dtype=bool)] = valid_matrix + else: + assert post_nan_mask is not None + post_arr = ( + post_nan_mask.to_numpy(dtype=bool) + if hasattr(post_nan_mask, "to_numpy") + else np.asarray(post_nan_mask, dtype=bool) + ) + full[post_arr] = valid_matrix + return full + def predict( self, X: pd.DataFrame, eras: pd.Series | None = None, + meta_model: np.ndarray | pd.Series | None = None, ) -> np.ndarray: """Generate predictions for new data. @@ -202,6 +306,8 @@ def predict( eras: Optional era labels for per-era feature neutralization. When *None* and neutralization is enabled, neutralization is applied across the entire batch. + meta_model: Optional Numerai meta model predictions aligned with *X* + for meta-model neutralization when enabled. Returns: 1-D array of predictions. Rank-normalized when neutralization @@ -259,19 +365,44 @@ def predict( raw_preds[post_arr] = valid_preds if self._neutralizer is None: - return raw_preds + output = raw_preds + else: + neutral_cols = ( + self.neutralize_features or self.feature_columns or list(X_orig.columns) + ) + available_cols = [c for c in neutral_cols if c in X_orig.columns] + if not available_cols: + output = rank_normalize(raw_preds) + return self._apply_meta_neutralization( + output, meta_model, eras, X_orig.index + ) - neutral_cols = ( - self.neutralize_features or self.feature_columns or list(X_orig.columns) - ) - available_cols = [c for c in neutral_cols if c in X_orig.columns] - if not available_cols: - return rank_normalize(raw_preds) - - neutralized = self._neutralizer.neutralize( - raw_preds, - X_orig[available_cols], - eras=eras, + neutralized = self._neutralizer.neutralize( + raw_preds, + X_orig[available_cols], + eras=eras, + ) + output = rank_normalize(neutralized) + + return self._apply_meta_neutralization(output, meta_model, eras, X_orig.index) + + def _apply_meta_neutralization( + self, + preds: np.ndarray, + meta_model: np.ndarray | pd.Series | None, + eras: pd.Series | None, + index: pd.Index, + ) -> np.ndarray: + if self._meta_neutralizer is None or meta_model is None: + return preds + meta_arr = np.asarray(meta_model, dtype=np.float64).reshape(-1) + if len(meta_arr) != len(preds): + raise ValueError( + f"meta_model length {len(meta_arr)} != predictions length {len(preds)}" + ) + era_series = eras if eras is not None else pd.Series(index=index) + neutralized = self._meta_neutralizer.neutralize( + preds, meta_arr, eras=era_series ) return rank_normalize(neutralized) @@ -316,6 +447,7 @@ def to_numerai_predict( blend_weight = self.benchmark_blend_weight bench_col = benchmark_col use_neutralization = pipeline._neutralizer is not None + use_meta_neutralization = pipeline._meta_neutralizer is not None def predict( live_features: pd.DataFrame, @@ -327,7 +459,12 @@ def predict( if use_neutralization and "era" in live_features.columns else None ) - raw_preds = pipeline.predict(X, eras=eras) + meta_model = ( + meta_model_from_benchmarks(live_benchmark_models, live_features.index) + if use_meta_neutralization + else None + ) + raw_preds = pipeline.predict(X, eras=eras, meta_model=meta_model) ranked = rank_normalize(raw_preds) if ( diff --git a/src/alphapulse/preprocessors/packboost.py b/src/alphapulse/preprocessors/packboost.py index 46db430..a92ae80 100644 --- a/src/alphapulse/preprocessors/packboost.py +++ b/src/alphapulse/preprocessors/packboost.py @@ -19,6 +19,13 @@ def __init__( early_stopping_rounds_base: int = 50, n_rounds_boost: int = 200, early_stopping_rounds_boost: int = 30, + device: str = "cuda", + max_depth: int = 7, + nfolds: int = 8, + lr: float = 0.07, + l2: float = 100_000.0, + nfeatsets: int = 32, + seed: int = 42, name: str = "PackboostPreprocessor", ) -> None: super().__init__(name) @@ -34,6 +41,13 @@ def __init__( early_stopping_rounds_base=early_stopping_rounds_base, n_rounds_boost=n_rounds_boost, early_stopping_rounds_boost=early_stopping_rounds_boost, + device=device, + max_depth=max_depth, + nfolds=nfolds, + lr=lr, + l2=l2, + nfeatsets=nfeatsets, + seed=seed, ) def fit(self, X: pd.DataFrame, y: pd.Series | None = None) -> Self: diff --git a/src/alphapulse/utils/gpu_cleanup.py b/src/alphapulse/utils/gpu_cleanup.py new file mode 100644 index 0000000..e7b42c1 --- /dev/null +++ b/src/alphapulse/utils/gpu_cleanup.py @@ -0,0 +1,179 @@ +"""GPU memory cleanup helpers for HPO trial subprocesses.""" + +from __future__ import annotations + +import gc +import os +import shutil +import signal +import subprocess +import time +from pathlib import Path + +_TRIAL_WORKER_MARKERS = ( + "scripts/hpo_pipeline.py", + "multiprocessing.spawn", + "alphapulse", +) + + +def release_cuda_memory() -> None: + """Release CUDA caches in the current process.""" + gc.collect() + try: + import torch + + if torch.cuda.is_available(): + torch.cuda.synchronize() + torch.cuda.empty_cache() + if hasattr(torch.cuda, "ipc_collect"): + torch.cuda.ipc_collect() + except Exception: # noqa: BLE001, S110 + pass + gc.collect() + + +def _read_cmdline(pid: int) -> str: + try: + return Path(f"/proc/{pid}/cmdline").read_bytes().replace(b"\x00", b" ").decode() + except OSError: + return "" + + +def _read_ppid(pid: int) -> int | None: + try: + for line in Path(f"/proc/{pid}/status").read_text().splitlines(): + if line.startswith("PPid:"): + return int(line.split()[1]) + except OSError: + return None + return None + + +def _collect_descendants(root_pid: int) -> list[int]: + by_ppid: dict[int, list[int]] = {} + for entry in Path("/proc").iterdir(): + if not entry.name.isdigit(): + continue + pid = int(entry.name) + ppid = _read_ppid(pid) + if ppid is not None: + by_ppid.setdefault(ppid, []).append(pid) + + ordered: list[int] = [] + stack = [root_pid] + while stack: + current = stack.pop() + ordered.append(current) + stack.extend(by_ppid.get(current, [])) + return ordered + + +def _signal_kill(pid: int) -> bool: + if pid <= 1: + return False + try: + os.kill(pid, signal.SIGKILL) + return True + except ProcessLookupError: + return False + except PermissionError: + return False + + +def nvidia_compute_pids() -> list[int]: + """Return PIDs with active CUDA contexts according to nvidia-smi.""" + nvidia_smi = shutil.which("nvidia-smi") + if nvidia_smi is None: + return [] + try: + proc = subprocess.run( # noqa: S603 + [ + nvidia_smi, + "--query-compute-apps=pid", + "--format=csv,noheader", + ], + capture_output=True, + text=True, + timeout=10, + check=False, + ) + except (FileNotFoundError, subprocess.TimeoutExpired): + return [] + + pids: list[int] = [] + for line in proc.stdout.strip().splitlines(): + token = line.strip().split(",")[0].strip() + if not token or not token.isdigit(): + continue + pids.append(int(token)) + return pids + + +def _is_hpo_trial_worker(pid: int, parent_pid: int) -> bool: + if pid == parent_pid: + return False + cmd = _read_cmdline(pid) + if not cmd: + return False + if any(marker in cmd for marker in _TRIAL_WORKER_MARKERS): + return True + ppid = _read_ppid(pid) + return ppid == parent_pid + + +def kill_process_tree(root_pid: int, *, except_pid: int | None = None) -> list[int]: + """SIGKILL a process and all descendants.""" + killed: list[int] = [] + for pid in reversed(_collect_descendants(root_pid)): + if except_pid is not None and pid == except_pid: + continue + if _signal_kill(pid): + killed.append(pid) + return killed + + +def cleanup_stale_gpu_processes( + *, + parent_pid: int | None = None, + worker_pid: int | None = None, + kill_worker_tree: bool = False, +) -> dict[str, object]: + """Kill orphaned HPO trial workers still holding GPU memory.""" + parent = parent_pid or os.getpid() + killed: list[int] = [] + + if worker_pid is not None and kill_worker_tree: + killed.extend(kill_process_tree(worker_pid, except_pid=parent)) + + for pid in nvidia_compute_pids(): + if pid == parent: + continue + if worker_pid is not None and pid in _collect_descendants(worker_pid): + if _signal_kill(pid): + killed.append(pid) + continue + if _is_hpo_trial_worker(pid, parent): + if _signal_kill(pid): + killed.append(pid) + + release_cuda_memory() + if killed: + time.sleep(0.5) + + remaining = [p for p in nvidia_compute_pids() if p != parent] + return {"killed_pids": sorted(set(killed)), "remaining_gpu_pids": remaining} + + +def cleanup_after_trial_subprocess( + worker_pid: int | None, + *, + parent_pid: int | None = None, + kill_worker_tree: bool = False, +) -> dict[str, object]: + """Run post-trial GPU cleanup in the HPO parent process.""" + return cleanup_stale_gpu_processes( + parent_pid=parent_pid, + worker_pid=worker_pid, + kill_worker_tree=kill_worker_tree, + ) diff --git a/tests/conftest.py b/tests/conftest.py new file mode 100644 index 0000000..e53ae26 --- /dev/null +++ b/tests/conftest.py @@ -0,0 +1,16 @@ +from collections.abc import Iterator + +import pytest + + +@pytest.fixture(autouse=True) +def _reset_matplotlib_state() -> Iterator[None]: + yield + try: + import matplotlib as mpl + import matplotlib.pyplot as plt + + plt.close("all") + mpl.use("Agg", force=True) + except ImportError: + pass diff --git a/tests/test_coverage_gaps.py b/tests/test_coverage_gaps.py index 9a84821..176ce37 100644 --- a/tests/test_coverage_gaps.py +++ b/tests/test_coverage_gaps.py @@ -1,5 +1,6 @@ """Tests covering previously uncovered code paths.""" +import warnings from pathlib import Path from typing import Any @@ -19,6 +20,7 @@ from alphapulse.hpo.builder import build_pipeline_or_multi, build_preprocessors from alphapulse.hpo.search_space import resolve_flat_config from alphapulse.models import XGBoostModel +from alphapulse.models.era_ensemble_model import EraEnsembleModel from alphapulse.pipeline.ensemble import EnsembleStrategy from alphapulse.pipeline.multihead import HeadSpec, MultiHeadPipeline from alphapulse.preprocessors import StandardScalerPreprocessor @@ -181,6 +183,46 @@ def test_save_load_roundtrip( np.testing.assert_array_almost_equal(preds_before, preds_after) + def test_head_matrix_preserves_era_for_era_ensemble( + self, toy_data: tuple[pd.DataFrame, pd.Series] + ) -> None: + X, y = toy_data + era = pd.Series(np.repeat([f"e{i:03d}" for i in range(20)], 10), index=X.index) + X = X.assign(era=era) + feature_groups = {"group_ab": ["a", "b"], "group_cd": ["c", "d"]} + + def factory(name: str) -> EraEnsembleModel: + return EraEnsembleModel( + base_model_factory=lambda: _xgb(name), + n_subs=4, + name=name, + ) + + heads = [ + HeadSpec( + model=factory("h1"), + input_columns=None, + input_group="group_ab", + local_preprocessors=[StandardScalerPreprocessor()], + feature_groups=feature_groups, + ), + HeadSpec( + model=factory("h2"), + input_columns=None, + input_group="group_cd", + local_preprocessors=[], + feature_groups=feature_groups, + ), + ] + pipe = MultiHeadPipeline(global_preprocessors=[], heads=heads) + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + pipe.fit(X, y, n_rounds=5) + assert not [w for w in caught if "falling back" in str(w.message)] + for head in pipe.heads: + assert isinstance(head.model, EraEnsembleModel) + assert len(head.model._sub_models) > 1 + class TestGroupedPreprocessor: def test_fit_transform_shape( @@ -341,7 +383,7 @@ def test_resolve_flat_two_models_weighted(self) -> None: cfg = resolve_flat_config(flat) assert len(cfg["models"]) == 2 assert cfg["ensemble_method"] == "weighted" - assert "weights" in cfg["ensemble_params"] + assert cfg["ensemble_params"]["optimize_weights"] is True def test_resolve_flat_one_model_forces_single_ensemble(self) -> None: flat = self._base_flat_config() @@ -380,7 +422,7 @@ def _make_experiment(self, tmp_path: Path) -> ExperimentV1: models=[ModelSpec(type="XGBoost", params={"max_depth": 3})], ensemble_method="single", train=TrainConfig(n_rounds=10, early_stopping_rounds=5), - evaluation=EvaluationConfig(primary_metric="sharpe"), + evaluation=EvaluationConfig(primary_metric="corr_sharpe"), ) def test_to_pipeline_config_keys(self, tmp_path: Path) -> None: diff --git a/tests/test_ensemble_postfit.py b/tests/test_ensemble_postfit.py new file mode 100644 index 0000000..50bbc55 --- /dev/null +++ b/tests/test_ensemble_postfit.py @@ -0,0 +1,149 @@ +import numpy as np +import pandas as pd +import pytest + +from alphapulse.models import XGBoostModel +from alphapulse.pipeline import Pipeline +from alphapulse.pipeline.ensemble import EnsembleStrategy +from alphapulse.pipeline.ensemble_optimizer import ( + DEFAULT_MAX_WEIGHT, + DEFAULT_MIN_WEIGHT, + EnsembleOptimizer, + project_weights_to_bounds, + project_weights_to_bounds_list, + validate_weight_bounds, + validate_weight_bounds_list, +) + + +def test_validate_weight_bounds_infeasible() -> None: + with pytest.raises(ValueError, match="infeasible"): + validate_weight_bounds(2, min_weight=0.6, max_weight=0.9) + + +def test_project_weights_respects_bounds() -> None: + w = project_weights_to_bounds( + np.array([0.99, 0.01]), min_weight=0.05, max_weight=0.90 + ) + assert w.sum() == pytest.approx(1.0) + assert all(DEFAULT_MIN_WEIGHT <= x <= DEFAULT_MAX_WEIGHT for x in w) + + +def test_ensemble_optimizer_bounded_weights_stay_in_box() -> None: + rng = np.random.RandomState(0) + n = 200 + eras = pd.Series(np.repeat(["e1", "e2", "e3", "e4"], n // 4)) + y = rng.randn(n) + oof = np.column_stack([rng.randn(n), y + rng.randn(n) * 0.05]) + + bounded = EnsembleOptimizer( + seed=0, min_weight=0.05, max_weight=0.90, objective="corr_sharpe" + ) + bounded.fit(oof, y, eras) + assert bounded.weights_ is not None + assert bounded.weights_.sum() == pytest.approx(1.0) + assert all(0.05 <= float(w) <= 0.90 for w in bounded.weights_) + + +def test_pipeline_optimize_weights_post_fit() -> None: + rng = np.random.RandomState(1) + n = 160 + eras = np.repeat([f"e{i}" for i in range(8)], n // 8) + X = pd.DataFrame(rng.randn(n, 3), columns=list("ABC")) + X["era"] = eras + y = pd.Series(X["A"] * 0.4 + rng.randn(n) * 0.2) + + n_train = 120 + X_train = X.iloc[:n_train] + y_train = y.iloc[:n_train] + X_val = X.iloc[n_train:] + y_val = y.iloc[n_train:] + era_val = X_val["era"] + + model_a = XGBoostModel( + params={"max_depth": 2, "learning_rate": 0.1, "tree_method": "hist"}, + name="m_a", + ) + model_b = XGBoostModel( + params={"max_depth": 2, "learning_rate": 0.1, "tree_method": "hist"}, + name="m_b", + ) + pipeline = Pipeline( + preprocessors=[], + models=[model_a, model_b], + ensemble_method="weighted", + ensemble_params={ + "optimize_weights": True, + "objective": "corr_sharpe", + "min_weight": 0.05, + "max_weight": 0.90, + }, + ) + pipeline.fit( + X_train.drop(columns=["era"]), + y_train, + X_val=X_val.drop(columns=["era"]), + y_val=y_val, + era_val=era_val, + n_rounds=30, + ) + weights = pipeline.ensemble_weights + assert weights is not None + assert len(weights) == 2 + assert sum(weights) == pytest.approx(1.0) + assert all(0.05 <= w <= 0.90 for w in weights) + + +def test_ensemble_optimizer_per_model_bounds() -> None: + rng = np.random.RandomState(3) + n = 200 + eras = pd.Series(np.repeat(["e1", "e2", "e3", "e4"], n // 4)) + y = rng.randn(n) + oof = np.column_stack([rng.randn(n), y + rng.randn(n) * 0.05]) + + optimizer = EnsembleOptimizer(seed=0, objective="corr_sharpe") + optimizer.fit( + oof, + y, + eras, + min_weights=[0.05, 0.05], + max_weights=[0.35, 0.90], + ) + assert optimizer.weights_ is not None + assert optimizer.weights_[0] <= 0.35 + 1e-6 + assert optimizer.weights_[1] <= 0.90 + 1e-6 + + +def test_project_weights_to_bounds_list() -> None: + w = project_weights_to_bounds_list( + np.array([0.9, 0.1]), + [0.05, 0.05], + [0.35, 0.90], + ) + validate_weight_bounds_list([0.05, 0.05], [0.35, 0.90]) + assert w.sum() == pytest.approx(1.0) + assert w[0] <= 0.35 + 1e-6 + + +def test_ensemble_strategy_fixed_weights_skip_optimizer() -> None: + strategy = EnsembleStrategy( + method="weighted", + params={"weights": [0.9, 0.1]}, + ) + strategy.fit(n_models=2) + assert strategy.weights is not None + assert strategy.weights[0] == pytest.approx(0.9) + + +def test_ensemble_strategy_rejects_optimize_and_fixed_weights() -> None: + strategy = EnsembleStrategy( + method="weighted", + params={"optimize_weights": True, "weights": [0.9, 0.1]}, + ) + with pytest.raises(ValueError, match="both optimize_weights and fixed weights"): + strategy.fit( + n_models=2, + get_val_predictions=lambda: np.zeros((10, 2)), + y_val=pd.Series(np.zeros(10)), + eras_val=pd.Series(["e"] * 10), + ) diff --git a/tests/test_feature_catalog.py b/tests/test_feature_catalog.py new file mode 100644 index 0000000..3ce5be5 --- /dev/null +++ b/tests/test_feature_catalog.py @@ -0,0 +1,54 @@ +import json +from pathlib import Path + +import pytest + +from alphapulse.features.catalog import ( + LEGACY_EXCLUDED, + load_feature_catalog, + load_target_catalog, +) + + +@pytest.fixture +def features_json(tmp_path: Path) -> Path: + data_dir = tmp_path / "data" + data_dir.mkdir() + payload = { + "feature_sets": { + "small": ["f_a", "f_b"], + "medium": ["f_a", "f_b", "f_c"], + "all": ["f_a", "f_b", "f_c", "f_d"], + "strength": ["f_a", "f_c"], + "v2_equivalent_features": ["legacy_1"], + }, + "targets": ["target", "target_alpha_20", "target_cyrusd_60"], + } + (data_dir / "features.json").write_text(json.dumps(payload), encoding="utf-8") + return data_dir + + +def test_feature_catalog_excludes_legacy(features_json: Path) -> None: + catalog = load_feature_catalog(features_json) + assert "v2_equivalent_features" not in catalog.feature_sets + assert "v2_equivalent_features" not in catalog.searchable_names + assert LEGACY_EXCLUDED.isdisjoint(catalog.searchable_names) + assert set(catalog.searchable_names) == { + "small", + "medium", + "all", + "strength", + } + + +def test_feature_catalog_union_dedup(features_json: Path) -> None: + catalog = load_feature_catalog(features_json) + union = catalog.union(["small", "strength"]) + assert union == ["f_a", "f_b", "f_c"] + + +def test_target_catalog_load(features_json: Path) -> None: + catalog = load_target_catalog(features_json) + assert "target" in catalog.targets + assert catalog.parse_horizon("target_alpha_20") == 20 + assert catalog.parse_horizon("target") == 20 diff --git a/tests/test_feature_routing.py b/tests/test_feature_routing.py new file mode 100644 index 0000000..0d6a4bd --- /dev/null +++ b/tests/test_feature_routing.py @@ -0,0 +1,136 @@ +import json +import random +from pathlib import Path + +import pytest + +from alphapulse.features.catalog import load_feature_catalog +from alphapulse.hpo.builder import build_pipeline_or_multi +from alphapulse.hpo.feature_routing import ( + MAX_ROUTED_FEATURES, + merge_routing_into_pipeline_config, + resolve_feature_routing, + sample_feature_routing, +) +from alphapulse.hpo.search_space import resolve_flat_config +from alphapulse.pipeline.multihead import MultiHeadPipeline + + +@pytest.fixture +def catalog_dir(tmp_path: Path) -> Path: + data_dir = tmp_path / "data" + data_dir.mkdir() + payload = { + "feature_sets": { + "small": ["f_a", "f_b"], + "medium": ["f_a", "f_b", "f_c"], + "strength": ["f_a", "f_c"], + "constitution": ["f_b", "f_c"], + }, + "targets": ["target"], + } + (data_dir / "features.json").write_text(json.dumps(payload), encoding="utf-8") + return data_dir + + +def _base_flat(num_models: int = 1) -> dict: + return { + "num_models": num_models, + "model_1_type": "XGBoost", + "model_2_type": "LightGBM", + "model_3_type": "XGBoost", + "scaler_type": "StandardScaler", + "use_packboost": False, + "ensemble_method": "single" if num_models == 1 else "weighted", + "xgb_max_depth": 3, + "xgb_learning_rate": 0.05, + "xgb_n_rounds": 10, + "xgb_early_stopping": 5, + "lgbm_num_leaves": 16, + "lgbm_learning_rate": 0.05, + "lgbm_n_rounds": 10, + "lgbm_early_stopping": 5, + "use_neutralization": False, + } + + +def test_resolve_simple_path(catalog_dir: Path) -> None: + catalog = load_feature_catalog(catalog_dir) + flat = { + **_base_flat(1), + "use_feature_routing": True, + "active_groups": ["small", "strength"], + "model_1_groups": ["small", "strength"], + "model_1_lane": 0, + "lane_0_steps": [], + } + routing = resolve_feature_routing(flat, catalog) + assert routing.build_path == "simple" + assert routing.feature_columns == ["f_a", "f_b", "f_c"] + + +def test_resolve_grouped_path_single_model_with_lane_steps(catalog_dir: Path) -> None: + catalog = load_feature_catalog(catalog_dir) + flat = { + **_base_flat(1), + "use_feature_routing": True, + "active_groups": ["small"], + "model_1_groups": ["small"], + "model_1_lane": 0, + "lane_0_steps": ["VarianceFeatureSelector"], + } + routing = resolve_feature_routing(flat, catalog) + assert routing.build_path == "grouped" + assert routing.pipeline_config_patch["preprocessors"][0]["type"] == "Grouped" + + +def test_resolve_multihead_path(catalog_dir: Path) -> None: + catalog = load_feature_catalog(catalog_dir) + flat = { + **_base_flat(2), + "use_feature_routing": True, + "active_groups": ["small", "strength"], + "model_1_groups": ["small"], + "model_2_groups": ["strength"], + "model_1_lane": 0, + "model_2_lane": 0, + "lane_0_steps": [], + } + routing = resolve_feature_routing(flat, catalog) + assert routing.build_path == "multihead" + cfg = merge_routing_into_pipeline_config(resolve_flat_config(flat), routing) + pipeline = build_pipeline_or_multi( + cfg, + feature_columns=routing.feature_columns, + feature_groups=routing.feature_groups, + ) + assert isinstance(pipeline, MultiHeadPipeline) + + +def test_sample_feature_routing_fragment(catalog_dir: Path) -> None: + catalog = load_feature_catalog(catalog_dir) + fragment = sample_feature_routing(random.Random(0), catalog, 2, fast=True) + assert fragment["use_feature_routing"] is True + assert fragment["active_groups"] + assert fragment["active_groups_count"] == len(fragment["active_groups"]) + assert fragment["routed_feature_count"] <= MAX_ROUTED_FEATURES + + +def test_sample_feature_routing_respects_feature_limit(tmp_path: Path) -> None: + data_dir = tmp_path / "data" + data_dir.mkdir() + payload = { + "feature_sets": { + "small": [f"f_{i}" for i in range(100)], + "medium": [f"f_{i}" for i in range(800)], + "all": [f"f_{i}" for i in range(1200)], + "strength": [f"f_{i}" for i in range(400, 900)], + "rain": [f"f_{i}" for i in range(850, 1150)], + }, + "targets": ["target"], + } + (data_dir / "features.json").write_text(json.dumps(payload), encoding="utf-8") + catalog = load_feature_catalog(data_dir) + fragment = sample_feature_routing(random.Random(7), catalog, 2, fast=True) + routing = resolve_feature_routing({**_base_flat(2), **fragment}, catalog) + assert len(routing.feature_columns) <= MAX_ROUTED_FEATURES diff --git a/tests/test_foundation_models.py b/tests/test_foundation_models.py index d22d960..7d06f78 100644 --- a/tests/test_foundation_models.py +++ b/tests/test_foundation_models.py @@ -8,7 +8,6 @@ from alphapulse.models.foundation_models import ( TabICLModel, TabPFN3Model, - TabPFN3ReasoningModel, TabPFNModel, ) @@ -270,34 +269,6 @@ def test_tabpfn3_not_in_tree_model_names() -> None: assert "TabPFN3" not in TREE_MODEL_NAMES -# --------------------------------------------------------------------------- -# TabPFN3ReasoningModel -# --------------------------------------------------------------------------- - - -def test_tabpfn3_reasoning_init_defaults() -> None: - model = TabPFN3ReasoningModel() - assert model.name == "TabPFN3Reasoning" - assert model.thinking_mode is True - assert not model.is_trained - - -def test_tabpfn3_reasoning_train_requires_api_key( - toy_data: tuple[pd.DataFrame, pd.Series], - monkeypatch: pytest.MonkeyPatch, -) -> None: - pytest.importorskip("tabpfn_client", reason="tabpfn-client not installed — skip") - monkeypatch.delenv("TABPFN_API_KEY", raising=False) - X, y = toy_data - with pytest.raises(ValueError, match="TABPFN_API_KEY"): - TabPFN3ReasoningModel().train(X, y) - - -def test_tabpfn3_reasoning_registry_returns_model() -> None: - models = build_models([{"type": "TabPFN3Reasoning", "params": {}}]) - assert isinstance(models[0], TabPFN3ReasoningModel) - - # --------------------------------------------------------------------------- # TabICLModel # --------------------------------------------------------------------------- diff --git a/tests/test_gpu_cleanup.py b/tests/test_gpu_cleanup.py new file mode 100644 index 0000000..0f041ba --- /dev/null +++ b/tests/test_gpu_cleanup.py @@ -0,0 +1,29 @@ +from __future__ import annotations + +import os + +from alphapulse.utils.gpu_cleanup import ( + _collect_descendants, + _is_hpo_trial_worker, + cleanup_after_trial_subprocess, + release_cuda_memory, +) + + +def test_release_cuda_memory_does_not_raise() -> None: + release_cuda_memory() + + +def test_collect_descendants_includes_parent() -> None: + descendants = _collect_descendants(os.getpid()) + assert os.getpid() in descendants + + +def test_is_hpo_trial_worker_false_for_parent() -> None: + assert not _is_hpo_trial_worker(os.getpid(), os.getpid()) + + +def test_cleanup_after_trial_subprocess_returns_shape() -> None: + result = cleanup_after_trial_subprocess(None) + assert "killed_pids" in result + assert "remaining_gpu_pids" in result diff --git a/tests/test_hpo_pipeline.py b/tests/test_hpo_pipeline.py index 2d0d86e..6df25d1 100644 --- a/tests/test_hpo_pipeline.py +++ b/tests/test_hpo_pipeline.py @@ -1,10 +1,13 @@ """Tests for HPO pipeline (run_trial with preloaded data).""" +import json +from pathlib import Path from typing import Any import numpy as np import pandas as pd import pytest +from scripts.hpo_pipeline import main as hpo_main from alphapulse.hpo import run_trial, sample_random_config @@ -55,6 +58,28 @@ def minimal_flat_config() -> dict[str, Any]: } +def test_sample_random_config_includes_targets_when_data_dir(tmp_path: Path) -> None: + data_dir = tmp_path / "data" + data_dir.mkdir() + payload = { + "feature_sets": {"medium": ["f_a"], "strength": ["f_a"]}, + "targets": ["target", "target_alpha_20"], + } + (data_dir / "features.json").write_text(json.dumps(payload), encoding="utf-8") + config = sample_random_config(seed=1, fast=True, data_dir=data_dir) + assert "target_mode" in config + assert "primary_target" in config + assert "use_feature_routing" in config + assert config["use_feature_routing"] is True + assert "active_groups" in config + assert "routed_feature_count" in config + + +def test_hpo_main_default_objective_is_payout_score() -> None: + assert hpo_main.__defaults__ is not None + assert "payout_score" in hpo_main.__defaults__ + + def test_run_trial_returns_metrics( toy_data_with_era: dict[str, Any], minimal_flat_config: dict[str, Any] ) -> None: @@ -80,6 +105,7 @@ def test_sample_random_config_fast_tighter_bounds() -> None: assert config["n_subs"] <= 5 assert config["xgb_n_rounds"] <= 400 assert config["num_models"] <= 2 + assert config.get("foundation_max_train_rows", 10_000) <= 5_000 assert "SyntheticDataAugmenter" not in ( config.get("model_1_type"), config.get("model_2_type"), @@ -87,17 +113,14 @@ def test_sample_random_config_fast_tighter_bounds() -> None: ) -def test_resolve_flat_config_fast_foundation_uses_autoencoder() -> None: +def test_resolve_flat_config_fast_foundation_defaults_to_pca() -> None: from alphapulse.hpo.search_space import ( _torch_available, resolve_flat_config, resolve_foundation_compression, ) - assert resolve_foundation_compression(None, hpo_fast=True) in { - "autoencoder", - "pca", - } + assert resolve_foundation_compression(None, hpo_fast=True) == "pca" if not _torch_available(): assert resolve_foundation_compression("autoencoder") == "pca" @@ -115,8 +138,7 @@ def test_resolve_flat_config_fast_foundation_uses_autoencoder() -> None: } cfg = resolve_flat_config(flat) params = cfg["models"][0]["params"] - expected = "autoencoder" if _torch_available() else "pca" - assert params["compression"] == expected + assert params["compression"] == "pca" assert params["n_estimators"] == 2 assert params["compression_epochs"] == 5 assert cfg["neutralize_proportion"] == 0.0 @@ -163,10 +185,30 @@ def test_resolve_flat_config_mixed_ensemble_uses_neutralization() -> None: } cfg = resolve_flat_config(flat) assert cfg["neutralize_proportion"] >= MIN_NEUTRALIZATION_PROPORTION + assert cfg["ensemble_params"].get("optimize_weights") is True + assert cfg["ensemble_params"].get("min_weight") == 0.05 + assert cfg["ensemble_params"].get("max_weight") == 0.90 + + +def test_resolve_flat_config_weighted_uses_saved_ensemble_weights() -> None: + from alphapulse.hpo.search_space import resolve_flat_config + + flat = { + "num_models": 2, + "model_1_type": "XGBoost", + "model_2_type": "LightGBM", + "scaler_type": "StandardScaler", + "use_packboost": False, + "ensemble_method": "weighted", + "ensemble_weights": [0.85, 0.15], + } + cfg = resolve_flat_config(flat) + assert cfg["ensemble_params"]["weights"] == [0.85, 0.15] + assert "optimize_weights" not in cfg["ensemble_params"] def test_sample_random_config_boosting_enables_neutralization() -> None: - foundation_types = {"TabPFN", "TabICL", "TabPFN3", "TabPFN3Reasoning"} + foundation_types = {"TabPFN", "TabICL", "TabPFN3"} for seed in range(20): config = sample_random_config(seed=seed, fast=True) types = [ @@ -266,3 +308,196 @@ def test_xgb_defaults_when_type_xgb() -> None: flat = {"num_models": 1, "model_1_type": "XGBoost", "xgb_n_rounds": 400} kw = get_train_kwargs_from_flat(flat) assert kw["n_rounds"] == 400 + + +def test_multi_blend_packboost_multihead_forwards_era( + tmp_path: Path, +) -> None: + from alphapulse.features.catalog import load_feature_catalog + from alphapulse.hpo.feature_routing import ( + merge_routing_into_pipeline_config, + resolve_feature_routing, + ) + from alphapulse.hpo.objective import _fit_pipeline + from alphapulse.hpo.search_space import ( + get_train_kwargs_from_flat, + resolve_flat_config, + ) + from alphapulse.models.foundation_models import TabPFNModel + from alphapulse.pipeline.multi_target import MultiTargetPipeline + + data_dir = tmp_path / "data" + data_dir.mkdir() + feature_cols = [f"f_{i}" for i in range(12)] + payload = { + "feature_sets": { + "medium": feature_cols[:8], + "agility": feature_cols[4:10], + "dexterity": feature_cols[6:12], + "serenity": feature_cols[2:8], + }, + "targets": ["target", "target_alpha_20"], + } + (data_dir / "features.json").write_text(json.dumps(payload), encoding="utf-8") + catalog = load_feature_catalog(data_dir) + + flat = { + "num_models": 2, + "model_1_type": "TabPFN", + "model_2_type": "Packboost", + "target_mode": "multi_blend", + "primary_target": "target", + "auxiliary_targets": ["target_alpha_20"], + "target_blend_method": "equal", + "foundation_max_train_rows": 200, + "foundation_compression": "pca", + "foundation_n_components": 4, + "scaler_type": "RobustScaler", + "use_packboost": False, + "ensemble_method": "single", + "use_feature_routing": True, + "active_groups": ["medium", "agility", "dexterity", "serenity"], + "model_1_groups": ["medium", "serenity"], + "model_2_groups": ["agility", "dexterity"], + "model_1_lane": 0, + "model_2_lane": 0, + "lane_0_steps": [], + "hpo_fast": True, + "packboost_model_n_worst_eras": 2, + "packboost_model_boost_weight": 0.3, + "packboost_model_n_rounds_base": 20, + "packboost_model_n_rounds_boost": 10, + } + routing = resolve_feature_routing(flat, catalog) + cfg = merge_routing_into_pipeline_config(resolve_flat_config(flat), routing) + + def fast_tabpfn_train( + self: TabPFNModel, + X_train: pd.DataFrame, + y_train: pd.Series, + X_val: pd.DataFrame | None = None, + y_val: pd.Series | None = None, + **kwargs: Any, + ) -> dict[str, float]: + self.is_trained = True + self.model = type("M", (), {"predict": lambda _s, x: np.zeros(len(x))})() + return {} + + TabPFNModel.train = fast_tabpfn_train # type: ignore[method-assign, assignment] + + rng = np.random.default_rng(0) + n_eras = 30 + rows_per_era = 10 + n = n_eras * rows_per_era + cols = routing.feature_columns + X = pd.DataFrame(rng.standard_normal((n, len(cols))), columns=cols) + X["era"] = np.repeat([f"era_{i:04d}" for i in range(n_eras)], rows_per_era) + targets = pd.DataFrame( + { + "target": pd.Series(rng.standard_normal(n)), + "target_alpha_20": pd.Series(rng.standard_normal(n)), + } + ) + pipeline = _fit_pipeline( + cfg, + cols, + X, + targets["target"], + get_train_kwargs_from_flat(flat), + flat_config=flat, + seed=42, + feature_groups=routing.feature_groups, + targets_df=targets, + ) + assert isinstance(pipeline, MultiTargetPipeline) + preds = pipeline.predict(X.drop(columns=["era"])) + assert preds.shape == (n,) + assert np.all(np.isfinite(preds)) + + +def test_load_diagnostics_train_data_multi_blend(tmp_path: Path) -> None: + from scripts.hpo_pipeline import _load_diagnostics_train_data + + from alphapulse.hpo.objective import _fit_pipeline + from alphapulse.hpo.search_space import ( + get_train_kwargs_from_flat, + resolve_flat_config, + ) + + data_dir = tmp_path / "data" + data_dir.mkdir() + feature_cols = [f"f_{i}" for i in range(6)] + payload = { + "feature_sets": {"medium": feature_cols}, + "targets": ["target", "target_alpha_20"], + } + (data_dir / "features.json").write_text(json.dumps(payload), encoding="utf-8") + + rng = np.random.default_rng(0) + n = 80 + df = pd.DataFrame(rng.standard_normal((n, len(feature_cols))), columns=feature_cols) + df["era"] = np.repeat([f"era_{i:04d}" for i in range(8)], 10) + df["target"] = rng.standard_normal(n) + df["target_alpha_20"] = rng.standard_normal(n) + df.to_parquet(data_dir / "train.parquet") + + best_config = { + "num_models": 1, + "model_1_type": "XGBoost", + "target_mode": "multi_blend", + "primary_target": "target", + "auxiliary_targets": ["target_alpha_20"], + "target_blend_method": "equal", + "scaler_type": "StandardScaler", + "ensemble_method": "single", + "use_feature_routing": False, + "xgb_n_rounds": 5, + "xgb_early_stopping": 3, + "xgb_max_depth": 2, + "xgb_learning_rate": 0.1, + } + X_train, y_train, targets_df, feat_cols, feature_groups = ( + _load_diagnostics_train_data( + best_config, + data_dir, + train_subsample=1.0, + target_col="target", + seed=42, + ) + ) + assert targets_df is not None + assert "target_alpha_20" in targets_df.columns + + era_train = X_train["era"] + eras_sorted = sorted(era_train.unique(), key=str) + train_mask = ~era_train.isin(set(eras_sorted[-2:])) + targets_train = targets_df.loc[train_mask] + + pipeline = _fit_pipeline( + resolve_flat_config(best_config), + feat_cols, + X_train.loc[train_mask], + y_train.loc[train_mask], + get_train_kwargs_from_flat(best_config), + flat_config=best_config, + seed=42, + feature_groups=feature_groups or None, + targets_df=targets_train, + ) + preds = pipeline.predict(X_train.drop(columns=["era"]).iloc[:5]) + assert len(preds) == 5 + assert np.all(np.isfinite(preds)) + + +def test_persistable_flat_config_strips_runtime_keys() -> None: + from scripts.hpo_pipeline import _persistable_flat_config + + flat = { + "model_1_type": "XGBoost", + "_data_dir": "/tmp/data", + "_train_subsample": 0.125, + "log_wandb_diagnostics": True, + "target_mode": "single", + } + persisted = _persistable_flat_config(flat) + assert persisted == {"model_1_type": "XGBoost", "target_mode": "single"} diff --git a/tests/test_leaderboard.py b/tests/test_leaderboard.py index 99e5b07..7740063 100644 --- a/tests/test_leaderboard.py +++ b/tests/test_leaderboard.py @@ -3,12 +3,14 @@ from alphapulse.logging_.leaderboard import ( TrialLeaderboardEntry, + compute_robust_payout_score, format_leaderboard, save_leaderboard, + selection_score_from_metrics, ) -def test_leaderboard_format_and_sort(tmp_path: Path) -> None: +def test_leaderboard_format_and_sort_by_sharpe(tmp_path: Path) -> None: entries = [ TrialLeaderboardEntry( trial_number=1, @@ -18,6 +20,7 @@ def test_leaderboard_format_and_sort(tmp_path: Path) -> None: max_drawdown=0.1, model_types="XGBoost", elapsed_seconds=10.0, + holdout_corr_sharpe=0.5, ), TrialLeaderboardEntry( trial_number=2, @@ -27,10 +30,13 @@ def test_leaderboard_format_and_sort(tmp_path: Path) -> None: max_drawdown=0.05, model_types="LightGBM+CatBoost", elapsed_seconds=25.0, + holdout_corr_sharpe=1.2, ), ] text = format_leaderboard(entries, current_trial=2) assert "LEADERBOARD" in text + assert "holdout corr_sharpe" in text + assert "HoldoutSharpe" in text assert "LightGBM+CatBoost" in text assert "*" in text @@ -38,3 +44,114 @@ def test_leaderboard_format_and_sort(tmp_path: Path) -> None: save_leaderboard(path, entries) data = json.loads(path.read_text(encoding="utf-8")) assert data[0]["trial_number"] == 2 + + +def test_leaderboard_format_and_sort_by_payout_score(tmp_path: Path) -> None: + entries = [ + TrialLeaderboardEntry( + trial_number=1, + sharpe=1.5, + mean_per_era_correlation=0.03, + std_per_era_correlation=0.01, + max_drawdown=0.1, + model_types="XGBoost", + elapsed_seconds=10.0, + payout_score=0.8, + mmc_sharpe=0.2, + val_corr_sharpe=0.1, + val_mean_per_era_correlation=0.01, + holdout_corr_sharpe=1.5, + robust_payout_score=0.8, + ), + TrialLeaderboardEntry( + trial_number=2, + sharpe=0.9, + mean_per_era_correlation=0.04, + std_per_era_correlation=0.02, + max_drawdown=0.05, + model_types="LightGBM", + elapsed_seconds=25.0, + payout_score=1.1, + mmc_sharpe=0.4, + val_corr_sharpe=0.2, + val_mean_per_era_correlation=0.02, + holdout_corr_sharpe=0.9, + robust_payout_score=1.1, + ), + ] + text = format_leaderboard(entries) + assert "by payout on validation" in text + assert "ValidationMmcSharpe" in text + assert "ValidationSharpe" in text + assert "HoldoutSharpe" in text + assert text.index("LightGBM") < text.index("XGBoost") + + path = tmp_path / "leaderboard.json" + save_leaderboard(path, entries) + data = json.loads(path.read_text(encoding="utf-8")) + assert data[0]["trial_number"] == 2 + assert data[0]["payout_score"] == 1.1 + assert data[0]["mmc_sharpe"] == 0.4 + + +def test_robust_payout_penalizes_negative_holdout() -> None: + payout = 1.0 + robust = compute_robust_payout_score( + payout, val_corr_sharpe=0.4, holdout_corr_sharpe=-0.02 + ) + assert robust is not None + assert robust == 0.25 + + +def test_robust_leaderboard_prefers_consistent_trial() -> None: + entries = [ + TrialLeaderboardEntry( + trial_number=35, + sharpe=-0.02, + mean_per_era_correlation=-0.001, + std_per_era_correlation=0.04, + max_drawdown=0.2, + model_types="XGBoost", + elapsed_seconds=300.0, + payout_score=0.997, + mmc_sharpe=0.32, + val_corr_sharpe=0.37, + val_mean_per_era_correlation=0.01, + holdout_corr_sharpe=-0.02, + robust_payout_score=compute_robust_payout_score(0.997, 0.37, -0.02), + ), + TrialLeaderboardEntry( + trial_number=104, + sharpe=0.62, + mean_per_era_correlation=0.024, + std_per_era_correlation=0.03, + max_drawdown=0.1, + model_types="LightGBM+XGBoost", + elapsed_seconds=900.0, + payout_score=0.916, + mmc_sharpe=0.25, + val_corr_sharpe=0.46, + val_mean_per_era_correlation=0.016, + holdout_corr_sharpe=0.615, + robust_payout_score=compute_robust_payout_score(0.916, 0.46, 0.615), + ), + ] + text = format_leaderboard(entries) + assert "robust payout" in text + assert text.index("104") < text.index("35", text.index("robust payout")) + + +def test_selection_score_from_metrics_uses_robust_payout() -> None: + metrics = { + "payout_score": 0.997, + "val_corr_sharpe": 0.37, + "holdout_corr_sharpe": -0.02, + } + objective_score = selection_score_from_metrics( + metrics, objective="payout_score", criteria="objective" + ) + robust_score = selection_score_from_metrics( + metrics, objective="payout_score", criteria="robust_payout" + ) + assert objective_score == 0.997 + assert robust_score == 0.997 * 0.25 diff --git a/tests/test_meta_neutralization.py b/tests/test_meta_neutralization.py new file mode 100644 index 0000000..d5b0e82 --- /dev/null +++ b/tests/test_meta_neutralization.py @@ -0,0 +1,163 @@ +from pathlib import Path +from typing import Any + +import numpy as np +import pandas as pd + +from alphapulse.experiments.data import META_MODEL_COLUMN, meta_model_from_benchmarks +from alphapulse.models.base import BaseModel +from alphapulse.pipeline.neutralizer import ( + MetaModelNeutralizer, + neutralize_against_meta, +) +from alphapulse.pipeline.pipeline import Pipeline +from alphapulse.preprocessors.base import BasePreprocessor + + +class _IdentityPreprocessor(BasePreprocessor): + def fit(self, X: pd.DataFrame, y: pd.Series | None = None) -> BasePreprocessor: + return self + + def transform(self, X: pd.DataFrame) -> pd.DataFrame: + return X + + +class _ConstModel(BaseModel): + def __init__(self, value: float, name: str = "m") -> None: + self.value = value + self.name = name + self.is_trained = True + + def train( + self, + X_train: pd.DataFrame, + y_train: pd.Series, + X_val: pd.DataFrame | None = None, + y_val: pd.Series | None = None, + **kwargs: Any, + ) -> dict[str, float]: + return {} + + def predict(self, X: pd.DataFrame) -> np.ndarray: + return np.full(len(X), self.value, dtype=np.float64) + + def save(self, path: Path) -> None: + pass + + def load(self, path: Path) -> BaseModel: + return self + + +def test_neutralize_against_meta_reduces_correlation() -> None: + rng = np.random.default_rng(0) + n = 200 + meta = rng.standard_normal(n) + preds = 0.9 * meta + 0.1 * rng.standard_normal(n) + eras = pd.Series(np.repeat([f"era_{i}" for i in range(10)], n // 10)) + neutral = neutralize_against_meta(preds, meta, eras=eras, proportion=1.0) + before = np.corrcoef(preds, meta)[0, 1] + after = np.corrcoef(neutral, meta)[0, 1] + assert abs(after) < abs(before) + + +def test_pipeline_meta_neutralization_in_predict() -> None: + rng = np.random.default_rng(1) + n = 80 + X = pd.DataFrame({"f1": rng.standard_normal(n), "f2": rng.standard_normal(n)}) + meta = rng.standard_normal(n) + pipeline = Pipeline( + preprocessors=[_IdentityPreprocessor()], + models=[_ConstModel(0.5)], + feature_columns=list(X.columns), + meta_neutralize_proportion=0.8, + ) + preds = pipeline.predict(X, meta_model=meta) + assert preds.shape == (n,) + assert np.all(np.isfinite(preds)) + + +def test_meta_neutralizer_optimize_proportion() -> None: + rng = np.random.default_rng(2) + n = 120 + eras = pd.Series(np.repeat([f"era_{i}" for i in range(6)], n // 6)) + meta = rng.standard_normal(n) + y = 0.2 * meta + rng.standard_normal(n) * 0.5 + preds = 0.85 * meta + rng.standard_normal(n) * 0.1 + neutralizer = MetaModelNeutralizer(proportion=0.5) + optimized = neutralizer.optimize_proportion( + preds, meta, pd.Series(y), eras, objective="mmc_sharpe" + ) + assert 0.0 <= optimized <= 1.0 + + +def test_meta_model_from_benchmarks_aligns_to_index() -> None: + rng = np.random.default_rng(3) + index = pd.Index(["a", "b", "c"]) + benchmarks = pd.DataFrame( + {META_MODEL_COLUMN: rng.standard_normal(3)}, + index=index, + ) + meta = meta_model_from_benchmarks(benchmarks, index) + assert meta is not None + assert meta.shape == (3,) + + +class _FeatureModel(BaseModel): + def __init__(self) -> None: + super().__init__("feature") + self.is_trained = True + + def train( + self, + X_train: pd.DataFrame, + y_train: pd.Series, + X_val: pd.DataFrame | None = None, + y_val: pd.Series | None = None, + **kwargs: Any, + ) -> dict[str, float]: + return {} + + def predict(self, X: pd.DataFrame) -> np.ndarray: + return np.asarray( + 0.9 * X["f1"].to_numpy(dtype=np.float64) + + 0.1 * X["f2"].to_numpy(dtype=np.float64), + dtype=np.float64, + ) + + def save(self, path: Path) -> None: + pass + + def load(self, path: Path) -> BaseModel: + return self + + +def test_to_numerai_predict_applies_meta_neutralization() -> None: + from alphapulse.evaluation.metrics import rank_normalize + + rng = np.random.default_rng(4) + n = 80 + X = pd.DataFrame( + { + "f1": rng.standard_normal(n), + "f2": rng.standard_normal(n), + } + ) + meta = rng.standard_normal(n) + pipeline = Pipeline( + preprocessors=[_IdentityPreprocessor()], + models=[_FeatureModel()], + feature_columns=list(X.columns), + meta_neutralize_proportion=0.8, + ) + direct_with = pipeline.predict(X, meta_model=meta) + direct_without = pipeline.predict(X, meta_model=None) + assert not np.allclose(direct_with, direct_without) + + predict_fn = pipeline.to_numerai_predict() + benchmarks = pd.DataFrame({META_MODEL_COLUMN: meta}, index=X.index) + exported_with = predict_fn(X, benchmarks)["prediction"].to_numpy() + exported_without = predict_fn(X, pd.DataFrame(index=X.index))[ + "prediction" + ].to_numpy() + assert np.allclose(exported_with, rank_normalize(direct_with)) + assert np.allclose(exported_without, rank_normalize(direct_without)) diff --git a/tests/test_mmc_validation_load.py b/tests/test_mmc_validation_load.py new file mode 100644 index 0000000..7150de0 --- /dev/null +++ b/tests/test_mmc_validation_load.py @@ -0,0 +1,97 @@ +from __future__ import annotations + +from pathlib import Path + +import numpy as np +import pandas as pd + +from alphapulse.experiments.data import load_mmc_validation_frame +from alphapulse.hpo.objective import _merge_validation_mmc_metrics +from alphapulse.models.xgboost_model import XGBoostModel +from alphapulse.pipeline.pipeline import Pipeline +from alphapulse.preprocessors.scaling import StandardScalerPreprocessor + + +def _write_mmc_dataset(data_dir: Path) -> list[str]: + data_dir.mkdir(parents=True, exist_ok=True) + rng = np.random.default_rng(0) + n = 400 + eras = np.repeat([f"{i:04d}" for i in range(1133, 1143)], n // 10) + ids = [f"n{i:016x}" for i in range(n)] + val_df = pd.DataFrame( + { + "era": eras, + "target": rng.standard_normal(n), + "f_a": rng.standard_normal(n), + "f_b": rng.standard_normal(n), + }, + index=ids, + ) + val_df.index.name = "id" + val_df.to_parquet(data_dir / "validation.parquet") + + meta_df = pd.DataFrame( + { + "era": eras, + "data_type": "validation", + "numerai_meta_model": rng.uniform(0.0, 1.0, n), + }, + index=ids, + ) + meta_df.index.name = "id" + meta_df.to_parquet(data_dir / "meta_model.parquet") + return ["f_a", "f_b"] + + +def test_load_mmc_validation_frame_returns_aligned_meta(tmp_path: Path) -> None: + feature_cols = _write_mmc_dataset(tmp_path) + frame = load_mmc_validation_frame( + tmp_path, + feature_cols=feature_cols, + target_col="target", + train_subsample=0.5, + seed=1, + ) + assert frame is not None + X_val, y_val, era_val, meta_preds = frame + assert len(X_val) == len(y_val) == len(era_val) == len(meta_preds) + assert np.isfinite(meta_preds).all() + + +def test_merge_validation_mmc_metrics_populates_mmc(tmp_path: Path) -> None: + feature_cols = _write_mmc_dataset(tmp_path) + frame = load_mmc_validation_frame( + tmp_path, + feature_cols=feature_cols, + target_col="target", + train_subsample=1.0, + seed=1, + ) + assert frame is not None + X_val, y_val, _, _ = frame + pipe = Pipeline( + preprocessors=[StandardScalerPreprocessor()], + model=XGBoostModel( + params={ + "max_depth": 3, + "learning_rate": 0.1, + "tree_method": "hist", + "objective": "reg:squarederror", + } + ), + ) + pipe.fit(X_val.iloc[:300], y_val.iloc[:300], n_rounds=8) + merged = _merge_validation_mmc_metrics( + {"corr_sharpe": 1.0}, + pipeline=pipe, + data_dir=tmp_path, + feature_cols=feature_cols, + target_col="target", + train_subsample=1.0, + seed=1, + )[0] + assert np.isfinite(merged["mmc"]) + assert np.isfinite(merged["mmc_sharpe"]) + assert np.isfinite(merged["payout_score"]) + assert merged["holdout_corr_sharpe"] == 1.0 + assert np.isfinite(merged["val_corr_sharpe"]) diff --git a/tests/test_models.py b/tests/test_models.py index 247b98f..0c96e71 100644 --- a/tests/test_models.py +++ b/tests/test_models.py @@ -58,12 +58,19 @@ def test_instantiate_model_matches_model_factory() -> None: params = {"params": {"max_depth": 3, "learning_rate": 0.1}, "name": "TestXGB"} from_builder = instantiate_model("XGBoost", params, index=0, n_subs=3) - from_factory = ModelFactory().suggest_fixed("xgboost", params, n_subs=3) + from_factory = ModelFactory().suggest_fixed("XGBoost", params, n_subs=3) assert isinstance(from_builder, EraEnsembleModel) assert isinstance(from_factory, EraEnsembleModel) assert from_builder.n_subs == from_factory.n_subs == 3 +def test_apply_gpu_model_params_packboost_sets_cuda() -> None: + from alphapulse.hpo.search_space import apply_gpu_model_params + + params = apply_gpu_model_params("Packboost", {"n_rounds_base": 100}) + assert params["device"] == "cuda" + + def test_apply_gpu_model_params_lightgbm_sets_device() -> None: from alphapulse.hpo.search_space import apply_gpu_model_params @@ -100,10 +107,11 @@ def test_ensemble_optimizer_fit_predict() -> None: y = rng.randn(n) oof = np.column_stack([y + rng.randn(n) * 0.5, y + rng.randn(n) * 0.8]) - optimizer = EnsembleOptimizer(seed=0) + optimizer = EnsembleOptimizer(seed=0, min_weight=0.05, max_weight=0.90) optimizer.fit(oof, y, eras) assert optimizer.weights_ is not None assert optimizer.weights_.sum() == pytest.approx(1.0) + assert all(0.05 <= w <= 0.90 for w in optimizer.weights_) blend = optimizer.predict(oof[:10]) assert blend.shape == (10,) diff --git a/tests/test_multitarget_diagnostics.py b/tests/test_multitarget_diagnostics.py new file mode 100644 index 0000000..614278d --- /dev/null +++ b/tests/test_multitarget_diagnostics.py @@ -0,0 +1,132 @@ +from __future__ import annotations + +from unittest.mock import MagicMock, patch + +import numpy as np +import pandas as pd + +from alphapulse.evaluation.shap_report import compute_universal_feature_importance +from alphapulse.evaluation.wandb_diagnostics import log_experiment_diagnostics +from alphapulse.models.xgboost_model import XGBoostModel +from alphapulse.pipeline.model_access import iter_trained_models, model_prediction_map +from alphapulse.pipeline.multi_target import MultiTargetPipeline +from alphapulse.preprocessors.scaling import StandardScalerPreprocessor + +FEATURE_COLS = ["f_a", "f_b"] +N = 120 + + +def _multitarget_pipeline() -> tuple[ + MultiTargetPipeline, pd.DataFrame, pd.Series, pd.Series +]: + rng = np.random.default_rng(3) + era = pd.Series(np.repeat([f"e{i:03d}" for i in range(12)], N // 12)) + X = pd.DataFrame( + { + "f_a": rng.standard_normal(N), + "f_b": rng.standard_normal(N), + "era": era.values, + } + ) + targets = pd.DataFrame( + { + "target": rng.standard_normal(N), + "target_alpha_20": rng.standard_normal(N), + } + ) + + def factory() -> XGBoostModel: + return XGBoostModel( + params={ + "max_depth": 3, + "learning_rate": 0.1, + "tree_method": "hist", + "objective": "reg:squarederror", + } + ) + + pipeline = MultiTargetPipeline( + preprocessors=[StandardScalerPreprocessor()], + model_factory=factory, + target_columns=["target", "target_alpha_20"], + primary_target="target", + ) + pipeline.fit(X.drop(columns=["era"]), targets, n_rounds=8) + y_val = targets["target"] + return pipeline, X, y_val, era + + +def test_iter_trained_models_multitarget() -> None: + pipeline, _, _, _ = _multitarget_pipeline() + models = iter_trained_models(pipeline) + assert len(models) == 2 + assert all(isinstance(m, XGBoostModel) for m in models) + + +def test_model_prediction_map_multitarget() -> None: + pipeline, X, _, _ = _multitarget_pipeline() + preds = model_prediction_map(pipeline, X, FEATURE_COLS) + assert set(preds.keys()) == {"target", "target_alpha_20"} + assert all(len(v) == N for v in preds.values()) + + +def test_compute_universal_feature_importance_multitarget() -> None: + pipeline, X, _, _ = _multitarget_pipeline() + importance, label = compute_universal_feature_importance( + pipeline, + X.drop(columns=["era"]), + feature_cols=FEATURE_COLS, + top_n=10, + ) + assert len(importance) > 0 + assert "XGBoost" in label + + +def test_log_experiment_diagnostics_multitarget() -> None: + pipeline, X, y_val, era_val = _multitarget_pipeline() + X_feat = X.drop(columns=["era"]) + metrics = { + "corr_sharpe": 1.0, + "mmc": 0.01, + "mmc_sharpe": 0.5, + "payout_score": 1.2, + "mean_per_era_correlation": 0.02, + "std_per_era_correlation": 0.01, + "max_drawdown": 0.05, + "pct_positive_eras": 0.8, + "n_valid_eras": 10, + } + + mock_wandb = MagicMock() + mock_wandb.run = MagicMock() + mock_wandb.Table = MagicMock(side_effect=lambda columns: MagicMock(columns=columns)) + mock_wandb.plot = MagicMock() + mock_wandb.Image = MagicMock() + + with ( + patch( + "alphapulse.evaluation.wandb_diagnostics._wandb_active", + return_value=True, + ), + patch.dict("sys.modules", {"wandb": mock_wandb}), + ): + log_experiment_diagnostics( + pipeline=pipeline, + X_val=X_feat, + y_val=y_val, + era_val=era_val, + feature_cols=FEATURE_COLS, + metrics=metrics, + log_shap=True, + log_feature_report=False, + log_era_importance=False, + split="validation", + ) + + keys: set[str] = set() + for call in mock_wandb.log.call_args_list: + keys.update(call.args[0].keys()) + assert mock_wandb.log.called + assert "diagnostics/validation/ValidationMmcSharpe" in keys + assert "diagnostics/validation/ValidationSharpe" in keys + assert not {k for k in keys if k.endswith(("_table", "_top"))} diff --git a/tests/test_numerai_export_e2e.py b/tests/test_numerai_export_e2e.py new file mode 100644 index 0000000..2bb250d --- /dev/null +++ b/tests/test_numerai_export_e2e.py @@ -0,0 +1,189 @@ +import json +import random +from pathlib import Path + +import cloudpickle +import numpy as np +import pandas as pd + +from alphapulse.evaluation.export_validation import smoke_test_predict_fn +from alphapulse.features.catalog import TargetCatalog +from alphapulse.hpo.export import ( + build_hpo_pipeline_from_flat, + prepare_hpo_flat, + resolve_hpo_build_context, +) +from alphapulse.hpo.target_strategy import sample_target_strategy +from alphapulse.pipeline.multi_target import MultiTargetPipeline +from alphapulse.pipeline.multihead import MultiHeadPipeline +from alphapulse.pipeline.pipeline import Pipeline + + +def _write_toy_dataset(data_dir: Path, *, n: int = 400) -> list[str]: + data_dir.mkdir(parents=True, exist_ok=True) + rng = np.random.default_rng(0) + eras = np.repeat([f"era_{i:04d}" for i in range(20)], n // 20) + df = pd.DataFrame( + { + "era": eras, + "target": rng.standard_normal(n), + "target_alpha_20": rng.standard_normal(n), + "f_a": rng.standard_normal(n), + "f_b": rng.standard_normal(n), + "f_c": rng.standard_normal(n), + } + ) + df.to_parquet(data_dir / "train.parquet", index=False) + features = { + "feature_sets": { + "small": ["f_a", "f_b"], + "medium": ["f_a", "f_b", "f_c"], + "strength": ["f_a", "f_c"], + }, + "targets": ["target", "target_alpha_20"], + } + (data_dir / "features.json").write_text(json.dumps(features), encoding="utf-8") + return ["f_a", "f_b", "f_c"] + + +def _minimal_flat() -> dict: + return { + "num_models": 1, + "model_1_type": "XGBoost", + "model_2_type": "XGBoost", + "model_3_type": "XGBoost", + "scaler_type": "StandardScaler", + "use_packboost": False, + "ensemble_method": "single", + "use_neutralization": False, + "xgb_max_depth": 3, + "xgb_learning_rate": 0.1, + "xgb_n_rounds": 10, + "xgb_early_stopping": 5, + "n_subs": 3, + "target_mode": "single", + "primary_target": "target", + "auxiliary_targets": [], + "use_feature_routing": False, + } + + +def test_hpo_primary_is_target() -> None: + catalog = TargetCatalog(targets=["target", "target_alpha_20"]) + for seed in range(30): + strategy = sample_target_strategy(random.Random(seed), catalog, fast=True) + assert strategy.primary_target == "target" + + +def test_export_matches_worker_build_context(tmp_path: Path) -> None: + data_dir = tmp_path / "data" + _write_toy_dataset(data_dir) + flat = prepare_hpo_flat( + { + **_minimal_flat(), + "use_feature_routing": True, + "active_groups": ["small", "strength"], + "model_1_groups": ["small", "strength"], + "model_1_lane": 0, + "lane_0_steps": [], + }, + data_dir, + ) + ctx = resolve_hpo_build_context(flat) + assert ctx.routing.build_path == "simple" + assert set(ctx.feature_columns) == {"f_a", "f_b", "f_c"} + + +def test_export_routed_config(tmp_path: Path) -> None: + data_dir = tmp_path / "data" + _write_toy_dataset(data_dir) + flat = { + **_minimal_flat(), + "use_feature_routing": True, + "active_groups": ["small"], + "model_1_groups": ["small"], + "model_1_lane": 0, + "lane_0_steps": [], + } + result = build_hpo_pipeline_from_flat( + flat, + data_dir, + train_subsample=0.5, + seed=42, + ) + assert isinstance(result.pipeline, Pipeline | MultiHeadPipeline) + pkl = tmp_path / "predict.pkl" + with open(pkl, "wb") as f: + cloudpickle.dump(result.pipeline.to_numerai_predict(), f) + smoke_test_predict_fn(pkl, result.feature_columns) + + +def test_export_multi_blend_config(tmp_path: Path) -> None: + data_dir = tmp_path / "data" + _write_toy_dataset(data_dir) + flat = { + **_minimal_flat(), + "target_mode": "multi_blend", + "primary_target": "target", + "auxiliary_targets": ["target_alpha_20"], + "target_blend_method": "equal", + } + result = build_hpo_pipeline_from_flat( + flat, + data_dir, + train_subsample=0.5, + seed=7, + ) + assert isinstance(result.pipeline, MultiTargetPipeline) + pkl = tmp_path / "predict_multi.pkl" + with open(pkl, "wb") as f: + cloudpickle.dump(result.pipeline.to_numerai_predict(), f) + smoke_test_predict_fn(pkl, result.feature_columns) + + +def test_multitarget_predict_reindex_missing_columns() -> None: + from alphapulse.models.xgboost_model import XGBoostModel + from alphapulse.preprocessors.scaling import StandardScalerPreprocessor + + rng = np.random.default_rng(1) + n = 80 + X = pd.DataFrame( + { + "f_a": rng.standard_normal(n), + "f_b": rng.standard_normal(n), + } + ) + targets = pd.DataFrame( + { + "target": rng.standard_normal(n), + "target_alpha_20": rng.standard_normal(n), + } + ) + + def factory() -> XGBoostModel: + return XGBoostModel( + params={ + "max_depth": 3, + "learning_rate": 0.1, + "tree_method": "hist", + "objective": "reg:squarederror", + } + ) + + pipeline = MultiTargetPipeline( + preprocessors=[StandardScalerPreprocessor()], + model_factory=factory, + target_columns=["target", "target_alpha_20"], + primary_target="target", + ) + pipeline.fit(X, targets, n_rounds=5) + live = pd.DataFrame( + { + "f_a": rng.standard_normal(10), + "extra_col": rng.standard_normal(10), + } + ) + predict_fn = pipeline.to_numerai_predict() + out = predict_fn(live, pd.DataFrame()) + assert "prediction" in out.columns + assert out["prediction"].between(0.0, 1.0).all() diff --git a/tests/test_optuna_search.py b/tests/test_optuna_search.py new file mode 100644 index 0000000..8573d5f --- /dev/null +++ b/tests/test_optuna_search.py @@ -0,0 +1,149 @@ +import json +from pathlib import Path +from unittest.mock import MagicMock + +import optuna + +from alphapulse.hpo.optuna_search import ( + DEFAULT_N_STARTUP_TRIALS, + _suggest_model_hyperparams, + create_hpo_study, + suggest_flat_config, + tell_trial_result, +) + + +def test_suggest_flat_config_returns_routing(tmp_path: Path) -> None: + data_dir = tmp_path / "data" + data_dir.mkdir() + payload = { + "feature_sets": { + "small": ["f_a", "f_b"], + "medium": ["f_a", "f_b", "f_c"], + "strength": ["f_a", "f_c"], + }, + "targets": ["target", "target_alpha_20"], + } + (data_dir / "features.json").write_text(json.dumps(payload), encoding="utf-8") + + study = optuna.create_study(direction="maximize") + trial = study.ask() + cfg = suggest_flat_config(trial, fast=True, data_dir=data_dir) + assert cfg["use_feature_routing"] is True + assert cfg["active_groups"] + assert cfg["routed_feature_count"] <= 1000 + tell_trial_result(study, trial, 0.5) + + +def test_create_hpo_study_persists_storage(tmp_path: Path) -> None: + out = tmp_path / "hpo" + out.mkdir() + study = create_hpo_study(out, seed=1, sampler="tpe", resume=False) + trial = study.ask() + tell_trial_result(study, trial, 1.0) + assert (out / "optuna.db").exists() + + +def test_create_hpo_study_uses_n_startup_trials(tmp_path: Path) -> None: + out = tmp_path / "hpo" + out.mkdir() + study = create_hpo_study( + out, seed=1, sampler="tpe", resume=False, n_startup_trials=25 + ) + sampler = study.sampler + assert isinstance(sampler, optuna.samplers.TPESampler) + assert sampler._n_startup_trials == 25 + + +def test_default_n_startup_trials_constant() -> None: + assert DEFAULT_N_STARTUP_TRIALS == 25 + + +def test_suggest_model_hyperparams_skips_unused_families() -> None: + trial = MagicMock() + params = _suggest_model_hyperparams(trial, {"CatBoost"}, fast=True) + assert "catboost_depth" in params + assert "lgbm_num_leaves" not in params + assert "xgb_max_depth" not in params + assert "foundation_max_train_rows" not in params + trial.suggest_categorical.assert_called() + trial.suggest_float.assert_called() + + +def test_suggest_model_hyperparams_includes_foundation_when_active() -> None: + trial = MagicMock() + params = _suggest_model_hyperparams(trial, {"TabPFN"}, fast=True) + assert "foundation_max_train_rows" in params + assert "catboost_depth" not in params + + +def test_suggest_flat_config_omits_lgbm_when_not_selected() -> None: + study = optuna.create_study( + direction="maximize", + sampler=optuna.samplers.RandomSampler(seed=0), + ) + + def objective(trial: optuna.Trial) -> float: + cfg = suggest_flat_config(trial, fast=True) + types = [ + cfg.get("model_1_type"), + cfg.get("model_2_type"), + cfg.get("model_3_type"), + ][: int(cfg.get("num_models", 1))] + if "LightGBM" not in types: + assert not any(k.startswith("lgbm_") for k in cfg) + if "XGBoost" not in types: + assert not any(k.startswith("xgb_") for k in cfg) + if "CatBoost" not in types: + assert not any(k.startswith("catboost_") for k in cfg) + return 0.0 + + study.optimize(objective, n_trials=30) + + +def test_suggest_flat_config_meta_neutralization_conditional() -> None: + study = optuna.create_study( + direction="maximize", + sampler=optuna.samplers.RandomSampler(seed=1), + ) + + def objective(trial: optuna.Trial) -> float: + cfg = suggest_flat_config(trial, fast=True) + if not cfg.get("use_meta_neutralization"): + assert "meta_neutralization_proportion" not in cfg + return 0.0 + + study.optimize(objective, n_trials=20) + + +def test_suggest_flat_config_single_model_no_ensemble_method_suggest() -> None: + study = optuna.create_study( + direction="maximize", + sampler=optuna.samplers.RandomSampler(seed=2), + ) + saw_single = False + for _ in range(40): + trial = study.ask() + cfg = suggest_flat_config(trial, fast=True) + tell_trial_result(study, trial, 0.0) + if int(cfg.get("num_models", 1)) == 1: + saw_single = True + assert cfg["ensemble_method"] == "single" + assert saw_single + + +def test_suggest_flat_config_fast_respects_max_models() -> None: + study = optuna.create_study( + direction="maximize", + sampler=optuna.samplers.RandomSampler(seed=3), + ) + saw_three = False + for _ in range(40): + trial = study.ask() + cfg = suggest_flat_config(trial, fast=True, max_models=3) + tell_trial_result(study, trial, 0.0) + assert int(cfg["num_models"]) <= 3 + if int(cfg["num_models"]) == 3: + saw_three = True + assert cfg.get("model_3_type") is not None + assert saw_three diff --git a/tests/test_packboost_cuda.py b/tests/test_packboost_cuda.py new file mode 100644 index 0000000..eeaba78 --- /dev/null +++ b/tests/test_packboost_cuda.py @@ -0,0 +1,82 @@ +from __future__ import annotations + +import numpy as np +import pandas as pd +import pytest + +from alphapulse.models.packboost_backend import packboost_cuda_available +from alphapulse.models.packboost_encoding import ( + bin_features_for_packboost, + default_nfeatsets, + encode_era_ids, +) +from alphapulse.models.packboost_model import PackboostModel + + +def test_bin_features_for_packboost_clips_integers() -> None: + frame = pd.DataFrame( + { + "a": [0.0, 1.0, 2.0, 4.0, 3.0], + "b": [0, 1, 2, 3, 4], + } + ) + binned = bin_features_for_packboost(frame) + assert binned.dtype == np.int8 + assert binned.max() <= 4 + assert binned.min() >= 0 + + +def test_bin_features_for_packboost_quantizes_scaled_values() -> None: + frame = pd.DataFrame({"a": [-2.5, -1.0, 0.0, 1.0, 2.5]}) + binned = bin_features_for_packboost(frame) + assert binned.dtype == np.int8 + assert sorted(binned[:, 0].tolist()) == [0, 1, 2, 3, 4] + + +def test_default_nfeatsets_scales_with_feature_count() -> None: + assert default_nfeatsets(20, requested=32) == 2 + assert default_nfeatsets(200, requested=32) == 25 + + +def test_encode_era_ids_preserves_chronological_order() -> None: + era = pd.Series(["era_0002", "era_0001", "era_0002", "era_0001"]) + encoded = encode_era_ids(era) + assert encoded.tolist() == [1, 0, 1, 0] + + +def test_packboost_model_rejects_non_cuda_device() -> None: + rng = np.random.default_rng(0) + n = 40 + cols = [f"f_{i}" for i in range(6)] + x = pd.DataFrame(rng.integers(0, 5, size=(n, len(cols))), columns=cols) + x["era"] = np.repeat(["era_0001", "era_0002"], n // 2) + y = pd.Series(rng.standard_normal(n)) + + model = PackboostModel(device="cpu", n_rounds_base=2, n_rounds_boost=2) + with pytest.raises(ValueError, match="only supports device='cuda'"): + model.train(x, y) + + +@pytest.mark.skipif(not packboost_cuda_available(), reason="PackBoost CUDA unavailable") +def test_packboost_model_trains_on_cuda() -> None: + rng = np.random.default_rng(1) + n = 200 + cols = [f"f_{i}" for i in range(24)] + x = pd.DataFrame(rng.integers(0, 5, size=(n, len(cols))), columns=cols) + x["era"] = np.repeat([f"era_{i:04d}" for i in range(10)], n // 10) + y = pd.Series(rng.standard_normal(n), dtype=np.float32) + + model = PackboostModel( + device="cuda", + n_rounds_base=5, + n_rounds_boost=3, + n_worst_eras=2, + nfolds=4, + max_depth=4, + nfeatsets=2, + ) + metrics = model.train(x, y) + preds = model.predict(x) + assert "n_boost_eras" in metrics + assert len(preds) == n + assert np.isfinite(preds).all() diff --git a/tests/test_runner_multitarget.py b/tests/test_runner_multitarget.py new file mode 100644 index 0000000..b580d19 --- /dev/null +++ b/tests/test_runner_multitarget.py @@ -0,0 +1,95 @@ +import json +from pathlib import Path + +import numpy as np +import pandas as pd +import pytest + +from alphapulse.experiments.runner import run_experiment +from alphapulse.experiments.schema import ExperimentV1 + + +@pytest.fixture +def multitarget_dataset_dir(tmp_path: Path) -> Path: + rng = np.random.RandomState(0) + n = 240 + eras = np.repeat([f"era_{i:04d}" for i in range(6)], n // 6) + df = pd.DataFrame( + { + "feature_a": rng.randn(n).astype(np.float32), + "feature_b": rng.randn(n).astype(np.float32), + "era": eras, + "target": rng.randn(n).astype(np.float32), + "target_aux": rng.randn(n).astype(np.float32), + "id": [f"id_{i}" for i in range(n)], + } + ) + df.to_parquet(tmp_path / "train.parquet", index=False) + (tmp_path / "validation.parquet").write_bytes( + (tmp_path / "train.parquet").read_bytes() + ) + (tmp_path / "features.json").write_text( + json.dumps( + { + "feature_sets": { + "small": ["feature_a"], + "all": ["feature_a", "feature_b"], + }, + "targets": ["target", "target_aux"], + } + ) + ) + return tmp_path + + +def test_run_experiment_multitarget_builds_pipeline( + multitarget_dataset_dir: Path, +) -> None: + exp = ExperimentV1.model_validate( + { + "version": "1", + "data": { + "data_dir": str(multitarget_dataset_dir), + "train_subsample": 1.0, + "target_col": "target", + "auxiliary_targets": ["target_aux"], + "target_blend_method": "equal", + "seed": 42, + }, + "features": {"columns": ["feature_a", "feature_b"], "groups": {}}, + "preprocessing": [], + "models": [ + { + "type": "XGBoost", + "params": { + "params": { + "max_depth": 2, + "learning_rate": 0.1, + "tree_method": "hist", + "objective": "reg:squarederror", + } + }, + } + ], + "ensemble_method": "single", + "train": {"n_rounds": 10, "early_stopping_rounds": 5}, + } + ) + result = run_experiment(exp, artifact_dir=multitarget_dataset_dir / "artifacts") + assert result.error is None + assert "corr_sharpe" in result.metrics + + +def test_schema_rejects_primary_in_auxiliary_targets(tmp_path: Path) -> None: + with pytest.raises(ValueError, match="auxiliary_targets must not include"): + ExperimentV1.model_validate( + { + "version": "1", + "data": { + "data_dir": str(tmp_path), + "target_col": "target", + "auxiliary_targets": ["target", "target_aux"], + }, + "models": [{"type": "XGBoost", "params": {}}], + } + ) diff --git a/tests/test_target_strategy.py b/tests/test_target_strategy.py new file mode 100644 index 0000000..f9d9abd --- /dev/null +++ b/tests/test_target_strategy.py @@ -0,0 +1,97 @@ +import random + +import numpy as np +import pandas as pd +import pytest + +from alphapulse.features.catalog import TargetCatalog +from alphapulse.hpo.target_strategy import ( + TargetStrategy, + sample_target_strategy, + validate_target_strategy_early, +) + + +@pytest.fixture +def target_catalog() -> TargetCatalog: + return TargetCatalog( + targets=["target", "target_alpha_20", "target_cyrusd_60", "target_sparse"] + ) + + +def test_sample_target_strategy_modes(target_catalog: TargetCatalog) -> None: + rng = random.Random(0) + modes = { + sample_target_strategy(rng, target_catalog, fast=True).target_mode + for _ in range(50) + } + assert "single" in modes + + +def test_validate_rejects_sparse_auxiliary(target_catalog: TargetCatalog) -> None: + n = 100 + targets_df = pd.DataFrame( + { + "target": np.random.randn(n), + "target_sparse": [np.nan] * 80 + list(np.random.randn(20)), + } + ) + strategy = TargetStrategy( + target_mode="multi_blend", + primary_target="target", + auxiliary_targets=["target_sparse"], + ) + result = validate_target_strategy_early( + targets_df, + strategy, + catalog=target_catalog, + rng=random.Random(1), + ) + assert result.ok + assert result.strategy.target_mode == "single" + assert result.strategy.auxiliary_targets == [] + + +def test_validate_accepts_valid_auxiliary(target_catalog: TargetCatalog) -> None: + n = 100 + targets_df = pd.DataFrame( + { + "target": np.random.randn(n), + "target_alpha_20": np.random.randn(n), + } + ) + strategy = TargetStrategy( + target_mode="multi_blend", + primary_target="target", + auxiliary_targets=["target_alpha_20"], + ) + result = validate_target_strategy_early( + targets_df, + strategy, + catalog=target_catalog, + rng=random.Random(2), + ) + assert result.ok + assert result.strategy.target_mode == "multi_blend" + + +def test_validate_does_not_invoke_build(target_catalog: TargetCatalog) -> None: + targets_df = pd.DataFrame( + { + "target": np.random.randn(50), + "target_sparse": [np.nan] * 40 + list(np.random.randn(10)), + } + ) + strategy = TargetStrategy( + target_mode="multi_blend", + primary_target="target", + auxiliary_targets=["target_sparse"], + ) + result = validate_target_strategy_early( + targets_df, + strategy, + catalog=target_catalog, + rng=random.Random(3), + ) + assert result.ok + assert result.strategy.target_mode == "single" diff --git a/tests/test_trial_db.py b/tests/test_trial_db.py index e8b6441..611df81 100644 --- a/tests/test_trial_db.py +++ b/tests/test_trial_db.py @@ -62,6 +62,16 @@ def test_wandb_group_file_persists_across_resume(tmp_path: Path) -> None: assert (output_dir / "wandb_group.txt").read_text(encoding="utf-8") == first +def test_resolve_wandb_project_persists_stamp(tmp_path: Path) -> None: + from alphapulse.logging_.wandb_utils import resolve_wandb_project + + first = resolve_wandb_project("alphapulse-hpo", output_dir=tmp_path) + second = resolve_wandb_project("alphapulse-hpo", output_dir=tmp_path) + assert first == second + assert first.startswith("alphapulse-hpo-") + assert (tmp_path / "wandb_project.txt").read_text(encoding="utf-8") == first + + def test_all_results_from_db_includes_all_trials(tmp_path: Path) -> None: from scripts.hpo_pipeline import _all_results_from_db diff --git a/tests/test_wandb_logging.py b/tests/test_wandb_logging.py new file mode 100644 index 0000000..f3aba68 --- /dev/null +++ b/tests/test_wandb_logging.py @@ -0,0 +1,50 @@ +from unittest.mock import MagicMock, patch + +import pytest +from loguru import logger + +from alphapulse.logging_.wandb_logging import ( + attach_wandb_loguru, + log_boosting_round_metrics, + parse_xgb_evals_log, +) + + +def test_parse_xgb_evals_log() -> None: + parsed = parse_xgb_evals_log( + { + "train": {"rmse": [0.22, 0.21]}, + "eval": {"rmse": [(0.23, 0.0), (0.22, 0.0)]}, + } + ) + assert parsed["train_rmse"] == pytest.approx(0.21) + assert parsed["eval_rmse"] == pytest.approx(0.22) + + +def test_log_boosting_round_metrics_logs_to_wandb() -> None: + mock_wandb = MagicMock() + with ( + patch("alphapulse.logging_.wandb_logging.wandb_run_active", return_value=True), + patch.dict("sys.modules", {"wandb": mock_wandb}), + ): + log_boosting_round_metrics( + model_name="XGBoost_0", + round_num=10, + metrics={"train_rmse": 0.2, "eval_rmse": 0.21}, + ) + mock_wandb.log.assert_called_once() + logged = mock_wandb.log.call_args.args[0] + assert logged["train/round"] == 10 + assert logged["train/XGBoost_0/train_rmse"] == 0.2 + + +def test_attach_wandb_loguru_adds_sink() -> None: + from alphapulse.logging_.wandb_logging import detach_wandb_loguru + + with patch("alphapulse.logging_.wandb_logging.wandb_run_active", return_value=True): + attach_wandb_loguru() + try: + logger.info("wandb sink smoke test") + finally: + detach_wandb_loguru() + logger.add(lambda msg: None, level="INFO") diff --git a/tests/test_wandb_utils_hpo.py b/tests/test_wandb_utils_hpo.py new file mode 100644 index 0000000..e902e5d --- /dev/null +++ b/tests/test_wandb_utils_hpo.py @@ -0,0 +1,39 @@ +from unittest.mock import MagicMock, patch + +from alphapulse.hpo.objective import TrialResult +from alphapulse.logging_.wandb_utils import log_hpo_trial_metrics + + +def test_log_hpo_trial_metrics_logs_feature_routing_fields() -> None: + mock_wandb = MagicMock() + result = TrialResult( + trial_number=1, + sharpe=1.0, + metrics={ + "corr_sharpe": 1.0, + "holdout_corr_sharpe": 1.0, + "val_corr_sharpe": 0.4, + "payout_score": 0.9, + "mmc_sharpe": 0.2, + }, + model_type="XGBoost", + elapsed_seconds=12.0, + params={ + "active_groups": ["small", "strength"], + "routed_feature_count": 512, + }, + corr_sharpe=1.0, + mmc_sharpe=0.2, + payout_score=0.9, + ) + with patch.dict("sys.modules", {"wandb": mock_wandb}): + log_hpo_trial_metrics(result, objective=0.9, model_types="XGBoost") + + logged = mock_wandb.log.call_args.args[0] + assert logged["active_groups"] == "small+strength" + assert logged["active_groups_count"] == 2 + assert logged["routed_feature_count"] == 512 + assert logged["holdout/HoldoutSharpe"] == 1.0 + assert logged["validation/ValidationSharpe"] == 0.4 + assert logged["validation/ValidationMmcSharpe"] == 0.2 + assert logged["validation/PayoutScore"] == 0.9 diff --git a/uv.lock b/uv.lock index 242c0b3..59e564a 100644 --- a/uv.lock +++ b/uv.lock @@ -129,7 +129,7 @@ wheels = [ [[package]] name = "alphapulse" -version = "0.5.0" +version = "0.6.0" source = { editable = "." } dependencies = [ { name = "catboost" }, @@ -176,12 +176,14 @@ foundation = [ { name = "tabicl" }, { name = "tabpfn" }, ] -foundation-api = [ - { name = "tabpfn-client" }, -] hpo = [ { name = "ray", extra = ["tune"] }, ] +packboost = [ + { name = "ninja" }, + { name = "packboost" }, + { name = "torch" }, +] [package.metadata] requires-dist = [ @@ -194,8 +196,10 @@ requires-dist = [ { name = "mypy", marker = "extra == 'dev'", specifier = ">=1.10" }, { name = "nbqa", marker = "extra == 'dev'", specifier = ">=1.9.1" }, { name = "networkx", marker = "extra == 'eda'", specifier = ">=3.2" }, + { name = "ninja", marker = "extra == 'packboost'" }, { name = "numerapi" }, { name = "optuna", specifier = ">=3,<5" }, + { name = "packboost", marker = "extra == 'packboost'", git = "https://github.com/Pranshu-Bahadur/PackBoost.git" }, { name = "pandas", specifier = ">=2,<3" }, { name = "plotly", marker = "extra == 'eda'", specifier = ">=5.18" }, { name = "pre-commit", marker = "extra == 'dev'", specifier = ">=3.7.0" }, @@ -213,15 +217,15 @@ requires-dist = [ { name = "streamlit", marker = "extra == 'eda'", specifier = ">=1.30" }, { name = "tabicl", marker = "extra == 'foundation'", specifier = ">=2.0" }, { name = "tabpfn", marker = "extra == 'foundation'", specifier = ">=7.0" }, - { name = "tabpfn-client", marker = "extra == 'foundation-api'", specifier = ">=0.3.0" }, { name = "torch", marker = "extra == 'deep'" }, + { name = "torch", marker = "extra == 'packboost'" }, { name = "types-pyyaml", marker = "extra == 'dev'", specifier = ">=6.0" }, { name = "tyro" }, { name = "vulture", marker = "extra == 'dev'", specifier = ">=2.14" }, { name = "wandb" }, { name = "xgboost", specifier = ">=2,<3" }, ] -provides-extras = ["dev", "hpo", "deep", "foundation", "foundation-api", "eda"] +provides-extras = ["dev", "hpo", "deep", "foundation", "packboost", "eda"] [[package]] name = "altair" @@ -956,29 +960,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/6a/09/e21df6aef1e1ffc0c816f0522ddc3f6dcded766c3261813131c78a704470/gitpython-3.1.46-py3-none-any.whl", hash = "sha256:79812ed143d9d25b6d176a10bb511de0f9c67b1fa641d82097b0ab90398a2058", size = 208620, upload-time = "2026-01-01T15:37:30.574Z" }, ] -[[package]] -name = "google-crc32c" -version = "1.8.0" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/03/41/4b9c02f99e4c5fb477122cd5437403b552873f014616ac1d19ac8221a58d/google_crc32c-1.8.0.tar.gz", hash = "sha256:a428e25fb7691024de47fecfbff7ff957214da51eddded0da0ae0e0f03a2cf79", size = 14192, upload-time = "2025-12-16T00:35:25.142Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/e9/5f/7307325b1198b59324c0fa9807cafb551afb65e831699f2ce211ad5c8240/google_crc32c-1.8.0-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:4b8286b659c1335172e39563ab0a768b8015e88e08329fa5321f774275fc3113", size = 31300, upload-time = "2025-12-16T00:21:56.723Z" }, - { url = "https://files.pythonhosted.org/packages/21/8e/58c0d5d86e2220e6a37befe7e6a94dd2f6006044b1a33edf1ff6d9f7e319/google_crc32c-1.8.0-cp312-cp312-macosx_12_0_x86_64.whl", hash = "sha256:2a3dc3318507de089c5384cc74d54318401410f82aa65b2d9cdde9d297aca7cb", size = 30867, upload-time = "2025-12-16T00:38:31.302Z" }, - { url = "https://files.pythonhosted.org/packages/ce/a9/a780cc66f86335a6019f557a8aaca8fbb970728f0efd2430d15ff1beae0e/google_crc32c-1.8.0-cp312-cp312-manylinux1_x86_64.manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:14f87e04d613dfa218d6135e81b78272c3b904e2a7053b841481b38a7d901411", size = 33364, upload-time = "2025-12-16T00:40:22.96Z" }, - { url = "https://files.pythonhosted.org/packages/21/3f/3457ea803db0198c9aaca2dd373750972ce28a26f00544b6b85088811939/google_crc32c-1.8.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:cb5c869c2923d56cb0c8e6bcdd73c009c36ae39b652dbe46a05eb4ef0ad01454", size = 33740, upload-time = "2025-12-16T00:40:23.96Z" }, - { url = "https://files.pythonhosted.org/packages/df/c0/87c2073e0c72515bb8733d4eef7b21548e8d189f094b5dad20b0ecaf64f6/google_crc32c-1.8.0-cp312-cp312-win_amd64.whl", hash = "sha256:3cc0c8912038065eafa603b238abf252e204accab2a704c63b9e14837a854962", size = 34437, upload-time = "2025-12-16T00:35:21.395Z" }, - { url = "https://files.pythonhosted.org/packages/d1/db/000f15b41724589b0e7bc24bc7a8967898d8d3bc8caf64c513d91ef1f6c0/google_crc32c-1.8.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:3ebb04528e83b2634857f43f9bb8ef5b2bbe7f10f140daeb01b58f972d04736b", size = 31297, upload-time = "2025-12-16T00:23:20.709Z" }, - { url = "https://files.pythonhosted.org/packages/d7/0d/8ebed0c39c53a7e838e2a486da8abb0e52de135f1b376ae2f0b160eb4c1a/google_crc32c-1.8.0-cp313-cp313-macosx_12_0_x86_64.whl", hash = "sha256:450dc98429d3e33ed2926fc99ee81001928d63460f8538f21a5d6060912a8e27", size = 30867, upload-time = "2025-12-16T00:43:14.628Z" }, - { url = "https://files.pythonhosted.org/packages/ce/42/b468aec74a0354b34c8cbf748db20d6e350a68a2b0912e128cabee49806c/google_crc32c-1.8.0-cp313-cp313-manylinux1_x86_64.manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:3b9776774b24ba76831609ffbabce8cdf6fa2bd5e9df37b594221c7e333a81fa", size = 33344, upload-time = "2025-12-16T00:40:24.742Z" }, - { url = "https://files.pythonhosted.org/packages/1c/e8/b33784d6fc77fb5062a8a7854e43e1e618b87d5ddf610a88025e4de6226e/google_crc32c-1.8.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:89c17d53d75562edfff86679244830599ee0a48efc216200691de8b02ab6b2b8", size = 33694, upload-time = "2025-12-16T00:40:25.505Z" }, - { url = "https://files.pythonhosted.org/packages/92/b1/d3cbd4d988afb3d8e4db94ca953df429ed6db7282ed0e700d25e6c7bfc8d/google_crc32c-1.8.0-cp313-cp313-win_amd64.whl", hash = "sha256:57a50a9035b75643996fbf224d6661e386c7162d1dfdab9bc4ca790947d1007f", size = 34435, upload-time = "2025-12-16T00:35:22.107Z" }, - { url = "https://files.pythonhosted.org/packages/21/88/8ecf3c2b864a490b9e7010c84fd203ec8cf3b280651106a3a74dd1b0ca72/google_crc32c-1.8.0-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:e6584b12cb06796d285d09e33f63309a09368b9d806a551d8036a4207ea43697", size = 31301, upload-time = "2025-12-16T00:24:48.527Z" }, - { url = "https://files.pythonhosted.org/packages/36/c6/f7ff6c11f5ca215d9f43d3629163727a272eabc356e5c9b2853df2bfe965/google_crc32c-1.8.0-cp314-cp314-macosx_12_0_x86_64.whl", hash = "sha256:f4b51844ef67d6cf2e9425983274da75f18b1597bb2c998e1c0a0e8d46f8f651", size = 30868, upload-time = "2025-12-16T00:48:12.163Z" }, - { url = "https://files.pythonhosted.org/packages/56/15/c25671c7aad70f8179d858c55a6ae8404902abe0cdcf32a29d581792b491/google_crc32c-1.8.0-cp314-cp314-manylinux1_x86_64.manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:b0d1a7afc6e8e4635564ba8aa5c0548e3173e41b6384d7711a9123165f582de2", size = 33381, upload-time = "2025-12-16T00:40:26.268Z" }, - { url = "https://files.pythonhosted.org/packages/42/fa/f50f51260d7b0ef5d4898af122d8a7ec5a84e2984f676f746445f783705f/google_crc32c-1.8.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:8b3f68782f3cbd1bce027e48768293072813469af6a61a86f6bb4977a4380f21", size = 33734, upload-time = "2025-12-16T00:40:27.028Z" }, - { url = "https://files.pythonhosted.org/packages/08/a5/7b059810934a09fb3ccb657e0843813c1fee1183d3bc2c8041800374aa2c/google_crc32c-1.8.0-cp314-cp314-win_amd64.whl", hash = "sha256:d511b3153e7011a27ab6ee6bb3a5404a55b994dc1a7322c0b87b29606d9790e2", size = 34878, upload-time = "2025-12-16T00:35:23.142Z" }, -] - [[package]] name = "graphviz" version = "0.21" @@ -1802,6 +1783,32 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/9e/c9/b2622292ea83fbb4ec318f5b9ab867d0a28ab43c5717bb85b0a5f6b3b0a4/networkx-3.6.1-py3-none-any.whl", hash = "sha256:d47fbf302e7d9cbbb9e2555a0d267983d2aa476bac30e90dfbe5669bd57f3762", size = 2068504, upload-time = "2025-12-08T17:02:38.159Z" }, ] +[[package]] +name = "ninja" +version = "1.13.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/43/73/79a0b22fc731989c708068427579e840a6cf4e937fe7ae5c5d0b7356ac22/ninja-1.13.0.tar.gz", hash = "sha256:4a40ce995ded54d9dc24f8ea37ff3bf62ad192b547f6c7126e7e25045e76f978", size = 242558, upload-time = "2025-08-11T15:10:19.421Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/3c/74/d02409ed2aa865e051b7edda22ad416a39d81a84980f544f8de717cab133/ninja-1.13.0-py3-none-macosx_10_9_universal2.whl", hash = "sha256:fa2a8bfc62e31b08f83127d1613d10821775a0eb334197154c4d6067b7068ff1", size = 310125, upload-time = "2025-08-11T15:09:50.971Z" }, + { url = "https://files.pythonhosted.org/packages/8e/de/6e1cd6b84b412ac1ef327b76f0641aeb5dcc01e9d3f9eee0286d0c34fd93/ninja-1.13.0-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:3d00c692fb717fd511abeb44b8c5d00340c36938c12d6538ba989fe764e79630", size = 177467, upload-time = "2025-08-11T15:09:52.767Z" }, + { url = "https://files.pythonhosted.org/packages/c8/83/49320fb6e58ae3c079381e333575fdbcf1cca3506ee160a2dcce775046fa/ninja-1.13.0-py3-none-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:be7f478ff9f96a128b599a964fc60a6a87b9fa332ee1bd44fa243ac88d50291c", size = 187834, upload-time = "2025-08-11T15:09:54.115Z" }, + { url = "https://files.pythonhosted.org/packages/56/c7/ba22748fb59f7f896b609cd3e568d28a0a367a6d953c24c461fe04fc4433/ninja-1.13.0-py3-none-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:60056592cf495e9a6a4bea3cd178903056ecb0943e4de45a2ea825edb6dc8d3e", size = 202736, upload-time = "2025-08-11T15:09:55.745Z" }, + { url = "https://files.pythonhosted.org/packages/79/22/d1de07632b78ac8e6b785f41fa9aad7a978ec8c0a1bf15772def36d77aac/ninja-1.13.0-py3-none-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:1c97223cdda0417f414bf864cfb73b72d8777e57ebb279c5f6de368de0062988", size = 179034, upload-time = "2025-08-11T15:09:57.394Z" }, + { url = "https://files.pythonhosted.org/packages/ed/de/0e6edf44d6a04dabd0318a519125ed0415ce437ad5a1ec9b9be03d9048cf/ninja-1.13.0-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:fb46acf6b93b8dd0322adc3a4945452a4e774b75b91293bafcc7b7f8e6517dfa", size = 180716, upload-time = "2025-08-11T15:09:58.696Z" }, + { url = "https://files.pythonhosted.org/packages/54/28/938b562f9057aaa4d6bfbeaa05e81899a47aebb3ba6751e36c027a7f5ff7/ninja-1.13.0-py3-none-manylinux_2_28_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:4be9c1b082d244b1ad7ef41eb8ab088aae8c109a9f3f0b3e56a252d3e00f42c1", size = 146843, upload-time = "2025-08-11T15:10:00.046Z" }, + { url = "https://files.pythonhosted.org/packages/2a/fb/d06a3838de4f8ab866e44ee52a797b5491df823901c54943b2adb0389fbb/ninja-1.13.0-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:6739d3352073341ad284246f81339a384eec091d9851a886dfa5b00a6d48b3e2", size = 154402, upload-time = "2025-08-11T15:10:01.657Z" }, + { url = "https://files.pythonhosted.org/packages/31/bf/0d7808af695ceddc763cf251b84a9892cd7f51622dc8b4c89d5012779f06/ninja-1.13.0-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:11be2d22027bde06f14c343f01d31446747dbb51e72d00decca2eb99be911e2f", size = 552388, upload-time = "2025-08-11T15:10:03.349Z" }, + { url = "https://files.pythonhosted.org/packages/9d/70/c99d0c2c809f992752453cce312848abb3b1607e56d4cd1b6cded317351a/ninja-1.13.0-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:aa45b4037b313c2f698bc13306239b8b93b4680eb47e287773156ac9e9304714", size = 472501, upload-time = "2025-08-11T15:10:04.735Z" }, + { url = "https://files.pythonhosted.org/packages/9f/43/c217b1153f0e499652f5e0766da8523ce3480f0a951039c7af115e224d55/ninja-1.13.0-py3-none-musllinux_1_2_i686.whl", hash = "sha256:5f8e1e8a1a30835eeb51db05cf5a67151ad37542f5a4af2a438e9490915e5b72", size = 638280, upload-time = "2025-08-11T15:10:06.512Z" }, + { url = "https://files.pythonhosted.org/packages/8c/45/9151bba2c8d0ae2b6260f71696330590de5850e5574b7b5694dce6023e20/ninja-1.13.0-py3-none-musllinux_1_2_ppc64le.whl", hash = "sha256:3d7d7779d12cb20c6d054c61b702139fd23a7a964ec8f2c823f1ab1b084150db", size = 642420, upload-time = "2025-08-11T15:10:08.35Z" }, + { url = "https://files.pythonhosted.org/packages/3c/fb/95752eb635bb8ad27d101d71bef15bc63049de23f299e312878fc21cb2da/ninja-1.13.0-py3-none-musllinux_1_2_riscv64.whl", hash = "sha256:d741a5e6754e0bda767e3274a0f0deeef4807f1fec6c0d7921a0244018926ae5", size = 585106, upload-time = "2025-08-11T15:10:09.818Z" }, + { url = "https://files.pythonhosted.org/packages/c1/31/aa56a1a286703800c0cbe39fb4e82811c277772dc8cd084f442dd8e2938a/ninja-1.13.0-py3-none-musllinux_1_2_s390x.whl", hash = "sha256:e8bad11f8a00b64137e9b315b137d8bb6cbf3086fbdc43bf1f90fd33324d2e96", size = 707138, upload-time = "2025-08-11T15:10:11.366Z" }, + { url = "https://files.pythonhosted.org/packages/34/6f/5f5a54a1041af945130abdb2b8529cbef0cdcbbf9bcf3f4195378319d29a/ninja-1.13.0-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:b4f2a072db3c0f944c32793e91532d8948d20d9ab83da9c0c7c15b5768072200", size = 581758, upload-time = "2025-08-11T15:10:13.295Z" }, + { url = "https://files.pythonhosted.org/packages/95/97/51359c77527d45943fe7a94d00a3843b81162e6c4244b3579fe8fc54cb9c/ninja-1.13.0-py3-none-win32.whl", hash = "sha256:8cfbb80b4a53456ae8a39f90ae3d7a2129f45ea164f43fadfa15dc38c4aef1c9", size = 267201, upload-time = "2025-08-11T15:10:15.158Z" }, + { url = "https://files.pythonhosted.org/packages/29/45/c0adfbfb0b5895aa18cec400c535b4f7ff3e52536e0403602fc1a23f7de9/ninja-1.13.0-py3-none-win_amd64.whl", hash = "sha256:fb8ee8719f8af47fed145cced4a85f0755dd55d45b2bddaf7431fa89803c5f3e", size = 309975, upload-time = "2025-08-11T15:10:16.697Z" }, + { url = "https://files.pythonhosted.org/packages/df/93/a7b983643d1253bb223234b5b226e69de6cda02b76cdca7770f684b795f5/ninja-1.13.0-py3-none-win_arm64.whl", hash = "sha256:3c0b40b1f0bba764644385319028650087b4c1b18cdfa6f45cb39a3669b81aa9", size = 290806, upload-time = "2025-08-11T15:10:18.018Z" }, +] + [[package]] name = "nodeenv" version = "1.9.1" @@ -2098,6 +2105,17 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/20/12/38679034af332785aac8774540895e234f4d07f7545804097de4b666afd8/packaging-25.0-py3-none-any.whl", hash = "sha256:29572ef2b1f17581046b3a2227d5c611fb25ec70ca1ba8554b24b0e69331a484", size = 66469, upload-time = "2025-04-19T11:48:57.875Z" }, ] +[[package]] +name = "packboost" +version = "1.0" +source = { git = "https://github.com/Pranshu-Bahadur/PackBoost.git#26992486ee99dfd86fd2145e2c9b574036ea277e" } +dependencies = [ + { name = "numpy" }, + { name = "pandas" }, + { name = "scikit-learn" }, + { name = "torch" }, +] + [[package]] name = "pandas" version = "2.3.2" @@ -2141,18 +2159,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/16/32/f8e3c85d1d5250232a5d3477a2a28cc291968ff175caeadaf3cc19ce0e4a/parso-0.8.5-py2.py3-none-any.whl", hash = "sha256:646204b5ee239c396d040b90f9e272e9a8017c630092bf59980beb62fd033887", size = 106668, upload-time = "2025-08-23T15:15:25.663Z" }, ] -[[package]] -name = "password-strength" -version = "0.0.3.post2" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "six" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/db/f1/6165ebcca27fca3f1d63f8c3a45805c2ed8568be4d09219a2aa45e792c14/password_strength-0.0.3.post2.tar.gz", hash = "sha256:bf4df10a58fcd3abfa182367307b4fd7b1cec518121dd83bf80c1c42ba796762", size = 12857, upload-time = "2019-01-04T13:41:29.711Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/1c/d6/08fd888c980589e4e27c2a4177e972481e8881600138e63afb785fe52630/password_strength-0.0.3.post2-py2.py3-none-any.whl", hash = "sha256:6739357c2863d707b7c7f247ff7c6882a70904a18d12c9aaf98f8b95da176fb9", size = 12167, upload-time = "2019-01-04T13:41:27.497Z" }, -] - [[package]] name = "pathspec" version = "0.12.1" @@ -3142,14 +3148,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/46/2c/9664130905f03db57961b8980b05cab624afd114bf2be2576628a9f22da4/sqlalchemy-2.0.48-py3-none-any.whl", hash = "sha256:a66fe406437dd65cacd96a72689a3aaaecaebbcd62d81c5ac1c0fdbeac835096", size = 1940202, upload-time = "2026-03-02T15:52:43.285Z" }, ] -[[package]] -name = "sseclient-py" -version = "1.9.0" -source = { registry = "https://pypi.org/simple" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/4d/2e/59920f7d66b7f9932a3d83dd0ec53fab001be1e058bf582606fe414a5198/sseclient_py-1.9.0-py3-none-any.whl", hash = "sha256:340062b1587fc2880892811e2ab5b176d98ef3eee98b3672ff3a3ba1e8ed0f6f", size = 8351, upload-time = "2026-01-02T23:39:30.995Z" }, -] - [[package]] name = "stack-data" version = "0.6.3" @@ -3268,32 +3266,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/1f/13/3f7fbe98e6ff733cc1a5bdb62a72130cee9b9a57e4438c8f008249c1fe01/tabpfn-7.1.1-py3-none-any.whl", hash = "sha256:7482516cc9be151e849353bb1e1d65a0fdcefa0a41e3c9e635f34da0c7a8a5bd", size = 660580, upload-time = "2026-04-09T12:05:53.611Z" }, ] -[[package]] -name = "tabpfn-client" -version = "0.3.0" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "backoff" }, - { name = "google-crc32c" }, - { name = "httpx" }, - { name = "omegaconf" }, - { name = "pandas" }, - { name = "password-strength" }, - { name = "pyarrow" }, - { name = "pydantic" }, - { name = "rich" }, - { name = "scikit-learn" }, - { name = "sseclient-py" }, - { name = "tabpfn-common-utils" }, - { name = "tqdm" }, - { name = "typing-extensions" }, - { name = "xxhash" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/2f/72/17691b5e0b2f414a8749f771a17432d85f6724a3a6ac4968bdaa36aa7d8f/tabpfn_client-0.3.0.tar.gz", hash = "sha256:bf81c9ebe7687e900ca3c50be2ac57c7ae5839897e384c91438a7275dc6e6e4b", size = 186983, upload-time = "2026-05-11T16:32:06.47Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/81/ed/8546c347f442e838b72bffb48a3df36cc9de633a7928a38b4356f32c0ac5/tabpfn_client-0.3.0-py3-none-any.whl", hash = "sha256:b96d06c402b75c03f1f9182fca57ca98bc3a67cc68ae0cd336b8465f633d3679", size = 46643, upload-time = "2026-05-11T16:32:04.829Z" }, -] - [[package]] name = "tabpfn-common-utils" version = "0.2.19" @@ -3745,89 +3717,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/43/80/0b5a2dfcf5b4da27b0b68d2833f05d77e1a374d43db951fca200a1f12a52/xgboost-2.1.4-py3-none-win_amd64.whl", hash = "sha256:8bbfe4fedc151b83a52edbf0de945fd94358b09a81998f2945ad330fd5f20cd6", size = 124910381, upload-time = "2025-02-06T18:17:43.202Z" }, ] -[[package]] -name = "xxhash" -version = "3.6.0" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/02/84/30869e01909fb37a6cc7e18688ee8bf1e42d57e7e0777636bd47524c43c7/xxhash-3.6.0.tar.gz", hash = "sha256:f0162a78b13a0d7617b2845b90c763339d1f1d82bb04a4b07f4ab535cc5e05d6", size = 85160, upload-time = "2025-10-02T14:37:08.097Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/9a/07/d9412f3d7d462347e4511181dea65e47e0d0e16e26fbee2ea86a2aefb657/xxhash-3.6.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:01362c4331775398e7bb34e3ab403bc9ee9f7c497bc7dee6272114055277dd3c", size = 32744, upload-time = "2025-10-02T14:34:34.622Z" }, - { url = "https://files.pythonhosted.org/packages/79/35/0429ee11d035fc33abe32dca1b2b69e8c18d236547b9a9b72c1929189b9a/xxhash-3.6.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:b7b2df81a23f8cb99656378e72501b2cb41b1827c0f5a86f87d6b06b69f9f204", size = 30816, upload-time = "2025-10-02T14:34:36.043Z" }, - { url = "https://files.pythonhosted.org/packages/b7/f2/57eb99aa0f7d98624c0932c5b9a170e1806406cdbcdb510546634a1359e0/xxhash-3.6.0-cp312-cp312-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:dc94790144e66b14f67b10ac8ed75b39ca47536bf8800eb7c24b50271ea0c490", size = 194035, upload-time = "2025-10-02T14:34:37.354Z" }, - { url = "https://files.pythonhosted.org/packages/4c/ed/6224ba353690d73af7a3f1c7cdb1fc1b002e38f783cb991ae338e1eb3d79/xxhash-3.6.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:93f107c673bccf0d592cdba077dedaf52fe7f42dcd7676eba1f6d6f0c3efffd2", size = 212914, upload-time = "2025-10-02T14:34:38.6Z" }, - { url = "https://files.pythonhosted.org/packages/38/86/fb6b6130d8dd6b8942cc17ab4d90e223653a89aa32ad2776f8af7064ed13/xxhash-3.6.0-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:2aa5ee3444c25b69813663c9f8067dcfaa2e126dc55e8dddf40f4d1c25d7effa", size = 212163, upload-time = "2025-10-02T14:34:39.872Z" }, - { url = "https://files.pythonhosted.org/packages/ee/dc/e84875682b0593e884ad73b2d40767b5790d417bde603cceb6878901d647/xxhash-3.6.0-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:f7f99123f0e1194fa59cc69ad46dbae2e07becec5df50a0509a808f90a0f03f0", size = 445411, upload-time = "2025-10-02T14:34:41.569Z" }, - { url = "https://files.pythonhosted.org/packages/11/4f/426f91b96701ec2f37bb2b8cec664eff4f658a11f3fa9d94f0a887ea6d2b/xxhash-3.6.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:49e03e6fe2cac4a1bc64952dd250cf0dbc5ef4ebb7b8d96bce82e2de163c82a2", size = 193883, upload-time = "2025-10-02T14:34:43.249Z" }, - { url = "https://files.pythonhosted.org/packages/53/5a/ddbb83eee8e28b778eacfc5a85c969673e4023cdeedcfcef61f36731610b/xxhash-3.6.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:bd17fede52a17a4f9a7bc4472a5867cb0b160deeb431795c0e4abe158bc784e9", size = 210392, upload-time = "2025-10-02T14:34:45.042Z" }, - { url = "https://files.pythonhosted.org/packages/1e/c2/ff69efd07c8c074ccdf0a4f36fcdd3d27363665bcdf4ba399abebe643465/xxhash-3.6.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:6fb5f5476bef678f69db04f2bd1efbed3030d2aba305b0fc1773645f187d6a4e", size = 197898, upload-time = "2025-10-02T14:34:46.302Z" }, - { url = "https://files.pythonhosted.org/packages/58/ca/faa05ac19b3b622c7c9317ac3e23954187516298a091eb02c976d0d3dd45/xxhash-3.6.0-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:843b52f6d88071f87eba1631b684fcb4b2068cd2180a0224122fe4ef011a9374", size = 210655, upload-time = "2025-10-02T14:34:47.571Z" }, - { url = "https://files.pythonhosted.org/packages/d4/7a/06aa7482345480cc0cb597f5c875b11a82c3953f534394f620b0be2f700c/xxhash-3.6.0-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:7d14a6cfaf03b1b6f5f9790f76880601ccc7896aff7ab9cd8978a939c1eb7e0d", size = 414001, upload-time = "2025-10-02T14:34:49.273Z" }, - { url = "https://files.pythonhosted.org/packages/23/07/63ffb386cd47029aa2916b3d2f454e6cc5b9f5c5ada3790377d5430084e7/xxhash-3.6.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:418daf3db71e1413cfe211c2f9a528456936645c17f46b5204705581a45390ae", size = 191431, upload-time = "2025-10-02T14:34:50.798Z" }, - { url = "https://files.pythonhosted.org/packages/0f/93/14fde614cadb4ddf5e7cebf8918b7e8fac5ae7861c1875964f17e678205c/xxhash-3.6.0-cp312-cp312-win32.whl", hash = "sha256:50fc255f39428a27299c20e280d6193d8b63b8ef8028995323bf834a026b4fbb", size = 30617, upload-time = "2025-10-02T14:34:51.954Z" }, - { url = "https://files.pythonhosted.org/packages/13/5d/0d125536cbe7565a83d06e43783389ecae0c0f2ed037b48ede185de477c0/xxhash-3.6.0-cp312-cp312-win_amd64.whl", hash = "sha256:c0f2ab8c715630565ab8991b536ecded9416d615538be8ecddce43ccf26cbc7c", size = 31534, upload-time = "2025-10-02T14:34:53.276Z" }, - { url = "https://files.pythonhosted.org/packages/54/85/6ec269b0952ec7e36ba019125982cf11d91256a778c7c3f98a4c5043d283/xxhash-3.6.0-cp312-cp312-win_arm64.whl", hash = "sha256:eae5c13f3bc455a3bbb68bdc513912dc7356de7e2280363ea235f71f54064829", size = 27876, upload-time = "2025-10-02T14:34:54.371Z" }, - { url = "https://files.pythonhosted.org/packages/33/76/35d05267ac82f53ae9b0e554da7c5e281ee61f3cad44c743f0fcd354f211/xxhash-3.6.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:599e64ba7f67472481ceb6ee80fa3bd828fd61ba59fb11475572cc5ee52b89ec", size = 32738, upload-time = "2025-10-02T14:34:55.839Z" }, - { url = "https://files.pythonhosted.org/packages/31/a8/3fbce1cd96534a95e35d5120637bf29b0d7f5d8fa2f6374e31b4156dd419/xxhash-3.6.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:7d8b8aaa30fca4f16f0c84a5c8d7ddee0e25250ec2796c973775373257dde8f1", size = 30821, upload-time = "2025-10-02T14:34:57.219Z" }, - { url = "https://files.pythonhosted.org/packages/0c/ea/d387530ca7ecfa183cb358027f1833297c6ac6098223fd14f9782cd0015c/xxhash-3.6.0-cp313-cp313-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:d597acf8506d6e7101a4a44a5e428977a51c0fadbbfd3c39650cca9253f6e5a6", size = 194127, upload-time = "2025-10-02T14:34:59.21Z" }, - { url = "https://files.pythonhosted.org/packages/ba/0c/71435dcb99874b09a43b8d7c54071e600a7481e42b3e3ce1eb5226a5711a/xxhash-3.6.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:858dc935963a33bc33490128edc1c12b0c14d9c7ebaa4e387a7869ecc4f3e263", size = 212975, upload-time = "2025-10-02T14:35:00.816Z" }, - { url = "https://files.pythonhosted.org/packages/84/7a/c2b3d071e4bb4a90b7057228a99b10d51744878f4a8a6dd643c8bd897620/xxhash-3.6.0-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:ba284920194615cb8edf73bf52236ce2e1664ccd4a38fdb543506413529cc546", size = 212241, upload-time = "2025-10-02T14:35:02.207Z" }, - { url = "https://files.pythonhosted.org/packages/81/5f/640b6eac0128e215f177df99eadcd0f1b7c42c274ab6a394a05059694c5a/xxhash-3.6.0-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:4b54219177f6c6674d5378bd862c6aedf64725f70dd29c472eaae154df1a2e89", size = 445471, upload-time = "2025-10-02T14:35:03.61Z" }, - { url = "https://files.pythonhosted.org/packages/5e/1e/3c3d3ef071b051cc3abbe3721ffb8365033a172613c04af2da89d5548a87/xxhash-3.6.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:42c36dd7dbad2f5238950c377fcbf6811b1cdb1c444fab447960030cea60504d", size = 193936, upload-time = "2025-10-02T14:35:05.013Z" }, - { url = "https://files.pythonhosted.org/packages/2c/bd/4a5f68381939219abfe1c22a9e3a5854a4f6f6f3c4983a87d255f21f2e5d/xxhash-3.6.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:f22927652cba98c44639ffdc7aaf35828dccf679b10b31c4ad72a5b530a18eb7", size = 210440, upload-time = "2025-10-02T14:35:06.239Z" }, - { url = "https://files.pythonhosted.org/packages/eb/37/b80fe3d5cfb9faff01a02121a0f4d565eb7237e9e5fc66e73017e74dcd36/xxhash-3.6.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:b45fad44d9c5c119e9c6fbf2e1c656a46dc68e280275007bbfd3d572b21426db", size = 197990, upload-time = "2025-10-02T14:35:07.735Z" }, - { url = "https://files.pythonhosted.org/packages/d7/fd/2c0a00c97b9e18f72e1f240ad4e8f8a90fd9d408289ba9c7c495ed7dc05c/xxhash-3.6.0-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:6f2580ffab1a8b68ef2b901cde7e55fa8da5e4be0977c68f78fc80f3c143de42", size = 210689, upload-time = "2025-10-02T14:35:09.438Z" }, - { url = "https://files.pythonhosted.org/packages/93/86/5dd8076a926b9a95db3206aba20d89a7fc14dd5aac16e5c4de4b56033140/xxhash-3.6.0-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:40c391dd3cd041ebc3ffe6f2c862f402e306eb571422e0aa918d8070ba31da11", size = 414068, upload-time = "2025-10-02T14:35:11.162Z" }, - { url = "https://files.pythonhosted.org/packages/af/3c/0bb129170ee8f3650f08e993baee550a09593462a5cddd8e44d0011102b1/xxhash-3.6.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:f205badabde7aafd1a31e8ca2a3e5a763107a71c397c4481d6a804eb5063d8bd", size = 191495, upload-time = "2025-10-02T14:35:12.971Z" }, - { url = "https://files.pythonhosted.org/packages/e9/3a/6797e0114c21d1725e2577508e24006fd7ff1d8c0c502d3b52e45c1771d8/xxhash-3.6.0-cp313-cp313-win32.whl", hash = "sha256:2577b276e060b73b73a53042ea5bd5203d3e6347ce0d09f98500f418a9fcf799", size = 30620, upload-time = "2025-10-02T14:35:14.129Z" }, - { url = "https://files.pythonhosted.org/packages/86/15/9bc32671e9a38b413a76d24722a2bf8784a132c043063a8f5152d390b0f9/xxhash-3.6.0-cp313-cp313-win_amd64.whl", hash = "sha256:757320d45d2fbcce8f30c42a6b2f47862967aea7bf458b9625b4bbe7ee390392", size = 31542, upload-time = "2025-10-02T14:35:15.21Z" }, - { url = "https://files.pythonhosted.org/packages/39/c5/cc01e4f6188656e56112d6a8e0dfe298a16934b8c47a247236549a3f7695/xxhash-3.6.0-cp313-cp313-win_arm64.whl", hash = "sha256:457b8f85dec5825eed7b69c11ae86834a018b8e3df5e77783c999663da2f96d6", size = 27880, upload-time = "2025-10-02T14:35:16.315Z" }, - { url = "https://files.pythonhosted.org/packages/f3/30/25e5321c8732759e930c555176d37e24ab84365482d257c3b16362235212/xxhash-3.6.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:a42e633d75cdad6d625434e3468126c73f13f7584545a9cf34e883aa1710e702", size = 32956, upload-time = "2025-10-02T14:35:17.413Z" }, - { url = "https://files.pythonhosted.org/packages/9f/3c/0573299560d7d9f8ab1838f1efc021a280b5ae5ae2e849034ef3dee18810/xxhash-3.6.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:568a6d743219e717b07b4e03b0a828ce593833e498c3b64752e0f5df6bfe84db", size = 31072, upload-time = "2025-10-02T14:35:18.844Z" }, - { url = "https://files.pythonhosted.org/packages/7a/1c/52d83a06e417cd9d4137722693424885cc9878249beb3a7c829e74bf7ce9/xxhash-3.6.0-cp313-cp313t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:bec91b562d8012dae276af8025a55811b875baace6af510412a5e58e3121bc54", size = 196409, upload-time = "2025-10-02T14:35:20.31Z" }, - { url = "https://files.pythonhosted.org/packages/e3/8e/c6d158d12a79bbd0b878f8355432075fc82759e356ab5a111463422a239b/xxhash-3.6.0-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:78e7f2f4c521c30ad5e786fdd6bae89d47a32672a80195467b5de0480aa97b1f", size = 215736, upload-time = "2025-10-02T14:35:21.616Z" }, - { url = "https://files.pythonhosted.org/packages/bc/68/c4c80614716345d55071a396cf03d06e34b5f4917a467faf43083c995155/xxhash-3.6.0-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:3ed0df1b11a79856df5ffcab572cbd6b9627034c1c748c5566fa79df9048a7c5", size = 214833, upload-time = "2025-10-02T14:35:23.32Z" }, - { url = "https://files.pythonhosted.org/packages/7e/e9/ae27c8ffec8b953efa84c7c4a6c6802c263d587b9fc0d6e7cea64e08c3af/xxhash-3.6.0-cp313-cp313t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:0e4edbfc7d420925b0dd5e792478ed393d6e75ff8fc219a6546fb446b6a417b1", size = 448348, upload-time = "2025-10-02T14:35:25.111Z" }, - { url = "https://files.pythonhosted.org/packages/d7/6b/33e21afb1b5b3f46b74b6bd1913639066af218d704cc0941404ca717fc57/xxhash-3.6.0-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:fba27a198363a7ef87f8c0f6b171ec36b674fe9053742c58dd7e3201c1ab30ee", size = 196070, upload-time = "2025-10-02T14:35:26.586Z" }, - { url = "https://files.pythonhosted.org/packages/96/b6/fcabd337bc5fa624e7203aa0fa7d0c49eed22f72e93229431752bddc83d9/xxhash-3.6.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:794fe9145fe60191c6532fa95063765529770edcdd67b3d537793e8004cabbfd", size = 212907, upload-time = "2025-10-02T14:35:28.087Z" }, - { url = "https://files.pythonhosted.org/packages/4b/d3/9ee6160e644d660fcf176c5825e61411c7f62648728f69c79ba237250143/xxhash-3.6.0-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:6105ef7e62b5ac73a837778efc331a591d8442f8ef5c7e102376506cb4ae2729", size = 200839, upload-time = "2025-10-02T14:35:29.857Z" }, - { url = "https://files.pythonhosted.org/packages/0d/98/e8de5baa5109394baf5118f5e72ab21a86387c4f89b0e77ef3e2f6b0327b/xxhash-3.6.0-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:f01375c0e55395b814a679b3eea205db7919ac2af213f4a6682e01220e5fe292", size = 213304, upload-time = "2025-10-02T14:35:31.222Z" }, - { url = "https://files.pythonhosted.org/packages/7b/1d/71056535dec5c3177eeb53e38e3d367dd1d16e024e63b1cee208d572a033/xxhash-3.6.0-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:d706dca2d24d834a4661619dcacf51a75c16d65985718d6a7d73c1eeeb903ddf", size = 416930, upload-time = "2025-10-02T14:35:32.517Z" }, - { url = "https://files.pythonhosted.org/packages/dc/6c/5cbde9de2cd967c322e651c65c543700b19e7ae3e0aae8ece3469bf9683d/xxhash-3.6.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:5f059d9faeacd49c0215d66f4056e1326c80503f51a1532ca336a385edadd033", size = 193787, upload-time = "2025-10-02T14:35:33.827Z" }, - { url = "https://files.pythonhosted.org/packages/19/fa/0172e350361d61febcea941b0cc541d6e6c8d65d153e85f850a7b256ff8a/xxhash-3.6.0-cp313-cp313t-win32.whl", hash = "sha256:1244460adc3a9be84731d72b8e80625788e5815b68da3da8b83f78115a40a7ec", size = 30916, upload-time = "2025-10-02T14:35:35.107Z" }, - { url = "https://files.pythonhosted.org/packages/ad/e6/e8cf858a2b19d6d45820f072eff1bea413910592ff17157cabc5f1227a16/xxhash-3.6.0-cp313-cp313t-win_amd64.whl", hash = "sha256:b1e420ef35c503869c4064f4a2f2b08ad6431ab7b229a05cce39d74268bca6b8", size = 31799, upload-time = "2025-10-02T14:35:36.165Z" }, - { url = "https://files.pythonhosted.org/packages/56/15/064b197e855bfb7b343210e82490ae672f8bc7cdf3ddb02e92f64304ee8a/xxhash-3.6.0-cp313-cp313t-win_arm64.whl", hash = "sha256:ec44b73a4220623235f67a996c862049f375df3b1052d9899f40a6382c32d746", size = 28044, upload-time = "2025-10-02T14:35:37.195Z" }, - { url = "https://files.pythonhosted.org/packages/7e/5e/0138bc4484ea9b897864d59fce9be9086030825bc778b76cb5a33a906d37/xxhash-3.6.0-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:a40a3d35b204b7cc7643cbcf8c9976d818cb47befcfac8bbefec8038ac363f3e", size = 32754, upload-time = "2025-10-02T14:35:38.245Z" }, - { url = "https://files.pythonhosted.org/packages/18/d7/5dac2eb2ec75fd771957a13e5dda560efb2176d5203f39502a5fc571f899/xxhash-3.6.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:a54844be970d3fc22630b32d515e79a90d0a3ddb2644d8d7402e3c4c8da61405", size = 30846, upload-time = "2025-10-02T14:35:39.6Z" }, - { url = "https://files.pythonhosted.org/packages/fe/71/8bc5be2bb00deb5682e92e8da955ebe5fa982da13a69da5a40a4c8db12fb/xxhash-3.6.0-cp314-cp314-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:016e9190af8f0a4e3741343777710e3d5717427f175adfdc3e72508f59e2a7f3", size = 194343, upload-time = "2025-10-02T14:35:40.69Z" }, - { url = "https://files.pythonhosted.org/packages/e7/3b/52badfb2aecec2c377ddf1ae75f55db3ba2d321c5e164f14461c90837ef3/xxhash-3.6.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4f6f72232f849eb9d0141e2ebe2677ece15adfd0fa599bc058aad83c714bb2c6", size = 213074, upload-time = "2025-10-02T14:35:42.29Z" }, - { url = "https://files.pythonhosted.org/packages/a2/2b/ae46b4e9b92e537fa30d03dbc19cdae57ed407e9c26d163895e968e3de85/xxhash-3.6.0-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:63275a8aba7865e44b1813d2177e0f5ea7eadad3dd063a21f7cf9afdc7054063", size = 212388, upload-time = "2025-10-02T14:35:43.929Z" }, - { url = "https://files.pythonhosted.org/packages/f5/80/49f88d3afc724b4ac7fbd664c8452d6db51b49915be48c6982659e0e7942/xxhash-3.6.0-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:3cd01fa2aa00d8b017c97eb46b9a794fbdca53fc14f845f5a328c71254b0abb7", size = 445614, upload-time = "2025-10-02T14:35:45.216Z" }, - { url = "https://files.pythonhosted.org/packages/ed/ba/603ce3961e339413543d8cd44f21f2c80e2a7c5cfe692a7b1f2cccf58f3c/xxhash-3.6.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0226aa89035b62b6a86d3c68df4d7c1f47a342b8683da2b60cedcddb46c4d95b", size = 194024, upload-time = "2025-10-02T14:35:46.959Z" }, - { url = "https://files.pythonhosted.org/packages/78/d1/8e225ff7113bf81545cfdcd79eef124a7b7064a0bba53605ff39590b95c2/xxhash-3.6.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:c6e193e9f56e4ca4923c61238cdaced324f0feac782544eb4c6d55ad5cc99ddd", size = 210541, upload-time = "2025-10-02T14:35:48.301Z" }, - { url = "https://files.pythonhosted.org/packages/6f/58/0f89d149f0bad89def1a8dd38feb50ccdeb643d9797ec84707091d4cb494/xxhash-3.6.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:9176dcaddf4ca963d4deb93866d739a343c01c969231dbe21680e13a5d1a5bf0", size = 198305, upload-time = "2025-10-02T14:35:49.584Z" }, - { url = "https://files.pythonhosted.org/packages/11/38/5eab81580703c4df93feb5f32ff8fa7fe1e2c51c1f183ee4e48d4bb9d3d7/xxhash-3.6.0-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:c1ce4009c97a752e682b897aa99aef84191077a9433eb237774689f14f8ec152", size = 210848, upload-time = "2025-10-02T14:35:50.877Z" }, - { url = "https://files.pythonhosted.org/packages/5e/6b/953dc4b05c3ce678abca756416e4c130d2382f877a9c30a20d08ee6a77c0/xxhash-3.6.0-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:8cb2f4f679b01513b7adbb9b1b2f0f9cdc31b70007eaf9d59d0878809f385b11", size = 414142, upload-time = "2025-10-02T14:35:52.15Z" }, - { url = "https://files.pythonhosted.org/packages/08/a9/238ec0d4e81a10eb5026d4a6972677cbc898ba6c8b9dbaec12ae001b1b35/xxhash-3.6.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:653a91d7c2ab54a92c19ccf43508b6a555440b9be1bc8be553376778be7f20b5", size = 191547, upload-time = "2025-10-02T14:35:53.547Z" }, - { url = "https://files.pythonhosted.org/packages/f1/ee/3cf8589e06c2164ac77c3bf0aa127012801128f1feebf2a079272da5737c/xxhash-3.6.0-cp314-cp314-win32.whl", hash = "sha256:a756fe893389483ee8c394d06b5ab765d96e68fbbfe6fde7aa17e11f5720559f", size = 31214, upload-time = "2025-10-02T14:35:54.746Z" }, - { url = "https://files.pythonhosted.org/packages/02/5d/a19552fbc6ad4cb54ff953c3908bbc095f4a921bc569433d791f755186f1/xxhash-3.6.0-cp314-cp314-win_amd64.whl", hash = "sha256:39be8e4e142550ef69629c9cd71b88c90e9a5db703fecbcf265546d9536ca4ad", size = 32290, upload-time = "2025-10-02T14:35:55.791Z" }, - { url = "https://files.pythonhosted.org/packages/b1/11/dafa0643bc30442c887b55baf8e73353a344ee89c1901b5a5c54a6c17d39/xxhash-3.6.0-cp314-cp314-win_arm64.whl", hash = "sha256:25915e6000338999236f1eb68a02a32c3275ac338628a7eaa5a269c401995679", size = 28795, upload-time = "2025-10-02T14:35:57.162Z" }, - { url = "https://files.pythonhosted.org/packages/2c/db/0e99732ed7f64182aef4a6fb145e1a295558deec2a746265dcdec12d191e/xxhash-3.6.0-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:c5294f596a9017ca5a3e3f8884c00b91ab2ad2933cf288f4923c3fd4346cf3d4", size = 32955, upload-time = "2025-10-02T14:35:58.267Z" }, - { url = "https://files.pythonhosted.org/packages/55/f4/2a7c3c68e564a099becfa44bb3d398810cc0ff6749b0d3cb8ccb93f23c14/xxhash-3.6.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:1cf9dcc4ab9cff01dfbba78544297a3a01dafd60f3bde4e2bfd016cf7e4ddc67", size = 31072, upload-time = "2025-10-02T14:35:59.382Z" }, - { url = "https://files.pythonhosted.org/packages/c6/d9/72a29cddc7250e8a5819dad5d466facb5dc4c802ce120645630149127e73/xxhash-3.6.0-cp314-cp314t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:01262da8798422d0685f7cef03b2bd3f4f46511b02830861df548d7def4402ad", size = 196579, upload-time = "2025-10-02T14:36:00.838Z" }, - { url = "https://files.pythonhosted.org/packages/63/93/b21590e1e381040e2ca305a884d89e1c345b347404f7780f07f2cdd47ef4/xxhash-3.6.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:51a73fb7cb3a3ead9f7a8b583ffd9b8038e277cdb8cb87cf890e88b3456afa0b", size = 215854, upload-time = "2025-10-02T14:36:02.207Z" }, - { url = "https://files.pythonhosted.org/packages/ce/b8/edab8a7d4fa14e924b29be877d54155dcbd8b80be85ea00d2be3413a9ed4/xxhash-3.6.0-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:b9c6df83594f7df8f7f708ce5ebeacfc69f72c9fbaaababf6cf4758eaada0c9b", size = 214965, upload-time = "2025-10-02T14:36:03.507Z" }, - { url = "https://files.pythonhosted.org/packages/27/67/dfa980ac7f0d509d54ea0d5a486d2bb4b80c3f1bb22b66e6a05d3efaf6c0/xxhash-3.6.0-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:627f0af069b0ea56f312fd5189001c24578868643203bca1abbc2c52d3a6f3ca", size = 448484, upload-time = "2025-10-02T14:36:04.828Z" }, - { url = "https://files.pythonhosted.org/packages/8c/63/8ffc2cc97e811c0ca5d00ab36604b3ea6f4254f20b7bc658ca825ce6c954/xxhash-3.6.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:aa912c62f842dfd013c5f21a642c9c10cd9f4c4e943e0af83618b4a404d9091a", size = 196162, upload-time = "2025-10-02T14:36:06.182Z" }, - { url = "https://files.pythonhosted.org/packages/4b/77/07f0e7a3edd11a6097e990f6e5b815b6592459cb16dae990d967693e6ea9/xxhash-3.6.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:b465afd7909db30168ab62afe40b2fcf79eedc0b89a6c0ab3123515dc0df8b99", size = 213007, upload-time = "2025-10-02T14:36:07.733Z" }, - { url = "https://files.pythonhosted.org/packages/ae/d8/bc5fa0d152837117eb0bef6f83f956c509332ce133c91c63ce07ee7c4873/xxhash-3.6.0-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:a881851cf38b0a70e7c4d3ce81fc7afd86fbc2a024f4cfb2a97cf49ce04b75d3", size = 200956, upload-time = "2025-10-02T14:36:09.106Z" }, - { url = "https://files.pythonhosted.org/packages/26/a5/d749334130de9411783873e9b98ecc46688dad5db64ca6e04b02acc8b473/xxhash-3.6.0-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:9b3222c686a919a0f3253cfc12bb118b8b103506612253b5baeaac10d8027cf6", size = 213401, upload-time = "2025-10-02T14:36:10.585Z" }, - { url = "https://files.pythonhosted.org/packages/89/72/abed959c956a4bfc72b58c0384bb7940663c678127538634d896b1195c10/xxhash-3.6.0-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:c5aa639bc113e9286137cec8fadc20e9cd732b2cc385c0b7fa673b84fc1f2a93", size = 417083, upload-time = "2025-10-02T14:36:12.276Z" }, - { url = "https://files.pythonhosted.org/packages/0c/b3/62fd2b586283b7d7d665fb98e266decadf31f058f1cf6c478741f68af0cb/xxhash-3.6.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:5c1343d49ac102799905e115aee590183c3921d475356cb24b4de29a4bc56518", size = 193913, upload-time = "2025-10-02T14:36:14.025Z" }, - { url = "https://files.pythonhosted.org/packages/9a/9a/c19c42c5b3f5a4aad748a6d5b4f23df3bed7ee5445accc65a0fb3ff03953/xxhash-3.6.0-cp314-cp314t-win32.whl", hash = "sha256:5851f033c3030dd95c086b4a36a2683c2ff4a799b23af60977188b057e467119", size = 31586, upload-time = "2025-10-02T14:36:15.603Z" }, - { url = "https://files.pythonhosted.org/packages/03/d6/4cc450345be9924fd5dc8c590ceda1db5b43a0a889587b0ae81a95511360/xxhash-3.6.0-cp314-cp314t-win_amd64.whl", hash = "sha256:0444e7967dac37569052d2409b00a8860c2135cff05502df4da80267d384849f", size = 32526, upload-time = "2025-10-02T14:36:16.708Z" }, - { url = "https://files.pythonhosted.org/packages/0f/c9/7243eb3f9eaabd1a88a5a5acadf06df2d83b100c62684b7425c6a11bcaa8/xxhash-3.6.0-cp314-cp314t-win_arm64.whl", hash = "sha256:bb79b1e63f6fd84ec778a4b1916dfe0a7c3fdb986c06addd5db3a0d413819d95", size = 28898, upload-time = "2025-10-02T14:36:17.843Z" }, -] - [[package]] name = "yarl" version = "1.23.0"