Repository navigation
Generalized ELPD (Beta-Divergence) for robust LOO cross-validation #2595
DJLacombeTTU
started this conversation in
Show and tell
Replies: 1 comment
|
I haven't had time to read the article yet, but it might be an interesting addition. |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Feature Request: Generalized ELPD (Beta-Divergence) for robust LOO cross-validation
Is your feature request related to a problem? Please describe.
Standard PSIS-LOO based on the logarithmic scoring rule is highly sensitive to influential observations and outliers. When dealing with heavy-tailed data or misspecified models, the standard expected log predictive density (ELPD) can be overly penalized by a few extreme points, leading to unstable model comparisons.
Describe the solution you'd like
I propose adding support for Generalized ELPD using beta-divergence (a type of Bregman divergence) to
az.loo. By allowing a tunablebetaparameter (typically between 1.01 and 1.05), we can smoothly downweight the influence of severe outliers, providing a more robust measure of predictive accuracy.This mathematical framework is formally detailed in:
Describe alternatives you've considered
Currently, researchers handling heavy outliers must either manually drop data points, switch to k-fold cross-validation (which is computationally expensive for large Bayesian models), or manually calculate the integral penalty outside of ArviZ.
Additional context
I have already implemented and tested this feature locally on my fork. The implementation:
score_type="beta"and abetakwarg to theloofunctions.posterior_predictivegroup to compute the necessary integral penalty usingxarray.integrate.psislwarchitecture.While running the test suite for this, I also caught and patched a
TypeError(unsupported operand type forintandNoneType) indiagnostics.pythat gets triggered when_get_r_effreturnsNoneon synthetic mock data.If the maintainers are open to this API approach, I have a clean, pre-commit-formatted branch with passing tests ready to submit as a Pull Request.
All reactions