Skip to content

refactor: Kriging based models - #606

Merged
ChrisRackauckas merged 12 commits into
SciML:masterfrom
sathvikbhagavan:sb/pr10-kriging
Sep 7, 2026
Merged

ChrisRackauckas merged 12 commits into
SciML:masterfrom
sathvikbhagavan:sb/pr10-kriging

Conversation

@sathvikbhagavan

@sathvikbhagavan sathvikbhagavan commented Sep 2, 2026 •

Copy link
Copy Markdown
Member

This PR is breaking. See Breaking changes below.

Rewrites the correctness-critical parts of Kriging, GEK, GEKPLS, KPLS and KPLSK. Several
of these models were not computing what their documentation claimed; the sections below state each
defect, what it cost, and how it was verified.

What was wrong

GEK was not a gradient-enhanced kriging model. Its d-dimensional value–value block was
exp(-Σ θ_l Δ_l) — the difference neither squared nor absolute-valued, so the matrix was not
symmetric, not bounded by 1, and not positive definite. Its derivative–derivative block was missing
the 2θ_l δ_lm term, leaving zeros on part of the diagonal where the true variance of ∂_l f is
2θ_l σ². And at prediction time it dropped theta from the kernel and summed only the first n
of n(1+d) weights, discarding every gradient. The model missed its own training points by
7.6e15.

The predictive variance squared the wrong residual. Both Kriging and GEK used
(1 - rᵀR⁻¹r)² where the ordinary-kriging MSE (Jones 2001, eq. 5) has (1 - 𝟙ᵀR⁻¹r)². The third
term is the variance from estimating the trend, so its numerator is the trend residual, not the
prediction residual. This feeds straight into every acquisition function in surrogate_optimize!.

theta was never fitted. It came from a fixed spread heuristic and stayed there. The
correlation scale is the parameter that determines everything about how a kriging model
interpolates.

Invalid input printed to stdout and returned nothing. Duplicate samples, out-of-bounds points,
mismatched theta lengths, and n_comp > d either produced an uncatchable diagnostic and a
nothing surrogate, or an opaque DimensionMismatch several frames deep, or — for n_comp > d — a
silently meaningless model whose correlation scales were fitted against all-zero PLS rotation
columns.

GEKPLS silently answered a different question. Passing several points returned the prediction
at the first one, with no error.

update! grew the caller's arrays. KPLS, KPLSK and GEKPLS all push!ed onto the vectors
the user had passed in.

What changed

  • GEK covariance rebuilt from the analytic derivatives of the Gaussian kernel, all four block
    types, verified exact to 1e-18 against ForwardDiff. p is now validated to be 2, since the
    derivative blocks are second derivatives that only exist for the Gaussian kernel.
  • Correct MSE in both models, with max(·, 0) instead of abs(·) so an ill-conditioned solve
    reports zero rather than a reflected error bar.
  • Maximum-likelihood fitting of theta in Kriging, GEK and (opt-in) GEKPLS, maximizing
    -n/2 log σ̂²(θ) - ½ log|R(θ)| with multi-start Nelder–Mead in log₁₀ θ over a ±5-decade
    relative box. Verified against a brute-force grid.
  • inv(R) → Cholesky throughout. inverse_of_R survives as a deprecated property.
  • Condition-number target raised from κ = 1e8 to 1e12, following Mohammadi et al.
    (arXiv:1602.00853); one eigvals call instead of two full eigensolves; LAPACKException handled
    so an extreme scale visited by the search cannot abort the fit.
  • GEKPLS nugget starts at 10·eps() and escalates only as far as the factorization needs,
    instead of a fixed 1e6·eps() on every problem — worth more than a factor of two in RMSE on
    welded beam. KPLS/KPLSK keep the old jitter, so their fits are unchanged.
  • Element types preserved. Float32 designs no longer promote to Float64 through unqualified
    literals; BigFloat exercises the generic path.
  • Consistent error signalling — ArgumentError for input the model cannot accept, @warn plus
    a no-op for a duplicate in update! (optimizers re-propose points near convergence, so erroring
    would break every optimization loop).
  • vcat instead of push! in every update!.

Tests

Kriging 138 → 149, GEK 116 → 119, GEKPLS 35 → 40, KPLS 5 → 47, KPLSK 5 → 50. All pass.

Tolerances tightened where the quantity is deterministic — interpolation at training points went
from < 1e-6 to < 1e-12, GEK values and derivatives from atol = 1e-2 to 1e-8. Several of
the bugs above survived because a tolerance of 1e-2 on an interpolant evaluated at its own
training points passes with almost anything.

New tests include independent-reference checks that share no code with the implementation: KPLS
and KPLSK predictions are compared against an ordinary kriging system rebuilt from the fitted
fields and solved directly, and the Kriging MSE against an augmented-Lagrangian solve of the BLUP
system.

Breaking changes

  1. inverse_of_R is no longer a field. It is a deprecated property that still returns R⁻¹
    with a depwarn; the Cholesky factorization lives in R_fact. Code that reads it keeps working.
    Code that constructs Kriging or GEK positionally will break — the field list changed.

  2. theta is fitted by default. Predictions change wherever theta was not supplied
    explicitly. They improved on every benchmark measured (by factors of 1.03 to 25.6 for Kriging,
    and 140× to 1500× for GEK on three of four problems), but they change. optimize_theta = false
    restores the previous behaviour of using the heuristic as given. An explicitly supplied theta
    is still honoured and is now also preserved across update!.

  3. Invalid input throws instead of returning nothing. Duplicate samples and out-of-bounds
    training points now raise ArgumentError from Kriging, GEK, KPLS, KPLSK and GEKPLS.
    Code that tested the return value for nothing needs a try/catch; code that did not was
    already broken further downstream.

  4. GEK predictions change completely, because the previous ones came from a covariance model
    that was not a covariance model. GEK now also requires p == 2 and rejects anything else,
    where the old default was p = 1.

  5. p = 0 is rejected for Kriging. The documented interval was the closed [0, 2]; at
    p = 0 every off-diagonal correlation is the constant exp(-θ), so R has rank 2 and is
    singular for more than two samples.

  6. n_comp is range-checked in KPLS and KPLSK: it must satisfy 1 ≤ n_comp ≤ d. Code that
    passed a larger value was building a model whose extra correlation scales were fitted against
    all-zero rotation columns.

  7. KPLS and KPLSK gained an optimize_theta::Bool field, so positional construction of
    those structs breaks. The new optimize_theta keyword defaults to true, which is the previous
    behaviour; false uses the supplied theta as given. Unlike Kriging and GEK the default
    cannot be derived from theta — there it is an optional keyword, here a required positional
    argument that is always supplied. For KPLSK, false skips both likelihood searches but still
    expands the reduced-space theta to full dimension, since that expansion is the model KPLSK fits
    rather than a search step. On a 300-point two-dimensional design the search costs 12.6s for
    KPLS and 17.2s for KPLSK, against milliseconds without it.

Made with claude code

@ChrisRackauckas
ChrisRackauckas merged commit 5f31874 into SciML:master Sep 7, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants