Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 13 additions & 13 deletions docs/pages.jl
Original file line number Diff line number Diff line change
Expand Up @@ -4,22 +4,22 @@ pages = [
"Basics" => "tutorials.md",
"Radials" => "radials.md",
"Kriging" => "kriging.md",
"Gaussian Process" => "abstractgps.md",
"Gaussian process" => "abstractgps.md",
"Lobachevsky" => "lobachevsky.md",
"Linear" => "LinearSurrogate.md",
"InverseDistance" => "InverseDistance.md",
"Inverse distance" => "InverseDistance.md",
"XGBoost" => "xgboost.md",
"SecondOrderPolynomial" => "secondorderpoly.md",
"Second order polynomial" => "secondorderpoly.md",
"Earth (MARS)" => "earth.md",
"NeuralSurrogate" => "neural.md",
"Neural network" => "neural.md",
"Wendland" => "wendland.md",
"Polynomial Chaos" => "polychaos.md",
"Variable Fidelity" => "variablefidelity.md",
"Gradient Enhanced Kriging" => "gek.md",
"Polynomial chaos" => "polychaos.md",
"Variable fidelity" => "variablefidelity.md",
"Gradient enhanced Kriging" => "gek.md",
"GEKPLS" => "gekpls.md",
"KPLS and KPLSK" => "kpls.md",
"MOE" => "moe.md",
"Parallel Optimization" => "parallel.md",
"Parallel optimization" => "parallel.md",
]
"User guide" => [
"Samples" => "samples.md",
Expand All @@ -32,14 +32,14 @@ pages = [
"Lp norm" => "lp.md",
"Rosenbrock" => "rosenbrock.md",
"Tensor product" => "tensor_prod.md",
"Cantilever beam" => "cantilever.md",
"Water Flow function" => "water_flow.md",
"Cantilever beam function" => "cantilever.md",
"Water flow function" => "water_flow.md",
"Welded beam function" => "welded_beam.md",
"Branin function" => "BraninFunction.md",
"Improved Branin function" => "ImprovedBraninFunction.md",
"Ackley function" => "ackley.md",
"Gramacy & Lee Function" => "gramacylee.md",
"Salustowicz Benchmark" => "Salustowicz.md",
"Multi objective optimization" => "multi_objective_opt.md",
"Gramacy & Lee function" => "gramacylee.md",
"Salustowicz benchmark" => "Salustowicz.md",
"Multi-objective optimization" => "multi_objective_opt.md",
]
]
19 changes: 10 additions & 9 deletions docs/src/BraninFunction.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,12 +40,13 @@ lower_bound = [-5, 0]
upper_bound = [10, 15]
xys = sample(n_samples, lower_bound, upper_bound, SobolSample())
zs = branin.(xys)
x, y = -5.00:10.00, 0.00:15.00
p1 = surface(x, y, (x1, x2) -> branin((x1, x2)))
xgrid = range(lower_bound[1], upper_bound[1], length = 100)
ygrid = range(lower_bound[2], upper_bound[2], length = 100)
p1 = surface(xgrid, ygrid, (x1, x2) -> branin((x1, x2)))
xs = [xy[1] for xy in xys]
ys = [xy[2] for xy in xys]
scatter!(xs, ys, zs)
p2 = contour(x, y, (x1, x2) -> branin((x1, x2)))
p2 = contour(xgrid, ygrid, (x1, x2) -> branin((x1, x2)))
scatter!(xs, ys)
plot(p1, p2, title = "True function")
```
Expand All @@ -58,9 +59,9 @@ radial_surrogate = RadialBasis(xys, zs, lower_bound, upper_bound)
```

```@example BraninFunction
p1 = surface(x, y, (x, y) -> radial_surrogate([x y]))
p1 = surface(xgrid, ygrid, (x, y) -> radial_surrogate([x y]))
scatter!(xs, ys, zs, marker_z = zs)
p2 = contour(x, y, (x, y) -> radial_surrogate([x y]))
p2 = contour(xgrid, ygrid, (x, y) -> radial_surrogate([x y]))
scatter!(xs, ys, marker_z = zs)
plot(p1, p2, title = "Radial Surrogate")
```
Expand All @@ -72,9 +73,9 @@ InverseDistance = InverseDistanceSurrogate(xys, zs, lower_bound, upper_bound)
```

```@example BraninFunction
p1 = surface(x, y, (x, y) -> InverseDistance([x y]))
p1 = surface(xgrid, ygrid, (x, y) -> InverseDistance([x y]))
scatter!(xs, ys, zs, marker_z = zs)
p2 = contour(x, y, (x, y) -> InverseDistance([x y]))
p2 = contour(xgrid, ygrid, (x, y) -> InverseDistance([x y]))
scatter!(xs, ys, marker_z = zs)
plot(p1, p2, title = "Inverse Distance Surrogate")
```
Expand All @@ -87,9 +88,9 @@ Lobachevsky = LobachevskySurrogate(
```

```@example BraninFunction
p1 = surface(x, y, (x, y) -> Lobachevsky([x y]))
p1 = surface(xgrid, ygrid, (x, y) -> Lobachevsky([x y]))
scatter!(xs, ys, zs, marker_z = zs)
p2 = contour(x, y, (x, y) -> Lobachevsky([x y]))
p2 = contour(xgrid, ygrid, (x, y) -> Lobachevsky([x y]))
scatter!(xs, ys, marker_z = zs)
plot(p1, p2, title = "Lobachevsky Surrogate")
```
67 changes: 46 additions & 21 deletions docs/src/ImprovedBraninFunction.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,26 @@
# Branin Function
# Improved Branin function

The Branin Function is commonly used as a test function for metamodelling in computer experiments, especially in the context of optimization.
The [Branin function](BraninFunction.md) is smooth, deterministic, and has three
global minima of equal value. That makes it a clean test, but an unrealistic one:
a real objective evaluated by simulation is rarely reproducible to the last bit.

## Modifications for Improved Branin Function:
The *improved* Branin function adds a time-varying perturbation inside the cosine
term, turning the landscape into something closer to what an optimizer meets in
practice:

To enhance the Branin function, changes were made to introduce irregularities, variability, and a dynamic aspect to its landscape. Here's an example:
```math
f(x_1, x_2, \tau) = \left(x_2 - \frac{5.1}{4\pi^2}x_1^2 + \frac{5}{\pi}x_1 - 6\right)^2
+ 10\left(1 - \frac{1}{8\pi}\right)\cos(x_1 + \tau\,\xi) + 10,
\qquad \xi \sim \mathcal{N}(0, 1)
```

where `time_step` is ``\tau``. Because ``\xi`` is drawn afresh on every call, the
objective is **stochastic**: two evaluations at the same point disagree.

```@example improved_branin
using Surrogates, Plots, Random
Random.seed!(42)

function improved_branin(x, time_step)
x1 = x[1]
x2 = x[2]
Expand All @@ -17,35 +31,46 @@ function improved_branin(x, time_step)
s = 10
t = 1 / (8 * pi)

# Adding noise to the function's output
noise = randn() * time_step # Simulating time-varying noise
# Time-varying perturbation of the phase, redrawn on every call.
noise = randn() * time_step
term1 = a * (x2 - b * x1^2 + c * x1 - r)^2
term2 = s * (1 - t) * cos(x1 + noise) # Introducing dynamic component
y = term1 + term2 + s
term2 = s * (1 - t) * cos(x1 + noise)
return term1 + term2 + s
end
```

This improved function now incorporates irregularities, variability, and a dynamic aspect. These changes aim to make the optimization landscape more challenging and realistic.
The seed matters here: without it, every run of this page — and every build of
these docs — produces a different surface.

## Using the Improved Branin Function:
```@example improved_branin
p = [2.5, 7.5]
repeats = [improved_branin(p, 0.1) for _ in 1:6]
extrema(repeats)
```

After defining the improved Branin function, you can proceed to test different surrogates and visualize their performance using the updated function. Here's an example of using the improved function with the Radial Basis surrogate:
## Fitting a surrogate

```@example improved_branin
using Surrogates, Plots

n_samples = 80
lower_bound = [-5, 0]
upper_bound = [10, 15]
lower_bound = [-5.0, 0.0]
upper_bound = [10.0, 15.0]
xys = sample(n_samples, lower_bound, upper_bound, SobolSample())
zs = [improved_branin(xy, 0.1) for xy in xys]
radial_surrogate = RadialBasis(xys, zs, lower_bound, upper_bound)
x, y = -5.00:10.00, 0.00:15.00
kriging_surrogate = Kriging(xys, zs, lower_bound, upper_bound)
```

```@example improved_branin
xgrid = range(lower_bound[1], upper_bound[1], length = 100)
ygrid = range(lower_bound[2], upper_bound[2], length = 100)
xs = [xy[1] for xy in xys]
ys = [xy[2] for xy in xys]
p1 = surface(x, y, (x, y) -> radial_surrogate([x, y]))
p1 = surface(xgrid, ygrid, (x, y) -> kriging_surrogate((x, y)))
scatter!(xs, ys, zs, marker_z = zs)
p2 = contour(xgrid, ygrid, (x, y) -> kriging_surrogate((x, y)))
scatter!(xs, ys, marker_z = zs)
p2 = contour(x, y, (x, y) -> radial_surrogate([x, y]))
scatter!(xs, ys, marker_z = zs)
plot(p1, p2, title = "Radial Surrogate")
plot(p1, p2, title = "Kriging surrogate")
```

Increasing `time_step` raises the noise level; at large enough values the fit is
dominated by the perturbation and the three basins of the original Branin function
stop being recoverable from this many samples.
27 changes: 15 additions & 12 deletions docs/src/InverseDistance.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# InverseDistance Surrogate Tutorial
# Inverse distance surrogate tutorial

The **Inverse Distance Surrogate** (Shepard's method) is an interpolating method, and in this method, the unknown points are calculated with a weighted average of the sampling points. This model uses the inverse distance between the unknown and training points to predict the unknown point. We do not need to fit this model because the response of an unknown point x is computed with respect to the distance between x and the training points.

Expand All @@ -19,6 +19,8 @@ using Surrogates
using Plots
```

## One dimension

### Sampling

We choose to sample f in 100 points between 0 and 10 using the `sample` function. The sampling points are chosen using a Low Discrepancy, this can be done by passing `HaltonSample()` to the `sample` function.
Expand All @@ -36,7 +38,7 @@ scatter(x, y, label = "Sampled points", xlims = (lower_bound, upper_bound), lege
plot!(f, label = "True function", xlims = (lower_bound, upper_bound), legend = :top)
```

## Building a Surrogate
### Building a Surrogate

```@example Inverse_Distance1D
InverseDistance = InverseDistanceSurrogate(x, y, lower_bound, upper_bound)
Expand Down Expand Up @@ -94,7 +96,7 @@ sampled response exactly.
maximum(abs(InverseDistance(x[i]) - y[i]) for i in eachindex(x))
```

## Optimizing
### Optimizing

Having built a surrogate, we can now use it to search for minima in our original function `f`.

Expand All @@ -109,7 +111,7 @@ plot!(InverseDistance, label = "Surrogate function",
xlims = (lower_bound, upper_bound), legend = :top)
```

## Inverse Distance Surrogate Tutorial (ND):
## Several dimensions

First of all we will define the `Schaffer` function we are going to build a surrogate for. Notice, how its argument is a vector of numbers, one for each coordinate, and its output is a scalar.

Expand Down Expand Up @@ -141,17 +143,18 @@ zs = schaffer.(xys);
```

```@example Inverse_DistanceND
x, y = -5:10, 0:15
p1 = surface(x, y, (x1, x2) -> schaffer((x1, x2)))
xgrid = range(lower_bound[1], upper_bound[1], length = 100)
ygrid = range(lower_bound[2], upper_bound[2], length = 100)
p1 = surface(xgrid, ygrid, (x1, x2) -> schaffer((x1, x2)))
xs = [xy[1] for xy in xys]
ys = [xy[2] for xy in xys]
scatter!(xs, ys, zs)
p2 = contour(x, y, (x1, x2) -> schaffer((x1, x2)))
p2 = contour(xgrid, ygrid, (x1, x2) -> schaffer((x1, x2)))
scatter!(xs, ys)
plot(p1, p2, title = "True function")
```

### Building a surrogate
### Building a Surrogate

Using the sampled points we build the surrogate, the steps are analogous to the 1-dimensional case.

Expand All @@ -160,9 +163,9 @@ InverseDistance = InverseDistanceSurrogate(xys, zs, lower_bound, upper_bound)
```

```@example Inverse_DistanceND
p1 = surface(x, y, (x, y) -> InverseDistance([x y]))
p1 = surface(xgrid, ygrid, (x, y) -> InverseDistance([x y]))
scatter!(xs, ys, zs, marker_z = zs)
p2 = contour(x, y, (x, y) -> InverseDistance([x y]))
p2 = contour(xgrid, ygrid, (x, y) -> InverseDistance([x y]))
scatter!(xs, ys, marker_z = zs)
plot(p1, p2, title = "Surrogate")
```
Expand All @@ -189,12 +192,12 @@ length(xys), length(InverseDistance.x)
```

```@example Inverse_DistanceND
p1 = surface(x, y, (x, y) -> InverseDistance([x y]))
p1 = surface(xgrid, ygrid, (x, y) -> InverseDistance([x y]))
xs = [xy[1] for xy in InverseDistance.x]
ys = [xy[2] for xy in InverseDistance.x]
zs = schaffer.(InverseDistance.x)
scatter!(xs, ys, zs, marker_z = zs)
p2 = contour(x, y, (x, y) -> InverseDistance([x y]))
p2 = contour(xgrid, ygrid, (x, y) -> InverseDistance([x y]))
scatter!(xs, ys, marker_z = zs)
plot(p1, p2)
```
16 changes: 9 additions & 7 deletions docs/src/LinearSurrogate.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
## Linear Surrogate
# Linear surrogate tutorial

The linear surrogate models a scalar response as an affine function of the
explanatory variables: an intercept plus one slope per input dimension, fitted
Expand All @@ -19,9 +19,11 @@ using Surrogates
using Plots
```

## One dimension

### Sampling

We choose to sample f in 100 points between 0 and 10 using the `sample` function. The sampling points are chosen using a Sobol sequence, this can be done by passing `SobolSample()` to the `sample` function.
We choose to sample f in 100 points between 5.2 and 12.5 using the `sample` function. The sampling points are chosen using a Sobol sequence, this can be done by passing `SobolSample()` to the `sample` function.

```@example linear_surrogate1D
f(x) = 2 * x + 10.0
Expand All @@ -34,7 +36,7 @@ scatter(x, y, label = "Sampled points", xlims = (lower_bound, upper_bound))
plot!(f, label = "True function", xlims = (lower_bound, upper_bound))
```

## Building a Surrogate
### Building a Surrogate

With our sampled points, we can build the **Linear Surrogate** using the `LinearSurrogate` function.

Expand All @@ -60,7 +62,7 @@ plot!(f, label = "True function", xlims = (lower_bound, upper_bound))
plot!(my_linear_surr_1D, label = "Surrogate function", xlims = (lower_bound, upper_bound))
```

## Optimizing
### Optimizing

Having built a surrogate, we can now use it to search for minima in our original function `f`.

Expand All @@ -74,7 +76,7 @@ plot!(f, label = "True function", xlims = (lower_bound, upper_bound))
plot!(my_linear_surr_1D, label = "Surrogate function", xlims = (lower_bound, upper_bound))
```

## Vector-valued responses
### Vector-valued responses

A single surrogate can model several outputs at once. Pass a vector of responses
per sample; the design matrix is shared, so each output is fitted independently
Expand All @@ -101,7 +103,7 @@ multi.coeff
multi(7.0), f(7.0)
```

## Linear Surrogate tutorial (ND)
## Several dimensions

A linear surrogate is the right tool when the response is dominated by a global
trend: it is cheap, needs few samples, and its coefficients are directly
Expand Down Expand Up @@ -137,7 +139,7 @@ y = otl_circuit.(x)
extrema(y)
```

### Building a surrogate
### Building a Surrogate

```@example linear_surrogateND
my_linear_ND = LinearSurrogate(x, y, lower_bound, upper_bound)
Expand Down
16 changes: 9 additions & 7 deletions docs/src/Salustowicz.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
## Salustowicz Benchmark Function
# Salustowicz Benchmark Function

The true underlying function HyGP had to approximate is the 1D Salustowicz function. The function can be evaluated in the given domain:
``x \in [0, 10]``.
The Salustowicz function is a one-dimensional benchmark, usually evaluated on
``x \in [0, 10]``. The ``e^{-x}x^3`` prefactor makes its amplitude vary by
several orders of magnitude across that domain — large oscillations near
``x \approx 3`` decaying to almost nothing by ``x = 10`` — so it separates
surrogates that adapt their length scale from those that do not.

The Salustowicz benchmark function is as follows:

Expand All @@ -18,9 +21,9 @@ Now, let's define our objective function:

```@example salustowicz1D
function salustowicz(x)
term1 = 2.72^(-x) * x^3 * cos(x) * sin(x)
term1 = exp(-x) * x^3 * cos(x) * sin(x)
term2 = (cos(x) * sin(x) * sin(x) - 1)
y = term1 * term2
return term1 * term2
end
```

Expand All @@ -30,7 +33,6 @@ Let's sample f in 30 points between 0 and 10 using the `sample` function. The sa
n_samples = 30
lower_bound = 0
upper_bound = 10
num_round = 2
x = sample(n_samples, lower_bound, upper_bound, SobolSample())
y = salustowicz.(x)
xs = lower_bound:0.001:upper_bound
Expand All @@ -51,7 +53,7 @@ plot!(xs, InverseDistance.(xs), label = "InverseDistanceSurrogate", legend = :to
plot!(xs, lobachevsky_surrogate.(xs), label = "Lobachevsky", legend = :topright)
```

Not's let's see Kriging Surrogate with different hyper parameter:
Now let's see the Kriging surrogate under different correlation exponents `p`. Smaller `p` gives rougher sample paths, and the ribbon shows the predicted standard error:

```@example salustowicz1D
kriging_surrogate1 = Kriging(x, y, lower_bound, upper_bound, p = 0.9);
Expand Down
Loading