gavindiaz/rust-optimizer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor(docs): Documentation Overhaul

Browse Source
This commit is contained in:
Manuel Raimann
2026-02-12 08:33:23 +01:00
parent 3581187cd3
commit 1f1e9d1779
56 changed files with 3298 additions and 1557 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/sampler/bohb.rs
+52 -6
View File
@@ -1,13 +1,13 @@
//! BOHB (Bayesian Optimization + `HyperBand`) sampler.
//!
//! BOHB combines TPE's model-guided sampling with Hyperband's budget-aware
//! BOHB combines TPE's model-guided sampling with `HyperBand`'s budget-aware
//! evaluation. Instead of building one global TPE model, BOHB conditions
//! its TPE model on trials evaluated at a specific budget level, giving
//! better-calibrated proposals for each rung of the Hyperband schedule.
//! better-calibrated proposals for each rung of the `HyperBand` schedule.
//!
//! # How it works
//!
//! 1. Compute all Hyperband rung steps (budget levels) from the config.
//! 1. Compute all `HyperBand` rung steps (budget levels) from the config.
//! 2. On each `sample()` call, scan the history's `intermediate_values`
//! to find the **largest budget level** with enough observations
//! (`>= min_points_in_model`).
@@ -16,6 +16,26 @@
//! 4. Delegate to an internal [`TpeSampler`] for the actual sampling.
//! 5. Fall back to random sampling if no budget level has enough data.
//!
//! # When to use
//!
//! - You are tuning hyperparameters for models that support early stopping
//! (e.g., neural networks with configurable epoch counts).
//! - You want to combine model-guided search with aggressive pruning of
//! unpromising configurations.
//! - Your objective has a natural "budget" axis (epochs, iterations, data
//! fraction) reported via [`Trial::report`](crate::Trial::report).
//!
//! Pair `BohbSampler` with [`matching_pruner`](BohbSampler::matching_pruner)
//! to get a `HyperBandPruner` whose budget schedule is consistent with
//! the sampler's conditioning levels.
//!
//! # Configuration
//!
//! - `min_resource` / `max_resource` — budget range (default: 1 … 81)
//! - `reduction_factor` (η) — successive halving factor (default: 3)
//! - `min_points_in_model` — observations needed before TPE replaces random (default: 10)
//! - All [`TpeSamplerBuilder`](super::tpe::TpeSamplerBuilder) options (gamma, seed, etc.)
//!
//! # Examples
//!
//! ```
@@ -27,7 +47,7 @@
//! let study: Study<f64> = Study::with_sampler_and_pruner(Direction::Minimize, bohb, pruner);
//! ```
//!
//! Using the builder for custom configuration:
//! Custom configuration via builder:
//!
//! ```
//! use optimizer::sampler::bohb::BohbSampler;
@@ -50,7 +70,7 @@ use crate::sampler::tpe::TpeSampler;
use crate::sampler::{CompletedTrial, Sampler};
use crate::types::Direction;
/// A BOHB sampler that combines TPE with Hyperband budget awareness.
/// A BOHB sampler that combines TPE with `HyperBand` budget awareness.
///
/// BOHB filters trial history by budget level before delegating to TPE,
/// so the surrogate model is conditioned on trials evaluated at the same
@@ -58,7 +78,25 @@ use crate::types::Direction;
/// than using a single global model across all budgets.
///
/// Use [`BohbSampler::matching_pruner`] to create a [`HyperbandPruner`]
/// with matching parameters.
/// with matching `HyperBand` parameters.
///
/// # Examples
///
/// ```
/// use optimizer::parameter::{FloatParam, Parameter};
/// use optimizer::sampler::bohb::BohbSampler;
/// use optimizer::{Direction, Study};
///
/// let bohb = BohbSampler::builder()
/// .min_resource(1)
/// .max_resource(27)
/// .reduction_factor(3)
/// .seed(42)
/// .build()
/// .unwrap();
/// let pruner = bohb.matching_pruner(Direction::Minimize);
/// let study: Study<f64> = Study::with_sampler_and_pruner(Direction::Minimize, bohb, pruner);
/// ```
pub struct BohbSampler {
min_resource: u64,
max_resource: u64,
@@ -225,6 +263,14 @@ impl Sampler for BohbSampler {
/// Builder for configuring a [`BohbSampler`].
///
/// # Defaults
///
/// - `min_resource`: 1
/// - `max_resource`: 81
/// - `reduction_factor`: 3 (η)
/// - `min_points_in_model`: 10
/// - TPE: default settings (gamma = 0.25, etc.)
///
/// # Examples
///
/// ```
src/sampler/cma_es.rs
+56 -10
View File
@@ -1,15 +1,55 @@
//! CMA-ES (Covariance Matrix Adaptation Evolution Strategy) sampler.
//!
//! CMA-ES maintains a multivariate Gaussian distribution over continuous
//! parameters and adapts its mean, covariance matrix, and step-size based
//! on trial rankings. It is one of the most effective derivative-free
//! optimizers for continuous search spaces.
//! CMA-ES is a stochastic, population-based optimizer that maintains a
//! multivariate Gaussian distribution over continuous parameters and adapts
//! its **mean**, **covariance matrix**, and **step-size** (σ) based on
//! trial rankings. It is widely regarded as one of the most effective
//! derivative-free optimizers for continuous search spaces.
//!
//! Categorical parameters are sampled uniformly at random (not part of
//! the CMA-ES vector). If all parameters are categorical, the sampler
//! falls back to pure random sampling.
//! # Algorithm overview
//!
//! Requires the `cma-es` feature flag.
//! Each generation:
//! 1. **Sample** λ (population size) candidates from N(m, σ²C).
//! 2. **Evaluate** and **rank** the candidates by objective value.
//! 3. **Update** the mean toward the best μ candidates (weighted recombination).
//! 4. **Adapt** the covariance matrix C via rank-one and rank-μ updates, and
//! adapt σ via cumulative step-size adaptation (CSA).
//!
//! Over time the search distribution narrows and rotates to align with the
//! landscape, efficiently exploiting structure in the objective function.
//!
//! # When to use
//!
//! - **Continuous parameters only** (float/int). Categorical parameters are
//! sampled uniformly at random and do not participate in the CMA-ES model.
//! - **Moderate dimensionality** — works well up to ~100 continuous dimensions.
//! Beyond that, the O(n²) covariance matrix becomes expensive to maintain.
//! - **Non-separable objectives** — CMA-ES learns parameter correlations
//! through the covariance matrix, making it especially effective on
//! rotated or ill-conditioned landscapes.
//! - **Moderate evaluation budgets** — typically needs ≈10×n to 100×n
//! evaluations to converge, where n is the number of continuous dimensions.
//!
//! For very cheap evaluations in low dimensions (d ≤ 20), consider
//! [`GpSampler`](super::gp::GpSampler) instead. For high-dimensional
//! separable problems, TPE may be more efficient.
//!
//! # Configuration
//!
//! | Option | Default | Description |
//! |--------|---------|-------------|
//! | `sigma0` | `avg_range / 4` | Initial step size — controls exploration breadth |
//! | `population_size` | `4 + ⌊3 ln n⌋` | Candidates per generation (λ) |
//! | `seed` | random | RNG seed for reproducibility |
//!
//! # Feature flag
//!
//! Requires the **`cma-es`** feature (adds the `nalgebra` dependency):
//!
//! ```toml
//! [dependencies]
//! optimizer = { version = "...", features = ["cma-es"] }
//! ```
//!
//! # Examples
//!
@@ -17,8 +57,14 @@
//! use optimizer::sampler::cma_es::CmaEsSampler;
//! use optimizer::{Direction, Study};
//!
//! let sampler = CmaEsSampler::with_seed(42);
//! let study: Study<f64> = Study::with_sampler(Direction::Minimize, sampler);
//! // Minimize a 2-D sphere function with CMA-ES
//! let sampler = CmaEsSampler::builder()
//! .sigma0(1.0)
//! .population_size(10)
//! .seed(42)
//! .build();
//!
//! let mut study: Study<f64> = Study::with_sampler(Direction::Minimize, sampler);
//! ```
use std::collections::HashMap;
src/sampler/differential_evolution.rs
+54 -10
View File
@@ -1,22 +1,66 @@
//! Differential Evolution (DE) sampler.
//!
//! DE is a population-based metaheuristic that maintains a population of
//! candidate solutions and creates new candidates by combining (mutating +
//! crossing over) existing ones. It is competitive with CMA-ES on many
//! problems and simpler to implement.
//! DE is a population-based metaheuristic that maintains a pool of candidate
//! solutions and creates new candidates through **mutation** (combining
//! difference vectors of existing members) and **binomial crossover**. A
//! trial vector replaces its parent only if it achieves a better objective
//! value, guaranteeing monotonic improvement of the population.
//!
//! Categorical parameters are sampled uniformly at random (not part of the
//! DE vector). If all parameters are categorical, the sampler falls back to
//! pure random sampling.
//! # Algorithm overview
//!
//! Each generation, for every population member *xᵢ*:
//! 1. **Mutation** — create a mutant vector *v* from other population
//! members using the selected [`DifferentialEvolutionStrategy`]:
//! - `Rand1`: `v = x_r1 + F * (x_r2 - x_r3)`
//! - `Best1`: `v = x_best + F * (x_r1 - x_r2)`
//! - `CurrentToBest1`: `v = x_i + F * (x_best - x_i) + F * (x_r1 - x_r2)`
//! 2. **Crossover** — create a trial vector *u* by mixing *v* and *xᵢ*
//! dimension-by-dimension with probability CR.
//! 3. **Selection** — replace *xᵢ* with *u* if `f(u) ≤ f(xᵢ)`.
//!
//! # When to use
//!
//! - **Continuous parameters** (float/int). Categorical parameters are
//! sampled uniformly at random and do not participate in DE.
//! - **Moderate to large search spaces** — DE scales better than GP-based
//! methods to higher dimensions, though it may need more evaluations.
//! - **Multi-modal landscapes** — the `Rand1` strategy maintains diversity
//! and avoids premature convergence.
//! - **No feature flags required** — DE is available with default features.
//!
//! For non-separable problems in moderate dimensions, consider
//! [`CmaEsSampler`](super::cma_es::CmaEsSampler) which learns parameter
//! correlations. For expensive functions with few dimensions, consider
//! [`GpSampler`](super::gp::GpSampler).
//!
//! # Configuration
//!
//! | Option | Default | Description |
//! |--------|---------|-------------|
//! | `population_size` | `max(10n, 15)` | Candidates per generation |
//! | `mutation_factor` (F) | 0.8 | Differential amplification — higher = more exploration |
//! | `crossover_rate` (CR) | 0.9 | Probability of taking a dimension from the mutant |
//! | `strategy` | `Rand1` | Mutation strategy (see [`DifferentialEvolutionStrategy`]) |
//! | `seed` | random | RNG seed for reproducibility |
//!
//! # Examples
//!
//! ```
//! use optimizer::sampler::differential_evolution::DifferentialEvolutionSampler;
//! use optimizer::sampler::differential_evolution::{
//! DifferentialEvolutionSampler, DifferentialEvolutionStrategy,
//! };
//! use optimizer::{Direction, Study};
//!
//! let sampler = DifferentialEvolutionSampler::with_seed(42);
//! let study: Study<f64> = Study::with_sampler(Direction::Minimize, sampler);
//! // Minimize with DE using the Best1 strategy for faster convergence
//! let sampler = DifferentialEvolutionSampler::builder()
//! .mutation_factor(0.7)
//! .crossover_rate(0.9)
//! .strategy(DifferentialEvolutionStrategy::Best1)
//! .population_size(20)
//! .seed(42)
//! .build();
//!
//! let mut study: Study<f64> = Study::with_sampler(Direction::Minimize, sampler);
//! ```
use std::collections::HashMap;
src/sampler/gp.rs
+59 -8
View File
@@ -1,15 +1,60 @@
//! Gaussian Process (GP) sampler with Expected Improvement acquisition.
//!
//! A classical Bayesian optimization sampler that uses a Gaussian Process
//! surrogate model with a Matérn 5/2 kernel and Expected Improvement (EI)
//! A classical Bayesian optimization sampler that builds a Gaussian Process
//! surrogate model with a **Matérn 5/2 kernel** (with ARD lengthscales) and
//! selects the next trial by maximizing the **Expected Improvement (EI)**
//! acquisition function. Best suited for small, expensive evaluations in
//! low-dimensional continuous spaces (d ≤ 20).
//!
//! Categorical parameters are sampled uniformly at random (not part of
//! the GP model). If all parameters are categorical, the sampler falls
//! back to pure random sampling.
//! # Algorithm overview
//!
//! Requires the `gp` feature flag.
//! 1. **Startup phase** — the first `n_startup_trials` trials are sampled
//! uniformly at random to build an initial dataset.
//! 2. **Fit GP** — training observations are standardized (zero mean, unit
//! variance) and a GP with Matérn 5/2 kernel is fitted via Cholesky
//! decomposition. ARD lengthscales are set to the per-dimension standard
//! deviation of the training inputs.
//! 3. **Maximize EI** — `n_candidates` random points are evaluated under
//! the GP posterior and the point with the highest Expected Improvement
//! is returned as the next trial.
//!
//! The GP uses at most 100 training points (the most recent ones) to keep
//! the O(n³) fitting cost manageable.
//!
//! # When to use
//!
//! - **Expensive objective functions** where every evaluation is costly
//! (e.g. physical experiments, large simulations). The GP surrogate
//! amortizes this cost by making fewer evaluations.
//! - **Low-dimensional continuous spaces** — typically d ≤ 20. Beyond that,
//! the GP becomes unreliable and alternatives like
//! [`CmaEsSampler`](super::cma_es::CmaEsSampler) or
//! [`TpeSampler`](super::tpe::TpeSampler) are preferable.
//! - **Smooth, low-noise objectives** — the GP assumes smoothness through
//! the Matérn 5/2 kernel. Very noisy objectives require increasing
//! `noise_variance`.
//!
//! Categorical parameters are sampled uniformly at random and do not
//! participate in the GP model. If all parameters are categorical, the
//! sampler falls back to pure random sampling.
//!
//! # Configuration
//!
//! | Option | Default | Description |
//! |--------|---------|-------------|
//! | `n_startup_trials` | 10 | Random trials before GP-guided sampling begins |
//! | `n_candidates` | 1000 | Random candidates for EI maximization |
//! | `noise_variance` | 1e-6 | Observation noise added to kernel diagonal |
//! | `seed` | random | RNG seed for reproducibility |
//!
//! # Feature flag
//!
//! Requires the **`gp`** feature (adds the `nalgebra` dependency):
//!
//! ```toml
//! [dependencies]
//! optimizer = { version = "...", features = ["gp"] }
//! ```
//!
//! # Examples
//!
@@ -17,8 +62,14 @@
//! use optimizer::sampler::gp::GpSampler;
//! use optimizer::{Direction, Study};
//!
//! let sampler = GpSampler::with_seed(42);
//! let study: Study<f64> = Study::with_sampler(Direction::Minimize, sampler);
//! // Minimize an expensive function with GP-based Bayesian optimization
//! let sampler = GpSampler::builder()
//! .n_startup_trials(5)
//! .n_candidates(500)
//! .seed(42)
//! .build();
//!
//! let mut study: Study<f64> = Study::with_sampler(Direction::Minimize, sampler);
//! ```
use std::collections::HashMap;
src/sampler/grid.rs
+53 -21
View File
@@ -1,7 +1,39 @@
//! Grid search sampler implementation.
//! Grid search sampler — exhaustive evaluation of discretized parameter spaces.
//!
//! `GridSearchSampler` performs exhaustive grid search over the parameter space,
//! systematically evaluating all combinations of discretized parameter values.
//! [`GridSearchSampler`] divides each parameter range into a fixed number of
//! evenly spaced points (or uses the explicit step size when defined) and
//! evaluates them sequentially. This guarantees complete coverage of the
//! search grid at the cost of scaling exponentially with the number of
//! parameters.
//!
//! # When to use
//!
//! - **Small, discrete spaces** — when you have a handful of categorical or
//! integer parameters and want to evaluate every combination.
//! - **Reproducibility** — grid search is fully deterministic with no random
//! component.
//! - **Benchmarking** — compare grid search results against adaptive samplers
//! to measure their benefit.
//!
//! Avoid grid search for high-dimensional or large continuous spaces;
//! prefer [`TpeSampler`](super::tpe::TpeSampler) or
//! [`RandomSampler`](super::random::RandomSampler) instead.
//!
//! # Configuration
//!
//! | Option | Default | Description |
//! |---|---|---|
//! | `n_points_per_param` | 10 | Points per continuous parameter (ignored when `step` is set) |
//!
//! # Example
//!
//! ```
//! use optimizer::prelude::*;
//! use optimizer::sampler::grid::GridSearchSampler;
//!
//! let sampler = GridSearchSampler::builder().n_points_per_param(5).build();
//! let study: Study<f64> = Study::with_sampler(Direction::Minimize, sampler);
//! ```
use std::collections::HashMap;
@@ -295,38 +327,38 @@ struct GridState {
grids: HashMap<String, CachedGrid>,
}
/// A grid search sampler that exhaustively evaluates all grid points.
/// Exhaustive grid search sampler.
///
/// `GridSearchSampler` divides the parameter space into a grid and systematically
/// samples each point. This is useful when you want to evaluate all combinations
/// of parameter values, especially for discrete or small parameter spaces.
/// Divide each parameter range into evenly spaced points and evaluate them
/// sequentially. When a parameter has an explicit `step` size, the step grid
/// is used instead of auto-discretization.
///
/// # Grid Exhaustion
/// Grid state is tracked **per distribution key** (bounds + step + log-scale).
/// Parameters with identical distributions share the same grid counter, so
/// use distinct ranges when multiple parameters span the same domain.
///
/// The sampler tracks its position in the grid for each distribution independently.
/// When all grid points for a distribution have been sampled, subsequent calls to
/// `sample()` for that distribution will **panic** with the message:
/// `"GridSearchSampler: all grid points exhausted"`.
/// # Grid exhaustion
///
/// To avoid panics, use [`is_exhausted()`](Self::is_exhausted) to check if all
/// points have been sampled before calling `sample()`. You can also use
/// [`grid_size()`](Self::grid_size) to determine the total number of grid points
/// that will be sampled.
/// When all grid points for a distribution have been sampled, the next
/// `sample()` call for that distribution **panics**. Use
/// [`is_exhausted()`](Self::is_exhausted) to check before sampling, or
/// set `n_points_per_param` high enough to cover the planned number of
/// trials.
///
/// # Thread Safety
/// # Thread safety
///
/// `GridSearchSampler` is thread-safe (`Send + Sync`) and uses internal locking
/// to ensure safe concurrent access to grid state.
/// `GridSearchSampler` is `Send + Sync` and uses internal locking for
/// safe concurrent access.
///
/// # Examples
///
/// ```
/// use optimizer::sampler::grid::GridSearchSampler;
///
/// // Create with default settings (10 points per parameter)
/// // Default: 10 points per parameter
/// let sampler = GridSearchSampler::new();
///
/// // Create with custom settings using the builder
/// // Custom grid density
/// let sampler = GridSearchSampler::builder().n_points_per_param(20).build();
/// ```
pub struct GridSearchSampler {
src/sampler/moead.rs
+79 -10
View File
@@ -1,8 +1,58 @@
//! MOEA/D (Multi-Objective Evolutionary Algorithm based on Decomposition) sampler.
//!
//! Decomposes a multi-objective problem into scalar subproblems using
//! weight vectors and solves them collaboratively. Supports Weighted Sum,
//! Tchebycheff, and Penalty-based Boundary Intersection (PBI) scalarization.
//! MOEA/D takes a fundamentally different approach from Pareto-based
//! algorithms like NSGA-II/III. It **decomposes** the multi-objective
//! problem into a set of scalar subproblems using evenly distributed
//! weight vectors (Das-Dennis points), then solves them collaboratively
//! through **neighborhood-based mating and replacement**.
//!
//! # Algorithm
//!
//! 1. **Decompose** — generate weight vectors on the unit simplex and
//! assign one scalar subproblem per weight vector.
//! 2. **Build neighborhoods** — for each subproblem, find its T nearest
//! neighbors by Euclidean distance between weight vectors.
//! 3. **Mate from neighborhood** — select parents from the neighborhood
//! of each subproblem and produce offspring via SBX crossover +
//! polynomial mutation.
//! 4. **Scalarize and update** — evaluate offspring using a scalarization
//! function and update neighboring subproblems if the offspring improves
//! their scalar value.
//! 5. **Update ideal point** — track the best value seen per objective.
//!
//! # Scalarization methods
//!
//! | Method | Formula | Best for |
//! |--------|---------|----------|
//! | [`Tchebycheff`](Decomposition::Tchebycheff) (default) | `max(wᵢ * \|fᵢ - zᵢ*\|)` | General purpose, handles non-convex fronts |
//! | [`WeightedSum`](Decomposition::WeightedSum) | `Σ(wᵢ * fᵢ)` | Convex Pareto fronts only |
//! | [`Pbi`](Decomposition::Pbi) | `d₁ + θ * d₂` | Fine-grained convergence/diversity control |
//!
//! # When to use
//!
//! - Problems where you want **evenly distributed** solutions along the
//! Pareto front (one solution per weight direction).
//! - Many-objective optimization (3+ objectives) — scales well because
//! each subproblem is a simple scalar optimization.
//! - Problems with **non-convex** Pareto fronts (use Tchebycheff or PBI).
//! - When you need explicit control over the trade-off distribution via
//! weight vectors.
//!
//! For Pareto-based approaches, see
//! [`Nsga2Sampler`](super::nsga2::Nsga2Sampler) (crowding distance) or
//! [`Nsga3Sampler`](super::nsga3::Nsga3Sampler) (reference-point niching).
//!
//! # Configuration
//!
//! | Parameter | Builder method | Default |
//! |-----------|---------------|---------|
//! | Population size | [`population_size`](MoeadSamplerBuilder::population_size) | Number of Das-Dennis weight vectors |
//! | Neighborhood size (T) | [`neighborhood_size`](MoeadSamplerBuilder::neighborhood_size) | `min(20, pop_size)` |
//! | Decomposition method | [`decomposition`](MoeadSamplerBuilder::decomposition) | Tchebycheff |
//! | Crossover probability | [`crossover_prob`](MoeadSamplerBuilder::crossover_prob) | 1.0 |
//! | SBX distribution index | [`crossover_eta`](MoeadSamplerBuilder::crossover_eta) | 20.0 |
//! | Mutation distribution index | [`mutation_eta`](MoeadSamplerBuilder::mutation_eta) | 20.0 |
//! | Random seed | [`seed`](MoeadSamplerBuilder::seed) | random |
//!
//! # Examples
//!
@@ -37,15 +87,29 @@ use crate::multi_objective::MultiObjectiveTrial;
use crate::param::ParamValue;
use crate::types::Direction;
/// Decomposition (scalarization) method for MOEA/D.
/// Decomposition (scalarization) method for [`MoeadSampler`].
///
/// Control how multi-objective values are reduced to a single scalar
/// for each subproblem. The default is [`Tchebycheff`](Self::Tchebycheff),
/// which handles both convex and non-convex Pareto fronts.
#[derive(Debug, Clone, Default)]
pub enum Decomposition {
/// Weighted sum: `sum(w_i * f_i)`.
/// Weighted sum: `Σ(wᵢ * fᵢ)`.
///
/// Simplest method but can only find solutions on convex regions
/// of the Pareto front.
WeightedSum,
/// Tchebycheff: `max(w_i * |f_i - z_i*|)`.
/// Tchebycheff: `max(wᵢ * |fᵢ - zᵢ*|)`.
///
/// Handles non-convex Pareto fronts. The most commonly used
/// decomposition method (default).
#[default]
Tchebycheff,
/// Penalty-based Boundary Intersection with parameter theta.
/// Penalty-based Boundary Intersection: `d₁ + θ * d₂`.
///
/// Provides fine-grained control over the convergence/diversity
/// balance via the penalty parameter `theta`. Higher `theta`
/// favors solutions closer to the weight direction.
Pbi {
/// Penalty parameter controlling the balance between convergence
/// and diversity. Default: 5.0.
@@ -55,9 +119,14 @@ pub enum Decomposition {
/// MOEA/D sampler for multi-objective optimization.
///
/// Decomposes the multi-objective problem into scalar subproblems
/// using weight vectors, solving them collaboratively via
/// neighborhood-based mating and replacement.
/// Decompose a multi-objective problem into scalar subproblems using
/// weight vectors and solve them collaboratively via neighborhood-based
/// mating. Supports [`Tchebycheff`](Decomposition::Tchebycheff),
/// [`WeightedSum`](Decomposition::WeightedSum), and
/// [`Pbi`](Decomposition::Pbi) scalarization.
///
/// Create with [`MoeadSampler::new`], [`MoeadSampler::with_seed`], or
/// [`MoeadSampler::builder`] for full configuration.
pub struct MoeadSampler {
state: Mutex<MoeadState>,
}
src/sampler/motpe.rs
+58 -14
View File
@@ -1,19 +1,38 @@
//! Multi-Objective Tree-Parzen Estimator (MOTPE) sampler.
//!
//! Extends TPE to handle multi-objective optimization by using Pareto
//! non-dominated sorting to define "good" vs "bad" trial regions for
//! the KDE models, replacing the single-objective gamma-based split.
//! MOTPE extends TPE to multi-objective optimization by replacing the gamma-based
//! split with Pareto non-dominated sorting. This lets the sampler propose
//! parameters that push the Pareto front forward across all objectives
//! simultaneously.
//!
//! # Algorithm
//!
//! In single-objective TPE, trials are sorted by value and split at a
//! gamma percentile into good/bad groups. MOTPE replaces this with:
//! In single-objective TPE, trials are sorted by value and split at a gamma
//! percentile into good/bad groups. MOTPE replaces this with:
//!
//! 1. Compute non-dominated sorting on all completed trials
//! 2. Use the Pareto front (rank 0) as "good" trials
//! 3. Use dominated trials as "bad" trials
//! 4. Build KDE l(x) from good, g(x) from bad
//! 5. Sample candidates and score by l(x)/g(x)
//! 1. Compute non-dominated sorting on all completed trials.
//! 2. Use the Pareto front (rank 0) as "good" trials.
//! 3. Use dominated trials (rank 1+) as "bad" trials.
//! 4. Build KDE l(x) from good, g(x) from bad.
//! 5. Sample candidates and score by l(x)/g(x).
//!
//! # When to use
//!
//! - You have 2+ objectives and want model-guided search (not pure evolutionary).
//! - Your objectives are relatively smooth and continuous.
//! - You want a Pareto-aware version of TPE without the overhead of full
//! population-based algorithms like NSGA-II or NSGA-III.
//!
//! For single-objective problems, use [`TpeSampler`](super::tpe::TpeSampler) instead.
//! For many-objective (3+) problems with reference-point decomposition, consider
//! NSGA-III or MOEA/D.
//!
//! # Configuration
//!
//! - `n_startup_trials` — number of random trials before MOTPE kicks in (default: 11)
//! - `n_ei_candidates` — candidates evaluated per sample (default: 24)
//! - `kde_bandwidth` — optional fixed KDE bandwidth; `None` uses Scott's rule
//! - `seed` — optional seed for reproducibility
//!
//! # Examples
//!
@@ -50,14 +69,21 @@ use crate::{pareto, rng_util};
/// Multi-Objective TPE (MOTPE) sampler for multi-objective Bayesian optimization.
///
/// Uses Pareto non-dominated sorting to split completed trials into
/// "good" (non-dominated, rank 0) and "bad" (dominated) groups, then
/// fits kernel density estimators to each group and samples new points
/// that maximize l(x)/g(x).
/// Use Pareto non-dominated sorting to split completed trials into "good"
/// (non-dominated, rank 0) and "bad" (dominated) groups, then fit kernel
/// density estimators to each group and sample new points that maximize
/// l(x)/g(x).
///
/// During the startup phase (fewer than `n_startup_trials` completed),
/// MOTPE falls back to random sampling.
///
/// # When to use
///
/// Use `MotpeSampler` when optimizing 2+ objectives and you want a
/// model-guided sampler that adapts proposals based on the current
/// Pareto front. For single-objective problems, use
/// [`TpeSampler`](super::tpe::TpeSampler) instead.
///
/// # Examples
///
/// ```
@@ -74,6 +100,17 @@ use crate::{pareto, rng_util};
///
/// let study =
/// MultiObjectiveStudy::with_sampler(vec![Direction::Minimize, Direction::Minimize], sampler);
///
/// let x = FloatParam::new(0.0, 1.0);
/// study
/// .optimize(30, |trial| {
/// let xv = x.suggest(trial)?;
/// Ok::<_, optimizer::Error>(vec![xv, 1.0 - xv])
/// })
/// .unwrap();
///
/// let front = study.pareto_front();
/// assert!(!front.is_empty());
/// ```
pub struct MotpeSampler {
/// Number of trials before MOTPE kicks in (uses random sampling before this).
@@ -520,6 +557,13 @@ impl MultiObjectiveSampler for MotpeSampler {
/// Builder for configuring a [`MotpeSampler`].
///
/// # Defaults
///
/// - `n_startup_trials`: 11
/// - `n_ei_candidates`: 24
/// - `kde_bandwidth`: None (Scott's rule)
/// - `seed`: None (OS entropy)
///
/// # Examples
///
/// ```
src/sampler/nsga2.rs
+46 -4
View File
@@ -1,7 +1,45 @@
//! NSGA-II (Non-dominated Sorting Genetic Algorithm II) sampler.
//!
//! Implements multi-objective optimization using non-dominated sorting,
//! crowding distance, SBX crossover, and polynomial mutation.
//! NSGA-II is one of the most widely used evolutionary multi-objective
//! optimization algorithms. It ranks the population using **non-dominated
//! sorting** (fast O(MN²) algorithm) and breaks ties within the same
//! Pareto front using **crowding distance**, which favors solutions in
//! less-crowded regions of the objective space.
//!
//! # Algorithm
//!
//! Each generation proceeds as follows:
//!
//! 1. **Non-dominated sorting** — partition the combined parent+offspring
//! population into Pareto fronts F₁, F₂, …
//! 2. **Crowding distance** — for each front, compute per-solution crowding
//! distance (sum of normalized neighbor gaps in each objective).
//! 3. **Selection** — fill the next population front-by-front. When a front
//! only partially fits, prefer solutions with higher crowding distance.
//! 4. **Binary tournament** — select parents using (rank, crowding distance)
//! comparisons.
//! 5. **SBX crossover + polynomial mutation** — generate offspring.
//!
//! # When to use
//!
//! - Two-objective problems where you want a well-spread Pareto front.
//! - General-purpose multi-objective optimization with moderate population
//! sizes.
//! - Problems that benefit from diversity preservation via crowding distance.
//!
//! For problems with **three or more objectives**, consider
//! [`Nsga3Sampler`](super::nsga3::Nsga3Sampler) (reference-point niching)
//! or [`MoeadSampler`](super::moead::MoeadSampler) (decomposition).
//!
//! # Configuration
//!
//! | Parameter | Builder method | Default |
//! |-----------|---------------|---------|
//! | Population size | [`population_size`](Nsga2SamplerBuilder::population_size) | `4 + floor(3 * ln(n_params))`, min 4 |
//! | Crossover probability | [`crossover_prob`](Nsga2SamplerBuilder::crossover_prob) | 0.9 |
//! | SBX distribution index | [`crossover_eta`](Nsga2SamplerBuilder::crossover_eta) | 20.0 |
//! | Mutation distribution index | [`mutation_eta`](Nsga2SamplerBuilder::mutation_eta) | 20.0 |
//! | Random seed | [`seed`](Nsga2SamplerBuilder::seed) | random |
//!
//! # Examples
//!
@@ -39,8 +77,12 @@ use crate::types::Direction;
/// NSGA-II sampler for multi-objective optimization.
///
/// Provides non-dominated sorting, crowding distance selection,
/// SBX crossover, and polynomial mutation.
/// Use non-dominated sorting with crowding-distance tie-breaking to
/// evolve a well-spread Pareto front. Best suited for bi-objective
/// problems; for 3+ objectives prefer [`Nsga3Sampler`](super::nsga3::Nsga3Sampler).
///
/// Create with [`Nsga2Sampler::new`], [`Nsga2Sampler::with_seed`], or
/// [`Nsga2Sampler::builder`] for full configuration.
pub struct Nsga2Sampler {
state: Mutex<Nsga2State>,
}
src/sampler/nsga3.rs
+50 -6
View File
@@ -1,9 +1,49 @@
//! NSGA-III (Non-dominated Sorting Genetic Algorithm III) sampler.
//!
//! Uses reference-point-based niching for better diversity in
//! many-objective (3+) optimization problems. Das-Dennis structured
//! reference points guide the search toward a well-distributed
//! Pareto front.
//! NSGA-III extends NSGA-II to handle **many-objective** (3+) problems
//! where crowding distance loses effectiveness. Instead of crowding
//! distance, it uses **reference-point-based niching** with structured
//! Das-Dennis reference points distributed on the unit simplex to guide
//! the population toward a well-diversified Pareto front.
//!
//! # Algorithm
//!
//! Each generation proceeds as follows:
//!
//! 1. **Non-dominated sorting** — same as NSGA-II, partition the
//! combined population into Pareto fronts F₁, F₂, …
//! 2. **Normalize objectives** — translate by ideal point and scale by
//! intercepts so all objectives lie in roughly \[0, 1\].
//! 3. **Associate with reference points** — assign each solution to the
//! closest Das-Dennis reference direction by perpendicular distance.
//! 4. **Niching selection** — when the last front only partially fits,
//! prefer solutions associated with under-represented reference points
//! (lowest niche count first, closest distance second).
//! 5. **SBX crossover + polynomial mutation** — generate offspring via
//! rank-based tournament selection.
//!
//! # When to use
//!
//! - **Three or more objectives** — NSGA-III maintains diversity far
//! better than NSGA-II as the number of objectives grows.
//! - Problems where you want a **uniformly distributed** Pareto front
//! guided by structured reference points.
//! - Scales well up to ~10 objectives with appropriate division settings.
//!
//! For bi-objective problems, [`Nsga2Sampler`](super::nsga2::Nsga2Sampler)
//! is simpler and equally effective. For decomposition-based optimization,
//! see [`MoeadSampler`](super::moead::MoeadSampler).
//!
//! # Configuration
//!
//! | Parameter | Builder method | Default |
//! |-----------|---------------|---------|
//! | Population size | [`population_size`](Nsga3SamplerBuilder::population_size) | Number of Das-Dennis reference points |
//! | Das-Dennis divisions (H) | [`n_divisions`](Nsga3SamplerBuilder::n_divisions) | Auto-chosen from population size and objectives |
//! | Crossover probability | [`crossover_prob`](Nsga3SamplerBuilder::crossover_prob) | 1.0 |
//! | SBX distribution index | [`crossover_eta`](Nsga3SamplerBuilder::crossover_eta) | 30.0 |
//! | Mutation distribution index | [`mutation_eta`](Nsga3SamplerBuilder::mutation_eta) | 20.0 |
//! | Random seed | [`seed`](Nsga3SamplerBuilder::seed) | random |
//!
//! # Examples
//!
@@ -49,8 +89,12 @@ use crate::types::Direction;
/// NSGA-III sampler for multi-objective optimization.
///
/// Uses reference-point-based niching to maintain diversity,
/// especially effective for problems with 3 or more objectives.
/// Use reference-point niching with Das-Dennis structured points to
/// maintain diversity in many-objective (3+) problems. For bi-objective
/// problems, [`Nsga2Sampler`](super::nsga2::Nsga2Sampler) is simpler.
///
/// Create with [`Nsga3Sampler::new`], [`Nsga3Sampler::with_seed`], or
/// [`Nsga3Sampler::builder`] for full configuration.
pub struct Nsga3Sampler {
state: Mutex<Nsga3State>,
}
src/sampler/random.rs
+36 -5
View File
@@ -1,4 +1,31 @@
//! Random sampler implementation.
//! Random sampler — uniform independent sampling.
//!
//! [`RandomSampler`] draws each parameter value independently and uniformly
//! at random, ignoring trial history entirely. It respects log-scale and
//! step-size constraints defined by the parameter distribution.
//!
//! # When to use
//!
//! - **Baseline comparison** — run Random alongside smarter samplers to
//! quantify their benefit.
//! - **Startup phase** — many model-based samplers (TPE, GP, CMA-ES) use
//! random sampling for their first *n* trials before fitting a surrogate.
//! - **Very high dimensions** — when the search space is too large for
//! structured exploration, random search with enough budget can be
//! surprisingly competitive.
//!
//! For better uniform coverage without model fitting, consider
//! [`SobolSampler`](super::sobol::SobolSampler) (requires the `sobol`
//! feature flag).
//!
//! # Example
//!
//! ```
//! use optimizer::prelude::*;
//! use optimizer::sampler::random::RandomSampler;
//!
//! let study: Study<f64> = Study::with_sampler(Direction::Minimize, RandomSampler::with_seed(42));
//! ```
use parking_lot::Mutex;
@@ -7,11 +34,15 @@ use crate::param::ParamValue;
use crate::rng_util;
use crate::sampler::{CompletedTrial, Sampler};
/// A simple random sampler that samples uniformly from distributions.
/// Uniform independent random sampler.
///
/// This sampler ignores the trial history and samples uniformly at random,
/// respecting log scale and step size constraints. It serves as a baseline
/// sampler and is used during the startup phase of more sophisticated samplers.
/// Sample each parameter value uniformly at random, respecting log-scale and
/// step-size constraints. Trial history is ignored — every sample is drawn
/// independently.
///
/// This is the default sampler used by [`Study::new`](crate::Study::new)
/// and during the startup phase of model-based samplers such as
/// [`TpeSampler`](super::tpe::TpeSampler).
///
/// # Examples
///
src/sampler/sobol.rs
+50 -8
View File
@@ -1,4 +1,48 @@
//! Quasi-random sampler using Sobol low-discrepancy sequences.
//!
//! [`SobolSampler`] generates points from a Sobol sequence (scrambled via the
//! Burley 2020 algorithm) to fill the parameter space more uniformly than
//! pure random sampling. Where [`RandomSampler`](super::random::RandomSampler)
//! may cluster points in some regions by chance, Sobol sequences are
//! constructed to spread points evenly across all dimensions.
//!
//! # When to use
//!
//! - **Better-than-random baseline** — when you want uniform coverage
//! without the cost of model fitting (TPE, GP, etc.).
//! - **Startup phase replacement** — use Sobol instead of random for the
//! initial exploration phase of adaptive samplers.
//! - **Moderate dimensionality** — Sobol uniformity is strongest up to
//! ~20 dimensions; beyond that the advantage over random sampling
//! diminishes.
//! - **Deterministic exploration** — Sobol sequences are fully deterministic
//! for a given seed, making experiments reproducible.
//!
//! # How it works
//!
//! Each trial maps to a Sobol sequence index, and each parameter within a
//! trial maps to a separate Sobol dimension. The resulting quasi-random
//! point in \[0, 1) is then scaled to the parameter's distribution (linear,
//! log-scale, or step grid).
//!
//! **Important:** parameters must be suggested in the same order across
//! trials for consistent dimension assignment.
//!
//! Requires the **`sobol`** feature flag:
//!
//! ```toml
//! [dependencies]
//! optimizer = { version = "...", features = ["sobol"] }
//! ```
//!
//! # Example
//!
//! ```
//! use optimizer::prelude::*;
//! use optimizer::sampler::sobol::SobolSampler;
//!
//! let study: Study<f64> = Study::with_sampler(Direction::Minimize, SobolSampler::with_seed(42));
//! ```
use parking_lot::Mutex;
use sobol_burley::sample;
@@ -17,13 +61,11 @@ struct SobolState {
/// Quasi-random sampler using Sobol low-discrepancy sequences.
///
/// Provides better uniform coverage of the parameter space than
/// [`RandomSampler`](super::random::RandomSampler). Useful as a baseline or
/// for the startup phase of model-based samplers.
///
/// Unlike random sampling, Sobol sequences are deterministic and fill the
/// space more evenly, reducing the number of trials needed to adequately
/// cover the search space.
/// Produce better uniform coverage of the parameter space than
/// [`RandomSampler`](super::random::RandomSampler) by using a
/// scrambled Sobol sequence (Burley 2020). Useful as a standalone
/// baseline or as a drop-in replacement for the random startup
/// phase of model-based samplers.
///
/// Each trial uses a different Sobol sequence index, and each parameter
/// within a trial maps to a different Sobol dimension. Parameters must be
@@ -33,7 +75,7 @@ struct SobolState {
/// Sobol sequences are most effective in moderate dimensions (up to ~20).
/// For very high dimensions, the uniformity advantage diminishes.
///
/// Requires the `sobol` feature flag.
/// Requires the **`sobol`** feature flag.
///
/// # Examples
///
src/sampler/tpe/mod.rs
+59 -3
View File
@@ -1,7 +1,63 @@
//! Tree-Parzen Estimator (TPE) sampler implementation and utilities.
//! Tree-Parzen Estimator (TPE) sampler family for Bayesian optimization.
//!
//! This module provides TPE-based sampling for Bayesian optimization,
//! including support for intersection search space calculation.
//! TPE is a sequential model-based optimization algorithm that models P(x|y) instead
//! of P(y|x). It splits completed trials into "good" (below the gamma quantile) and
//! "bad" groups, fits a kernel density estimator (KDE) to each, and proposes new
//! points by maximizing the l(x)/g(x) ratio — an approximation of Expected Improvement.
//!
//! # Samplers
//!
//! | Sampler | Models parameters | Best for |
//! |---------|-------------------|----------|
//! | [`TpeSampler`] | Independently | General-purpose single-objective optimization |
//! | [`MultivariateTpeSampler`] | Jointly | Problems with correlated parameters |
//!
//! # Gamma strategies
//!
//! The gamma quantile controls how many trials are considered "good". This module
//! provides four built-in strategies via the [`GammaStrategy`] trait:
//!
//! | Strategy | Formula | Default |
//! |----------|---------|---------|
//! | [`FixedGamma`] | Constant value | gamma = 0.25 |
//! | [`LinearGamma`] | Linear ramp from min to max | 0.10 → 0.25 over 100 trials |
//! | [`SqrtGamma`] | 1/√n decay (Optuna-style) | factor = 1.0, max = 0.25 |
//! | [`HyperoptGamma`] | (base+1)/n (Hyperopt-style) | base = 24, max = 0.25 |
//!
//! You can also implement [`GammaStrategy`] for a custom splitting rule.
//!
//! # Search-space utilities
//!
//! The [`search_space`] submodule provides [`IntersectionSearchSpace`] for computing
//! the common parameter set across trials, and [`GroupDecomposedSearchSpace`] for
//! splitting parameters into independent groups based on co-occurrence.
//!
//! # Examples
//!
//! Basic TPE with default settings:
//!
//! ```
//! use optimizer::sampler::tpe::TpeSampler;
//! use optimizer::{Direction, Study};
//!
//! let study: Study<f64> = Study::with_sampler(Direction::Minimize, TpeSampler::new());
//! ```
//!
//! Multivariate TPE for correlated parameters:
//!
//! ```
//! use optimizer::sampler::tpe::MultivariateTpeSampler;
//! use optimizer::{Direction, Study};
//!
//! let sampler = MultivariateTpeSampler::builder()
//! .gamma(0.15)
//! .n_startup_trials(20)
//! .group(true)
//! .seed(42)
//! .build()
//! .unwrap();
//! let study: Study<f64> = Study::with_sampler(Direction::Minimize, sampler);
//! ```
mod gamma;
mod multivariate;
src/sampler/tpe/multivariate.rs
+45 -7
View File
@@ -167,15 +167,53 @@ pub enum ConstantLiarStrategy {
/// A Multivariate Tree-Parzen Estimator (TPE) sampler for Bayesian optimization.
///
/// This sampler extends the standard TPE approach by modeling joint distributions
/// over all parameters, allowing it to capture parameter correlations.
/// Unlike the standard [`super::TpeSampler`], which samples each parameter
/// independently, this sampler models joint distributions over all parameters
/// using multivariate KDE. This captures correlations between parameters and
/// can significantly improve optimization on problems where parameters interact
/// (e.g., Rosenbrock, coupled hyperparameters).
///
/// # Fields
/// When the search space varies between trials (conditional parameters), the
/// sampler automatically falls back to independent TPE or uniform sampling for
/// parameters outside the intersection search space.
///
/// - `gamma_strategy`: Strategy for computing the gamma quantile
/// - `n_startup_trials`: Number of random trials before TPE sampling begins
/// - `n_ei_candidates`: Number of candidates to evaluate per joint sample
/// - `group`: Whether to decompose search space into independent groups
/// # When to use
///
/// - Parameters are correlated or interact with each other.
/// - The search space is mostly fixed across trials.
/// - You need parallel optimization (enable [`ConstantLiarStrategy`]).
///
/// Prefer [`super::TpeSampler`] when parameters are independent or the search space changes
/// dynamically.
///
/// # Examples
///
/// ```
/// use optimizer::parameter::{FloatParam, Parameter};
/// use optimizer::sampler::tpe::MultivariateTpeSampler;
/// use optimizer::{Direction, Study};
///
/// let sampler = MultivariateTpeSampler::builder()
/// .gamma(0.15)
/// .n_startup_trials(20)
/// .seed(42)
/// .build()
/// .unwrap();
///
/// let study: Study<f64> = Study::with_sampler(Direction::Minimize, sampler);
/// let x = FloatParam::new(-5.0, 5.0);
/// let y = FloatParam::new(-5.0, 5.0);
///
/// study
/// .optimize(30, |trial| {
/// let xv = x.suggest(trial)?;
/// let yv = y.suggest(trial)?;
/// Ok::<_, optimizer::Error>(xv * xv + yv * yv)
/// })
/// .unwrap();
///
/// assert!(study.best_value().unwrap() < 1.0);
/// ```
pub struct MultivariateTpeSampler {
/// Strategy for computing the gamma quantile.
gamma_strategy: Arc<dyn GammaStrategy>,
src/sampler/tpe/sampler.rs
+1 -1
View File
@@ -102,7 +102,7 @@ use crate::sampler::{CompletedTrial, Sampler};
///
/// // Create with custom settings using the builder
/// let sampler = TpeSampler::builder()
/// .gamma(0.15) // Shorthand for Fixednew(0.15)
/// .gamma(0.15) // Shorthand for FixedGamma::new(0.15)
/// .n_startup_trials(20)
/// .n_ei_candidates(32)
/// .seed(42)