optimiz-rs/docs/differential_evolution.md

Differential Evolution API

Overview

The Differential Evolution (DE) module provides a global optimization algorithm for non-convex, multimodal objective functions. It's particularly effective for problems where gradient information is unavailable or unreliable, and for escaping local optima.

Function: differential_evolution

from optimizr import differential_evolution

Signature

differential_evolution(
    objective_fn: Callable[[np.ndarray], float],
    bounds: List[Tuple[float, float]],
    popsize: int = 15,
    maxiter: int = 1000,
    f: float = 0.8,
    cr: float = 0.7,
) -> Tuple[np.ndarray, float]

Parameters

• objective_fn (callable): Function to minimize.

  • Signature: objective_fn(x: np.ndarray) -> float
  • Takes a 1D array of parameters and returns a scalar objective value.
  • Lower values are better.

• bounds (List[Tuple[float, float]]): List of (min, max) bounds for each parameter dimension.

• popsize (int, optional): Population size multiplier. Total population size will be popsize × n_params. Default is 15.

• maxiter (int, optional): Maximum number of generations. Default is 1,000.

• f (float, optional): Mutation factor, typically in range [0.5, 2.0]. Controls the amplification of differential variation. Default is 0.8.

• cr (float, optional): Crossover probability, typically in range [0.1, 0.9]. Controls the fraction of parameter values copied from the mutant. Default is 0.7.

Returns

Returns a tuple (x, fun):

• x (np.ndarray): Best parameters found (minimum).
• fun (float): Best objective value (minimum).

Alternatively, when using the Rust backend directly, returns a DEResult object with attributes:

• x: Best parameters
• fun: Best objective value
• nfev: Number of function evaluations

Basic Example

import numpy as np
from optimizr import differential_evolution

# Define the Rosenbrock function (global minimum at [1, 1, ..., 1])
def rosenbrock(x):
    return sum(100.0 * (x[i+1] - x[i]**2)**2 + (1 - x[i])**2 
               for i in range(len(x) - 1))

# Optimize
x_opt, f_min = differential_evolution(
    objective_fn=rosenbrock,
    bounds=[(-5, 5)] * 10,
    popsize=15,
    maxiter=1000
)

print(f"Optimal parameters: {x_opt}")
print(f"Minimum value: {f_min:.6f}")
print(f"Expected: {rosenbrock(np.ones(10)):.6f}")

Advanced Examples

1. Rastrigin Function (Many Local Minima)

import numpy as np
from optimizr import differential_evolution

def rastrigin(x):
    """Highly multimodal function with many local minima"""
    A = 10
    n = len(x)
    return A * n + sum(xi**2 - A * np.cos(2 * np.pi * xi) for xi in x)

# True global minimum is at origin with f(0, ..., 0) = 0
x_opt, f_min = differential_evolution(
    objective_fn=rastrigin,
    bounds=[(-5.12, 5.12)] * 10,
    popsize=20,
    maxiter=2000,
    f=0.8,
    cr=0.9
)

print(f"Minimum found: {f_min:.6f}")
print(f"Distance from optimum: {np.linalg.norm(x_opt):.6f}")

2. Constrained Optimization

import numpy as np
from optimizr import differential_evolution

def constrained_objective(x):
    """Minimize x^2 + y^2 subject to x + y >= 1"""
    obj = x[0]**2 + x[1]**2
    
    # Add penalty for constraint violation
    constraint = x[0] + x[1] - 1
    if constraint < 0:
        obj += 1000 * constraint**2  # Penalty term
    
    return obj

x_opt, f_min = differential_evolution(
    objective_fn=constrained_objective,
    bounds=[(-5, 5), (-5, 5)],
    popsize=15,
    maxiter=500
)

print(f"Optimal point: ({x_opt[0]:.3f}, {x_opt[1]:.3f})")
print(f"Constraint: x + y = {x_opt[0] + x_opt[1]:.3f} (should be ≥ 1)")
print(f"Objective: {f_min:.3f}")

3. Hyperparameter Tuning

import numpy as np
from sklearn.model_selection import cross_val_score
from sklearn.svm import SVC
from sklearn.datasets import load_digits
from optimizr import differential_evolution

# Load data
X, y = load_digits(return_X_y=True)

def svm_objective(params):
    """Optimize SVM hyperparameters"""
    C, gamma = params
    
    # Convert to log scale
    C = 10 ** C
    gamma = 10 ** gamma
    
    # Cross-validation score (negative because we minimize)
    model = SVC(C=C, gamma=gamma)
    score = cross_val_score(model, X, y, cv=3, scoring='accuracy')
    
    return -score.mean()  # Negative because we minimize

# Optimize
params_opt, score_min = differential_evolution(
    objective_fn=svm_objective,
    bounds=[(-3, 3), (-5, 1)],  # log10 scale for C and gamma
    popsize=10,
    maxiter=30
)

C_opt = 10 ** params_opt[0]
gamma_opt = 10 ** params_opt[1]

print(f"Best C: {C_opt:.4f}")
print(f"Best gamma: {gamma_opt:.6f}")
print(f"Best CV accuracy: {-score_min:.4f}")

4. Portfolio Optimization

import numpy as np
from optimizr import differential_evolution

# Sample returns (rows = assets, columns = time periods)
returns = np.random.randn(5, 1000) * 0.01
returns += np.array([0.08, 0.10, 0.12, 0.06, 0.09])[:, np.newaxis] / 252

def portfolio_objective(weights):
    """Maximize Sharpe ratio (minimize negative Sharpe)"""
    # Ensure weights sum to 1
    weights = weights / weights.sum()
    
    # Calculate portfolio return and volatility
    portfolio_return = np.sum(returns.mean(axis=1) * weights) * 252
    portfolio_vol = np.sqrt(
        np.dot(weights, np.dot(np.cov(returns), weights))
    ) * np.sqrt(252)
    
    # Sharpe ratio (assuming risk-free rate = 2%)
    sharpe = (portfolio_return - 0.02) / portfolio_vol
    
    return -sharpe  # Negative because we minimize

# Optimize
n_assets = 5
weights_opt, sharpe_neg = differential_evolution(
    objective_fn=portfolio_objective,
    bounds=[(0, 1)] * n_assets,  # Long-only portfolio
    popsize=20,
    maxiter=500
)

# Normalize weights
weights_opt = weights_opt / weights_opt.sum()

print("Optimal Portfolio Weights:")
for i, w in enumerate(weights_opt):
    print(f"  Asset {i+1}: {w:.2%}")

print(f"\nSharpe Ratio: {-sharpe_neg:.3f}")

5. Function Fitting

import numpy as np
import matplotlib.pyplot as plt
from optimizr import differential_evolution

# Generate noisy data
x_data = np.linspace(0, 10, 100)
y_true = 2.5 * np.sin(0.8 * x_data + 1.2) + 1.5
y_data = y_true + np.random.normal(0, 0.3, len(x_data))

def fitting_objective(params):
    """Fit y = A * sin(B * x + C) + D"""
    A, B, C, D = params
    y_pred = A * np.sin(B * x_data + C) + D
    mse = np.mean((y_data - y_pred)**2)
    return mse

# Optimize
params_opt, mse_min = differential_evolution(
    objective_fn=fitting_objective,
    bounds=[(0, 10), (0, 2), (0, 2*np.pi), (-5, 5)],
    popsize=15,
    maxiter=1000
)

A, B, C, D = params_opt
print(f"Fitted parameters: A={A:.2f}, B={B:.2f}, C={C:.2f}, D={D:.2f}")
print(f"MSE: {mse_min:.4f}")

# Plot
y_fitted = A * np.sin(B * x_data + C) + D
plt.figure(figsize=(10, 6))
plt.scatter(x_data, y_data, alpha=0.5, label='Data')
plt.plot(x_data, y_true, 'g--', label='True', linewidth=2)
plt.plot(x_data, y_fitted, 'r-', label='Fitted', linewidth=2)
plt.legend()
plt.title('Differential Evolution Function Fitting')
plt.show()

Parameter Tuning Guide

Population Size (popsize)

• Small (5-10): Fast but may converge prematurely
• Medium (15-20): Good balance for most problems
• Large (30+): Better exploration, slower convergence

Rule of thumb: popsize ≥ 10 for problems with up to 10 parameters.

Mutation Factor (f)

• Low (0.4-0.6): Conservative, good for fine-tuning
• Medium (0.7-0.9): Standard, works for most problems
• High (1.0-2.0): Aggressive exploration, avoids local minima

Crossover Probability (cr)

• Low (0.1-0.3): Preserves more of original vector
• Medium (0.5-0.7): Balanced mixing
• High (0.8-1.0): Aggressive recombination

Maximum Iterations (maxiter)

• Depends on problem difficulty and dimensions
• Monitor convergence: if still improving at maxiter, increase it
• Typical values: 500-5000

Convergence Analysis

# Track convergence history (requires modification to return history)
import matplotlib.pyplot as plt

history = []

def tracked_objective(x):
    result = objective_fn(x)
    history.append(result)
    return result

x_opt, f_min = differential_evolution(
    objective_fn=tracked_objective,
    bounds=bounds,
    popsize=15,
    maxiter=1000
)

# Plot convergence
plt.figure(figsize=(10, 6))
plt.semilogy(history)
plt.xlabel('Function Evaluation')
plt.ylabel('Objective Value')
plt.title('Convergence History')
plt.grid(True)
plt.show()

Performance Notes

• Rust Backend: 50-100x faster than pure Python implementations for compute-intensive objectives.

• Python Fallback: Falls back to scipy.optimize.differential_evolution if Rust is unavailable.

• Parallelization: Population evaluations are independent and can be parallelized (future enhancement).

• Complexity: O(popsize × n_params × maxiter × cost_per_eval)

Common Use Cases

Application	Typical Settings	Notes
Hyperparameter tuning	popsize=10-15, maxiter=50-200	Fast evaluations
Engineering design	popsize=20-30, maxiter=500-2000	Complex constraints
Function fitting	popsize=15-20, maxiter=500-1000	Multiple local minima
Portfolio optimization	popsize=15-20, maxiter=200-500	Moderate dimensions
Neural network training	popsize=30-50, maxiter=1000+	High dimensions

Tips and Best Practices

1. Scaling: Normalize parameters to similar ranges for better performance.

2. Bounds: Set reasonable bounds based on domain knowledge.

3. Stochastic Objectives: For noisy functions, use larger population and more iterations.

4. Warm Start: Use results from previous runs as initial population.

5. Hybrid Approach: Use DE for global search, then local optimizer for refinement.

6. Early Stopping: Implement custom stopping criteria based on improvement rate.

Comparison with Other Optimizers

Method	Pros	Cons	When to Use
Differential Evolution	No gradients needed, global search, robust	Slow for high dimensions	Non-convex, derivative-free
Gradient Descent	Fast, precise	Needs gradients, local only	Smooth, differentiable
Genetic Algorithm	Very flexible	Slower convergence	Discrete, combinatorial
Simulated Annealing	Simple, global search	Sensitive to temperature schedule	Simple problems
Grid Search	Guaranteed coverage	Exponential cost	Few dimensions only

See Also

• Grid Search API - For exhaustive parameter search
• MCMC API - For Bayesian parameter estimation
• Differential Evolution Theory - Mathematical background
• Examples - Complete working examples