Improve code design

2025-12-03 22:08:08 +01:00
parent 923d27e87b
commit a62ceaa64b
18 changed files with 6613 additions and 4 deletions
@@ -0,0 +1,371 @@
+# Differential Evolution API
+
+## Overview
+
+The Differential Evolution (DE) module provides a global optimization algorithm for non-convex, multimodal objective functions. It's particularly effective for problems where gradient information is unavailable or unreliable, and for escaping local optima.
+
+## Function: `differential_evolution`
+
+```python
+from optimizr import differential_evolution
+```
+
+### Signature
+
+```python
+differential_evolution(
+    objective_fn: Callable[[np.ndarray], float],
+    bounds: List[Tuple[float, float]],
+    popsize: int = 15,
+    maxiter: int = 1000,
+    f: float = 0.8,
+    cr: float = 0.7,
+) -> Tuple[np.ndarray, float]
+```
+
+### Parameters
+
+- **`objective_fn`** (callable): Function to minimize.
+  - **Signature**: `objective_fn(x: np.ndarray) -> float`
+  - Takes a 1D array of parameters and returns a scalar objective value.
+  - Lower values are better.
+
+- **`bounds`** (List[Tuple[float, float]]): List of (min, max) bounds for each parameter dimension.
+
+- **`popsize`** (int, optional): Population size multiplier. Total population size will be `popsize × n_params`. Default is 15.
+
+- **`maxiter`** (int, optional): Maximum number of generations. Default is 1,000.
+
+- **`f`** (float, optional): Mutation factor, typically in range [0.5, 2.0]. Controls the amplification of differential variation. Default is 0.8.
+
+- **`cr`** (float, optional): Crossover probability, typically in range [0.1, 0.9]. Controls the fraction of parameter values copied from the mutant. Default is 0.7.
+
+### Returns
+
+Returns a tuple `(x, fun)`:
+- **`x`** (np.ndarray): Best parameters found (minimum).
+- **`fun`** (float): Best objective value (minimum).
+
+Alternatively, when using the Rust backend directly, returns a `DEResult` object with attributes:
+- `x`: Best parameters
+- `fun`: Best objective value
+- `nfev`: Number of function evaluations
+
+## Basic Example
+
+```python
+import numpy as np
+from optimizr import differential_evolution
+
+# Define the Rosenbrock function (global minimum at [1, 1, ..., 1])
+def rosenbrock(x):
+    return sum(100.0 * (x[i+1] - x[i]**2)**2 + (1 - x[i])**2 
+               for i in range(len(x) - 1))
+
+# Optimize
+x_opt, f_min = differential_evolution(
+    objective_fn=rosenbrock,
+    bounds=[(-5, 5)] * 10,
+    popsize=15,
+    maxiter=1000
+)
+
+print(f"Optimal parameters: {x_opt}")
+print(f"Minimum value: {f_min:.6f}")
+print(f"Expected: {rosenbrock(np.ones(10)):.6f}")
+```
+
+## Advanced Examples
+
+### 1. Rastrigin Function (Many Local Minima)
+
+```python
+import numpy as np
+from optimizr import differential_evolution
+
+def rastrigin(x):
+    """Highly multimodal function with many local minima"""
+    A = 10
+    n = len(x)
+    return A * n + sum(xi**2 - A * np.cos(2 * np.pi * xi) for xi in x)
+
+# True global minimum is at origin with f(0, ..., 0) = 0
+x_opt, f_min = differential_evolution(
+    objective_fn=rastrigin,
+    bounds=[(-5.12, 5.12)] * 10,
+    popsize=20,
+    maxiter=2000,
+    f=0.8,
+    cr=0.9
+)
+
+print(f"Minimum found: {f_min:.6f}")
+print(f"Distance from optimum: {np.linalg.norm(x_opt):.6f}")
+```
+
+### 2. Constrained Optimization
+
+```python
+import numpy as np
+from optimizr import differential_evolution
+
+def constrained_objective(x):
+    """Minimize x^2 + y^2 subject to x + y >= 1"""
+    obj = x[0]**2 + x[1]**2
+    
+    # Add penalty for constraint violation
+    constraint = x[0] + x[1] - 1
+    if constraint < 0:
+        obj += 1000 * constraint**2  # Penalty term
+    
+    return obj
+
+x_opt, f_min = differential_evolution(
+    objective_fn=constrained_objective,
+    bounds=[(-5, 5), (-5, 5)],
+    popsize=15,
+    maxiter=500
+)
+
+print(f"Optimal point: ({x_opt[0]:.3f}, {x_opt[1]:.3f})")
+print(f"Constraint: x + y = {x_opt[0] + x_opt[1]:.3f} (should be ≥ 1)")
+print(f"Objective: {f_min:.3f}")
+```
+
+### 3. Hyperparameter Tuning
+
+```python
+import numpy as np
+from sklearn.model_selection import cross_val_score
+from sklearn.svm import SVC
+from sklearn.datasets import load_digits
+from optimizr import differential_evolution
+
+# Load data
+X, y = load_digits(return_X_y=True)
+
+def svm_objective(params):
+    """Optimize SVM hyperparameters"""
+    C, gamma = params
+    
+    # Convert to log scale
+    C = 10 ** C
+    gamma = 10 ** gamma
+    
+    # Cross-validation score (negative because we minimize)
+    model = SVC(C=C, gamma=gamma)
+    score = cross_val_score(model, X, y, cv=3, scoring='accuracy')
+    
+    return -score.mean()  # Negative because we minimize
+
+# Optimize
+params_opt, score_min = differential_evolution(
+    objective_fn=svm_objective,
+    bounds=[(-3, 3), (-5, 1)],  # log10 scale for C and gamma
+    popsize=10,
+    maxiter=30
+)
+
+C_opt = 10 ** params_opt[0]
+gamma_opt = 10 ** params_opt[1]
+
+print(f"Best C: {C_opt:.4f}")
+print(f"Best gamma: {gamma_opt:.6f}")
+print(f"Best CV accuracy: {-score_min:.4f}")
+```
+
+### 4. Portfolio Optimization
+
+```python
+import numpy as np
+from optimizr import differential_evolution
+
+# Sample returns (rows = assets, columns = time periods)
+returns = np.random.randn(5, 1000) * 0.01
+returns += np.array([0.08, 0.10, 0.12, 0.06, 0.09])[:, np.newaxis] / 252
+
+def portfolio_objective(weights):
+    """Maximize Sharpe ratio (minimize negative Sharpe)"""
+    # Ensure weights sum to 1
+    weights = weights / weights.sum()
+    
+    # Calculate portfolio return and volatility
+    portfolio_return = np.sum(returns.mean(axis=1) * weights) * 252
+    portfolio_vol = np.sqrt(
+        np.dot(weights, np.dot(np.cov(returns), weights))
+    ) * np.sqrt(252)
+    
+    # Sharpe ratio (assuming risk-free rate = 2%)
+    sharpe = (portfolio_return - 0.02) / portfolio_vol
+    
+    return -sharpe  # Negative because we minimize
+
+# Optimize
+n_assets = 5
+weights_opt, sharpe_neg = differential_evolution(
+    objective_fn=portfolio_objective,
+    bounds=[(0, 1)] * n_assets,  # Long-only portfolio
+    popsize=20,
+    maxiter=500
+)
+
+# Normalize weights
+weights_opt = weights_opt / weights_opt.sum()
+
+print("Optimal Portfolio Weights:")
+for i, w in enumerate(weights_opt):
+    print(f"  Asset {i+1}: {w:.2%}")
+
+print(f"\nSharpe Ratio: {-sharpe_neg:.3f}")
+```
+
+### 5. Function Fitting
+
+```python
+import numpy as np
+import matplotlib.pyplot as plt
+from optimizr import differential_evolution
+
+# Generate noisy data
+x_data = np.linspace(0, 10, 100)
+y_true = 2.5 * np.sin(0.8 * x_data + 1.2) + 1.5
+y_data = y_true + np.random.normal(0, 0.3, len(x_data))
+
+def fitting_objective(params):
+    """Fit y = A * sin(B * x + C) + D"""
+    A, B, C, D = params
+    y_pred = A * np.sin(B * x_data + C) + D
+    mse = np.mean((y_data - y_pred)**2)
+    return mse
+
+# Optimize
+params_opt, mse_min = differential_evolution(
+    objective_fn=fitting_objective,
+    bounds=[(0, 10), (0, 2), (0, 2*np.pi), (-5, 5)],
+    popsize=15,
+    maxiter=1000
+)
+
+A, B, C, D = params_opt
+print(f"Fitted parameters: A={A:.2f}, B={B:.2f}, C={C:.2f}, D={D:.2f}")
+print(f"MSE: {mse_min:.4f}")
+
+# Plot
+y_fitted = A * np.sin(B * x_data + C) + D
+plt.figure(figsize=(10, 6))
+plt.scatter(x_data, y_data, alpha=0.5, label='Data')
+plt.plot(x_data, y_true, 'g--', label='True', linewidth=2)
+plt.plot(x_data, y_fitted, 'r-', label='Fitted', linewidth=2)
+plt.legend()
+plt.title('Differential Evolution Function Fitting')
+plt.show()
+```
+
+## Parameter Tuning Guide
+
+### Population Size (`popsize`)
+
+- **Small (5-10)**: Fast but may converge prematurely
+- **Medium (15-20)**: Good balance for most problems
+- **Large (30+)**: Better exploration, slower convergence
+
+Rule of thumb: `popsize ≥ 10` for problems with up to 10 parameters.
+
+### Mutation Factor (`f`)
+
+- **Low (0.4-0.6)**: Conservative, good for fine-tuning
+- **Medium (0.7-0.9)**: Standard, works for most problems
+- **High (1.0-2.0)**: Aggressive exploration, avoids local minima
+
+### Crossover Probability (`cr`)
+
+- **Low (0.1-0.3)**: Preserves more of original vector
+- **Medium (0.5-0.7)**: Balanced mixing
+- **High (0.8-1.0)**: Aggressive recombination
+
+### Maximum Iterations (`maxiter`)
+
+- Depends on problem difficulty and dimensions
+- Monitor convergence: if still improving at `maxiter`, increase it
+- Typical values: 500-5000
+
+## Convergence Analysis
+
+```python
+# Track convergence history (requires modification to return history)
+import matplotlib.pyplot as plt
+
+history = []
+
+def tracked_objective(x):
+    result = objective_fn(x)
+    history.append(result)
+    return result
+
+x_opt, f_min = differential_evolution(
+    objective_fn=tracked_objective,
+    bounds=bounds,
+    popsize=15,
+    maxiter=1000
+)
+
+# Plot convergence
+plt.figure(figsize=(10, 6))
+plt.semilogy(history)
+plt.xlabel('Function Evaluation')
+plt.ylabel('Objective Value')
+plt.title('Convergence History')
+plt.grid(True)
+plt.show()
+```
+
+## Performance Notes
+
+- **Rust Backend**: 50-100x faster than pure Python implementations for compute-intensive objectives.
+
+- **Python Fallback**: Falls back to `scipy.optimize.differential_evolution` if Rust is unavailable.
+
+- **Parallelization**: Population evaluations are independent and can be parallelized (future enhancement).
+
+- **Complexity**: O(`popsize` × `n_params` × `maxiter` × cost_per_eval)
+
+## Common Use Cases
+
+| Application | Typical Settings | Notes |
+|-------------|------------------|-------|
+| Hyperparameter tuning | popsize=10-15, maxiter=50-200 | Fast evaluations |
+| Engineering design | popsize=20-30, maxiter=500-2000 | Complex constraints |
+| Function fitting | popsize=15-20, maxiter=500-1000 | Multiple local minima |
+| Portfolio optimization | popsize=15-20, maxiter=200-500 | Moderate dimensions |
+| Neural network training | popsize=30-50, maxiter=1000+ | High dimensions |
+
+## Tips and Best Practices
+
+1. **Scaling**: Normalize parameters to similar ranges for better performance.
+
+2. **Bounds**: Set reasonable bounds based on domain knowledge.
+
+3. **Stochastic Objectives**: For noisy functions, use larger population and more iterations.
+
+4. **Warm Start**: Use results from previous runs as initial population.
+
+5. **Hybrid Approach**: Use DE for global search, then local optimizer for refinement.
+
+6. **Early Stopping**: Implement custom stopping criteria based on improvement rate.
+
+## Comparison with Other Optimizers
+
+| Method | Pros | Cons | When to Use |
+|--------|------|------|-------------|
+| **Differential Evolution** | No gradients needed, global search, robust | Slow for high dimensions | Non-convex, derivative-free |
+| Gradient Descent | Fast, precise | Needs gradients, local only | Smooth, differentiable |
+| Genetic Algorithm | Very flexible | Slower convergence | Discrete, combinatorial |
+| Simulated Annealing | Simple, global search | Sensitive to temperature schedule | Simple problems |
+| Grid Search | Guaranteed coverage | Exponential cost | Few dimensions only |
+
+## See Also
+
+- [Grid Search API](grid_search.md) - For exhaustive parameter search
+- [MCMC API](mcmc.md) - For Bayesian parameter estimation
+- [Differential Evolution Theory](theory/differential_evolution.md) - Mathematical background
+- [Examples](../examples/) - Complete working examples