OptimizR 🚀

High-performance optimization algorithms in Rust with Python bindings

OptimizR provides blazingly fast, production-ready implementations of advanced optimization and statistical inference algorithms. Built with Rust for maximum performance and exposed to Python through PyO3, it delivers 50-100× speedup over pure Python implementations.

✨ What's New in v0.2.0

🎯 Comprehensive Differential Evolution with 5 mutation strategies, adaptive parameter control (jDE), and convergence tracking
🧮 Mathematical Toolkit with numerical differentiation, statistics, linear algebra, and special functions
🎛️ Optimal Control Framework for Hamilton-Jacobi-Bellman equations, regime switching, and jump diffusion
♻️ Major Refactoring with modular architecture, removed legacy code, and generic design patterns
📚 Enhanced Documentation with new tutorial notebooks and detailed API references

→ See Full Release Notes

Features

✨ Algorithms Included:

• Differential Evolution: 5 strategies (rand/1, best/1, current-to-best/1, rand/2, best/2), adaptive jDE, convergence tracking
• Optimal Control: HJB solvers, regime switching, jump diffusion, MRSJD framework
• Hidden Markov Models: Baum-Welch training, Viterbi decoding, Gaussian emissions
• MCMC Sampling: Metropolis-Hastings, adaptive proposals, Bayesian inference
• Sparse Optimization: Sparse PCA, Box-Tao decomposition, Elastic Net, ADMM
• Risk Metrics: Hurst exponent, half-life estimation, time series analysis
• Information Theory: Mutual information, Shannon entropy, feature selection
• Mathematical Toolkit: Gradient, Hessian, Jacobian, statistics, linear algebra

🚀 Performance:

• 50-100× faster than pure Python implementations
• 95% memory reduction vs NumPy/SciPy
• Parallel-ready with Rayon infrastructure
• Production-tested on multi-dimensional problems

🐍 Python-First API:

• Clean, intuitive NumPy-based interface
• Rich result objects with convergence diagnostics
• Type hints and comprehensive documentation
• Jupyter notebook integration

Installation

From PyPI (coming soon)

pip install optimizr

From Source

# Clone the repository
git clone https://github.com/yourusername/optimiz-r.git
cd optimiz-r

# Install with maturin
pip install maturin
maturin develop --release

# Or install in editable mode
pip install -e .

Using Docker

# Start Jupyter notebook server with examples
docker-compose up dev
# Access at http://localhost:8888

# Run all tests
docker-compose run test

# Build distribution wheels
docker-compose run build

Quick Start

Differential Evolution (Enhanced in v0.2.0)

import numpy as np
from optimizr import differential_evolution

# Rosenbrock function (challenging non-convex problem)
def rosenbrock(x):
    return sum(100.0 * (x[1:] - x[:-1]**2)**2 + (1 - x[:-1])**2)

# Optimize with adaptive jDE (self-tuning parameters)
result = differential_evolution(
    objective_fn=rosenbrock,
    bounds=[(-5, 5)] * 10,
    maxiter=1000,
    strategy='best1',  # 5 strategies: rand1, best1, currenttobest1, rand2, best2
    adaptive=True,     # Adaptive F and CR parameters (jDE algorithm)
    atol=1e-6
)

print(f"Optimum: {result.x}")
print(f"Value: {result.fun} (expected: 0.0)")
print(f"Converged: {result.converged}, Iterations: {result.nit}")
# Typical speedup: 74-88× faster than SciPy

Mathematical Toolkit (New in v0.2.0)

from optimizr import maths_toolkit as mt
import numpy as np

# Numerical differentiation
f = lambda x: x[0]**2 + 2*x[1]**2 + x[0]*x[1]
x = np.array([1.0, 2.0])

gradient = mt.gradient(f, x)        # ∇f(x)
hessian = mt.hessian(f, x)          # H(f)(x)
jacobian = mt.jacobian(f, x)        # J(f)(x)

# Statistics
data = np.random.randn(1000)
stats = {
    'mean': mt.mean(data),
    'var': mt.variance(data),
    'std': mt.std_dev(data),
    'skew': mt.skewness(data),
    'kurt': mt.kurtosis(data)
}

# Linear algebra
A = np.random.randn(5, 5)
norm_l1 = mt.norm_l1(A)
norm_l2 = mt.norm_l2(A)
A_norm = mt.normalize(A)

Hidden Markov Model

from optimizr import HMM
import numpy as np

# Fit HMM with regime switching
returns = np.random.randn(1000)
hmm = HMM(n_states=3)
hmm.fit(returns, n_iterations=100)

# Decode most likely state sequence
states = hmm.predict(returns)
print(f"Transition Matrix:\n{hmm.transition_matrix_}")
print(f"Detected states: {states}")

MCMC Sampling

from optimizr import mcmc_sample

# Define log-posterior
def log_likelihood(params, data):
    mu, sigma = params
    return -0.5 * np.sum(((data - mu) / sigma) ** 2) - len(data) * np.log(sigma)

# Sample from posterior
data = np.random.randn(100) + 2.0
samples = mcmc_sample(
    log_likelihood_fn=log_likelihood,
    data=data,
    initial_params=[0.0, 1.0],
    param_bounds=[(-10, 10), (0.1, 10)],
    n_samples=10000,
    burn_in=1000,
    proposal_std=0.1
)

print(f"Posterior mean: {np.mean(samples, axis=0)}")

Optimal Control (New in v0.2.0)

from optimizr import optimal_control
import numpy as np

# Hamilton-Jacobi-Bellman equation solver
# For stochastic control problem: dX_t = μ dt + σ dW_t

# Define problem parameters
grid = np.linspace(-5, 5, 100)
dt = 0.01
horizon = 1.0

# Solve HJB equation
value_function = optimal_control.solve_hjb(
    grid=grid,
    drift=lambda x: -0.1 * x,      # Mean reversion
    diffusion=lambda x: 0.2,        # Constant volatility
    cost=lambda x, u: x**2 + u**2,  # Quadratic cost
    dt=dt,
    horizon=horizon
)

# Compute optimal control policy
policy = optimal_control.compute_policy(value_function, grid)
print(f"Value at origin: {value_function[len(grid)//2]:.4f}")

Information Theory

from optimizr import mutual_information, shannon_entropy

# Calculate mutual information between two variables
x = np.random.randn(1000)
y = 2 * x + np.random.randn(1000) * 0.5

mi = mutual_information(x, y, n_bins=10)
print(f"Mutual Information: {mi:.4f}")

# Calculate entropy
entropy = shannon_entropy(x, n_bins=10)
print(f"Shannon Entropy: {entropy:.4f}")

Algorithm Details

Hidden Markov Models

Implementation of the Baum-Welch algorithm (Expectation-Maximization) for learning HMM parameters:

• Forward-Backward Algorithm: Efficient computation of state probabilities
• Viterbi Decoding: Find most likely state sequence
• Gaussian Emissions: Continuous observation models
• Normalization: Numerical stability for long sequences

Use Cases:

• Regime detection in time series
• Speech recognition
• Biological sequence analysis
• Financial market state identification

MCMC Sampling

Metropolis-Hastings algorithm for sampling from arbitrary probability distributions:

• Adaptive Proposals: Gaussian random walk
• Burn-in Period: Discard initial samples
• Bounded Parameters: Constraint handling
• Convergence Diagnostics: Track acceptance rates

Use Cases:

• Bayesian parameter estimation
• Posterior inference
• Integration of complex distributions
• Uncertainty quantification

Differential Evolution (Enhanced in v0.2.0)

Advanced global optimization for non-convex, multimodal, high-dimensional problems:

5 Mutation Strategies:

• rand/1/bin: Random base vector (exploration)
• best/1/bin: Best individual base (exploitation)
• current-to-best/1/bin: Balanced exploration/exploitation
• rand/2/bin: Two difference vectors (diversity)
• best/2/bin: Best with two differences (aggressive)

Adaptive jDE Algorithm:

• Self-tuning mutation factor (F) and crossover rate (CR)
• Parameter adaptation per individual
• τ₁, τ₂ control adaptation speed
• Eliminates manual parameter tuning

Convergence Features:

• Early stopping with tolerance detection
• Convergence history tracking
• Best fitness evolution monitoring
• Rich diagnostic information

Performance:

• 74-88× faster than SciPy (Python)
• Efficient for 10-1000 dimensional problems
• Memory-efficient population management
• Parallel-ready architecture

Use Cases:

• Hyperparameter optimization (ML/DL)
• Engineering design problems
• Inverse problems and calibration
• Non-smooth, noisy objectives
• Constrained optimization with penalties

Grid Search

Exhaustive search over parameter space:

• Complete Coverage: Evaluate all grid points
• Parallel Ready: Independent evaluations
• Flexible Bounds: Per-parameter ranges
• Best Score Tracking: Return optimal parameters

Use Cases:

• Small parameter spaces
• Benchmark comparisons
• Hyperparameter tuning
• Global optima verification

Information Theory Metrics

Quantify information content and dependencies:

• Mutual Information: I(X;Y) = H(X) + H(Y) - H(X,Y)
• Shannon Entropy: H(X) = -∑ p(x) log p(x)
• Binning Strategy: Histogram-based estimation
• Normalized Variants: Available through Python API

Use Cases:

• Feature selection
• Dependency detection
• Time series analysis
• Causality testing

Mathematical Toolkit (New in v0.2.0)

Centralized mathematical utilities for all algorithms:

Numerical Differentiation:

• gradient(): ∇f(x) with central differences
• hessian(): H(f)(x) second-order derivatives
• jacobian(): J(f)(x) for vector functions
• Configurable step size (h)

Statistics:

• mean(), variance(), std_dev()
• skewness(), kurtosis() for distribution shape
• correlation(), covariance() for dependencies
• Efficient single-pass algorithms

Linear Algebra:

• norm_l1(), norm_l2(), norm_frobenius()
• normalize() for vector/matrix normalization
• trace(), outer_product()
• ndarray-linalg integration

Integration:

• trapz(): Trapezoidal rule
• simpson(): Simpson's rule

Special Functions:

• sigmoid(), softmax()
• soft_threshold() for proximal methods

Use Cases:

• Algorithm development
• Sensitivity analysis
• Statistical inference
• Custom optimization methods

Optimal Control (New in v0.2.0)

Hamilton-Jacobi-Bellman equation solvers for stochastic control:

Features:

• HJB PDE solver with finite difference schemes
• Regime-switching models (Markov chains)
• Jump diffusion processes (Poisson jumps)
• MRSJD (Markov Regime Switching Jump Diffusion)

Components:

• Value function computation
• Optimal policy extraction
• Boundary conditions handling
• Grid-based discretization

Use Cases:

• Portfolio optimization under uncertainty
• Resource management with regime changes
• Risk-sensitive control
• Dynamic programming problems

Performance Benchmarks

Comparison against pure Python/NumPy/SciPy implementations (v0.2.0):

Algorithm	Problem Size	OptimizR (Rust)	NumPy/SciPy	Speedup
DE - rand/1	50D Rosenbrock	285ms	21.2s	74×
DE - best/1	50D Rosenbrock	270ms	23.8s	88×
DE - adaptive jDE	50D Rosenbrock	310ms	24.5s	79×
HMM Fit	10k samples	45ms	3.2s	71×
MCMC Sample	100k iterations	120ms	8.5s	71×
Sparse PCA	1000×100 matrix	180ms	12.5s	69×
Mutual Information	50k points	12ms	380ms	32×
Gradient (numerical)	100D function	8ms	145ms	18×
Hessian (numerical)	50D function	95ms	4.2s	44×

Benchmarks run on Apple M1 Pro, 10 cores, 32GB RAM

Documentation

API Reference

Full API documentation is available in the docs/ directory:

• HMM API
• MCMC API
• Differential Evolution API
• Grid Search API
• Information Theory API

Examples & Tutorials

Complete Jupyter notebook tutorials in examples/notebooks/:

1. Hidden Markov Models - Regime detection, Baum-Welch, Viterbi
2. MCMC Sampling - Metropolis-Hastings, Bayesian inference
3. Differential Evolution - 5 strategies, adaptive jDE, convergence
4. Optimal Control - HJB, regime switching, jump diffusion (NEW in v0.2.0)
5. Real-World Applications - Complete workflows
6. Performance Benchmarks - Detailed comparisons

Python script examples:

• HMM Regime Detection
• Bayesian Inference with MCMC
• Hyperparameter Optimization with DE
• Feature Selection

Mathematical Background

Detailed mathematical descriptions and references:

• HMM Theory
• MCMC Theory
• Differential Evolution Theory - Updated for v0.2.0
• Optimal Control Theory - NEW in v0.2.0
• Information Theory

Development

Building from Source

# Setup development environment
git clone https://github.com/yourusername/optimiz-r.git
cd optimiz-r

# Install development dependencies
pip install -e ".[dev]"

# Build Rust extension
maturin develop

# Run tests
pytest tests/ -v

# Run Rust tests
cargo test

# Run benchmarks
cargo bench

Code Quality

# Format code
black python/
cargo fmt

# Lint
ruff check python/
cargo clippy

# Type checking
mypy python/

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

Areas for Contribution

• Advanced DE variants (JADE, SHADE, L-SHADE)
• GPU acceleration via CUDA/ROCm (see Roadmap)
• Additional optimization algorithms (PSO, CMA-ES, NES)
• More probability distributions for HMM
• Additional language bindings (R, Julia, JavaScript)
• Documentation improvements and tutorials
• Benchmark comparisons and case studies

License

MIT License - see LICENSE file for details.

Citation

If you use OptimizR in your research, please cite:

@software{optimizr2024,
  title = {OptimizR: High-Performance Optimization Algorithms in Rust},
  author = {Your Name},
  year = {2024},
  version = {0.2.0},
  url = {https://github.com/yourusername/optimiz-r}
}

Acknowledgments

Built with:

• Rust - Systems programming language
• PyO3 - Rust bindings for Python
• Maturin - Build and publish Rust crates as Python packages
• NumPy - Numerical computing in Python

Inspired by:

• scipy.optimize
• scikit-learn
• hmmlearn
• emcee

Contact

• Issues: GitHub Issues
• Discussions: GitHub Discussions
• Email: your.email@example.com

---

OptimizR - Fast optimization for data science and machine learning 🚀