gavindiaz/optimiz-rs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Improve code design

Browse Source
This commit is contained in:
Melvin Avarez
2025-12-03 22:08:08 +01:00
parent 923d27e87b
commit a62ceaa64b
18 changed files with 6613 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/theory/differential_evolution.md
+495
View File
@@ -0,0 +1,495 @@
# Differential Evolution: Mathematical Theory
## Introduction
Differential Evolution (DE) is a population-based metaheuristic optimization algorithm introduced by Storn and Price (1997). It is particularly effective for continuous, non-convex, multimodal optimization problems where gradient information is unavailable or unreliable.
## Problem Formulation
### Objective
Minimize $f: \mathbb{R}^D \rightarrow \mathbb{R}$:
$$\min_{\mathbf{x} \in \mathbb{R}^D} f(\mathbf{x})$$
subject to box constraints:
$$x_j \in [l_j, u_j], \quad j = 1, ..., D$$
### Characteristics
**DE is suitable when**:
- $f$ is continuous but non-differentiable
- Multiple local minima exist
- Gradient information is unavailable or expensive
- Problem dimension is moderate ($D < 100$)
## Algorithm Overview
### Population
Maintain a population of $N_P$ candidate solutions:
$$P_g = \{\mathbf{x}_{1,g}, \mathbf{x}_{2,g}, ..., \mathbf{x}_{N_P,g}\}$$
where $g$ is the generation number and $\mathbf{x}_{i,g} \in \mathbb{R}^D$.
### Main Loop
For each generation $g = 0, 1, 2, ...$:
1. **Mutation**: Create mutant vectors
2. **Crossover**: Create trial vectors
3. **Selection**: Keep better solutions
## Mutation Strategies
### DE/rand/1 (Classic)
For each target vector $\mathbf{x}_{i,g}$, create mutant:
$$\mathbf{v}_{i,g+1} = \mathbf{x}_{r_1,g} + F \cdot (\mathbf{x}_{r_2,g} - \mathbf{x}_{r_3,g})$$
where:
- $r_1, r_2, r_3 \in \{1, ..., N_P\}$ are randomly chosen, distinct, and $\neq i$
- $F \in (0, 2]$ is the **mutation factor** (typically 0.5-1.0)
**Interpretation**:
- Start from a random population member $\mathbf{x}_{r_1}$
- Move in direction given by difference $(\mathbf{x}_{r_2} - \mathbf{x}_{r_3})$
- Scale movement by $F$
### DE/best/1
$$\mathbf{v}_{i,g+1} = \mathbf{x}_{\text{best},g} + F \cdot (\mathbf{x}_{r_1,g} - \mathbf{x}_{r_2,g})$$
**Advantage**: Faster convergence
**Disadvantage**: More likely to get stuck in local minima
### DE/current-to-best/1
$$\mathbf{v}_{i,g+1} = \mathbf{x}_{i,g} + F \cdot (\mathbf{x}_{\text{best},g} - \mathbf{x}_{i,g}) + F \cdot (\mathbf{x}_{r_1,g} - \mathbf{x}_{r_2,g})$$
**Interpretation**: Move current solution toward best while exploring
### DE/rand/2
$$\mathbf{v}_{i,g+1} = \mathbf{x}_{r_1,g} + F \cdot (\mathbf{x}_{r_2,g} - \mathbf{x}_{r_3,g}) + F \cdot (\mathbf{x}_{r_4,g} - \mathbf{x}_{r_5,g})$$
More disruptive, better for highly multimodal problems.
## Crossover
### Binomial Crossover
For each component $j = 1, ..., D$:
$$u_{i,j,g+1} = \begin{cases}
v_{i,j,g+1} & \text{if } \text{rand}(0,1) \leq CR \text{ or } j = j_{\text{rand}} \\
x_{i,j,g} & \text{otherwise}
\end{cases}$$
where:
- $CR \in [0, 1]$ is the **crossover probability**
- $j_{\text{rand}} \in \{1, ..., D\}$ ensures at least one component is from mutant
**Effect**: Controls how much of the mutant vector is used
### Exponential Crossover
Copy consecutive components from mutant with probability $CR$.
Less common, similar performance to binomial.
## Selection
Greedy selection (for minimization):
$$\mathbf{x}_{i,g+1} = \begin{cases}
\mathbf{u}_{i,g+1} & \text{if } f(\mathbf{u}_{i,g+1}) \leq f(\mathbf{x}_{i,g}) \\
\mathbf{x}_{i,g} & \text{otherwise}
\end{cases}$$
**Property**: Population quality never decreases:
$$f(\mathbf{x}_{\text{best},g+1}) \leq f(\mathbf{x}_{\text{best},g})$$
## Complete Algorithm
```
1. Initialize population:
For i = 1 to N_P:
x_{i,0} = l + rand(0,1) · (u - l)
2. Evaluate fitness:
f_i = f(x_{i,0}) for all i
3. While stopping criterion not met:
a. For i = 1 to N_P:
i. Mutation:
Select r_1, r_2, r_3 distinct and ≠ i
v_{i,g+1} = x_{r_1,g} + F · (x_{r_2,g} - x_{r_3,g})
ii. Crossover:
j_rand = randint(1, D)
For j = 1 to D:
if rand(0,1) ≤ CR or j = j_rand:
u_{i,j,g+1} = v_{i,j,g+1}
else:
u_{i,j,g+1} = x_{i,j,g}
iii. Boundary handling:
Clip u_{i,g+1} to [l, u]
iv. Selection:
if f(u_{i,g+1}) ≤ f(x_{i,g}):
x_{i,g+1} = u_{i,g+1}
else:
x_{i,g+1} = x_{i,g}
b. g = g + 1
4. Return x_best and f(x_best)
```
## Parameter Selection
### Population Size ($N_P$)
**Rule of thumb**: $N_P = 10D$ where $D$ is problem dimension
**Small population** (< 4D):
- Faster convergence
- Risk premature convergence
- Use for: simple unimodal problems
**Large population** (> 20D):
- Better exploration
- Slower convergence
- Use for: highly multimodal problems
**Minimum**: $N_P \geq 4$ (needed for mutation)
### Mutation Factor ($F$)
**Typical range**: $F \in [0.4, 1.0]$
**Low F** (0.4-0.6):
- Fine-tuning, local search
- Use near end of optimization
- Safer, less disruptive
**High F** (0.8-1.2):
- Exploration, global search
- Escape local minima
- More aggressive
**Adaptive F**: Some variants adjust $F$ during optimization
### Crossover Probability ($CR$)
**Typical range**: $CR \in [0.1, 0.9]$
**Low CR** (0.1-0.3):
- Less information exchange
- Slower convergence
- Use for: separable problems
**High CR** (0.7-0.9):
- More information exchange
- Faster convergence
- Use for: non-separable problems
**Special cases**:
- $CR = 0$: Pure mutation (except $j_{\text{rand}}$)
- $CR = 1$: Full crossover
### Stopping Criteria
1. **Maximum generations**: $g_{\max}$
2. **Function evaluations**: $FE_{\max}$
3. **Target fitness**: $f(\mathbf{x}_{\text{best}}) \leq f_{\text{target}}$
4. **Stagnation**: No improvement for $G_{\text{stag}}$ generations
5. **Diversity loss**: Population variance below threshold
## Convergence Analysis
### Theoretical Results
**Theorem** (Zaharie, 2002): Under certain conditions on $F$ and $CR$, DE converges to a stationary point.
**Conditions**:
- Bounded search space
- Continuous objective function
- Appropriate parameter settings
### Convergence Rate
**Empirical observations**:
- Linear convergence in early stages
- Slows down near optimum
- Faster than genetic algorithms for many problems
- Slower than gradient methods (when gradients available)
### No Free Lunch
DE is not universally optimal. Performance depends on:
- Problem landscape
- Parameter settings
- Population size
## Variants and Extensions
### Self-Adaptive DE (jDE)
Parameters $F$ and $CR$ evolve with the population:
$$F_{i,g+1} = \begin{cases}
F_l + \text{rand}(0,1) \cdot (F_u - F_l) & \text{if } \text{rand}(0,1) < \tau_1 \\
F_{i,g} & \text{otherwise}
\end{cases}$$
$$CR_{i,g+1} = \begin{cases}
\text{rand}(0,1) & \text{if } \text{rand}(0,1) < \tau_2 \\
CR_{i,g} & \text{otherwise}
\end{cases}$$
### SHADE (Success-History Adaptive DE)
Uses historical information about successful parameters.
### L-SHADE
SHADE with linear population size reduction.
### CoDE (Composite DE)
Uses multiple mutation strategies simultaneously.
### Opposition-Based DE
Initialize with both random solutions and their opposites.
### Constraint Handling
For constrained optimization:
1. **Penalty method**: Add penalty to objective
2. **Feasibility rules**: Prefer feasible solutions
3. **ε-constrained**: Relax constraints gradually
## Theoretical Properties
### Global Convergence
**Sufficient conditions** (Lampinen, 2001):
- Population size $N_P > 3$
- Mutation factor $F > 0$
- At least one component crossed over ($j_{\text{rand}}$)
Then DE is a **global optimization method**: Can reach any point with positive probability.
### Diversity Maintenance
Mutation creates diversity, selection reduces it. Balance determines exploration vs. exploitation.
**Diversity measure**:
$$D_g = \frac{1}{N_P D} \sum_{i=1}^{N_P} \sum_{j=1}^D |x_{i,j,g} - \bar{x}_{j,g}|$$
High diversity → exploration
Low diversity → exploitation
### Convergence Speed
**Expected number of generations** to reach near-optimum depends on:
- Problem difficulty (number of local minima, basin sizes)
- Population size
- Parameter settings
**Empirical rule**: Budget $10^4 D$ function evaluations for moderately difficult problems.
## Comparison with Other Algorithms
| Algorithm | Gradient | Global | Constraints | Speed | Best For |
|-----------|----------|--------|-------------|-------|----------|
| **DE** | No | Yes | Penalty | Medium | Non-convex, continuous |
| Gradient Descent | Yes | No | Yes | Fast | Smooth, convex |
| Genetic Algorithm | No | Yes | Yes | Slow | Discrete, combinatorial |
| Particle Swarm | No | Yes | Penalty | Fast | Continuous, many dims |
| Simulated Annealing | No | Yes | Penalty | Slow | Small problems |
| CMA-ES | No | Yes | Penalty | Fast | Continuous, noisy |
## Applications
### 1. Engineering Design
**Example**: Antenna design
- Objective: Maximize gain, minimize side lobes
- Constraints: Physical realizability
- High-dimensional, non-convex
### 2. Machine Learning
**Example**: Neural network hyperparameter tuning
- Objective: Validation accuracy
- Parameters: Learning rate, regularization, architecture
- Noisy, expensive evaluations
### 3. Chemical Engineering
**Example**: Reactor optimization
- Objective: Maximize yield, minimize cost
- Constraints: Safety, temperature, pressure
- Nonlinear dynamics
### 4. Portfolio Optimization
**Example**: Asset allocation
- Objective: Maximize Sharpe ratio
- Constraints: Budget, diversification
- Non-convex risk measures
### 5. System Identification
**Example**: Parameter estimation
- Objective: Minimize prediction error
- Parameters: Model coefficients
- Multimodal likelihood surface
## Computational Complexity
### Time Complexity
Per generation: $O(N_P \cdot D \cdot T_f)$
where $T_f$ is cost of evaluating $f$.
Total: $O(G_{\max} \cdot N_P \cdot D \cdot T_f)$
### Space Complexity
$O(N_P \cdot D)$ for population storage.
### Parallelization
**Embarrassingly parallel**: Each trial vector evaluation is independent.
**Speedup**: Near-linear with number of processors (up to $N_P$ processors).
## Practical Tips
### 1. Start Simple
Use default parameters: $N_P = 10D$, $F = 0.8$, $CR = 0.7$
### 2. Scale Variables
Normalize parameters to similar ranges for better performance.
### 3. Warm Start
If you have a good initial guess, seed population around it.
### 4. Hybrid Approach
Use DE for global search, then local optimizer for refinement:
```
1. Run DE for G_global generations
2. Take best solution x_best
3. Run local optimizer starting from x_best
```
### 5. Monitor Convergence
Plot:
- Best fitness vs. generation
- Average fitness vs. generation
- Population diversity vs. generation
### 6. Restarts
If premature convergence detected, restart with new random population.
## Advantages and Limitations
### Advantages
✅ No gradient information needed
✅ Handles non-convex, multimodal functions well
✅ Few parameters to tune
✅ Simple to implement
✅ Robust across problem types
✅ Naturally handles box constraints
✅ Population maintains diversity
### Limitations
❌ Slower than gradient methods (when gradients available)
❌ Scales poorly to high dimensions ($D > 100$)
❌ No convergence guarantees for finite time
❌ Requires many function evaluations
❌ Performance sensitive to parameters
❌ Difficult to handle complex constraints
❌ No theoretical optimal parameter settings
## Key References
1. **Storn, R., & Price, K.** (1997). *Differential evolution - A simple and efficient heuristic for global optimization over continuous spaces*. Journal of Global Optimization, 11(4), 341-359.
- Original DE paper
2. **Price, K., Storn, R. M., & Lampinen, J. A.** (2005). *Differential Evolution: A Practical Approach to Global Optimization*. Springer.
- Comprehensive book on DE
3. **Das, S., & Suganthan, P. N.** (2011). *Differential evolution: A survey of the state-of-the-art*. IEEE Transactions on Evolutionary Computation, 15(1), 4-31.
- Survey of DE variants and applications
4. **Brest, J., Greiner, S., Bošković, B., Mernik, M., & Žumer, V.** (2006). *Self-adapting control parameters in differential evolution: A comparative study on numerical benchmark problems*. IEEE Transactions on Evolutionary Computation, 10(6), 646-657.
- jDE algorithm
5. **Tanabe, R., & Fukunaga, A.** (2013). *Success-history based parameter adaptation for differential evolution*. In IEEE Congress on Evolutionary Computation (pp. 71-78).
- SHADE algorithm
6. **Qin, A. K., Huang, V. L., & Suganthan, P. N.** (2009). *Differential evolution algorithm with strategy adaptation for global numerical optimization*. IEEE Transactions on Evolutionary Computation, 13(2), 398-417.
- Self-adaptive DE
## Summary
Differential Evolution is a powerful metaheuristic for global optimization:
**Key Features**:
- Population-based search
- Mutation, crossover, selection operators
- Self-organizing behavior
**Best suited for**:
- Non-convex, multimodal problems
- Moderate dimensions (< 100)
- When gradients unavailable
- Robust optimization needed
**Success factors**:
- Appropriate parameter settings
- Sufficient population size
- Adequate function evaluation budget
## See Also
- [Differential Evolution API Documentation](../differential_evolution.md) - Implementation and usage
- [Grid Search Theory](../grid_search.md) - Alternative for small spaces
- [MCMC Theory](mcmc.md) - Sampling-based inference
docs/theory/hmm.md
+367
View File
@@ -0,0 +1,367 @@
# Hidden Markov Models: Mathematical Theory
## Introduction
Hidden Markov Models (HMMs) are probabilistic models for sequential data where we observe a sequence of outputs generated by a system that transitions between hidden (latent) states. HMMs are widely used in speech recognition, biological sequence analysis, financial time series, and many other domains.
## Model Definition
A Hidden Markov Model $\lambda$ is defined by:
### 1. States
- **Number of states**: $N$
- **State at time $t$**: $q_t \in \{1, 2, ..., N\}$
- **State sequence**: $Q = q_1, q_2, ..., q_T$
### 2. Observations
- **Number of possible observations**: $M$ (discrete) or $\mathbb{R}$ (continuous)
- **Observation at time $t$**: $o_t$
- **Observation sequence**: $O = o_1, o_2, ..., o_T$
### 3. Parameters
**Initial State Distribution**:
$$\pi_i = P(q_1 = i), \quad 1 \leq i \leq N$$
$$\sum_{i=1}^N \pi_i = 1$$
**State Transition Probabilities**:
$$a_{ij} = P(q_{t+1} = j \mid q_t = i), \quad 1 \leq i,j \leq N$$
$$\sum_{j=1}^N a_{ij} = 1 \text{ for all } i$$
**Emission Probabilities** (continuous case - Gaussian):
$$b_j(o_t) = P(o_t \mid q_t = j) = \mathcal{N}(o_t; \mu_j, \sigma_j^2)$$
$$b_j(o_t) = \frac{1}{\sigma_j\sqrt{2\pi}} \exp\left(-\frac{(o_t - \mu_j)^2}{2\sigma_j^2}\right)$$
Complete model: $\lambda = (\pi, A, B)$ where:
- $\pi$ is the initial state distribution
- $A = \{a_{ij}\}$ is the transition matrix
- $B = \{b_j(o)\}$ is the emission distribution
## Markov Assumptions
### First-Order Markov Property
The future state depends only on the current state, not the history:
$$P(q_{t+1} \mid q_1, q_2, ..., q_t) = P(q_{t+1} \mid q_t)$$
### Output Independence
Observations are conditionally independent given the state:
$$P(o_t \mid q_1, ..., q_T, o_1, ..., o_{t-1}, o_{t+1}, ..., o_T) = P(o_t \mid q_t)$$
## Fundamental Problems
### 1. Evaluation Problem
**Given**: Model $\lambda = (\pi, A, B)$ and observation sequence $O$
**Find**: $P(O \mid \lambda)$, the probability that the model generated the observations
**Solution**: Forward Algorithm (or Backward Algorithm)
### 2. Decoding Problem
**Given**: Model $\lambda$ and observation sequence $O$
**Find**: $Q^* = \arg\max_Q P(Q \mid O, \lambda)$, the most likely state sequence
**Solution**: Viterbi Algorithm
### 3. Learning Problem
**Given**: Observation sequence $O$
**Find**: $\lambda^* = \arg\max_\lambda P(O \mid \lambda)$, the model parameters that best explain $O$
**Solution**: Baum-Welch Algorithm (Expectation-Maximization)
## Forward Algorithm
Computes $P(O \mid \lambda)$ efficiently using dynamic programming.
### Forward Variable
$$\alpha_t(i) = P(o_1, o_2, ..., o_t, q_t = i \mid \lambda)$$
The probability of observing the first $t$ observations and being in state $i$ at time $t$.
### Algorithm
**Initialization** ($t = 1$):
$$\alpha_1(i) = \pi_i b_i(o_1), \quad 1 \leq i \leq N$$
**Recursion** ($1 \leq t < T$):
$$\alpha_{t+1}(j) = \left[\sum_{i=1}^N \alpha_t(i) a_{ij}\right] b_j(o_{t+1})$$
**Termination**:
$$P(O \mid \lambda) = \sum_{i=1}^N \alpha_T(i)$$
### Complexity
- **Time**: $O(N^2 T)$
- **Space**: $O(NT)$
Without dynamic programming: $O(N^T)$ - exponential!
## Backward Algorithm
Alternative computation of $P(O \mid \lambda)$.
### Backward Variable
$$\beta_t(i) = P(o_{t+1}, o_{t+2}, ..., o_T \mid q_t = i, \lambda)$$
The probability of observing the remaining observations given state $i$ at time $t$.
### Algorithm
**Initialization** ($t = T$):
$$\beta_T(i) = 1, \quad 1 \leq i \leq N$$
**Recursion** ($t = T-1, T-2, ..., 1$):
$$\beta_t(i) = \sum_{j=1}^N a_{ij} b_j(o_{t+1}) \beta_{t+1}(j)$$
**Termination**:
$$P(O \mid \lambda) = \sum_{i=1}^N \pi_i b_i(o_1) \beta_1(i)$$
## Viterbi Algorithm
Finds the single most likely state sequence.
### Objective
$$Q^* = \arg\max_Q P(Q \mid O, \lambda) = \arg\max_Q P(Q, O \mid \lambda)$$
### Viterbi Variable
$$\delta_t(i) = \max_{q_1, ..., q_{t-1}} P(q_1, ..., q_{t-1}, q_t = i, o_1, ..., o_t \mid \lambda)$$
The maximum probability of any path ending in state $i$ at time $t$.
### Algorithm
**Initialization** ($t = 1$):
$$\delta_1(i) = \pi_i b_i(o_1)$$
$$\psi_1(i) = 0$$
**Recursion** ($2 \leq t \leq T$):
$$\delta_t(j) = \max_{1 \leq i \leq N} [\delta_{t-1}(i) a_{ij}] b_j(o_t)$$
$$\psi_t(j) = \arg\max_{1 \leq i \leq N} [\delta_{t-1}(i) a_{ij}]$$
**Termination**:
$$P^* = \max_{1 \leq i \leq N} \delta_T(i)$$
$$q_T^* = \arg\max_{1 \leq i \leq N} \delta_T(i)$$
**Backtracking** ($t = T-1, T-2, ..., 1$):
$$q_t^* = \psi_{t+1}(q_{t+1}^*)$$
### Complexity
- **Time**: $O(N^2 T)$
- **Space**: $O(NT)$
## Baum-Welch Algorithm
An Expectation-Maximization (EM) algorithm for learning HMM parameters.
### Auxiliary Variables
**State occupation probability**:
$$\gamma_t(i) = P(q_t = i \mid O, \lambda) = \frac{\alpha_t(i)\beta_t(i)}{\sum_{j=1}^N \alpha_t(j)\beta_t(j)}$$
**Transition probability**:
$$\xi_t(i,j) = P(q_t = i, q_{t+1} = j \mid O, \lambda)$$
$$= \frac{\alpha_t(i) a_{ij} b_j(o_{t+1}) \beta_{t+1}(j)}{\sum_{i=1}^N \sum_{j=1}^N \alpha_t(i) a_{ij} b_j(o_{t+1}) \beta_{t+1}(j)}$$
### E-Step
Compute $\gamma_t(i)$ and $\xi_t(i,j)$ for all $t$, $i$, $j$ using current parameters.
### M-Step
Update parameters to maximize expected log-likelihood:
**Initial state probabilities**:
$$\bar{\pi}_i = \gamma_1(i)$$
**Transition probabilities**:
$$\bar{a}_{ij} = \frac{\sum_{t=1}^{T-1} \xi_t(i,j)}{\sum_{t=1}^{T-1} \gamma_t(i)}$$
**Emission parameters** (Gaussian):
$$\bar{\mu}_j = \frac{\sum_{t=1}^T \gamma_t(j) o_t}{\sum_{t=1}^T \gamma_t(j)}$$
$$\bar{\sigma}_j^2 = \frac{\sum_{t=1}^T \gamma_t(j) (o_t - \bar{\mu}_j)^2}{\sum_{t=1}^T \gamma_t(j)}$$
### Convergence
Iterate E-step and M-step until:
$$|L(\lambda^{(k+1)}) - L(\lambda^{(k)})| < \epsilon$$
where $L(\lambda) = \log P(O \mid \lambda)$ is the log-likelihood.
### Properties
- Guaranteed to converge to a **local maximum**
- May converge to different solutions depending on initialization
- Multiple random restarts recommended
## Numerical Stability
### Scaling
Raw probabilities can underflow for long sequences. Use **scaling factors**:
$$c_t = \frac{1}{\sum_{i=1}^N \alpha_t(i)}$$
Scaled forward variables:
$$\hat{\alpha}_t(i) = c_t \alpha_t(i)$$
### Log-Space Computation
For Viterbi, work in log-space:
$$\log \delta_t(j) = \max_{1 \leq i \leq N} [\log \delta_{t-1}(i) + \log a_{ij}] + \log b_j(o_t)$$
Use log-sum-exp trick for additions:
$$\log(e^a + e^b) = \max(a,b) + \log(1 + e^{-|a-b|})$$
## Model Selection
### Number of States
**Information Criteria**:
- **AIC** (Akaike): $-2\log L + 2k$
- **BIC** (Bayesian): $-2\log L + k\log n$
where $k$ is the number of parameters and $n$ is the sample size.
Lower values indicate better models (penalized for complexity).
### Cross-Validation
Split data into training and validation sets. Choose $N$ that maximizes validation log-likelihood.
## Extensions
### Multiple Observation Sequences
Train on multiple sequences $O^{(1)}, ..., O^{(K)}$:
$$\lambda^* = \arg\max_\lambda \prod_{k=1}^K P(O^{(k)} \mid \lambda)$$
Modify M-step to sum statistics across sequences.
### Continuous Observation Mixtures
Use mixture of Gaussians for emissions:
$$b_j(o_t) = \sum_{m=1}^M c_{jm} \mathcal{N}(o_t; \mu_{jm}, \sigma_{jm}^2)$$
where $\sum_{m=1}^M c_{jm} = 1$.
### Higher-Order HMMs
Second-order: $P(q_t \mid q_{t-1}, q_{t-2})$
Increases state space from $N$ to $N^2$.
### Semi-Markov Models
Allow state durations to have explicit distributions.
## Applications
### 1. Financial Markets
**Regime Detection**:
- States: bull market, bear market, high volatility, etc.
- Observations: returns, volatility measures
- Identify market regime changes
### 2. Speech Recognition
**Phoneme Recognition**:
- States: phonemes or sub-phoneme states
- Observations: acoustic features (MFCC)
- Decode speech to text
### 3. Bioinformatics
**Gene Prediction**:
- States: exon, intron, intergenic
- Observations: DNA nucleotides
- Identify gene locations
### 4. Natural Language Processing
**Part-of-Speech Tagging**:
- States: noun, verb, adjective, etc.
- Observations: words
- Tag each word with its grammatical role
## Computational Considerations
### Parallel Forward-Backward
States at each time step can be computed independently within the time step.
### Sparse Transitions
If transition matrix is sparse, exploit sparsity:
- Only store non-zero transitions
- Skip zero-probability paths
### GPU Acceleration
Matrix operations in forward-backward and Viterbi are parallelizable on GPUs.
## Theoretical Properties
### Ergodicity
An HMM is **ergodic** if every state can be reached from every other state.
For ergodic HMMs:
- Unique stationary distribution exists
- Baum-Welch converges to global maximum (under certain conditions)
### Identifiability
HMMs are **not identifiable**: different parameter sets can produce same observations.
**Label switching**: permuting states gives equivalent model.
## Key References
1. **Rabiner, L. R.** (1989). *A tutorial on hidden Markov models and selected applications in speech recognition*. Proceedings of the IEEE, 77(2), 257-286.
- Seminal tutorial paper
2. **Baum, L. E., & Petrie, T.** (1966). *Statistical inference for probabilistic functions of finite state Markov chains*. The Annals of Mathematical Statistics, 37(6), 1554-1563.
- Original Baum-Welch algorithm
3. **Viterbi, A.** (1967). *Error bounds for convolutional codes and an asymptotically optimum decoding algorithm*. IEEE Transactions on Information Theory, 13(2), 260-269.
- Viterbi algorithm
4. **Durbin, R., Eddy, S. R., Krogh, A., & Mitchison, G.** (1998). *Biological Sequence Analysis: Probabilistic Models of Proteins and Nucleic Acids*. Cambridge University Press.
- HMMs for bioinformatics
5. **Cappé, O., Moulines, E., & Rydén, T.** (2005). *Inference in Hidden Markov Models*. Springer.
- Comprehensive mathematical treatment
## Summary
Hidden Markov Models provide a powerful framework for modeling sequential data with latent structure. The three fundamental algorithms:
1. **Forward-Backward**: Compute probabilities efficiently
2. **Viterbi**: Find most likely state sequence
3. **Baum-Welch**: Learn parameters from data
Together, these enable HMMs to solve a wide range of pattern recognition and time series problems.
## See Also
- [HMM API Documentation](../hmm.md) - Implementation details and usage
- [MCMC Theory](mcmc.md) - Alternative inference method for more complex models
- [Information Theory](information_theory.md) - Theoretical foundations for measuring information
docs/theory/information_theory.md
+512
View File
@@ -0,0 +1,512 @@
# Information Theory: Mathematical Foundations
## Introduction
Information theory, founded by Claude Shannon in 1948, provides a mathematical framework for quantifying information, uncertainty, and communication. It has applications in data compression, communication, cryptography, machine learning, and statistical inference.
## Shannon Entropy
### Definition
For a discrete random variable $X$ with probability mass function $p(x)$:
$$H(X) = -\sum_{x \in \mathcal{X}} p(x) \log p(x)$$
**Convention**: $0 \log 0 = 0$ (limit as $p \to 0$)
### Units
- **Nats**: Natural logarithm (base $e$)
- **Bits**: Logarithm base 2
- **Dits**: Logarithm base 10
**Conversion**: $H_{\text{bits}} = H_{\text{nats}} / \ln(2) \approx 1.4427 \cdot H_{\text{nats}}$
### Interpretation
**Shannon entropy measures**:
1. **Uncertainty** about $X$ before observation
2. **Information content** of a sample from $X$
3. **Average code length** (optimal compression)
4. **Unpredictability** of $X$
### Properties
**Non-negativity**:
$$H(X) \geq 0$$
Equality iff $X$ is deterministic (probability 1 on one outcome).
**Maximum entropy**:
$$H(X) \leq \log |\mathcal{X}|$$
Achieved by uniform distribution: $p(x) = 1/|\mathcal{X}|$ for all $x$.
**Concavity**:
$H$ is a concave function of the distribution $p$.
### Examples
**Binary variable** ($p$ = probability of success):
$$H(X) = -p\log p - (1-p)\log(1-p)$$
Maximum at $p = 0.5$: $H_{\text{bits}} = 1$ bit.
**Fair die**:
$$H(X) = -\sum_{i=1}^6 \frac{1}{6}\log\frac{1}{6} = \log 6 \approx 1.79 \text{ bits}$$
**Biased die** (probability 0.5 for face 1, 0.1 for others):
$$H(X) = -0.5\log(0.5) - 5 \times 0.1\log(0.1) \approx 1.36 \text{ bits}$$
Less entropy than fair die (more predictable).
## Continuous Entropy (Differential Entropy)
### Definition
For continuous random variable $X$ with density $f(x)$:
$$h(X) = -\int f(x) \log f(x) dx$$
### Differences from Discrete Case
- Can be **negative**
- Not invariant under coordinate transformations
- Measures relative information (to uniform over support)
### Gaussian Distribution
For $X \sim \mathcal{N}(\mu, \sigma^2)$:
$$h(X) = \frac{1}{2}\log(2\pi e \sigma^2)$$
**Maximal entropy** among all distributions with variance $\sigma^2$.
### Multivariate Gaussian
For $\mathbf{X} \sim \mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma})$:
$$h(\mathbf{X}) = \frac{1}{2}\log\det(2\pi e \boldsymbol{\Sigma})$$
## Joint and Conditional Entropy
### Joint Entropy
For pair $(X, Y)$:
$$H(X, Y) = -\sum_{x,y} p(x,y) \log p(x,y)$$
**Chain rule**:
$$H(X, Y) = H(X) + H(Y|X)$$
### Conditional Entropy
$$H(Y|X) = -\sum_{x,y} p(x,y) \log p(y|x)$$
**Interpretation**: Average uncertainty in $Y$ given $X$.
**Property**:
$$H(Y|X) \leq H(Y)$$
Conditioning reduces entropy (information never increases uncertainty).
Equality iff $X$ and $Y$ are independent.
## Mutual Information
### Definition
$$I(X; Y) = H(X) + H(Y) - H(X, Y)$$
Alternatively:
$$I(X; Y) = \sum_{x,y} p(x,y) \log \frac{p(x,y)}{p(x)p(y)}$$
Or:
$$I(X; Y) = H(X) - H(X|Y) = H(Y) - H(Y|X)$$
### Interpretation
**Mutual information measures**:
1. **Reduction** in uncertainty about $X$ given $Y$
2. **Shared information** between $X$ and $Y$
3. **Dependence** between $X$ and $Y$
4. **Distance** from independence
### Properties
**Non-negativity**:
$$I(X; Y) \geq 0$$
Equality iff $X$ and $Y$ are independent.
**Symmetry**:
$$I(X; Y) = I(Y; X)$$
**Bounded**:
$$I(X; Y) \leq \min(H(X), H(Y))$$
Equality when one variable completely determines the other.
**Data processing inequality**:
If $X \to Y \to Z$ form a Markov chain:
$$I(X; Z) \leq I(X; Y)$$
Processing can't increase information.
### Relationship to Correlation
For bivariate Gaussian $(X, Y)$ with correlation $\rho$:
$$I(X; Y) = -\frac{1}{2}\log(1 - \rho^2)$$
**Mutual information** detects both linear and nonlinear dependencies, while **Pearson correlation** only detects linear.
## Kullback-Leibler Divergence
### Definition
For distributions $p$ and $q$ over $\mathcal{X}$:
$$D_{KL}(p \| q) = \sum_{x \in \mathcal{X}} p(x) \log \frac{p(x)}{q(x)}$$
**Continuous case**:
$$D_{KL}(p \| q) = \int p(x) \log \frac{p(x)}{q(x)} dx$$
### Interpretation
- **Relative entropy**: Information gain when updating from $q$ to $p$
- **Divergence**: How much $p$ differs from $q$
- **Inefficiency**: Extra bits needed when using code for $q$ to encode $p$
### Properties
**Non-negativity** (Gibbs' inequality):
$$D_{KL}(p \| q) \geq 0$$
Equality iff $p = q$ (almost everywhere).
**Asymmetry**:
$$D_{KL}(p \| q) \neq D_{KL}(q \| p)$$
Not a true distance metric (doesn't satisfy triangle inequality).
**Connection to MI**:
$$I(X; Y) = D_{KL}(p(x,y) \| p(x)p(y))$$
MI is the KL divergence from joint to product of marginals.
## Cross Entropy
### Definition
$$H(p, q) = -\sum_x p(x) \log q(x)$$
**Relationship to KL divergence**:
$$H(p, q) = H(p) + D_{KL}(p \| q)$$
### Machine Learning Application
**Loss function** in classification:
For true distribution $p$ (one-hot) and predicted $q$ (softmax):
$$\text{Loss} = H(p, q)$$
Minimizing cross-entropy ≡ minimizing KL divergence ≡ maximizing likelihood.
## Estimation from Data
### Histogram Method
Given samples $x_1, ..., x_n$ from continuous distribution:
1. **Discretize**: Create histogram with $m$ bins
2. **Estimate probabilities**: $\hat{p}_i = n_i / n$ where $n_i$ is count in bin $i$
3. **Compute entropy**: $\hat{H}(X) = -\sum_{i=1}^m \hat{p}_i \log \hat{p}_i$
### Bin Selection
**Too few bins**: Underestimates entropy (over-smoothing)
**Too many bins**: Overestimates entropy (noise)
**Rules of thumb**:
- Sturges: $m = \lceil \log_2 n + 1 \rceil$
- Scott: $m = \lceil (x_{\max} - x_{\min}) / (3.5 \sigma n^{-1/3}) \rceil$
- Square root: $m = \lceil \sqrt{n} \rceil$
### Bias Correction
Histogram estimator is **biased** (tends to overestimate).
**Miller-Madow correction**:
$$\hat{H}_{\text{corrected}} = \hat{H} - \frac{m - 1}{2n}$$
where $m$ is number of non-empty bins.
### Mutual Information Estimation
For samples $(x_i, y_i)$, $i = 1, ..., n$:
1. Create 2D histogram (or separate 1D histograms)
2. Estimate joint and marginal probabilities
3. Compute:
$$\hat{I}(X; Y) = \sum_{i,j} \hat{p}_{ij} \log \frac{\hat{p}_{ij}}{\hat{p}_i \hat{p}_j}$$
**Alternative estimators**:
- k-nearest neighbors (Kraskov et al., 2004)
- Kernel density estimation
- Copula-based methods
## Information-Theoretic Principles
### Maximum Entropy Principle
**Given**: Constraints on moments or expectations
**Find**: Distribution with maximum entropy satisfying constraints
**Result**: Least informative distribution consistent with knowledge
**Example**: Max entropy with mean $\mu$ and variance $\sigma^2$ → Gaussian $\mathcal{N}(\mu, \sigma^2)$
### Minimum Description Length (MDL)
**Model selection**: Choose model that minimizes:
$$\text{Description Length} = \text{Data encoding cost} + \text{Model encoding cost}$$
Related to Bayesian Information Criterion (BIC).
### Information Bottleneck
**Goal**: Compress $X$ to $T$ while preserving information about $Y$
**Objective**:
$$\min_{p(t|x)} I(X; T) - \beta I(T; Y)$$
Trade-off between compression and relevance.
## Applications
### 1. Feature Selection
**Goal**: Select features most informative about target
**Method**: Rank features by $I(X_i; Y)$
**Advantages over correlation**:
- Detects nonlinear relationships
- Handles categorical variables naturally
**Example**:
```
Features: X₁, X₂, X₃, X₄
Target: Y
I(X₁; Y) = 0.8
I(X₂; Y) = 0.3
I(X₃; Y) = 1.2 ← most informative
I(X₄; Y) = 0.1
Select X₃, then X₁
```
### 2. Dependency Detection
**Test independence**: $X \perp Y$ iff $I(X; Y) = 0$
**Hypothesis test**:
- Null: $I(X; Y) = 0$ (independent)
- Alternative: $I(X; Y) > 0$ (dependent)
**Test statistic**: $2n \cdot I(X; Y) / \ln(2)$ approximately $\chi^2$ distributed.
### 3. Clustering
**Information-theoretic clustering** minimizes within-cluster entropy.
**Objective**:
$$\min \sum_{k=1}^K \pi_k H(X | C=k)$$
where $\pi_k$ is cluster proportion.
### 4. Transfer Entropy
**Causality detection** in time series:
$$TE_{X \to Y} = I(Y_{t+1}; X_t | Y_t)$$
Measures information flow from $X$ to $Y$.
### 5. Data Compression
**Shannon's source coding theorem**:
Expected code length $\geq H(X)$ (entropy is fundamental limit).
**Huffman coding**, **arithmetic coding** approach this limit.
### 6. Neural Network Analysis
**Information plane**: Track $I(X; T)$ and $I(T; Y)$ during training
where $T$ is hidden layer representation.
**Observations**:
- Initial phase: Increase both (fitting)
- Later phase: Decrease $I(X; T)$, maintain $I(T; Y)$ (compression)
## Multivariate Extensions
### Joint Mutual Information
$$I(X_1, X_2; Y) = H(Y) - H(Y | X_1, X_2)$$
### Conditional Mutual Information
$$I(X; Y | Z) = H(X|Z) - H(X|Y,Z)$$
**Interpretation**: Information shared by $X$ and $Y$ not contained in $Z$
### Total Correlation
$$C(X_1, ..., X_n) = \sum_{i=1}^n H(X_i) - H(X_1, ..., X_n)$$
Measures total dependence among variables.
### Interaction Information
For three variables:
$$I(X; Y; Z) = I(X; Y|Z) - I(X; Y)$$
Can be positive (synergy) or negative (redundancy).
## Relationship to Other Concepts
### Information and Probability
$$I(E) = -\log p(E)$$
**Self-information** of event $E$.
Rare events carry more information.
### Fisher Information
For parameter estimation:
$$\mathcal{I}(\theta) = \mathbb{E}\left[\left(\frac{\partial \log p(X|\theta)}{\partial \theta}\right)^2\right]$$
Measures precision of estimating $\theta$.
**Cramér-Rao bound**: Variance of any unbiased estimator $\geq 1/\mathcal{I}(\theta)$
### Entropy and Thermodynamics
**Boltzmann entropy**: $S = k_B \ln W$
**Connection**: Statistical mechanics entropy ≈ Shannon entropy of microstates.
### Entropy Rate
For stochastic process $\{X_t\}$:
$$h = \lim_{n \to \infty} \frac{1}{n} H(X_1, ..., X_n)$$
**For Markov chains**: $h = -\sum_{i,j} \pi_i p_{ij} \log p_{ij}$
## Theoretical Results
### Source Coding Theorem
Expected code length $L \geq H(X)$
Equality achieved by Shannon coding.
### Channel Capacity
Maximum rate of reliable communication:
$$C = \max_{p(x)} I(X; Y)$$
where $Y$ is channel output given input $X$.
### Data Processing Inequality
If $X \to Y \to Z$ (Markov chain):
$$I(X; Y) \geq I(X; Z)$$
Processing cannot increase mutual information.
### Fano's Inequality
For estimating $X$ from $Y$ with error probability $P_e$:
$$H(X|Y) \leq H(P_e) + P_e \log(|\mathcal{X}| - 1)$$
Lower bound on conditional entropy given error rate.
## Computational Considerations
### Complexity
**Entropy estimation**: $O(n + m)$ where $n$ = samples, $m$ = bins
**Mutual information**: $O(n + m^2)$ for 2D histogram
**High dimensions**: Curse of dimensionality (need $m^d$ bins for $d$ dimensions)
### Numerical Stability
**Issue**: $\log 0$ is undefined
**Solutions**:
- Add small constant: $p + \epsilon$
- Use convention: $0 \log 0 = 0$
- Laplace smoothing: $(n_i + \alpha) / (n + \alpha m)$
## Key References
1. **Shannon, C. E.** (1948). *A mathematical theory of communication*. Bell System Technical Journal, 27(3), 379-423.
- Foundational paper
2. **Cover, T. M., & Thomas, J. A.** (2006). *Elements of Information Theory* (2nd ed.). Wiley.
- Comprehensive textbook
3. **MacKay, D. J.** (2003). *Information Theory, Inference and Learning Algorithms*. Cambridge University Press.
- Applications to machine learning
4. **Kraskov, A., Stögbauer, H., & Grassberger, P.** (2004). *Estimating mutual information*. Physical Review E, 69(6), 066138.
- k-NN based MI estimation
5. **Paninski, L.** (2003). *Estimation of entropy and mutual information*. Neural Computation, 15(6), 1191-1253.
- Bias correction methods
## Summary
Information theory provides fundamental limits and tools for:
**Core concepts**:
- **Entropy**: Uncertainty/information content
- **Mutual Information**: Shared information/dependence
- **KL Divergence**: Difference between distributions
**Key properties**:
- Entropy is maximized by uniform distribution
- Conditioning reduces entropy
- Mutual information detects any dependency
**Applications**:
- Feature selection and dimensionality reduction
- Model selection and compression
- Causality and dependency detection
- Machine learning (cross-entropy loss)
## See Also
- [Information Theory API Documentation](../information_theory.md) - Implementation and usage
- [HMM Theory](hmm.md) - Applications to sequential models
- [MCMC Theory](mcmc.md) - Sampling and inference methods
docs/theory/mcmc.md
+467
View File
@@ -0,0 +1,467 @@
# Markov Chain Monte Carlo: Mathematical Theory
## Introduction
Markov Chain Monte Carlo (MCMC) methods are a class of algorithms for sampling from probability distributions based on constructing a Markov chain that has the desired distribution as its equilibrium distribution. MCMC is fundamental to Bayesian inference, computational physics, and many areas of computational statistics.
## The Monte Carlo Method
### Goal
Sample from a target distribution $\pi(\theta)$ where:
- Direct sampling is difficult or impossible
- We can evaluate $\pi(\theta)$ up to a normalization constant
### Why Monte Carlo?
Given samples $\theta^{(1)}, ..., \theta^{(N)} \sim \pi(\theta)$, we can approximate:
**Expectations**:
$$\mathbb{E}_\pi[f(\theta)] \approx \frac{1}{N}\sum_{i=1}^N f(\theta^{(i)})$$
**Probabilities**:
$$P(\theta \in A) \approx \frac{1}{N}\sum_{i=1}^N \mathbb{1}[\theta^{(i)} \in A]$$
**Quantiles**, **distributions**, and other properties of $\pi(\theta)$.
## Markov Chains
### Definition
A sequence $\theta^{(0)}, \theta^{(1)}, \theta^{(2)}, ...$ is a Markov chain if:
$$P(\theta^{(t+1)} \mid \theta^{(0)}, ..., \theta^{(t)}) = P(\theta^{(t+1)} \mid \theta^{(t)})$$
The next state depends only on the current state.
### Transition Kernel
$$K(\theta' \mid \theta) = P(\theta^{(t+1)} = \theta' \mid \theta^{(t)} = \theta)$$
### Stationary Distribution
A distribution $\pi(\theta)$ is **stationary** if:
$$\pi(\theta') = \int K(\theta' \mid \theta) \pi(\theta) d\theta$$
If we start with $\theta^{(0)} \sim \pi$, then $\theta^{(t)} \sim \pi$ for all $t$.
### Ergodicity
A Markov chain is **ergodic** if:
1. **Irreducible**: Can reach any state from any state
2. **Aperiodic**: No cyclic behavior
For ergodic chains with stationary distribution $\pi$:
$$\lim_{t \to \infty} P(\theta^{(t)} \in A) = \pi(A)$$
regardless of initial state $\theta^{(0)}$.
### Detailed Balance
A sufficient (but not necessary) condition for $\pi$ to be stationary:
$$\pi(\theta) K(\theta' \mid \theta) = \pi(\theta') K(\theta \mid \theta')$$
**Reversibility**: The probability of going from $\theta$ to $\theta'$ equals the probability of the reverse transition.
## Metropolis-Hastings Algorithm
### Overview
The Metropolis-Hastings (MH) algorithm constructs a Markov chain whose stationary distribution is the target $\pi(\theta)$.
### Algorithm
**Input**: Target distribution $\pi(\theta)$, proposal distribution $q(\theta' \mid \theta)$
1. Initialize $\theta^{(0)}$
2. For $t = 0, 1, 2, ...$:
a. **Propose**: Draw $\theta^* \sim q(\theta^* \mid \theta^{(t)})$
b. **Compute acceptance probability**:
$$\alpha = \min\left(1, \frac{\pi(\theta^*) q(\theta^{(t)} \mid \theta^*)}{\pi(\theta^{(t)}) q(\theta^* \mid \theta^{(t)})}\right)$$
c. **Accept/Reject**:
$$\theta^{(t+1)} = \begin{cases}
\theta^* & \text{with probability } \alpha \\
\theta^{(t)} & \text{with probability } 1-\alpha
\end{cases}$$
### Why It Works
**Theorem**: The MH algorithm produces a Markov chain with stationary distribution $\pi(\theta)$.
**Proof sketch**: Show detailed balance holds.
For accepted moves:
$$\pi(\theta) q(\theta' \mid \theta) \alpha(\theta' \mid \theta) = \pi(\theta') q(\theta \mid \theta') \alpha(\theta \mid \theta')$$
For rejected moves, transitions to same state also balance.
### Special Cases
#### Metropolis Algorithm
When proposal is **symmetric**: $q(\theta' \mid \theta) = q(\theta \mid \theta')$
Acceptance probability simplifies:
$$\alpha = \min\left(1, \frac{\pi(\theta^*)}{\pi(\theta^{(t)})}\right)$$
#### Random Walk Metropolis
Use Gaussian proposal:
$$q(\theta' \mid \theta) = \mathcal{N}(\theta' \mid \theta, \sigma^2 I)$$
Symmetric, so use Metropolis acceptance.
#### Independence Sampler
Proposal doesn't depend on current state:
$$q(\theta' \mid \theta) = g(\theta')$$
Good if $g$ approximates $\pi$ well.
## Gibbs Sampling
### Motivation
For multivariate distributions, updating all dimensions at once can be inefficient.
### Algorithm
For $\theta = (\theta_1, ..., \theta_d)$:
1. Initialize $\theta^{(0)} = (\theta_1^{(0)}, ..., \theta_d^{(0)})$
2. For $t = 0, 1, 2, ...$:
Sample each component from its conditional distribution:
$$\theta_1^{(t+1)} \sim \pi(\theta_1 \mid \theta_2^{(t)}, ..., \theta_d^{(t)})$$
$$\theta_2^{(t+1)} \sim \pi(\theta_2 \mid \theta_1^{(t+1)}, \theta_3^{(t)}, ..., \theta_d^{(t)})$$
$$\vdots$$
$$\theta_d^{(t+1)} \sim \pi(\theta_d \mid \theta_1^{(t+1)}, ..., \theta_{d-1}^{(t+1)})$$
### Properties
- Special case of Metropolis-Hastings with acceptance probability = 1
- Requires knowing conditional distributions
- Can be slow if variables are highly correlated
## Bayesian Inference with MCMC
### Bayes' Theorem
$$p(\theta \mid D) = \frac{p(D \mid \theta) p(\theta)}{p(D)}$$
where:
- $p(\theta \mid D)$ is the **posterior** (what we want)
- $p(D \mid \theta)$ is the **likelihood**
- $p(\theta)$ is the **prior**
- $p(D) = \int p(D \mid \theta) p(\theta) d\theta$ is the **evidence** (normalizing constant)
### MCMC for Posterior Sampling
The evidence $p(D)$ is often intractable, but we can evaluate:
$$\pi(\theta) \propto p(D \mid \theta) p(\theta)$$
MCMC only needs $\pi$ up to a constant, so we can sample from the posterior!
### Metropolis-Hastings for Bayesian Inference
Target: $\pi(\theta) = p(D \mid \theta) p(\theta)$ (unnormalized posterior)
Acceptance probability:
$$\alpha = \min\left(1, \frac{p(D \mid \theta^*) p(\theta^*)}{p(D \mid \theta^{(t)}) p(\theta^{(t)})} \cdot \frac{q(\theta^{(t)} \mid \theta^*)}{q(\theta^* \mid \theta^{(t)})}\right)$$
For symmetric proposals:
$$\alpha = \min\left(1, \frac{p(D \mid \theta^*) p(\theta^*)}{p(D \mid \theta^{(t)}) p(\theta^{(t)})}\right)$$
### Example: Normal Mean and Variance
**Model**: $y_i \sim \mathcal{N}(\mu, \sigma^2)$, $i = 1, ..., n$
**Prior**: $p(\mu, \sigma^2) = p(\mu) p(\sigma^2)$
- $p(\mu) = \mathcal{N}(0, 100)$
- $p(\sigma^2) = \text{InvGamma}(0.01, 0.01)$
**Likelihood**:
$$p(D \mid \mu, \sigma^2) = \prod_{i=1}^n \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(y_i - \mu)^2}{2\sigma^2}\right)$$
**Log-posterior** (up to constant):
$$\log \pi(\mu, \sigma^2) = \log p(D \mid \mu, \sigma^2) + \log p(\mu) + \log p(\sigma^2)$$
Sample using MH with Gaussian random walk proposals.
## Convergence Diagnostics
### Burn-in Period
Discard initial samples before the chain has converged to the stationary distribution.
**How to choose?**
- Plot trace plots and look for stabilization
- Typically 1000-10000 iterations
- Conservative: discard first 50% of samples
### Effective Sample Size (ESS)
Due to autocorrelation, MCMC samples are not independent.
$$\text{ESS} = \frac{N}{1 + 2\sum_{k=1}^\infty \rho_k}$$
where $\rho_k$ is the autocorrelation at lag $k$.
**Interpretation**: ESS ≈ number of independent samples
### Autocorrelation
$$\rho_k = \frac{\text{Cov}(\theta^{(t)}, \theta^{(t+k)})}{\text{Var}(\theta^{(t)})}$$
**Goal**: Low autocorrelation (faster mixing)
**Solutions**:
- Tune proposal distribution
- Thinning (keep every $k$-th sample)
- Advanced methods (HMC, parallel tempering)
### Gelman-Rubin Diagnostic ($\hat{R}$)
Run multiple chains with different starting points.
$$\hat{R} = \sqrt{\frac{\text{Var}^+}{\text{Within-chain variance}}}$$
**Interpretation**:
- $\hat{R} \approx 1$: Chains have converged
- $\hat{R} > 1.1$: Chains have not mixed
### Geweke Diagnostic
Compare means of first 10% and last 50% of chain.
$$Z = \frac{\bar{\theta}_A - \bar{\theta}_B}{\sqrt{\text{SE}_A^2 + \text{SE}_B^2}}$$
Under null hypothesis of convergence, $Z \sim \mathcal{N}(0, 1)$.
## Proposal Tuning
### Acceptance Rate
**Optimal acceptance rate** (for random walk Metropolis in high dimensions):
- 1D: 44%
- ∞-D: 23.4%
- Practical: 20-40%
**Too high** (> 50%): Proposals too small, slow exploration
**Too low** (< 10%): Proposals too large, many rejections
### Adaptive Metropolis
Automatically tune proposal covariance during burn-in:
$$\Sigma^{(t+1)} = \text{Cov}(\theta^{(1)}, ..., \theta^{(t)})$$
Proposal:
$$q(\theta' \mid \theta) = \mathcal{N}(\theta', \theta, 2.38^2 \Sigma^{(t)} / d)$$
where $d$ is dimension.
### Optimal Scaling
Roberts and Rosenthal (2001): For Gaussian targets in $d$ dimensions, optimal variance:
$$\sigma^2 = \frac{2.38^2}{d} \Sigma$$
where $\Sigma$ is posterior covariance.
## Advanced MCMC Methods
### Hamiltonian Monte Carlo (HMC)
Uses gradient information to propose distant states with high acceptance.
**Advantages**:
- Efficient for high-dimensional problems
- Low autocorrelation
**Disadvantages**:
- Requires gradient computation
- More complex to implement
### Parallel Tempering
Run multiple chains at different "temperatures":
$$\pi_\beta(\theta) \propto \pi(\theta)^\beta$$
Exchange states between chains to improve mixing.
### Reversible Jump MCMC
For problems where dimension changes (model selection).
### Sequential Monte Carlo (SMC)
Particle filters for sequential data.
## Practical Considerations
### Initialization
**Strategies**:
1. **Random**: From prior or broad distribution
2. **MAP estimate**: From optimization
3. **Overdispersed**: Multiple chains, widely separated
### Thinning
Keep every $k$-th sample to reduce autocorrelation and storage.
**Debate**: Some argue thinning wastes information. Better to run longer and keep all samples (if storage permits).
### Reparameterization
Transform parameters to reduce correlation:
**Example**: Instead of $(\mu, \sigma^2)$, use $(\mu, \log\sigma)$.
Better geometry → better sampling.
### Multimodal Distributions
**Challenge**: Single chain may get stuck in one mode.
**Solutions**:
- Multiple independent chains
- Parallel tempering
- Simulated annealing
## Theoretical Guarantees
### Central Limit Theorem
For ergodic chains:
$$\sqrt{N}(\bar{\theta} - \mathbb{E}[\theta]) \xrightarrow{d} \mathcal{N}(0, \sigma^2)$$
where $\sigma^2$ depends on autocorrelation.
**Implication**: Monte Carlo estimates are asymptotically normal.
### Law of Large Numbers
$$\bar{\theta} = \frac{1}{N}\sum_{i=1}^N \theta^{(i)} \xrightarrow{a.s.} \mathbb{E}_\pi[\theta]$$
**Implication**: Estimates converge to true values.
### Convergence Rate
Geometric ergodicity: $\|P^t(\theta, \cdot) - \pi\| \leq C \rho^t$
for some $C > 0$ and $\rho < 1$.
Faster convergence → fewer samples needed.
## MCMC vs. Alternatives
| Method | Pros | Cons |
|--------|------|------|
| **MCMC** | General, exact (asymptotically) | Slow convergence, diagnostics needed |
| **Variational Inference** | Fast, scalable | Approximate, may be biased |
| **Importance Sampling** | Simple, independent samples | Requires good proposal |
| **Rejection Sampling** | Independent samples | Inefficient in high dimensions |
| **Grid/Quadrature** | Deterministic | Exponential in dimension |
## Applications
### 1. Bayesian Regression
Posterior inference for regression coefficients and variance.
### 2. Hierarchical Models
Multi-level models with group-specific and population parameters.
### 3. Mixture Models
Cluster analysis with unknown number of components.
### 4. Time Series
State space models, GARCH, stochastic volatility.
### 5. Spatial Statistics
Gaussian processes, kriging, disease mapping.
### 6. Computational Biology
Phylogenetic inference, population genetics.
## Software Implementations
### Stan
- Hamiltonian Monte Carlo (NUTS)
- Automatic differentiation
- Interfaces: R, Python, Julia, etc.
### PyMC
- Python library
- Variety of samplers
- Integrates with NumPy, Theano
### JAGS
- Just Another Gibbs Sampler
- BUGS-like syntax
- Interfaces: R (rjags), Python
### TensorFlow Probability / PyTorch
- Probabilistic programming on GPUs
- Integration with deep learning
## Key References
1. **Metropolis, N., et al.** (1953). *Equation of state calculations by fast computing machines*. The Journal of Chemical Physics, 21(6), 1087-1092.
- Original Metropolis algorithm
2. **Hastings, W. K.** (1970). *Monte Carlo sampling methods using Markov chains and their applications*. Biometrika, 57(1), 97-109.
- Generalization to Metropolis-Hastings
3. **Geman, S., & Geman, D.** (1984). *Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images*. IEEE Transactions on Pattern Analysis and Machine Intelligence, 6, 721-741.
- Gibbs sampling
4. **Gelfand, A. E., & Smith, A. F. M.** (1990). *Sampling-based approaches to calculating marginal densities*. Journal of the American Statistical Association, 85(410), 398-409.
- Popularized MCMC for Bayesian inference
5. **Brooks, S., Gelman, A., Jones, G., & Meng, X. L.** (Eds.). (2011). *Handbook of Markov Chain Monte Carlo*. CRC Press.
- Comprehensive reference
6. **Robert, C. P., & Casella, G.** (2004). *Monte Carlo Statistical Methods*. Springer.
- Mathematical treatment
## Summary
MCMC provides a powerful framework for:
- Sampling from complex, high-dimensional distributions
- Bayesian inference when posteriors are intractable
- Computing expectations and quantiles
**Key components**:
1. **Markov chain**: Generates dependent samples
2. **Stationary distribution**: Chain converges to target
3. **Metropolis-Hastings**: General acceptance/rejection scheme
4. **Diagnostics**: Ensure convergence and adequate mixing
## See Also
- [MCMC API Documentation](../mcmc.md) - Implementation details and usage
- [HMM Theory](hmm.md) - Alternative for sequential latent variable models
- [Differential Evolution Theory](differential_evolution.md) - Optimization methods