gavindiaz/optimiz-rs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: upgrade all ASCII diagrams to Unicode box-drawing art

Browse Source 
Replace ~20 ASCII diagrams across §1–§10 with polished Unicode art:
- Proper axes: ╰──→ ▲ ╌ with tick marks and labels
- Curves: ╭─╮ ╰─╯ ╲ ╱ for smooth paths and bells
- Steps: ─── ┐ └ for Poisson staircase
- Fills: ▓ ░ · ◦ ● for density and scatter plots
- Tables: ┌┬─┤├┼┘ Unicode box drawing for HMM state machine
- Remove duplicate BM fan + GBM diagrams (were repeated verbatim)
- Consistent 44-char width, title banners with ┄ separators
This commit is contained in:
ThotDjehuty
2026-03-07 10:38:38 +01:00
parent d24b8b1036
commit e80d717aa7
1 changed files with 376 additions and 303 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source/theory/mathematical_foundations.md
+373 -300
View File
@@ -22,17 +22,21 @@ without Jacobians.
### 1.1 Geometric Intuition — Mutation in $\mathbb{R}^2$ ### 1.1 Geometric Intuition — Mutation in $\mathbb{R}^2$
``` ```
x_r3 * Mutation geometry in ℝ²
| ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
| F*(x_r2 - x_r3) F = scale factor in [0,2]
| ------------------>
x_r2 * * v_i (mutant)
\\ /
\\________________/
(difference vec)
x_r1 *------------------------------------------------> v_i ◆ x_r3
base vector mutation vector added ╲
╲ F·(x_r2 − x_r3) F ∈ [0, 2]
╲────────────────────────▶ ◆ v_i ← mutant
◆ x_r2 ╱
╲___________________╱
└── difference vec ┘
◆ x_r1 ─────────────────────────────────▶ ◆ v_i
└─ base └── mutation vector added ──┘
v_i = x_r1 + F · (x_r2 − x_r3)
``` ```
- $\mathbf{r}_1, \mathbf{r}_2, \mathbf{r}_3$ are three **distinct** randomly selected parents. - $\mathbf{r}_1, \mathbf{r}_2, \mathbf{r}_3$ are three **distinct** randomly selected parents.
@@ -77,17 +81,22 @@ oscillates rapidly — any gradient step hops between basins.
spans the characteristic basin width (~1.0), enabling inter-basin jumps. spans the characteristic basin width (~1.0), enabling inter-basin jumps.
``` ```
Rastrigin 1D sketch (d=1): Rastrigin 1D ─ f(x) = 10 + x² − 10·cos(2πx)
┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
f(x) f(x) ▲
| * * * <- local minima (many) 20 │ ● ● ● ● ●
| * * * * * * │ ╱ ╲ ╱ ╲ ╱ ╲ ╱ ╲ ╱ ╲
|* * 10 │╱ ╲ ╱ ╲ ╱ ╲ ╱ ╲ ╱ ╲
| * * │ ╲ ╱ ╲ ╱ ╲ ╱ ╲ ╱
+------|------+------|-- x 0 │───────●───────────────●───────────────▶ x
-1 0 1 │ -2 -1 ★ 0 1 2
^ ↑
global min f=0 f*=0 (global min)
✦ ~10^d local minima for d dimensions
✦ Gradient oscillates rapidly → gradient descent fails
✦ DE difference-vector ~spans basin width ~1.0 → can escape
``` ```
**Typical jDE convergence** ($d=10$, $N=100$, $\tau_1=\tau_2=0.1$): **Typical jDE convergence** ($d=10$, $N=100$, $\tau_1=\tau_2=0.1$):
@@ -137,18 +146,19 @@ $$S^{(n)}_t = \frac{1}{\sqrt{n}}\sum_{k=1}^{\lfloor nt \rfloor} \xi_k.$$
By the **Central Limit Theorem**, as $n\to\infty$: $S^{(n)}_t \xrightarrow{d} W_t \sim \mathcal{N}(0,t)$. By the **Central Limit Theorem**, as $n\to\infty$: $S^{(n)}_t \xrightarrow{d} W_t \sim \mathcal{N}(0,t)$.
``` ```
Coin-flip random walk (n=20 steps per unit time): Coin-flip random walk (n = 20 steps per unit time)
┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
W_t W_t ▲
+2 | * * +2 │ ◦ ◦
| * * * │ ◦ ◦ ◦
0 | * * * * * 0 ┼──◦──────────◦◦────◦ ◦─────────▶ t
|* * * │◦ ◦ ◦
-2 | * -2 │ ◦
+----------------------------> t └────┬──────────┬──────────┬────
0 0.5 1.0 0 0.5 1.0
n → ∞ ("zoom out"): jagged → smooth BM fan n → ∞ ──▶ jagged path smooths into BM fan
``` ```
**Step 2 — Scaling limit.** The normalization $1/\sqrt{n}$ is crucial: **Step 2 — Scaling limit.** The normalization $1/\sqrt{n}$ is crucial:
@@ -212,19 +222,20 @@ This is the **only** reason Itō's lemma has an extra term.
**Multiple sample paths** — the fan widens as $\propto\sqrt{t}$: **Multiple sample paths** — the fan widens as $\propto\sqrt{t}$:
``` ```
W_t Brownian motion — multiple sample paths ("trumpet fan")
+2 | ........... ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
| .... .....
0 |.. ..... <- E[W_t] = 0 (all paths centered)
| ....--......
-2 | ........
+--------------------------> t
0 T
Width ~ 2*sqrt(t) (95% of paths stay within +/- 2*sqrt(t)) W_t ▲
+2σ │╌╌╌╌╌╌╌╮ ╭─────── 95% band ≈ ±2√t
│ ╰─╮ ╭──╮ ╭─────╯
0 ┼────────────╲──╱────╲────╱──────────────▶ t
│ ╭──╯ ╰╮ ╰╮
-2σ │╌╌╌╌╌╌╌╯ ╰────╯ 95% band ≈ −2√t
└──────────────────────────────────────
0 T/2 T
Each dot is a BM path — different because different coin flips. ← narrow ─────────── trumpet opens as √t ──────── wide →
As T grows, the "trumpet" opens wider. 𝔼[W_t] = 0 for all t (all paths oscillate around zero)
``` ```
**Example — Geometric BM:** **Example — Geometric BM:**
@@ -232,42 +243,20 @@ $S_t = S_0 \exp\!\bigl((\mu-\tfrac12\sigma^2)t + \sigma W_t\bigr)$
is the Black-Scholes price model. Log-normal marginals; continuous, nowhere-differentiable paths: is the Black-Scholes price model. Log-normal marginals; continuous, nowhere-differentiable paths:
``` ```
S_t Geometric BM — log-normal price path S_t = S_0 · exp(·)
| .---. ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
| .--./ \----.
| / \---------.
|/
+-------------------------------> t
0 T
```
**Multiple sample paths** — the fan widens as $\propto\sqrt{t}$: S_t ▲
1.3 │ ╭──╮
1.1 │ ╭──╮ ╱ ╲──╮
1.0 │──╱ ╲╱ ╲────────╮
0.9 │ ╲─────
0.7 │
└──────────────────────────────────▶ t
0 T/2 T
``` 𝔼[S_t] = S_0·e^{μt} (grows at rate μ)
W_t 𝔼[log S_t] = log S_0 + (μ − σ²/2)·t (Itō correction!)
+2 | ...........
| .... .....
0 |.. ..... <- E[W_t] = 0 (all paths centered)
| ....--......
-2 | ........
+--------------------------> t
0 T
Width grows as sqrt(t) (68% of paths stay within +/- sqrt(t))
```
**Example — Geometric BM:**
$S_t = S_0 \exp\!\bigl((\mu-\tfrac12\sigma^2)t + \sigma W_t\bigr)$
is the Black-Scholes price model. Log-normal marginals; continuous, nowhere-differentiable paths:
```
S_t
| .---.
| .--./ \\----.
| / \\---------.
|/
+-------------------------------> t
0 T
``` ```
### 2.2 Itō Calculus ### 2.2 Itō Calculus
@@ -386,15 +375,20 @@ but $\mathbb{E}[S_T] = S_0 e^{\mu T}$ (Jensen's inequality explains the gap:
$e^{\mathbb{E}[X]} < \mathbb{E}[e^X]$ for non-degenerate $X$). $e^{\mathbb{E}[X]} < \mathbb{E}[e^X]$ for non-degenerate $X$).
``` ```
log S_t Itō correction: 𝔼[log Sₜ] vs naive slope μ
| Ito correction: slope = mu - sigma^2/2 (lower than mu!) ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
| _.-'
| / <- Without Ito correction, slope would be mu (wrong!)
|/
+----------------------------> t
E[log S_t] = log S_0 + (mu - sigma^2/2)*t log Sₜ ▲
(time-averaged growth, always less than mu for sigma > 0) │ ╭──── slope μ (naive, WRONG)
│ ╭───╯
│ ╭───╯ ╌╌slope μ−σ²/2 (Itō, correct)
│╭──╯╌╌╌╌╌╌╌╌╌
┼────────────────────────────────────▶ t
0 T
Gap = σ²·T/2 (Jensen's inequality: e^{𝔼[X]} ≤ 𝔼[e^X])
Grows with volatility σ and horizon T
Itō correction always lowers expected log-return
``` ```
**Example 2 — Itō product rule ($d(X_t Y_t)$):** **Example 2 — Itō product rule ($d(X_t Y_t)$):**
@@ -470,15 +464,21 @@ Geometric series → $X^{(n)}$ is Cauchy in $L^2$ → converges to the unique so
**Intuition:** **Intuition:**
``` ```
Picard iteration (simplest case: dx = f(x) dt): Picard iteration (dx = f(x) dt, simplest case)
┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
X^(0): --------- x_0 (constant) X_t ▲
X^(1): x_0 + int_0^t f(x_0) ds = x_0 + f(x_0)*t (linear approx) │ ╭── X^(∞) = true solution
X^(2): x_0 + int_0^t f(X^(1)) ds (use X^(1) in drift) │ ╭───╯
X^(3): ... │ ╭───╯ ╌╌ X^(3)
│ ╭───╯ ╌╌╌╌╌╌ X^(2)
x_0 ┼──────────────────╌╌╌╌╌╌╌╌╌╌ X^(1) linear
│────────────────────────────── X^(0) constant
└──────────────────────────────▶ t
Each iteration captures one more "correction layer": Each iteration adds one correction layer:
n=0: constant n=1: linear n=2: quadratic ... n=0 ──▶ constant n=1 ──▶ linear n=2 ──▶ quadratic …
ε_n(t) ≤ C·(2L²(T+1)t)ⁿ/n! → 0 (factorial decay)
``` ```
#### 2.3.1 The Fokker-Planck Equation — How Densities Evolve #### 2.3.1 The Fokker-Planck Equation — How Densities Evolve
@@ -497,28 +497,31 @@ derivatives from $\phi$ to $p$, giving the Fokker-Planck equation.
**Visual — density flows rightward (positive drift) and spreads (positive diffusion):** **Visual — density flows rightward (positive drift) and spreads (positive diffusion):**
``` ```
p(t, x) — probability density at time t Fokker-Planck evolution — density drifts and spreads
┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
t=0: | peak at x_0 p(x) ▲
| *** │
| * * <- narrow initial density t=0 │ ▐█▌ narrow spike at x₀
| * * │ ▐███▌
+-----------------> x │ ▐█████▌
x_0 └──────────────────────────────────▶ x
x₀
t=T/2: | peak shifted right t=T/2│ ╭──╮ drift right + widen
| *** │ ╭─╯ ╰─╮
| * * <- wider (diffusion spreads it) │ ╱ ╲
| * * └──────────────────────────────────▶ x
+-----------------> x x₀ + μT/2
x_0 + mu*T/2
t=T: | peak drifted further t=T │ ╭────╮ even wider
| *** │ ╭──╯ ╰──╮
| * * <- even wider │ ╱ ╲
| * * └──────────────────────────────────▶ x
+--------------------------> x x₀ + μT
x_0 + mu*T
Drift term −∂ₓ[b·p] ──▶ shifts peak rightward
Diffusion +½∂ₓₓ[σ²p] ──▶ broadens the bell
``` ```
**For OU: $b = \kappa(\theta-x)$, $\sigma$ = const** → **For OU: $b = \kappa(\theta-x)$, $\sigma$ = const** →
@@ -551,18 +554,22 @@ $$X_{t+\Delta t} \approx X_t + b\,\Delta t + \sigma\,\Delta W_t + \tfrac12\sigma
The extra term $\tfrac12\sigma\sigma_x[(\Delta W_t)^2 - \Delta t]$ comes from applying Itō's lemma to $\sigma(X_t)dW_t$. The extra term $\tfrac12\sigma\sigma_x[(\Delta W_t)^2 - \Delta t]$ comes from applying Itō's lemma to $\sigma(X_t)dW_t$.
``` ```
Error comparison (log scale): Strong error ‖X_T − X̂_T‖ vs step size Δt (log–log scale)
┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
| Euler-Maruyama (slope = 0.5) log ▲
| * err │ ● Euler-Maruyama (order ½)
| * │ ●
| * │ ●
| * Milstein (slope = 1.0) │ ● ◆ Milstein (order 1)
| o │ ◆
| o │ ◆
| o │ ◆
+----------------------------> log(Delta_t) └──────────────────────────────▶ log Δt
(Milstein converges MUCH faster: halving dt reduces error by 4x instead of 2x) Δt=0.1 Δt=0.001
Halve Δt ──▶ Euler: error ÷√2 ≈ 0.71×
Milstein: error ÷4 = 0.25× ✓ much faster!
``` ```
### 2.4 Ornstein-Uhlenbeck (Mean-Reversion) ### 2.4 Ornstein-Uhlenbeck (Mean-Reversion)
@@ -574,21 +581,23 @@ $$dX_t = \kappa(\theta - X_t)\,dt + \sigma\,dW_t.$$
**Intuition — restoring force:** The drift is a spring pulling $X_t$ back to $\theta$: **Intuition — restoring force:** The drift is a spring pulling $X_t$ back to $\theta$:
``` ```
X_t Ornstein-Uhlenbeck — mean-reversion dX = κ(θ−X)dt + σdW
| upper band: theta + sigma/sqrt(2*kappa) ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
| .--. . ___ <-- random excursions
| / v /\ / \
|/ v / \ ----- long-run mean: theta (equilibrium)
| \ \....
| \___/
| lower band: theta - sigma/sqrt(2*kappa)
+-------------------------------> t
Arrows: mean-reversion pull toward theta with speed kappa. X_t ▲
Half-life = ln(2) / kappa. +2σ∞│╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ ← upper ±2σ∞ band
│ ╭─╮ ╭──╮
│ ╱ ╲ ╭──╯ ╲
θ ┼─╯ ╲─╯ ╲──╭─╮────────────── ← long-run mean θ
│ ╰─╯
−2σ∞│╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ ← lower ±2σ∞ band
└──────────────────────────────────────▶ t
Strong kappa: tight oscillations around theta (faster spring) σ∞ = σ/√(2κ) (stationary std dev)
Weak kappa: slow drift back (loose spring, more "random walk" like) τ½ = ln2/κ (half-life of displacement)
↓ Strong κ: tight, rapid oscillations (stiff spring)
↓ Weak κ: slow drift back (loose spring ≈ random walk)
``` ```
#### 2.4.1 Closed-Form Solution — Step by Step #### 2.4.1 Closed-Form Solution — Step by Step
@@ -641,21 +650,20 @@ This is exact (no approximation) because the OU process is **linear**. Key form
$$\hat\mu(\tau) = \theta + (X_s-\theta)e^{-\kappa\tau}, \qquad \hat\sigma^2(\tau) = \frac{\sigma^2}{2\kappa}(1-e^{-2\kappa\tau}), \quad \tau=t-s.$$ $$\hat\mu(\tau) = \theta + (X_s-\theta)e^{-\kappa\tau}, \qquad \hat\sigma^2(\tau) = \frac{\sigma^2}{2\kappa}(1-e^{-2\kappa\tau}), \quad \tau=t-s.$$
``` ```
Transition density spreading over time: OU transition density p(xₜ | x₀) spreading toward θ
┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
p(x_t | x_0 = x_0) p ▲
│ t=0: spike t=τ½: shifted + wider
│ t=∞: centred on θ (stationary)
│ │ ╭╮ ╭──────╮
│ │ ╱ ╲ ╭─╯ ╰─╮
│ █ ────╱ ╲── ──╯ ╰──
└──┼──────────────────────────────────────────▶ x
x₀ μ̂(τ½) θ
t=0: delta function at x_0 mean: μ̂(τ) = θ + (x₀−θ)·e^{−κτ} ───▶ θ as τ→∞
| var: σ̂²(τ) = (σ²/2κ)·(1−e^{−2κτ}) ───▶ σ²/2κ
|* <- spike
t=0.2: | ****
|* * <- bell curve, mean shifted toward theta
t=T: | ****
| * * <- centered at theta, wider
| * *
+-----------> x
``` ```
#### 2.4.3 Half-Life and Mean-Reversion Speed #### 2.4.3 Half-Life and Mean-Reversion Speed
@@ -694,19 +702,20 @@ $n=250$ observations, $\Delta t=1/252$ years.
**Step 2 — Intermediate verification:** The OU log-likelihood surface: **Step 2 — Intermediate verification:** The OU log-likelihood surface:
``` ```
ell(kappa, theta | sigma_hat) Log-likelihood surface ℓ(κ, θ | σ̂) ─ contour plot
┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
kappa κ ▲
^ 80 │ · · ·
| +++ <- log-lik thickening near true kappa 65 │ · · ◎ · · ◎ = MLE optimum
| +++++ 55 │ · · ◎◎◎ · · contours: ─── ℓ = const
| +++++++ 45 │ · · ◎ · ·
| +++++ 30 │ · · ·
| +++ └─────────────────────────────────────────▶ θ
| 0.000 0.003 0.006
+-------------------------> theta
(theta_hat is the sample mean of X_t, very well identified) θ is tightly identified (≈ sample mean of Xₜ)
(kappa is harder: need long series to identify mean-reversion speed) κ needs long series (eigenvalue of autocorrelation)
``` ```
**Typical results:** **Typical results:**
@@ -722,15 +731,19 @@ kappa
Standardized residuals: $r_i = (X_{t_i} - \hat\mu_i)/\hat\sigma$ should be $\mathcal{N}(0,1)$. Standardized residuals: $r_i = (X_{t_i} - \hat\mu_i)/\hat\sigma$ should be $\mathcal{N}(0,1)$.
``` ```
r_i histogram vs N(0,1) Residual diagnostic: rᵢ = (Xₜᵢ − μ̂ᵢ)/σ̂ vs 𝒩(0,1)
┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
| . density ▲
| . . <- histogram bars (sample) 0.4 │ ╭───╮
|. . ......... 0.3 │ ╭─╯ ╰─╮ ─── 𝒩(0,1) theory
| .......... <- N(0,1) curve (theory) 0.2 │ ╱ ▓ ▓ ▓ ╲ ▓▓▓ sample histogram
| . . 0.1 │ ╱ ▓▓▓▓▓▓▓▓▓ ╲
+------|------|-----> r_i 0.0 └────────────────────────────────▶ rᵢ
-2 +2 -3 -2 -1 0 1 2 3
✓ bars hug the curve → OU model fits
✗ heavy tails / skew → consider jump-diffusion
``` ```
Ljung-Box test: checks for remaining autocorrelation in $r_i$. Ljung-Box test: checks for remaining autocorrelation in $r_i$.
@@ -770,16 +783,21 @@ is a martingale.
**Sample path — step function with random jumps ($\lambda=2$ per unit time):** **Sample path — step function with random jumps ($\lambda=2$ per unit time):**
``` ```
N_t Poisson process Nₜ ~ Poisson(λt) (λ = 2 jumps/unit)
5 | ___________ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
4 | ___________
3 | ______ Nₜ ▲
2 | ______ 5 │ ┌─────────
1 |____ 4 │ ┌────────────┘ ↑
0 | 3 │ ┌────────┘ τ₄ ~ Exp(2)
+-----|------|-------|------|---------> t 2 │ ┌─────┘ ↑
tau1 tau2 tau3 tau4 1 ├──┘ τ₂ ~ Exp(2)
(Exp(lambda) inter-arrivals, each tau_i ~ Exp(2)) 0 │
└────┬────┬────┬────┬───────────────────▶ t
τ₁ τ₂ τ₃ τ₄
Each inter-arrival τₖ ∼ Exp(λ) ─ memoryless!
Compensated: Ñₜ = Nₜ − λt is a martingale
``` ```
### 3.2 Compound Poisson Jump-Diffusion (Merton 1976) ### 3.2 Compound Poisson Jump-Diffusion (Merton 1976)
@@ -791,18 +809,22 @@ with $N_t$ Poisson($\lambda$) and $J_k \sim \mathcal{N}(\mu_J, \sigma_J^2)$.
**Sample path — smooth diffusion interrupted by sudden jumps:** **Sample path — smooth diffusion interrupted by sudden jumps:**
``` ```
S_t Merton jump-diffusion — Sₜ path (μ=0.05, σ=0.18, λ=2/yr)
| ^ jump +15% ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
| /|
| / | S_t ▲
| / | v jump -20% 1.25│ ↑ +15% jump
| / \\ /| 1.15│ ╱▕
| / \\_______/ | 1.05│ ╭────╯ ▕
| / \\____ 1.00│───╯ ╲▕ ↓ −20% jump
| / \\... 0.85│ ╰──────╮▕
| / 0.75│ ╰─────╮
+---------------------------------> t 0.65│ ╰────────
(Brownian between jumps; jump times ~ Poisson) └────────────────────────────────────▶ t
──── smooth Brownian diffusion between jumps
▕ jump discontinuity (Poisson arrival)
Each segment: dS = μS dt + σS dW (GBM)
``` ```
**Merton option price** — Poisson mixture of Black-Scholes prices: **Merton option price** — Poisson mixture of Black-Scholes prices:
@@ -860,24 +882,27 @@ satisfying $\int(1\wedge z^2)\nu(dz)<\infty$.
**Levy measure tail shapes:** **Levy measure tail shapes:**
``` ```
nu(dz)/dz Lévy measure tails ν(dz)/dz ─ log scale
┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
Gaussian BM: nu = 0 (no jump component, only diffusion) ν ▲
│ Compound Poisson: point masses ▼ ▼
│ ● ●
│
│ Variance Gamma: ν ∝ e^{−c|z|}/|z|
│ ╲
│ ╲
│ ╲───────────────___________
│
│ α-stable: ν ∝ |z|^{−1−α} (heavier)
│ ╲
│ ╲____
│ ╲_______________________
└──────────────────────────────────▶ z
−2 −1 0 1 2
Compound Poisson (rare large jumps): Gaussian BM: ν ≡ 0 (no jump component at all)
| ^ ^ (discrete point masses at fixed jump sizes) Heavier ν tail ──▶ more frequent/larger jumps
+--*----*---> z
Variance Gamma (smooth exponential decay):
|\\
| \\.
| \\...___
+---------> z (heavier left tail than Gaussian)
alpha-stable (power-law heavy tail):
|\\
| \\.....
+-----------> z (infinite variance, very fat tail)
``` ```
**Levy Process Zoo** **Levy Process Zoo**
@@ -1273,14 +1298,20 @@ Steady-state: $P_\infty \approx 0.17$, so $K_\infty \approx 0.15$.
Kalman weights the new observation at 15%, prior at 85%. Kalman weights the new observation at 15%, prior at 85%.
``` ```
P_t Kalman error covariance convergence P_t → P∞
1.0 |* ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
0.8 | \\
0.6 | \\ P_t ▲
0.4 | \\___ 1.0 │●
0.2 | \\__________________ P_inf ~ 0.17 0.8 │ ╲
+--------------------------------> t 0.6 │ ╲
(fast convergence to steady-state uncertainty) 0.4 │ ╲──╮
0.2 │ ╰─────────╌╌╌╌╌╌╌╌╌╌╌╌ P∞ ≈ 0.17
0.0 └────────────────────────────────▶ t
0 5 10 15 20 ∞
Fast decay (exponential rate ∝ spectral gap of Riccati)
R/Q = 100 → heavy smoothing, Kalman gain ≈ 0.15
``` ```
**Implication:** With $R/Q = 100$ (much noisier obs than process), the filter heavily **Implication:** With $R/Q = 100$ (much noisier obs than process), the filter heavily
@@ -1306,32 +1337,39 @@ ensures $\pi$ is the unique stationary distribution.
**Energy landscape and accept/reject:** **Energy landscape and accept/reject:**
``` ```
U(x) (negative log-posterior) Energy landscape U(x) = −log π(x) (bimodal example)
┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
| * * <- local minima (modes of pi) U ▲
| / \\ / \\ │ ● ● ← local maxima (low π)
| / \\ / \\ │ ╱ ╲ ╱ ╲
| / *___* \\ <- saddle between modes │ ╱ ╲ ╱ ╲
| / \\ │╱ ╲ ╱ ╲
+-------------------------> x │ ╲──○────╱ ╲ ← saddle
│ mode A mode B ╲───
└──────────────────────────────────▶ x
Proposal x' = x + h * xi: Proposal x’ = x + h·ξ, ξ∼𝒩(0,1):
If U(x') < U(x): always accept (moving downhill) U(x’) < U(x) → accept always (step downhill)
If U(x') > U(x): accept with prob exp(-(U(x')-U(x))) U(x’) > U(x) → accept with exp(−ΔU) (sometimes climb)
Prevents getting trapped in local minima. ↳ prevents permanent trapping in one mode
``` ```
**Trace plot of a well-mixed chain:** **Trace plot of a well-mixed chain:**
``` ```
x_t MCMC trace plot — well-mixed chain (bimodal π)
+2 | . . . . . . <- upper mode ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
| . . . . . .
0 |... ... <- transitions between modes x_t ▲
| . . . . . +2 │ · · · · · · ← upper mode
-2 | . . .. <- lower mode │ · · · · · ·
+-----------------------------> t 0 ┼─···─────────────···──────────────▶ t
(crossing between modes = good mixing) │ · · · · ·
-2 │ · · ·· ← lower mode
✓ frequent crossings → good mixing (both modes visited)
✗ stuck in one band → poor mixing (reduce h or use MALA)
``` ```
### 7.2 Langevin Dynamics (MALA) ### 7.2 Langevin Dynamics (MALA)
@@ -1352,11 +1390,18 @@ for high-dimensional posteriors.
**MALA vs RW-MH trajectory comparison:** **MALA vs RW-MH trajectory comparison:**
``` ```
RW-MH vs MALA trajectories
┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
RW-MH (random walk): MALA (gradient-guided): RW-MH (random walk): MALA (gradient-guided):
. . . ^ -grad U (towards mode)
. . . . --> / ◦ ◦ ◦ ▲ −∇U ← toward mode
. . . /. . ◦ ◦ ◦ ◦ vs ╱
(diffusive, slow mixing) (directed, fast mixing) ◦ ◦ ◦ ╱ ◦ ◦
◦ ◦ ◦ ◦ ◦
diffusive, O(d) steps directed, O(d^{1/3}) steps
each step ∼ isotropic ξ each step biased by −∇U(x)
``` ```
::::{admonition} Example — Calibrating OU Parameters via MCMC ::::{admonition} Example — Calibrating OU Parameters via MCMC
@@ -1394,23 +1439,25 @@ $Y_t \mid Z_t=k \sim B_k(y)$.
**State machine diagram ($K=3$ regimes):** **State machine diagram ($K=3$ regimes):**
``` ```
A_12 A_23 HMM regime state machine (K = 3)
+------------+ +------------+ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
| State 1 |------->| State 2 |-------> State 3
| (Bull) | | (Neutral) | (Bear/Crash)
+------------+<-------+------------+<--------
A_21 A_32
Each state k emits Y_t ~ B_k(y): A₁₂ → A₂₃ →
B_1: N(mu=+0.05, sigma=0.12) high return, low vol ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
B_2: N(mu=+0.00, sigma=0.18) flat, medium vol │ State 1 │──────▶│ State 2 │──────▶│ State 3 │
B_3: N(mu=-0.08, sigma=0.35) crash regime, high vol │ Bull │◀──────│ Neutral │◀──────│ Bear │
└─────────────┘ └─────────────┘ └─────────────┘
← A₂₁ ← A₃₂
Transition matrix A: Emission B_k(y) = 𝒩(μ_k, σ_k²):
A = [ 0.97 0.02 0.01 ] (row 1: from State 1) ┌────────┬────────┬────────┬──────────────────┐
[ 0.01 0.97 0.02 ] (row 2: from State 2) │ State │ μ │ σ │ Character │
[ 0.05 0.05 0.90 ] (row 3: from State 3) ├────────┼────────┼────────┼──────────────────┤
(rows sum to 1; diagonal = regime persistence) │ Bull │ +0.05 │ 0.12 │ high return, low vol │
│ Neutral│ 0.00 │ 0.18 │ flat, medium vol │
│ Bear │ -0.08 │ 0.35 │ crash, high vol │
└────────┴────────┴────────┴──────────────────┘
(self-transition: A₁₁=0.97, A₂₂=0.97, A₃₃=0.90)
``` ```
### 8.2 Baum-Welch (EM) ### 8.2 Baum-Welch (EM)
@@ -1434,18 +1481,21 @@ iteration monotonically increases $\mathcal{L}(\theta)$ by Jensen's inequality.
**Viterbi trellis diagram ($K=3$, $T=4$):** **Viterbi trellis diagram ($K=3$, $T=4$):**
``` ```
State t=1 t=2 t=3 t=4 Viterbi trellis (K=3, T=4)
1 delta1(1) --> delta2(1) --> delta3(1) --> delta4(1) ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
\ \\ //
2 delta1(2) --> delta2(2) --> delta3(2) --> delta4(2)
\ \ \
3 delta1(3) --> delta2(3) --> delta3(3) --> delta4(3)
delta_t(k) = max_j [ delta_{t-1}(j) * A_jk * B_k(y_t) ] State t=1 t=2 t=3 t=4
psi_t(k) = argmax (backtrack pointer -> records best prev state)
After forward pass: trace back from t=T to t=1 via psi to get MAP sequence 1 ○─────────▶○─────────▶○─────────▶○
z_1*, z_2*, z_3*, z_4* ╲ ╳
2 ○─────────▶●─────────▶●─────────▶○ ● = MAP path
╲ ╲ ╲
3 ○─────────▶○─────────▶○─────────▶○
δ_t(k) = max_j [δ_{t−1}(j) · A_jk · B_k(y_t)]
ψ_t(k) = argmax_j ← backtrack pointer
Traceback: z_4★ ← z_3★ ← z_2★ ← z_1★ via ψ
``` ```
**Viterbi (MAP path):** $\delta_t(k) = \max_j \delta_{t-1}(j)A_{jk} \cdot B_k(y_t)$, $O(TK^2)$. **Viterbi (MAP path):** $\delta_t(k) = \max_j \delta_{t-1}(j)A_{jk} \cdot B_k(y_t)$, $O(TK^2)$.
@@ -1535,21 +1585,22 @@ $$\mathcal{I}(\theta)_{ij}
**Fisher information as curvature of the log-likelihood:** **Fisher information as curvature of the log-likelihood:**
``` ```
log L(theta | x_obs) Fisher information = curvature of log-likelihood
┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
| .----. log ℒ ▲
| ./ \. │ ╭──╮
| ./ \. │ ╭─╯ ╰─╮ High ℐ: sharp peak
| ./ \. sharp → │ ╱ ╲ tight C-R bound
+-----------------------> theta ┼─────────────────────▶ θ
theta* wide peak examples:│
flat → │ ╭───────────╮ Low ℐ: flat peak
│╱ ╲ loose C-R bound
└─────────────────────▶ θ
θ★
I(theta*) = -d^2/dtheta^2 log L at the peak ℐ(θ★) = −∂²θθ log ℒ at the peak
Cramér-Rao: Var(θ̂) ≥ 1/ℐ(θ) ∀ unbiased θ̂
High I (sharp peak): theta well-identified, low estimation variance
Low I (flat peak): theta hard to identify, high estimation variance
Cramer-Rao: Var(theta_hat) >= 1 / I(theta) for any unbiased estimator
``` ```
**Cramer-Rao bound:** Any unbiased estimator $\hat\theta$ satisfies **Cramer-Rao bound:** Any unbiased estimator $\hat\theta$ satisfies
@@ -1613,12 +1664,19 @@ tangent space $T_p M$.
**Three canonical curvatures:** **Three canonical curvatures:**
``` ```
K > 0 (sphere S2) K = 0 (flat R2) K < 0 (hyperbolic H2) Three canonical curvatures
┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
(N) | / K > 0 (sphere S²) K = 0 (flat ℝ²) K < 0 (hyperbolic H²)
/|\ | /
/ | \ geodesics --+-- parallel / geodesics ▲N │ ╱ ╲
/ | \ reconverge (N-S) | lines / diverge exponentially ╱│╲ │ ╱ ╲
╱ │ ╲ geodesics ──┼── parallel ╱ ╲ exponential
╱ │ ╲ reconverge │ lines ╱ ╲ divergence
╱ ╲
exponential families → K=0 → Newton / natural gradient exact
portfolio sphere → K>0 → geodesics curve back (compact orbits)
``` ```
**Tangent space — linear approximation at $p$:** **Tangent space — linear approximation at $p$:**
@@ -1650,13 +1708,20 @@ The statistical manifold $\mathcal{M} = \{p(\cdot;\theta)\}$ carries the
**Standard vs natural gradient:** **Standard vs natural gradient:**
``` ```
Standard gradient descent: Natural gradient descent: Standard vs natural gradient
theta_{k+1} = theta_k - eta * grad L theta_{k+1} = theta_k - eta * I^{-1} grad L ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
Parameter space = flat R^d. Parameter space = Riemannian (metric I(theta)). Standard: θ_{k+1} = θ_k − η·∇ℒ Natural: θ_{k+1} = θ_k − η·ℐ(θ)^{−1}∇ℒ
Ignores curvature. Adapts step to local geometry. ────────────────────────────────────────────
Slow on ill-conditioned I. Invariant to reparametrisation.
O(kappa(I)) iterations. O(1) iterations on exponential families. ┌────────────────────┐ ┌────────────────────┐
│ Flat ℝᵈ geometry │ │ Riemannian metric ℐ(θ) │
│ Ignores curvature │ │ Adapts to geometry │
│ Slow on ill-cond ℐ │ │ Reparam invariant │
│ O(κ(ℐ)) iters │ │ O(1) on exp families │
└────────────────────┘ └────────────────────┘
On Gaussian / exponential family: ℐ⁻¹∇ℒ = MLE step → 1 iteration!
``` ```
**Natural gradient (Amari 1998):** **Natural gradient (Amari 1998):**
@@ -1699,20 +1764,22 @@ linearises the group at the identity.
**Matrix Lie group hierarchy:** **Matrix Lie group hierarchy:**
``` ```
GL(n,R) general linear group (all invertible n x n real matrices) Matrix Lie group hierarchy
| ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
|--- SL(n,R) special linear (det = 1)
|
|--- O(n) orthogonal (R'R = I)
| |
| +---- SO(n) special orthogonal (det = +1, pure rotations)
| Used in: portfolio factor rotation, PCA constraints
|
+--- Sp(2n,R) symplectic (preserves symplectic form omega)
Used in: Hamiltonian mechanics, PMP sections 4.2 and 10.4
Heisenberg group H(n): upper triangular with 1s on diagonal. GL(n,ℝ) ─ all invertible n×n real matrices
Used in: path-signature feature maps (lab_signature_methods) │
├──▶ SL(n,ℝ) det = 1
│
├──▶ O(n) RᵀR = I (orthogonal)
│ └─▶ SO(n) det = +1 (pure rotations)
│ ↳ portfolio factor rotation, PCA constraints
│
└──▶ Sp(2n,ℝ) preserves symplectic form ω
↳ Hamiltonian mechanics, PMP §4.2 / §10.4
H(n) Heisenberg ─ upper triangular, 1s on diagonal
↳ path-signature feature maps
``` ```
**Left-invariant control system on $G$:** **Left-invariant control system on $G$:**
@@ -1740,9 +1807,15 @@ allocation back-tests in Optimiz-rs.
The sectional curvature $K(\sigma)$ governs how quickly nearby geodesics diverge: The sectional curvature $K(\sigma)$ governs how quickly nearby geodesics diverge:
``` ```
K > 0 (sphere): geodesics converge -> compact optimiser trajectories Sectional curvature and optimiser geometry
K = 0 (flat ): Euclidean behaviour -> Newton / natural gradient exact ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
K < 0 (hyper.): exponential spread -> efficient landscape exploration
K > 0 (sphere) → geodesics converge → compact optimiser orbits
K = 0 (flat) → Euclidean behaviour → Newton / nat. grad. exact
K < 0 (hyper.) → exponential spread → fast landscape exploration
Exponential families live on K = 0 manifold (dually flat).
DE explores K < 0 terrain: difference vectors "fan out" exponentially.
``` ```
For exponential families in natural/mean parameters $K=0$ — explaining exact For exponential families in natural/mean parameters $K=0$ — explaining exact