Melvin Avarez
513 lines
12 KiB
Markdown
513 lines
12 KiB
Markdown
# Information Theory: Mathematical Foundations
|
|
|
|
## Introduction
|
|
|
|
Information theory, founded by Claude Shannon in 1948, provides a mathematical framework for quantifying information, uncertainty, and communication. It has applications in data compression, communication, cryptography, machine learning, and statistical inference.
|
|
|
|
## Shannon Entropy
|
|
|
|
### Definition
|
|
|
|
For a discrete random variable $X$ with probability mass function $p(x)$:
|
|
|
|
$$H(X) = -\sum_{x \in \mathcal{X}} p(x) \log p(x)$$
|
|
|
|
**Convention**: $0 \log 0 = 0$ (limit as $p \to 0$)
|
|
|
|
### Units
|
|
|
|
- **Nats**: Natural logarithm (base $e$)
|
|
- **Bits**: Logarithm base 2
|
|
- **Dits**: Logarithm base 10
|
|
|
|
**Conversion**: $H_{\text{bits}} = H_{\text{nats}} / \ln(2) \approx 1.4427 \cdot H_{\text{nats}}$
|
|
|
|
### Interpretation
|
|
|
|
**Shannon entropy measures**:
|
|
1. **Uncertainty** about $X$ before observation
|
|
2. **Information content** of a sample from $X$
|
|
3. **Average code length** (optimal compression)
|
|
4. **Unpredictability** of $X$
|
|
|
|
### Properties
|
|
|
|
**Non-negativity**:
|
|
$$H(X) \geq 0$$
|
|
|
|
Equality iff $X$ is deterministic (probability 1 on one outcome).
|
|
|
|
**Maximum entropy**:
|
|
$$H(X) \leq \log |\mathcal{X}|$$
|
|
|
|
Achieved by uniform distribution: $p(x) = 1/|\mathcal{X}|$ for all $x$.
|
|
|
|
**Concavity**:
|
|
$H$ is a concave function of the distribution $p$.
|
|
|
|
### Examples
|
|
|
|
**Binary variable** ($p$ = probability of success):
|
|
$$H(X) = -p\log p - (1-p)\log(1-p)$$
|
|
|
|
Maximum at $p = 0.5$: $H_{\text{bits}} = 1$ bit.
|
|
|
|
**Fair die**:
|
|
$$H(X) = -\sum_{i=1}^6 \frac{1}{6}\log\frac{1}{6} = \log 6 \approx 1.79 \text{ bits}$$
|
|
|
|
**Biased die** (probability 0.5 for face 1, 0.1 for others):
|
|
$$H(X) = -0.5\log(0.5) - 5 \times 0.1\log(0.1) \approx 1.36 \text{ bits}$$
|
|
|
|
Less entropy than fair die (more predictable).
|
|
|
|
## Continuous Entropy (Differential Entropy)
|
|
|
|
### Definition
|
|
|
|
For continuous random variable $X$ with density $f(x)$:
|
|
|
|
$$h(X) = -\int f(x) \log f(x) dx$$
|
|
|
|
### Differences from Discrete Case
|
|
|
|
- Can be **negative**
|
|
- Not invariant under coordinate transformations
|
|
- Measures relative information (to uniform over support)
|
|
|
|
### Gaussian Distribution
|
|
|
|
For $X \sim \mathcal{N}(\mu, \sigma^2)$:
|
|
|
|
$$h(X) = \frac{1}{2}\log(2\pi e \sigma^2)$$
|
|
|
|
**Maximal entropy** among all distributions with variance $\sigma^2$.
|
|
|
|
### Multivariate Gaussian
|
|
|
|
For $\mathbf{X} \sim \mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma})$:
|
|
|
|
$$h(\mathbf{X}) = \frac{1}{2}\log\det(2\pi e \boldsymbol{\Sigma})$$
|
|
|
|
## Joint and Conditional Entropy
|
|
|
|
### Joint Entropy
|
|
|
|
For pair $(X, Y)$:
|
|
|
|
$$H(X, Y) = -\sum_{x,y} p(x,y) \log p(x,y)$$
|
|
|
|
**Chain rule**:
|
|
$$H(X, Y) = H(X) + H(Y|X)$$
|
|
|
|
### Conditional Entropy
|
|
|
|
$$H(Y|X) = -\sum_{x,y} p(x,y) \log p(y|x)$$
|
|
|
|
**Interpretation**: Average uncertainty in $Y$ given $X$.
|
|
|
|
**Property**:
|
|
$$H(Y|X) \leq H(Y)$$
|
|
|
|
Conditioning reduces entropy (information never increases uncertainty).
|
|
|
|
Equality iff $X$ and $Y$ are independent.
|
|
|
|
## Mutual Information
|
|
|
|
### Definition
|
|
|
|
$$I(X; Y) = H(X) + H(Y) - H(X, Y)$$
|
|
|
|
Alternatively:
|
|
|
|
$$I(X; Y) = \sum_{x,y} p(x,y) \log \frac{p(x,y)}{p(x)p(y)}$$
|
|
|
|
Or:
|
|
|
|
$$I(X; Y) = H(X) - H(X|Y) = H(Y) - H(Y|X)$$
|
|
|
|
### Interpretation
|
|
|
|
**Mutual information measures**:
|
|
1. **Reduction** in uncertainty about $X$ given $Y$
|
|
2. **Shared information** between $X$ and $Y$
|
|
3. **Dependence** between $X$ and $Y$
|
|
4. **Distance** from independence
|
|
|
|
### Properties
|
|
|
|
**Non-negativity**:
|
|
$$I(X; Y) \geq 0$$
|
|
|
|
Equality iff $X$ and $Y$ are independent.
|
|
|
|
**Symmetry**:
|
|
$$I(X; Y) = I(Y; X)$$
|
|
|
|
**Bounded**:
|
|
$$I(X; Y) \leq \min(H(X), H(Y))$$
|
|
|
|
Equality when one variable completely determines the other.
|
|
|
|
**Data processing inequality**:
|
|
|
|
If $X \to Y \to Z$ form a Markov chain:
|
|
$$I(X; Z) \leq I(X; Y)$$
|
|
|
|
Processing can't increase information.
|
|
|
|
### Relationship to Correlation
|
|
|
|
For bivariate Gaussian $(X, Y)$ with correlation $\rho$:
|
|
|
|
$$I(X; Y) = -\frac{1}{2}\log(1 - \rho^2)$$
|
|
|
|
**Mutual information** detects both linear and nonlinear dependencies, while **Pearson correlation** only detects linear.
|
|
|
|
## Kullback-Leibler Divergence
|
|
|
|
### Definition
|
|
|
|
For distributions $p$ and $q$ over $\mathcal{X}$:
|
|
|
|
$$D_{KL}(p \| q) = \sum_{x \in \mathcal{X}} p(x) \log \frac{p(x)}{q(x)}$$
|
|
|
|
**Continuous case**:
|
|
$$D_{KL}(p \| q) = \int p(x) \log \frac{p(x)}{q(x)} dx$$
|
|
|
|
### Interpretation
|
|
|
|
- **Relative entropy**: Information gain when updating from $q$ to $p$
|
|
- **Divergence**: How much $p$ differs from $q$
|
|
- **Inefficiency**: Extra bits needed when using code for $q$ to encode $p$
|
|
|
|
### Properties
|
|
|
|
**Non-negativity** (Gibbs' inequality):
|
|
$$D_{KL}(p \| q) \geq 0$$
|
|
|
|
Equality iff $p = q$ (almost everywhere).
|
|
|
|
**Asymmetry**:
|
|
$$D_{KL}(p \| q) \neq D_{KL}(q \| p)$$
|
|
|
|
Not a true distance metric (doesn't satisfy triangle inequality).
|
|
|
|
**Connection to MI**:
|
|
$$I(X; Y) = D_{KL}(p(x,y) \| p(x)p(y))$$
|
|
|
|
MI is the KL divergence from joint to product of marginals.
|
|
|
|
## Cross Entropy
|
|
|
|
### Definition
|
|
|
|
$$H(p, q) = -\sum_x p(x) \log q(x)$$
|
|
|
|
**Relationship to KL divergence**:
|
|
$$H(p, q) = H(p) + D_{KL}(p \| q)$$
|
|
|
|
### Machine Learning Application
|
|
|
|
**Loss function** in classification:
|
|
|
|
For true distribution $p$ (one-hot) and predicted $q$ (softmax):
|
|
$$\text{Loss} = H(p, q)$$
|
|
|
|
Minimizing cross-entropy ≡ minimizing KL divergence ≡ maximizing likelihood.
|
|
|
|
## Estimation from Data
|
|
|
|
### Histogram Method
|
|
|
|
Given samples $x_1, ..., x_n$ from continuous distribution:
|
|
|
|
1. **Discretize**: Create histogram with $m$ bins
|
|
2. **Estimate probabilities**: $\hat{p}_i = n_i / n$ where $n_i$ is count in bin $i$
|
|
3. **Compute entropy**: $\hat{H}(X) = -\sum_{i=1}^m \hat{p}_i \log \hat{p}_i$
|
|
|
|
### Bin Selection
|
|
|
|
**Too few bins**: Underestimates entropy (over-smoothing)
|
|
|
|
**Too many bins**: Overestimates entropy (noise)
|
|
|
|
**Rules of thumb**:
|
|
- Sturges: $m = \lceil \log_2 n + 1 \rceil$
|
|
- Scott: $m = \lceil (x_{\max} - x_{\min}) / (3.5 \sigma n^{-1/3}) \rceil$
|
|
- Square root: $m = \lceil \sqrt{n} \rceil$
|
|
|
|
### Bias Correction
|
|
|
|
Histogram estimator is **biased** (tends to overestimate).
|
|
|
|
**Miller-Madow correction**:
|
|
$$\hat{H}_{\text{corrected}} = \hat{H} - \frac{m - 1}{2n}$$
|
|
|
|
where $m$ is number of non-empty bins.
|
|
|
|
### Mutual Information Estimation
|
|
|
|
For samples $(x_i, y_i)$, $i = 1, ..., n$:
|
|
|
|
1. Create 2D histogram (or separate 1D histograms)
|
|
2. Estimate joint and marginal probabilities
|
|
3. Compute:
|
|
$$\hat{I}(X; Y) = \sum_{i,j} \hat{p}_{ij} \log \frac{\hat{p}_{ij}}{\hat{p}_i \hat{p}_j}$$
|
|
|
|
**Alternative estimators**:
|
|
- k-nearest neighbors (Kraskov et al., 2004)
|
|
- Kernel density estimation
|
|
- Copula-based methods
|
|
|
|
## Information-Theoretic Principles
|
|
|
|
### Maximum Entropy Principle
|
|
|
|
**Given**: Constraints on moments or expectations
|
|
|
|
**Find**: Distribution with maximum entropy satisfying constraints
|
|
|
|
**Result**: Least informative distribution consistent with knowledge
|
|
|
|
**Example**: Max entropy with mean $\mu$ and variance $\sigma^2$ → Gaussian $\mathcal{N}(\mu, \sigma^2)$
|
|
|
|
### Minimum Description Length (MDL)
|
|
|
|
**Model selection**: Choose model that minimizes:
|
|
$$\text{Description Length} = \text{Data encoding cost} + \text{Model encoding cost}$$
|
|
|
|
Related to Bayesian Information Criterion (BIC).
|
|
|
|
### Information Bottleneck
|
|
|
|
**Goal**: Compress $X$ to $T$ while preserving information about $Y$
|
|
|
|
**Objective**:
|
|
$$\min_{p(t|x)} I(X; T) - \beta I(T; Y)$$
|
|
|
|
Trade-off between compression and relevance.
|
|
|
|
## Applications
|
|
|
|
### 1. Feature Selection
|
|
|
|
**Goal**: Select features most informative about target
|
|
|
|
**Method**: Rank features by $I(X_i; Y)$
|
|
|
|
**Advantages over correlation**:
|
|
- Detects nonlinear relationships
|
|
- Handles categorical variables naturally
|
|
|
|
**Example**:
|
|
```
|
|
Features: X₁, X₂, X₃, X₄
|
|
Target: Y
|
|
|
|
I(X₁; Y) = 0.8
|
|
I(X₂; Y) = 0.3
|
|
I(X₃; Y) = 1.2 ← most informative
|
|
I(X₄; Y) = 0.1
|
|
|
|
Select X₃, then X₁
|
|
```
|
|
|
|
### 2. Dependency Detection
|
|
|
|
**Test independence**: $X \perp Y$ iff $I(X; Y) = 0$
|
|
|
|
**Hypothesis test**:
|
|
- Null: $I(X; Y) = 0$ (independent)
|
|
- Alternative: $I(X; Y) > 0$ (dependent)
|
|
|
|
**Test statistic**: $2n \cdot I(X; Y) / \ln(2)$ approximately $\chi^2$ distributed.
|
|
|
|
### 3. Clustering
|
|
|
|
**Information-theoretic clustering** minimizes within-cluster entropy.
|
|
|
|
**Objective**:
|
|
$$\min \sum_{k=1}^K \pi_k H(X | C=k)$$
|
|
|
|
where $\pi_k$ is cluster proportion.
|
|
|
|
### 4. Transfer Entropy
|
|
|
|
**Causality detection** in time series:
|
|
|
|
$$TE_{X \to Y} = I(Y_{t+1}; X_t | Y_t)$$
|
|
|
|
Measures information flow from $X$ to $Y$.
|
|
|
|
### 5. Data Compression
|
|
|
|
**Shannon's source coding theorem**:
|
|
|
|
Expected code length $\geq H(X)$ (entropy is fundamental limit).
|
|
|
|
**Huffman coding**, **arithmetic coding** approach this limit.
|
|
|
|
### 6. Neural Network Analysis
|
|
|
|
**Information plane**: Track $I(X; T)$ and $I(T; Y)$ during training
|
|
|
|
where $T$ is hidden layer representation.
|
|
|
|
**Observations**:
|
|
- Initial phase: Increase both (fitting)
|
|
- Later phase: Decrease $I(X; T)$, maintain $I(T; Y)$ (compression)
|
|
|
|
## Multivariate Extensions
|
|
|
|
### Joint Mutual Information
|
|
|
|
$$I(X_1, X_2; Y) = H(Y) - H(Y | X_1, X_2)$$
|
|
|
|
### Conditional Mutual Information
|
|
|
|
$$I(X; Y | Z) = H(X|Z) - H(X|Y,Z)$$
|
|
|
|
**Interpretation**: Information shared by $X$ and $Y$ not contained in $Z$
|
|
|
|
### Total Correlation
|
|
|
|
$$C(X_1, ..., X_n) = \sum_{i=1}^n H(X_i) - H(X_1, ..., X_n)$$
|
|
|
|
Measures total dependence among variables.
|
|
|
|
### Interaction Information
|
|
|
|
For three variables:
|
|
$$I(X; Y; Z) = I(X; Y|Z) - I(X; Y)$$
|
|
|
|
Can be positive (synergy) or negative (redundancy).
|
|
|
|
## Relationship to Other Concepts
|
|
|
|
### Information and Probability
|
|
|
|
$$I(E) = -\log p(E)$$
|
|
|
|
**Self-information** of event $E$.
|
|
|
|
Rare events carry more information.
|
|
|
|
### Fisher Information
|
|
|
|
For parameter estimation:
|
|
|
|
$$\mathcal{I}(\theta) = \mathbb{E}\left[\left(\frac{\partial \log p(X|\theta)}{\partial \theta}\right)^2\right]$$
|
|
|
|
Measures precision of estimating $\theta$.
|
|
|
|
**Cramér-Rao bound**: Variance of any unbiased estimator $\geq 1/\mathcal{I}(\theta)$
|
|
|
|
### Entropy and Thermodynamics
|
|
|
|
**Boltzmann entropy**: $S = k_B \ln W$
|
|
|
|
**Connection**: Statistical mechanics entropy ≈ Shannon entropy of microstates.
|
|
|
|
### Entropy Rate
|
|
|
|
For stochastic process $\{X_t\}$:
|
|
|
|
$$h = \lim_{n \to \infty} \frac{1}{n} H(X_1, ..., X_n)$$
|
|
|
|
**For Markov chains**: $h = -\sum_{i,j} \pi_i p_{ij} \log p_{ij}$
|
|
|
|
## Theoretical Results
|
|
|
|
### Source Coding Theorem
|
|
|
|
Expected code length $L \geq H(X)$
|
|
|
|
Equality achieved by Shannon coding.
|
|
|
|
### Channel Capacity
|
|
|
|
Maximum rate of reliable communication:
|
|
|
|
$$C = \max_{p(x)} I(X; Y)$$
|
|
|
|
where $Y$ is channel output given input $X$.
|
|
|
|
### Data Processing Inequality
|
|
|
|
If $X \to Y \to Z$ (Markov chain):
|
|
|
|
$$I(X; Y) \geq I(X; Z)$$
|
|
|
|
Processing cannot increase mutual information.
|
|
|
|
### Fano's Inequality
|
|
|
|
For estimating $X$ from $Y$ with error probability $P_e$:
|
|
|
|
$$H(X|Y) \leq H(P_e) + P_e \log(|\mathcal{X}| - 1)$$
|
|
|
|
Lower bound on conditional entropy given error rate.
|
|
|
|
## Computational Considerations
|
|
|
|
### Complexity
|
|
|
|
**Entropy estimation**: $O(n + m)$ where $n$ = samples, $m$ = bins
|
|
|
|
**Mutual information**: $O(n + m^2)$ for 2D histogram
|
|
|
|
**High dimensions**: Curse of dimensionality (need $m^d$ bins for $d$ dimensions)
|
|
|
|
### Numerical Stability
|
|
|
|
**Issue**: $\log 0$ is undefined
|
|
|
|
**Solutions**:
|
|
- Add small constant: $p + \epsilon$
|
|
- Use convention: $0 \log 0 = 0$
|
|
- Laplace smoothing: $(n_i + \alpha) / (n + \alpha m)$
|
|
|
|
## Key References
|
|
|
|
1. **Shannon, C. E.** (1948). *A mathematical theory of communication*. Bell System Technical Journal, 27(3), 379-423.
|
|
- Foundational paper
|
|
|
|
2. **Cover, T. M., & Thomas, J. A.** (2006). *Elements of Information Theory* (2nd ed.). Wiley.
|
|
- Comprehensive textbook
|
|
|
|
3. **MacKay, D. J.** (2003). *Information Theory, Inference and Learning Algorithms*. Cambridge University Press.
|
|
- Applications to machine learning
|
|
|
|
4. **Kraskov, A., Stögbauer, H., & Grassberger, P.** (2004). *Estimating mutual information*. Physical Review E, 69(6), 066138.
|
|
- k-NN based MI estimation
|
|
|
|
5. **Paninski, L.** (2003). *Estimation of entropy and mutual information*. Neural Computation, 15(6), 1191-1253.
|
|
- Bias correction methods
|
|
|
|
## Summary
|
|
|
|
Information theory provides fundamental limits and tools for:
|
|
|
|
**Core concepts**:
|
|
- **Entropy**: Uncertainty/information content
|
|
- **Mutual Information**: Shared information/dependence
|
|
- **KL Divergence**: Difference between distributions
|
|
|
|
**Key properties**:
|
|
- Entropy is maximized by uniform distribution
|
|
- Conditioning reduces entropy
|
|
- Mutual information detects any dependency
|
|
|
|
**Applications**:
|
|
- Feature selection and dimensionality reduction
|
|
- Model selection and compression
|
|
- Causality and dependency detection
|
|
- Machine learning (cross-entropy loss)
|
|
|
|
## See Also
|
|
|
|
- [Information Theory API Documentation](../information_theory.md) - Implementation and usage
|
|
- [HMM Theory](hmm.md) - Applications to sequential models
|
|
- [MCMC Theory](mcmc.md) - Sampling and inference methods
|