The Theil T Index is an information-theoretic measure of inequality (or concentration) within a distribution of positive values. Originally developed for income inequality analysis, it quantifies how far a set of values deviates from perfect equality. In financial contexts, it measures the concentration of returns or price magnitudes within a sliding window, producing values ranging from 0 (perfect equality, all values identical) upward with no fixed upper bound. The Theil T Index belongs to the family of generalized entropy indices and is notable for its decomposability property: total inequality can be additively decomposed into between-group and within-group components.
## Historical Context
Henri Theil introduced the T Index in 1967 in his work "Economics and Information Theory," borrowing Shannon's entropy framework to measure economic inequality. Where Shannon entropy measures information content, Theil's adaptation measures the "information content" of observing a particular share of total resources relative to equal shares. The measure gained prominence alongside the Gini coefficient and Atkinson index as a standard tool in welfare economics.
For financial markets, the Theil T Index serves as a concentration detector. A window of prices with roughly equal magnitudes yields T near 0; a window dominated by one extreme value (a spike or crash) produces high T. This makes it useful for detecting regime changes, volatility clustering, and abnormal price behavior that other measures (like standard deviation) may underweight due to squaring.
The key advantage over Gini: decomposability. The key advantage over variance-based measures: scale invariance. Multiplying all prices by a constant leaves T unchanged, measuring only the relative distribution structure.
## Architecture and Physics
### 1. Core Algorithm
The implementation uses a sliding window (RingBuffer) of size `period`. On each update:
1. Add the new value to the buffer (substituting last-valid for NaN/Infinity/non-positive)
2. Compute the mean of all valid positive values in the buffer
3. For each value, compute the ratio $r_i = x_i / \mu$ and accumulate $r_i \cdot \ln(r_i)$
4. Divide the sum by the count of valid values
### 2. Complexity
- **Update:** O(n) per tick where n = period (must scan buffer for mean, then for Theil sum)
- **Memory:** O(period) for the RingBuffer
- No O(1) streaming shortcut exists because the mean changes with every update, invalidating cached ratio computations
### 3. Value Domain
- **Input:** Positive values only (prices, volumes). Non-positive values and NaN/Infinity are replaced with last-valid substitution.
- **Output:** T >= 0. T = 0 for perfect equality. No fixed upper bound; maximum depends on window size and value distribution.
### 4. NaN/Infinity Handling
Non-finite or non-positive inputs are replaced with the last valid positive value. If no valid value has been seen, the value defaults to 0 (which is filtered out in the Theil computation).
## Mathematical Foundation
The Theil T Index (also called GE(1), generalized entropy with parameter 1) is defined as:
O(N^2 log N) per update — expensive for N > 30. Use only where robustness to outliers justifies compute cost. Batch pre-computation strongly preferred for historical analysis.
| Higher inequality → higher T | Uniform vs skewed distribution | Verified |
## Common Pitfalls
1.**Non-positive values:** The Theil T Index requires strictly positive inputs. Zero or negative values produce undefined logarithms. The implementation substitutes last-valid values, but feeding predominantly non-positive data yields meaningless results.
2.**Confusing T and L:** Theil's T (GE(1)) is sensitive to the upper tail; Theil's L (GE(0), mean log deviation) is sensitive to the lower tail. This implementation computes T only.
3.**Interpreting magnitude:** Unlike Gini (bounded [0,1]), Theil T has no fixed upper bound. Values must be interpreted relative to the data's own history, not against absolute thresholds.
4.**Small windows:** With period=2, only two values are compared. The index becomes highly volatile and loses statistical meaning. Recommend period >= 10 for meaningful results.
5.**All-equal series:** Returns exactly 0. This is correct behavior, not a bug. A constant price series has zero inequality by definition.
6.**Log(1) = 0 effect:** When a value equals the mean exactly, its contribution to T is zero (ratio=1, ln(1)=0). This is mathematically correct but means the index is insensitive to values near the mean.
7.**Comparison with Shannon Entropy:** Shannon entropy measures histogram-based randomness; Theil T measures value-based concentration. They answer different questions about the same data.
## References
- Theil, H. (1967). *Economics and Information Theory*. North-Holland Publishing Company.
