QuanTAlib/lib/errors/theilu/TheilU.md

# Theil's U: Theil's U Statistic

> "The forecast that matters is the one that beats a naive guess."

Theil's U Statistic measures forecast accuracy relative to a naive no-change forecast. A value below 1 indicates the model outperforms simply predicting that tomorrow equals today; above 1 means you'd be better off not forecasting at all.

## Historical Context

Developed by Dutch econometrician Henri Theil in the 1960s, Theil's U was designed to evaluate economic forecasts against the simplest possible benchmark: the assumption of no change. This was revolutionary because many sophisticated models fail to beat this naive approach, especially in financial markets.

## Architecture & Physics

Theil's U computes two parallel error metrics: one for the forecast and one for a naive prediction. The ratio reveals whether the forecasting effort adds value. A forecast might have low absolute error but still be worse than doing nothing.

### Properties

* **Relative benchmark**: Compares against naive no-change forecast
* **Scale-independent**: Ratio is unitless
* **Interpretable threshold**: U = 1 is the break-even point
* **Range**: 0 to ∞, with 0 being perfect and > 1 being worse than naive

## Mathematical Foundation

### 1. Forecast Error

Calculate squared errors for the actual forecast:

$$FPE = \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$$

Where:
* $y_i$ = actual value at time i
* $\hat{y}_i$ = predicted value at time i

### 2. Naive Error

Calculate squared errors for naive prediction (previous actual):

$$NPE = \sum_{i=1}^{n} (y_i - y_{i-1})^2$$

### 3. Theil's U Calculation

Take the ratio of forecast to naive:

$$U = \sqrt{\frac{FPE}{NPE}} = \sqrt{\frac{\sum_{i=1}^{n} (y_i - \hat{y}_i)^2}{\sum_{i=1}^{n} (y_i - y_{i-1})^2}}$$

### 4. Running Update (O(1))

QuanTAlib maintains running sums of both squared error terms:

$$S_{f,new} = S_{f,old} - e_{f,oldest}^2 + e_{f,newest}^2$$

$$S_{n,new} = S_{n,old} - e_{n,oldest}^2 + e_{n,newest}^2$$

$$U = \sqrt{\frac{S_{f,new}}{S_{n,new}}}$$

## Implementation Details

### Usage Patterns

```csharp
// Streaming mode - update with each new observation
var theilU = new TheilU(period: 20);
var result = theilU.Update(actualValue, predictedValue);

// Batch mode - calculate for entire series
var results = TheilU.Calculate(actualSeries, predictedSeries, period: 20);

// Span mode - zero-allocation for high performance
TheilU.Batch(actualSpan, predictedSpan, outputSpan, period: 20);
```

### Parameters

| Parameter | Type | Description |
| :--- | :--- | :--- |
| **period** | int | Lookback window for calculation (must be > 0) |

### Properties

| Property | Type | Description |
| :--- | :--- | :--- |
| **Last** | TValue | Most recent Theil's U value |
| **IsHot** | bool | True when buffer is full |
| **Name** | string | Indicator name (e.g., "TheilU(20)") |
| **WarmupPeriod** | int | Number of periods before valid output |

## Performance Profile

### Operation Count (Streaming Mode)

Theil's U statistic: U = sqrt(MSE_forecast) / sqrt(MSE_naive). Requires two running mean-squared-error accumulators.

| Operation | Count | Cost (cycles) | Subtotal |
| :--- | :---: | :---: | :---: |
| Forecast MSE update (e^2 + EMA) | 2 | ~5 cy | ~10 cy |
| Naive MSE update (naive_e^2 + EMA) | 2 | ~5 cy | ~10 cy |
| U = sqrt(MSE_f) / sqrt(MSE_n) | 2 | ~15 cy | ~30 cy |
| **Total** | **~6** | — | **~50 cycles** |

O(1) per bar. Two parallel EMA accumulators + ratio with sqrt. ~50 cycles/bar.

### Batch Mode (SIMD Analysis)

| Operation | Vectorizable? | Notes |
| :--- | :---: | :--- |
| Squared error accumulation | Yes | Element-wise squares + reduction |
| sqrt ratio | No | Single scalar at end |

Batch MSE accumulation vectorizable; final ratio is scalar. ~8 cy/bar for squared-error accumulation.

| Metric | Score | Notes |
| :--- | :--- | :--- |
| **Throughput** | ~15 ns/bar | O(1) update complexity |
| **Allocations** | 0 | Uses pre-allocated ring buffers |
| **Complexity** | O(1) | Constant time per update |
| **Accuracy** | 10/10 | Exact calculation |
| **Timeliness** | 9/10 | No lag beyond the period |
| **Interpretability** | 10/10 | Clear benchmark comparison |

## Interpretation

| Theil's U | Interpretation |
| :--- | :--- |
| **0** | Perfect prediction |
| **< 0.5** | Excellent (error < 50% of naive) |
| **0.5 - 0.8** | Good forecasting skill |
| **0.8 - 1.0** | Marginal improvement over naive |
| **= 1.0** | Equal to naive forecast |
| **> 1.0** | Worse than naive (model adds noise) |

## Why Use Theil's U?

| Scenario | Low MAE but High U | High MAE but Low U |
| :--- | :--- | :--- |
| **Meaning** | Series is easy to predict | Model adds value despite errors |
| **Example** | Stable prices, any model works | Volatile prices, model captures moves |
| **Recommendation** | Use simpler model | Keep using the model |

## Common Use Cases

1. **Economic Forecasting**: Evaluate macro predictions against random walk
2. **Financial Markets**: Test trading signals against buy-and-hold
3. **Model Selection**: Choose models that beat naive benchmarks
4. **Forecast Validation**: Ensure forecasting effort is worthwhile

## Edge Cases

* **Zero Naive Error**: Returns infinity when series is perfectly flat (naive is perfect)
* **NaN Handling**: Uses last valid value substitution
* **Single Input**: Not supported (requires two series)
* **Period = 1**: Returns 0 (insufficient data for naive comparison)
* **First Value**: Needs at least 2 values for naive benchmark

## Related Indicators

* [RMSE](../rmse/Rmse.md) - Root Mean Squared Error (absolute, not relative)
* [MASE](../mase/Mase.md) - Mean Absolute Scaled Error (similar concept)
* [R-Squared](../rsquared/RSquared.md) - Coefficient of Determination