mirror of
https://github.com/mihakralj/QuanTAlib.git
synced 2026-08-25 05:48:06 +00:00
Implement ZTEST: One-Sample t-Test Statistic with validation tests
- Added Ztest class to compute the one-sample t-statistic using sample standard deviation with Bessel correction. - Implemented validation tests for Ztest to ensure accuracy against manual calculations and PineScript. - Updated documentation for Ztest, detailing its mathematical foundation, performance profile, and common pitfalls. - Adjusted NDepend badges to reflect changes in code metrics after implementation. - Updated missing indicators report to reflect the completion of statistical indicators, including ZTEST.
This commit is contained in:
Miha Kralj
@@ -0,0 +1,135 @@
|
||||
# JB: Jarque-Bera Test
|
||||
|
||||
> "The assumption of normality is the most dangerous assumption in all of statistics." — George Box (paraphrased)
|
||||
|
||||
The Jarque-Bera test quantifies departure from normality by combining skewness and excess kurtosis into a single chi-squared statistic. A rolling JB value near zero means the window looks Gaussian. Values exceeding 5.991 (5% significance) reject normality. Financial returns almost always fail this test, which is precisely why the test matters.
|
||||
|
||||
## Historical Context
|
||||
|
||||
Carlos Jarque and Anil Bera published the test in 1980, building on earlier work by Bowman and Shenton (1975). The insight was elegant: under normality, skewness is zero and kurtosis is three, so any deviation from these values indicates non-Gaussianity. The test statistic combines both deviations into a single number that follows a chi-squared distribution with two degrees of freedom.
|
||||
|
||||
Most implementations compute JB on static samples. This rolling implementation maintains O(1) updates by tracking running sums of powers (x, x², x³, x⁴), matching the approach used in the companion Skew indicator but extended to the fourth moment.
|
||||
|
||||
## Architecture
|
||||
|
||||
### 1. Running Power Sums
|
||||
|
||||
Four accumulators track $\sum x_i$, $\sum x_i^2$, $\sum x_i^3$, $\sum x_i^4$ over a sliding window of size $n$. When a new value enters and the oldest exits, each accumulator updates via simple addition/subtraction. This yields O(1) complexity per update.
|
||||
|
||||
### 2. Central Moments from Power Sums
|
||||
|
||||
Central moments are computed from raw power sums without explicitly centering each value:
|
||||
|
||||
$$m_2 = \frac{\sum x_i^2 - \frac{(\sum x_i)^2}{n}}{n}$$
|
||||
|
||||
$$m_3 = \frac{\sum x_i^3 - 3\bar{x}\sum x_i^2 + 2n\bar{x}^3}{n}$$
|
||||
|
||||
$$m_4 = \frac{\sum x_i^4 - 4\bar{x}\sum x_i^3 + 6\bar{x}^2\sum x_i^2 - 3n\bar{x}^4}{n}$$
|
||||
|
||||
### 3. Periodic Resync
|
||||
|
||||
Floating-point drift accumulates in running sums. Every 1000 ticks, the accumulator is rebuilt from the buffer contents. This bounds error growth without degrading amortized complexity.
|
||||
|
||||
## Mathematical Foundation
|
||||
|
||||
### Skewness
|
||||
|
||||
$$S = \frac{m_3}{m_2^{3/2}}$$
|
||||
|
||||
### Excess Kurtosis
|
||||
|
||||
$$K = \frac{m_4}{m_2^2} - 3$$
|
||||
|
||||
### Jarque-Bera Statistic
|
||||
|
||||
$$JB = \frac{n}{6}\left(S^2 + \frac{K^2}{4}\right)$$
|
||||
|
||||
Under $H_0$ (normality), $JB \sim \chi^2(2)$.
|
||||
|
||||
### Critical Values
|
||||
|
||||
| Significance | Critical Value |
|
||||
|:-------------|:---------------|
|
||||
| 10% (0.10) | 4.605 |
|
||||
| 5% (0.05) | 5.991 |
|
||||
| 1% (0.01) | 9.210 |
|
||||
|
||||
### Parameter Mapping
|
||||
|
||||
| Parameter | PineScript | QuanTAlib |
|
||||
|:----------|:-----------|:----------|
|
||||
| Window | `length` | `period` |
|
||||
| Min Value | 10 | 3 |
|
||||
|
||||
QuanTAlib allows period >= 3 (minimum for meaningful moments), though periods below 10 produce unstable estimates.
|
||||
|
||||
## Performance Profile
|
||||
|
||||
### Operation Count (Scalar, per bar)
|
||||
|
||||
| Operation | Count | Cycle Cost |
|
||||
|:----------|:------|:-----------|
|
||||
| ADD/SUB | 20 | 1 |
|
||||
| MUL | 16 | 3 |
|
||||
| DIV | 5 | 15 |
|
||||
| SQRT | 1 | 15 |
|
||||
| FMA | 1 | 4 |
|
||||
|
||||
### Batch Mode (SIMD/AVX2)
|
||||
|
||||
Vectorized path processes 4 bars per iteration using prefix-sum accumulators for all four power sums. Available when `Avx2.IsSupported` and input contains no NaN values.
|
||||
|
||||
| Metric | Scalar | AVX2 |
|
||||
|:------------|:-------|:-------|
|
||||
| Bars/cycle | 1 | ~3.2 |
|
||||
| Throughput | 1x | ~3.2x |
|
||||
|
||||
### Quality Metrics
|
||||
|
||||
| Metric | Score | Notes |
|
||||
|:------------|:------|:------|
|
||||
| Accuracy | 8/10 | Running sums accumulate FP drift; resync every 1000 ticks |
|
||||
| Timeliness | 9/10 | No lag beyond window fill |
|
||||
| Sensitivity | 7/10 | Responds to both skewness and kurtosis changes |
|
||||
| Robustness | 8/10 | NaN/Infinity guarded; non-negative by construction |
|
||||
|
||||
## Validation
|
||||
|
||||
No external library implements rolling Jarque-Bera with matching methodology. Validation relies on mathematical properties.
|
||||
|
||||
| Library | Status | Notes |
|
||||
|:---------|:------:|:------|
|
||||
| TA-Lib | - | Not implemented |
|
||||
| Skender | - | Not implemented |
|
||||
| Tulip | - | Not implemented |
|
||||
| Ooples | - | Not implemented |
|
||||
|
||||
Self-validation:
|
||||
|
||||
- Constant series produces JB = 0
|
||||
- Linear sequence {1..20} produces JB = 1.2 (analytical: uniform excess kurtosis = -6/5)
|
||||
- Skewed data produces larger JB than symmetric data
|
||||
- JB is always non-negative (sum of squares)
|
||||
- Batch, streaming, span, and event modes produce identical results
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
1. **Small windows inflate JB.** With n < 10, moment estimates are noisy. The test's chi-squared approximation requires n >= 30 for reliable p-values. QuanTAlib allows n >= 3 for computation but interprets results cautiously below n = 20.
|
||||
|
||||
2. **JB tests population skewness, not sample.** This implementation uses population moments (dividing by n, not n-1), matching the original Jarque-Bera formulation and the PineScript reference. Sample-adjusted versions exist but produce different critical values.
|
||||
|
||||
3. **Zero variance data returns JB = 0.** When all values in the window are identical, m2 = 0 and the formula is undefined. The implementation returns 0, which correctly indicates no evidence against normality (a degenerate distribution is trivially "normal-shaped").
|
||||
|
||||
4. **Financial returns almost always reject normality.** Fat tails (positive excess kurtosis) are universal in financial data. A persistently high JB is normal for markets. The indicator is most useful for detecting *changes* in the degree of non-normality.
|
||||
|
||||
5. **FP drift in x⁴ accumulator.** The fourth power amplifies floating-point errors more than lower moments. The resync interval of 1000 ticks keeps drift bounded, but for very long-running streams (>100k ticks), consider shorter resync intervals.
|
||||
|
||||
6. **NaN handling substitutes last valid.** Non-finite inputs are replaced with the most recent finite value. This maintains continuity but can mask data quality issues. Monitor NaN frequency separately.
|
||||
|
||||
7. **Memory: 4 doubles of running state.** The O(1) update carries sum, sumSq, sumCu, sumQu plus previous-state copies for bar correction. Total state footprint is ~128 bytes excluding the RingBuffer.
|
||||
|
||||
## References
|
||||
|
||||
- Jarque, C. M.; Bera, A. K. (1980). "Efficient tests for normality, homoscedasticity and serial independence of regression residuals." *Economics Letters*, 6(3), 255-259.
|
||||
- Bowman, K. O.; Shenton, L. R. (1975). "Omnibus test contours for departures from normality based on √b₁ and b₂." *Biometrika*, 62(2), 243-250.
|
||||
- PineScript reference: `lib/statistics/jb/jb.pine`
|
||||
Reference in New Issue
Block a user