gavindiaz/drift
1
0
Fork 0
mirror of https://github.com/webclinic017/drift.git synced 2026-08-04 22:57:46 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(Project): use SKLearn models directly, removed custom ensembling, use 5 minute data, batch inference, numba cusum filter (#192)

Browse Source 
* feat(Project): use 5 minute data, running training in parallel, sped up cusum filter by 10x with numba

* fix(WalkForward): inference mini-batch parallelization

* fix(WalkForward): don't use the parallel version of any of the functions

* feat(CI): download the data required

* fix(Project): 5min_crypto folder added

* fix(Evaluate): make sure we have numerical stability in returns

* feat(Models): use SKLearn models directly to enable composability

* feat(Inference): batched inference now working, added forecasting_horizon

* fix(Inference): works again

* fix(Inference)

* chore(Models): remove unused Ensemble model

* fix(Labeller): don't just forward shift returns, also take the sum of the data happened until then

* Update test.yml
This commit is contained in:
Mark Aron Szulyovszky
2022-02-17 16:36:35 +01:00
committed by GitHub
parent 5c94af8b01
commit 9d47ee942d
52 changed files with 470 additions and 628 deletions
Download Patch File Download Diff File Expand all files Collapse all files
training/walk_forward/process_transformations_parallel.py
+2 -1
View File
@@ -6,6 +6,7 @@ from transformations.base import Transformation
from typing import Optional
from data_loader.types import ForwardReturnSeries, XDataFrame, ySeries
import ray
from utils.parallel import parallel_compute_with_bar
def walk_forward_process_transformations(
X: XDataFrame,
@@ -23,7 +24,7 @@ def walk_forward_process_transformations(
train_from = first_nonzero_return + window_size + 1 if from_index is None else X.index.to_list().index(from_index)
train_till = len(y)
processed_transformations = ray.get([preprocess_transformations_window.remote(X, y, expanding_window, window_size, transformations, first_nonzero_return, index) for index in range(train_from, train_till, retrain_every)])
processed_transformations = parallel_compute_with_bar([preprocess_transformations_window.remote(X, y, expanding_window, window_size, transformations, first_nonzero_return, index) for index in range(train_from, train_till, retrain_every)])
for transformation, index_time in processed_transformations:
for transformation_index, transformation in enumerate(transformation):