gavindiaz/drift
1
0
Fork 0
mirror of https://github.com/webclinic017/drift.git synced 2026-08-18 13:28:22 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(DataLoader): caching MVP, added ability to use standard scaling for exogenous data, scaling is now also done before feature selection (#105)

Browse Source 
* fix(FeatureExtractor): apply log to transform some series to normality

* feat(DataLoader): add ability of not returning returns when they're not needed (exogenous data), applied log to certain features

* feat(FeatureExtractors): added standard scaling for exogenous data

* feat(FeatureSelection): scale data with the passed in scaler before doing feature-selection

* fix(Config): sweep config

* feat(Models): output probability, store it

* feat(Core): added caching to select_features() and load_data()

* fix(Dependencies): added diskcache

* fix(Training): error when creating results DF

* feat(Models): added xgboost, fixed tests

* refactor(Cache): moved hashing to a separate function, created wrapper functions to separate business logic and caching

* fix(Tests): new syntax

* fix(Model): XGboost can't handle -1 class, so we'll use the deprecated label_encoder fornow

* fix(Model): XGBoost config

* feat(Cache): add run_clear_cache script

* fix(Pipeline) accidentally re-instatiating all_predictions for each asset
This commit is contained in:
Mark Aron Szulyovszky
2022-01-04 11:44:35 +01:00
committed by GitHub
parent 867269df2b
commit 1cd0119589
27 changed files with 324 additions and 206 deletions
Download Patch File Download Diff File Expand all files Collapse all files
training/walk_forward.py
+11 -6
View File
@@ -16,9 +16,10 @@ def walk_forward_train_test(
window_size: int,
retrain_every: int,
scaler,
) -> tuple[pd.Series, pd.Series]:
) -> tuple[pd.Series, pd.Series, pd.DataFrame]:
assert len(X) == len(y)
predictions = pd.Series(index=y.index).rename(model_name)
probabilities = pd.DataFrame(index=y.index)
models = pd.Series(index=y.index).rename(model_name)
first_nonzero_return = max(get_first_valid_return_index(target_returns), get_first_valid_return_index(X.iloc[:,0]))
@@ -45,8 +46,8 @@ def walk_forward_train_test(
train_window_end = index - 1
if is_scaling_on:
# First we need to fit on the expanding window data slice
# This is our only way to avoid lookahead bia
# We need to fit on the expanding window data slice
# This is our only way to avoid lookahead bias
X_expanding_window = X[first_nonzero_return:train_window_end]
scaler.fit(X_expanding_window.values)
@@ -59,7 +60,7 @@ def walk_forward_train_test(
X_slice = X_slice.to_numpy()
current_model = model.clone()
current_model.fit(X_slice, y_slice.to_numpy(), models[index-1])
current_model.fit(X_slice, y_slice.to_numpy())
iterations_before_retrain = retrain_every
else:
current_model = models[index-1]
@@ -70,8 +71,12 @@ def walk_forward_train_test(
if is_scaling_on:
next_timestep = scaler.transform(next_timestep)
prediction = current_model.predict(next_timestep).item()
prediction, probs = current_model.predict(next_timestep)
predictions[index] = prediction
if len(probabilities.columns) != len(probs):
probabilities = probabilities.reindex(columns = ["prob_" + str(num) for num in range(0, len(probs.T))])
probabilities.iloc[index] = probs
iterations_before_retrain -= 1
return models, predictions
return models, predictions, probabilities