gavindiaz/drift
1
0
Fork 0
mirror of https://github.com/webclinic017/drift.git synced 2026-07-27 18:57:55 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(Transformations): replaced feature selection pre-processing step with online version (with cache) (#170)

Browse Source 
* feat(Transformations): removed feature-selection pre-processing step completely

* fix(Core): removed unnecessary `original_X`

* fix(Transformations): use the X_expanding_window to transform subsequent data

* fix(RFE): should check for model correctly

* fix(Config): only re-train the model every 40 timestamp

* fix(MetaLabeling): pass in the correct X to meta-labeling step

* fix(Transformation): PCA should at least keep as many features as sliding_window_size

* feat(Transformations): cache transformations across the same asset

* fix(Tests): missing preloaded_transformations arg

* chore(Config): got rid of unnecessary 'classification_models' and 'regression_models' dictionary keys
This commit is contained in:
Mark Aron Szulyovszky
2022-01-17 11:43:51 +01:00
committed by GitHub
parent 31dc847be1
commit 6b26643ece
27 changed files with 240 additions and 278 deletions
Download Patch File Download Diff File Expand all files Collapse all files
data_loader/process_data.py
-23
View File
@@ -1,31 +1,8 @@
import pandas as pd
from operator import itemgetter
from utils.helpers import has_enough_samples_to_train
from feature_selection.dim_reduction import reduce_dimensionality
from models.model_map import get_model_map
from feature_selection.feature_selection import select_features
import warnings
def process_data(X:pd.DataFrame, y:pd.Series, configs: dict) -> tuple[pd.DataFrame,pd.DataFrame]:
model_config, training_config, data_config = itemgetter('model_config', 'training_config', 'data_config')(configs)
_, _, _, default_feature_selector_regression, default_feature_selector_classification = get_model_map(model_config)
original_X = X.copy()
# 2b. Feature Selection
print("Feature Selection started")
# TODO: this needs to be done per model!
backup_model = default_feature_selector_regression if data_config['method'] == 'regression' else default_feature_selector_classification
X = select_features(X = X, y = y, model = model_config['primary_models'][0][1], n_features_to_select = training_config['n_features_to_select'], backup_model = backup_model, scaling = training_config['scaler'])
return X, original_X
def check_data(X:pd.DataFrame, y:pd.Series, training_config:dict):
""" Returns True if data is valid, else returns False."""