{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"#ignore\n",
"from IPython.core.display import HTML,Image\n",
"import sys"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Markets are, in my view, mostly random. However, they're not _completely_ random. Many small inefficiencies and patterns exist in markets which can be identified and used to gain slight edge on the market. \n",
"\n",
"These edges are rarely large enough to trade in isolation - transaction costs and overhead can easily exceed the expected profits offered. But when we are able to combine many such small edges together, the rewards can be great. \n",
"\n",
"In this article, I'll present a framework for blending together outputs from multiple models using a type of ensemble modeling known as _stacked generalization_. This approach excels at creating models which \"generalize\" well to unknown future data, making them an excellent choice for the financial domain, where overfitting to past data is a major challenge. \n",
"\n",
"This post is the sixth and final installment in my tutorial series on applying machine learning to financial time series data. If you haven't already read the prior articles, you may want to do that before starting this one. \n",
"* [Data management](ML_data_management.html)\n",
"* [Feature engineering](feature_engineering.html)\n",
"* [Feature selection](feature_selection.html)\n",
"* [Walk-forward modeling](walk_forward_model_building.html)\n",
"* [Model evaluation](model_evaluation.html)\n",
"\n",
"### Ensemble Learning \n",
"\n",
"_Ensemble learning_ is a powerful - and widely used - technique for improving model performance (especially it's _generalization_) by combining predictions made by multiple different machine learning models. The idea behind ensemble learning is not dissimilar from the concept [\"wisdom of the crowd\"](https://en.wikipedia.org/wiki/Wisdom_of_the_crowd), which posits that the aggregated/consensus answer of several diverse, well-informed individuals is typically better than any one individual within the group. \n",
"\n",
"In the world of machine learning, this concept of combining multiple models takes many forms. The first form appears _within_ a number of commonly used algorithms such as [Random Forests](https://en.wikipedia.org/wiki/Random_forest), [Bagging](https://en.wikipedia.org/wiki/Bootstrap_aggregating), and [Boosting](https://en.wikipedia.org/wiki/Boosting_(machine_learning) (though this one works somewhat differently). Each of these algorithms takes a single base model (e.g., a decision tree) and trains many versions of that single algorithm on differing sets of features or samples. The resulting collection of trained models are often more robust out of sample because they're likely to be less overfitted to certain features or samples in the training data.\n",
"\n",
"A second form of ensembling methods involves aggregating across multiple _different model types_ (e.g., an SVM, a logistic regression, and a decision tree) - or with _different hyperparameters_. Since each learning algorithm or set of hyperparameters tends to have different biases, it will tend to make different prediction errors - and extract different signals - from the same set of data. Assuming all models are reasonably good - and that the errors are reasonably uncorrelated to one another - they will partially cancel each other out and the aggregated predictions will be more useful than any single model's predictions. \n",
"\n",
"One particularly flexible approach to this latter type of ensemble modeling is \"stacked generalization\", or \"stacking\". In this post, I will walk through a simple example of stacked generalization applied to time series data. If you'd like to replicate and experiment with the below code, _you can download the source notebook for this post by right-clicking on the below button and choosing \"save link as\"_ \n",
"\n",
"
\n",
"\n",
"\n",
"\n",
"### Overview of Stacked Generalization \n",
"The \"stacked generalization\" framework was initially proposed by Wolpert in a [1992 academic paper](http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.56.1533). Since it was first proposed, stacked generalization (aka \"stacking\") has received a modest but consistent amount of attention from the ML research community. \n",
"\n",
"Stacked generalization is an ensemble modeling technique. The core concept of stacked generalization is to generate a single, optimally robust prediction for a regression or classification task by (a) building multiple different models (with varying learning algorithms, varying hyperparameters, and/or different features) to make predictions then (b) training a \"meta-model\" or \"blending model\" to determine how to combine the predictions of each of these multiple models. \n",
"\n",
"A nice way to visualize this (borrowed from documentation for Sebastian Rashka's excellent [mlxtend package](http://rasbt.github.io/mlxtend/user_guide/regressor/StackingRegressor/)) is shown below. Each model R1 thru Rm is trained on historical data and used to make predictions P1 thru Pm. Those predictions then become the features used to train a meta-model to determine how to combine these predictions. \n",
"\n",
"\n",
"
\n",
"\n",
"I think of this using an analogy. Imagine that there is a team of investment analysts whose manager has asked each of them to make earnings forecasts for the same set of companies across many quarters. The manager \"learns\" which analysts have historically been most accurate, somewhat accurate, and inaccurate. When future predictions are needed, the manager can assign greater and lesser (and in some cases, zero) weighting to each analyst's prediction.\n",
"\n",
"It's clear why it's referred to as \"stacked\". But why \"generalization\"? The principal motivation for applying this technique is to achieve greater \"generalization\" of models to out-of-sample (i.e., unseen) data by de-emphasizing models which appear to be overfitted to the data. This is achieved by allowing the meta-model to learn which of the base models' predictions have held up well (and poorly) out-of-sample and to weight models appropriately. \n",
"\n",
"\n",
"### Motivations\n",
"In my view, stacked generalization is perfectly suited to the challenges we face when making predictions in noisy, non-stationary, regime-switching financial markets. When properly implemented (see next section), stacking help to defend against the scourge of overfitting - something which virtually all practitioners of investing ML will agree is a major challenge. \n",
"\n",
"Better yet, stacking allows us to blend together relatively weak (but orthogonal and additive) signals together in a way that doesn't get drowned out by stronger signals. \n",
"\n",
"To illustrate, consider a canonical trend-following strategy which is predicated on 12 month minus 1 month price change. Perhaps we also believe that month-of-year or recent IBIS earnings trend have a weak, but still useful effect on price changes. If we were to train a model that lumped together dominant features (12 minus 1 momentum) and weaker features (seasonality or IBIS trend), our model may miss the subtle information because the dominant features overshadow them. \n",
"\n",
"A stacked model, which has one component (i.e., a base model) focused on solely momentum features, another component focused on solely seasonality features, and a third one focused on analyst revisions features can capture and use the more subtle effects alongside the more dominant momentum effect.\n",
"\n",
"\n",
"### Keys to Success\n",
"Stacked generalization is sometimes referred to as a \"black art\" and there is truth to that view. However, there are also two concrete principles that will get you a long way towards robust results.\n",
"\n",
"__1. Out of Sample Training__ \n",
"First, it's _absolutely critical_ that the predictions P1 thru Pm used to train the meta-model are exclusively _out of sample_ predictions. Why? Because in order to determine which models are likely to generalize best to out of sample (ie those with least overfit), we must judge that based on past predictions which were themselves made out-of-sample. \n",
"\n",
"Imagine that you trained two models using different algorithms, say logistic regression and decision trees. Both could be very useful (out of sample) but decision trees have a greater tendency to overfit training data. If we used in-sample predictions as features to our meta-learner, we'd likely give much more weight to the model with a tendancy to overfit the most. \n",
"\n",
"Several methods can be used for this purpose. Some advise splitting training data into Train1 and Train2 sets so base models can be trained on Train1 and then can make predictions on Train2 data for use in training the ensemble model. Predictions of the ensemble model must, of course, be evaluated on yet another dataset. \n",
"\n",
"Others use K-fold cross-validation prediction (such as scikit's `cross_val_predict`) on base models to simulate out-of-sample(ish) predictions to feed into the ensemble layer. \n",
"\n",
"However, in my view, the best method for financial time series data is to use walk-forward training and prediction on the base models, as described in my [Walk-forward modeling](walk_forward_model_building.html) post. In addition to ensuring that every base prediction is true out-of-sample, it simulates the impact of non-stationarity (a.k.a. regime change) over time. \n",
"\n",
"__2. Non-Negativity__ \n",
"Second - and this is less of a hard-and-fast rule - is to constrain the meta-model to learning non-negative coefficients only, using an algorithm like ElasticNet or lasso which allows non-negativity constraints. \n",
"\n",
"This technique is important because quite often (and sometimes by design) there will be very high collinearity of the \"features\" fed into the meta-model (P1 thru Pm). In periods of high collinearity, learning algorithms can do funky things, such as finding a slightly better fit to past data by assigning a high positive coefficient to one model and a large negative coefficient to another. This is rarely what we really want. \n",
"\n",
"Call me crazy, but if a model is useful only in that it consistently predicts the wrong outcome, it's probably not a model I want to trust. \n",
"\n",
"\n",
"\n",
"### Further Reading\n",
"That's enough (too much?) background for now. Those interested in more about the theory and practice of stacked generalization should check out the below research papers:\n",
"* [Wolpert, 1992](http://www.machine-learning.martinsewell.com/ensembles/stacking/Wolpert1992.pdf)\n",
"* [Ting and Witten, 1997](https://pdfs.semanticscholar.org/fd97/e40ef6c310213fae017fdbf328c8bdf5cb68.pdf)\n",
"* [Ting and Witten, 1999](https://arxiv.org/pdf/1105.5466.pdf)\n",
"* [Breiman, 1996](https://link.springer.com/content/pdf/10.1007/BF00117832.pdf)\n",
"* [Sigletos et al, 2005](http://www.jmlr.org/papers/volume6/sigletos05a/sigletos05a.pdf)\n",
"* [Why do stacked ensemble models win data science competitions?](https://blogs.sas.com/content/subconsciousmusings/2017/05/18/stacked-ensemble-models-win-data-science-competitions/)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Preparing the Data\n",
"For this simple example, I will create synthetic data rather than using real market prices to remove the ambiguity about what features and transformations may be necessary to extract maximum value from the model. \n",
"\n",
"Note: to make the dataset more realistic, I will extract an _index_ from actual stock prices using quandl's API, but all features and target values will be constructed below. \n",
"\n",
"With index in hand, we will generate four \"hidden factors\". These are the non-random drivers of the target variable, and are the \"signal\" we ideally want to learn. \n",
"\n",
"To ensure that these factors are meaningful, we will _create the target variable (`y`) using combinations of these factors_. The first two hidden factors have a __linear__ relationship to the target. The second two hidden factors have a more complex relationship involving interaction effects between variables. Lastly, we will add a noise component to make our learners work for it. \n",
"\n",
"Finally, we'll create several features that are each related to one or more hidden factors, including generous amounts of noise and bias. \n",
"\n",
"Key point: we've created X and y data which we know is related by several different linkages, some of which are linear and some of which aren't. This is what our modeling will seek to learn. "
]
},
{
"cell_type": "code",
"execution_count": 142,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
| \n", " | \n", " | f1 | \n", "f2 | \n", "f3 | \n", "f4 | \n", "f5 | \n", "f6 | \n", "f7 | \n", "f8 | \n", "f9 | \n", "f10 | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|
| date | \n", "symbol | \n", "\n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " |
| 2012-01-03 | \n", "AAPL | \n", "0.290710 | \n", "-0.891838 | \n", "2.716568 | \n", "-2.150783 | \n", "-0.705256 | \n", "-0.452650 | \n", "-1.681323 | \n", "2.315719 | \n", "-2.074949 | \n", "0.195330 | \n", "
| CSCO | \n", "0.684334 | \n", "-0.243884 | \n", "1.427696 | \n", "-1.840774 | \n", "-0.060560 | \n", "2.298205 | \n", "-1.252799 | \n", "1.553487 | \n", "-3.292124 | \n", "-1.953685 | \n", "|
| INTC | \n", "0.261258 | \n", "0.520605 | \n", "1.846479 | \n", "-1.650393 | \n", "0.651065 | \n", "0.178868 | \n", "0.051921 | \n", "1.194534 | \n", "-1.717330 | \n", "4.514711 | \n", "|
| MSFT | \n", "1.026038 | \n", "0.122784 | \n", "-0.091722 | \n", "-4.462153 | \n", "0.032104 | \n", "1.076377 | \n", "0.288104 | \n", "0.504962 | \n", "-2.324226 | \n", "-0.570928 | \n", "|
| 2012-01-04 | \n", "AAPL | \n", "-1.182916 | \n", "-2.676495 | \n", "2.087499 | \n", "-2.474870 | \n", "-0.968875 | \n", "0.655539 | \n", "0.567816 | \n", "4.224116 | \n", "-0.748807 | \n", "1.383618 | \n", "
| ... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
| 2018-03-26 | \n", "MSFT | \n", "-1.678099 | \n", "-2.300153 | \n", "2.644520 | \n", "-2.186149 | \n", "-2.277767 | \n", "-0.177903 | \n", "-3.435729 | \n", "5.015973 | \n", "-2.384433 | \n", "-2.592677 | \n", "
| 2018-03-27 | \n", "AAPL | \n", "2.425041 | \n", "0.408673 | \n", "1.817243 | \n", "-3.249701 | \n", "-0.675492 | \n", "0.393031 | \n", "1.332007 | \n", "2.384243 | \n", "-3.372531 | \n", "-1.509449 | \n", "
| CSCO | \n", "0.743400 | \n", "0.284298 | \n", "1.787818 | \n", "-3.457743 | \n", "0.545880 | \n", "0.890898 | \n", "-2.346138 | \n", "2.344296 | \n", "-0.666132 | \n", "2.443742 | \n", "|
| INTC | \n", "1.992259 | \n", "-0.691250 | \n", "-0.086637 | \n", "-2.517395 | \n", "-0.913152 | \n", "1.121162 | \n", "-0.618323 | \n", "1.892296 | \n", "-2.282686 | \n", "-2.508796 | \n", "|
| MSFT | \n", "1.097342 | \n", "-0.552628 | \n", "0.211335 | \n", "1.053797 | \n", "-0.462206 | \n", "1.230167 | \n", "0.150094 | \n", "-0.408570 | \n", "-3.068049 | \n", "0.807234 | \n", "
6266 rows × 10 columns
\n", "