Bump requests from 2.7.0 to 2.20.0

Bumps [requests](https://github.com/requests/requests) from 2.7.0 to 2.20.0.
- [Release notes](https://github.com/requests/requests/releases)
- [Changelog](https://github.com/psf/requests/blob/master/HISTORY.md)
- [Commits](https://github.com/requests/requests/compare/v2.7.0...v2.20.0)

Signed-off-by: dependabot[bot] <support@github.com>

README.md

@@ -1,30 +1,129 @@
 # QuantStart Forex
 
-QSForex is an open-source work-in-progress event-driven backtesting and live trading platform for use in the foreign exchange ("forex") markets. 
+QSForex is an open-source event-driven backtesting and live trading platform for use in the foreign exchange ("forex") markets, currently in an "alpha" state.
 
-It has been created as part of the Forex Trading Diary series on QuantStart.com, predominantly for education purposes, but also to provide the systematic trading community with a robust trading engine that allows straightforward forex strategy implementation and testing. 
+It has been created as part of the Forex Trading Diary series on QuantStart.com to provide the systematic trading community with a robust trading engine that allows straightforward forex strategy implementation and testing. 
 
 The software is provided under a permissive "MIT" license (see below).
 
 # Current Features
 
-* Currently in alpha mode - very early stage!
-* Live trading with one particular forex broker, OANDA, via their Rest API
-* Event-driven architecture with price streaming (via the OANDA API)
-* Basic local portfolio replication of live trades (ultimately for backtesting purposes)
+* **Open-Source** - QSForex has been released under an extremely permissive open-source MIT License, which allows full usage in both research and commercial applications, without restriction, but with no warranty of any kind whatsoever.
+* **Free** - QSForex is completely free and costs nothing to download or use.
+* **Collaboration** - As QSForex is open-source many developers collaborate to improve the software. New features are added frequently. Any bugs are quickly determined and fixed.
+* **Software Development** - QSForex is written in the Python programming language for straightforward cross-platform support. QSForex contains a suite of unit tests for the majority of its calculation code and new tests are constantly added for new features.</li>
+* **Event-Driven Architecture** - QSForex is completely event-driven both for backtesting and live trading, which leads to straightforward transitioning of strategies from a research/testing phase to a live trading implementation.
+* **Transaction Costs** - Spread costs are included by default for all backtested strategies.
+* **Backtesting** - QSForex features intraday tick-resolution multi-day multi-currency pair backtesting.
+* **Trading** - QSForex currently supports live intraday trading using the OANDA Brokerage API across a portfolio of pairs.
+* **Performance Metrics** - QSForex currently supports basic performance measurement and equity visualisation via the Matplotlib and Seaborn visualisation libraries.
 
 # Installation and Usage
 
-At this stage, since the software is in alpha mode, the installation instructions are somewhat more involved. As the software evolves they will become more straightforward and documentation will become more extensive. However, the basic approach is as follows:
+1) Visit http://www.oanda.com/ and setup an account to obtain the API authentication credentials, which you will need to carry out live trading. I explain how to carry this out in this article: https://www.quantstart.com/articles/Forex-Trading-Diary-1-Automated-Forex-Trading-with-the-OANDA-API.
 
-1. Setup an account with OANDA and obtain the API authentication credentials
-2. Clone the repository into a suitable location on your machine
-3. Create two environment variables: OANDA_API_ACCESS_TOKEN and OANDA_API_ACCOUNT_ID, which must contain, respectively, the OANDA API Access Token and the OANDA API Account ID, as provided by OANDA
-4. Create a virtual environment ("virtualenv") for the QSForex code and utilise pip to install the requirements - `pip install -r requirements.txt`
-5. Modify `strategy.py` to create an event-driven strategy class
-6. Execute `trading.py` to carry out practice/live trading
+2) Clone this git repository into a suitable location on your machine using the following command in your terminal: ```git clone https://github.com/mhallsmoore/qsforex.git```. Alternative you can download the zip file of the current master branch at https://github.com/mhallsmoore/qsforex/archive/master.zip.
 
-If you have any questions about the installation then please feel free to email me at mike AT quantstart DOT com. You might also wish to have a look at the Forex Trading Diary series on QuantStart in order to gain some intuition about how the system is built.
+3) Create a set of environment variables for all of the settings found in the ```settings.py``` file in the application root directory. Alternatively, you can "hard code" your specific settings by overwriting the ```os.environ.get(...)``` calls for each setting:
+
+```
+# The data directory used to store your backtesting CSV files
+CSV_DATA_DIR = "/path/to/your/csv/data/dir"
+
+# The directory where the backtest.csv and equity.csv files 
+# will be stored after a backtest is carried out
+OUTPUT_RESULTS_DIR = "/path/to/your/output/results/dir"
+
+# Change DOMAIN to "real" if you wish to carry out live trading
+DOMAIN = "practice"
+
+# Your OANDA API Access Token (found in your Account Details on their website)
+ACCESS_TOKEN = "1234123412341234"
+
+# Your OANDA Account ID (found in your Account Details on their website)
+ACCOUNT_ID = "1234123412341234"
+
+# Your base currency (e.g. "GBP", "USD", "EUR" etc.)
+BASE_CURRENCY = "GBP"
+
+# Your account equity in the base currency (for backtesting)
+EQUITY = Decimal("100000.00")
+```
+
+4) Create a virtual environment ("virtualenv") for the QSForex code and utilise pip to install the requirements. For instance in a Unix-based system (Mac or Linux) you might create such a directory as follows by entering the following commands in the terminal:
+
+```
+mkdir -p ~/venv/qsforex
+cd ~/venv/qsforex
+virtualenv .
+```
+
+This will create a new virtual environment to install the packages into. Assuming you downloaded the QSForex git repository into an example directory such as ```~/projects/qsforex/``` (change this directory below to wherever you installed QSForex), then in order to install the packages you will need to run the following commands:
+
+```
+source ~/venv/qsforex/bin/activate
+pip install -r ~/projects/qsforex/requirements.txt
+```
+
+This will take some time as NumPy, SciPy, Pandas, Scikit-Learn and Matplotlib must be compiled. There are many packages required for this to work, so please take a look at these two articles for more information:
+
+* https://www.quantstart.com/articles/Quick-Start-Python-Quantitative-Research-Environment-on-Ubuntu-14-04
+* https://www.quantstart.com/articles/Easy-Multi-Platform-Installation-of-a-Scientific-Python-Stack-Using-Anaconda
+
+You will also need to create a symbolic link from your ```site-packages``` directory to your QSForex installation directory in order to be able to call ```import qsforex``` within the code. To do this you will need a command similar to the following:
+
+```
+ln -s ~/projects/qsforex/ ~/venv/qsforex/lib/python2.7/site-packages/qsforex
+```
+
+Make sure to change ```~/projects/qsforex``` to your installation directory and ```~/venv/qsforex/lib/python2.7/site-packages/``` to your virtualenv site packages directory.
+
+You will now be able to run the subsequent commands correctly.
+
+## Practice/Live Trading
+
+5) At this stage, if you simply wish to carry out practice or live trading then you can run ```python trading/trading.py```, which will use the default ```TestStrategy``` trading strategy. This simply buys or sells a currency pair every 5th tick. It is purely for testing - do not use it in a live trading environment!
+
+If you wish to create a more useful strategy, then simply create a new class with a descriptive name, e.g. ```MeanReversionMultiPairStrategy``` and ensure it has a ```calculate_signals``` method. You will need to pass this class the ```pairs``` list as well as the ```events``` queue, as in ```trading/trading.py```.
+
+Please look at ```strategy/strategy.py``` for details.
+
+## Backtesting
+
+6) In order to carry out any backtesting it is necessary to generate simulated forex data or download historic tick data. If you wish to simply try the software out, the quickest way to generate an example backtest is to generate some simulated data. The current data format used by QSForex is the same as that provided by the DukasCopy Historical Data Feed at https://www.dukascopy.com/swiss/english/marketwatch/historical/.
+
+To generate some historical data, make sure that the ```CSV_DATA_DIR``` setting in ```settings.py``` is to set to a directory where you want the historical data to live. You then need to run ```generate_simulated_pair.py```, which is under the ```scripts/``` directory. It expects a single command line argument, which in this case is the currency pair in ```BBBQQQ``` format. For example:
+
+```
+cd ~/projects/qsforex
+python scripts/generate_simulated_pair.py GBPUSD
+```
+
+At this stage the script is hardcoded to create a single month's data for January 2014. That is, you will see individual files, of the format ```BBBQQQ_YYYYMMDD.csv``` (e.g. ```GBPUSD_20140112.csv```) appear in your ```CSV_DATA_DIR``` for all business days in that month. If you wish to change the month/year of the data output, simply modify the file and re-run.
+
+7) Now that the historical data has been generated it is possible to carry out a backtest. The backtest file itself is stored in ```backtest/backtest.py```, but this only contains the ```Backtest``` class. To actually execute a backtest you need to instantiate this class and provide it with the necessary modules. 
+
+The best way to see how this is done is to look at the example Moving Average Crossover implementation in the ```examples/mac.py``` file and use this as a template. This makes use of the ```MovingAverageCrossStrategy``` which is found in ```strategy/strategy.py```. This defaults to trading both GBP/USD and EUR/USD to demonstrate multiple currency pair usage. It uses data found in ```CSV_DATA_DIR```.
+
+To execute the example backtest, simply run the following:
+
+```
+python examples/mac.py
+```
+
+**This will take some time.** On my Ubuntu desktop system at home, with the historical data generated via ```generate_simulated_pair.py```, it takes around 5-10 mins to run. A large part of this calculation occurs at the end of the actual backtest, when the drawdown is being calculated, so please remember that the code has not hung up! Please leave it until completion.
+
+8) If you wish to view the performance of the backtest you can simply use ```output.py``` to view an equity curve, period returns (i.e. tick-to-tick returns) and a drawdown curve:
+
+```
+python backtest/output.py
+```
+
+And that's it! At this stage you are ready to begin creating your own backtests by modifying or appending strategies in ```strategy/strategy.py``` and using real data downloaded from DukasCopy (https://www.dukascopy.com/swiss/english/marketwatch/historical/).
+
+If you have any questions about the installation then please feel free to email me at mike@quantstart.com.
+
+If you have any bugs or other issues that you think may be due to the codebase specifically, feel free to open a Github issue here: https://github.com/mhallsmoore/qsforex/issues
 
 # License Terms
 

backtest/backtest.py

@@ -1,82 +1,83 @@
 from __future__ import print_function
 
-import copy
 try:
     import Queue as queue
 except ImportError:
     import queue
-import threading
 import time
-from decimal import Decimal, getcontext
 
-from qsforex.execution.execution import SimulatedExecution
-from qsforex.portfolio.portfolio import Portfolio
 from qsforex import settings
-from qsforex.strategy.strategy import TestStrategy, MovingAverageCrossStrategy
-from qsforex.data.price import HistoricCSVPriceHandler
 
 
-def backtest(
-        events, ticker, strategy, portfolio, 
-        execution, heartbeat, max_iters=200000
+class Backtest(object):
+    """
+    Enscapsulates the settings and components for carrying out
+    an event-driven backtest on the foreign exchange markets.
+    """
+    def __init__(
+        self, pairs, data_handler, strategy, 
+        strategy_params, portfolio, execution, 
+        equity=100000.0, heartbeat=0.0, 
+        max_iters=10000000000
     ):
-    """
-    Carries out an infinite while loop that polls the 
-    events queue and directs each event to either the
-    strategy component of the execution handler. The
-    loop will then pause for "heartbeat" seconds and
-    continue unti the maximum number of iterations is
-    exceeded.
-    """
-    iters = 0
-    while True and iters < max_iters:
-        ticker.stream_next_tick()
-        try:
-            event = events.get(False)
-        except queue.Empty:
-            pass
-        else:
-            if event is not None:
-                if event.type == 'TICK':
-                    strategy.calculate_signals(event)
-                    portfolio.update_portfolio(event)
-                elif event.type == 'SIGNAL':
-                    portfolio.execute_signal(event)
-                elif event.type == 'ORDER':
-                    execution.execute_order(event)
-        time.sleep(heartbeat)
-        iters += 1
-    portfolio.output_results()
+        """
+        Initialises the backtest.
+        """
+        self.pairs = pairs
+        self.events = queue.Queue()
+        self.csv_dir = settings.CSV_DATA_DIR
+        self.ticker = data_handler(self.pairs, self.events, self.csv_dir)
+        self.strategy_params = strategy_params
+        self.strategy = strategy(
+            self.pairs, self.events, **self.strategy_params
+        )
+        self.equity = equity
+        self.heartbeat = heartbeat
+        self.max_iters = max_iters
+        self.portfolio = portfolio(
+            self.ticker, self.events, equity=self.equity, backtest=True
+        )
+        self.execution = execution()
 
+    def _run_backtest(self):
+        """
+        Carries out an infinite while loop that polls the 
+        events queue and directs each event to either the
+        strategy component of the execution handler. The
+        loop will then pause for "heartbeat" seconds and
+        continue unti the maximum number of iterations is
+        exceeded.
+        """
+        print("Running Backtest...")
+        iters = 0
+        while iters < self.max_iters and self.ticker.continue_backtest:
+            try:
+                event = self.events.get(False)
+            except queue.Empty:
+                self.ticker.stream_next_tick()
+            else:
+                if event is not None:
+                    if event.type == 'TICK':
+                        self.strategy.calculate_signals(event)
+                        self.portfolio.update_portfolio(event)
+                    elif event.type == 'SIGNAL':
+                        self.portfolio.execute_signal(event)
+                    elif event.type == 'ORDER':
+                        self.execution.execute_order(event)
+            time.sleep(self.heartbeat)
+            iters += 1
 
-if __name__ == "__main__":
-    heartbeat = 0.0
-    events = queue.Queue()
-    equity = settings.EQUITY
+    def _output_performance(self):
+        """
+        Outputs the strategy performance from the backtest.
+        """
+        print("Calculating Performance Metrics...")
+        self.portfolio.output_results()
 
-    # Load the historic CSV tick data files
-    pairs = ["GBPUSD"]
-    csv_dir = settings.CSV_DATA_DIR
-    if csv_dir is None:
-        print("No historic data directory provided - backtest terminating.")
-        sys.exit()
-
-    # Create the historic tick data streaming class
-    ticker = HistoricCSVPriceHandler(pairs, events, csv_dir)
-
-    # Create the strategy/signal generator, passing the 
-    # instrument and the events queue
-    strategy = MovingAverageCrossStrategy(
-        pairs, events, 500, 2000
-    )
-
-    # Create the portfolio object to track trades
-    portfolio = Portfolio(
-        ticker, events, equity=equity, backtest=True
-    )
-
-    # Create the simulated execution handler
-    execution = SimulatedExecution()
-    
-    # Carry out the backtest loop
-    backtest(events, ticker, strategy, portfolio, execution, heartbeat)
+    def simulate_trading(self):
+        """
+        Simulates the backtest and outputs portfolio performance.
+        """
+        self._run_backtest()
+        self._output_performance()
+        print("Backtest complete.")

data/price.py

@@ -1,12 +1,16 @@
+from __future__ import print_function
+
 import datetime
 from decimal import Decimal, getcontext, ROUND_HALF_DOWN
 import os
 import os.path
+import re
 import time
 
 import numpy as np
 import pandas as pd
 
+from qsforex import settings
 from qsforex.event.event import TickEvent
 
 
@@ -96,9 +100,32 @@ class HistoricCSVPriceHandler(PriceHandler):
         self.csv_dir = csv_dir
         self.prices = self._set_up_prices_dict()
         self.pair_frames = {}
-        self._open_convert_csv_files()
+        self.file_dates = self._list_all_file_dates()
+        self.continue_backtest = True
+        self.cur_date_idx = 0
+        self.cur_date_pairs = self._open_convert_csv_files_for_day(
+            self.file_dates[self.cur_date_idx]
+        )
 
-    def _open_convert_csv_files(self):
+    def _list_all_csv_files(self):
+        files = os.listdir(settings.CSV_DATA_DIR)
+        pattern = re.compile("[A-Z]{6}_\d{8}.csv")
+        matching_files = [f for f in files if pattern.search(f)]
+        matching_files.sort()
+        return matching_files
+
+    def _list_all_file_dates(self):
+        """
+        Removes the pair, underscore and '.csv' from the
+        dates and eliminates duplicates. Returns a list
+        of date strings of the form "YYYYMMDD". 
+        """
+        csv_files = self._list_all_csv_files()
+        de_dup_csv = list(set([d[7:-4] for d in csv_files]))
+        de_dup_csv.sort()
+        return de_dup_csv
+
+    def _open_convert_csv_files_for_day(self, date_str):
         """
         Opens the CSV files from the data directory, converting
         them into pandas DataFrames within a pairs dictionary.
@@ -109,13 +136,24 @@ class HistoricCSVPriceHandler(PriceHandler):
         in a chronological fashion.
         """
         for p in self.pairs:
-            pair_path = os.path.join(self.csv_dir, '%s.csv' % p)
+            pair_path = os.path.join(self.csv_dir, '%s_%s.csv' % (p, date_str))
             self.pair_frames[p] = pd.io.parsers.read_csv(
-                pair_path, header=True, index_col=0, parse_dates=True,
+                pair_path, header=True, index_col=0, 
+                parse_dates=True, dayfirst=True,
                 names=("Time", "Ask", "Bid", "AskVolume", "BidVolume")
             )
             self.pair_frames[p]["Pair"] = p
-        self.all_pairs = pd.concat(self.pair_frames.values()).sort().iterrows()
+        return pd.concat(self.pair_frames.values()).sort().iterrows()
+
+    def _update_csv_for_day(self):
+        try:
+            dt = self.file_dates[self.cur_date_idx+1]
+        except IndexError:  # End of file dates
+            return False
+        else:
+            self.cur_date_pairs = self._open_convert_csv_files_for_day(dt)
+            self.cur_date_idx += 1
+            return True
 
     def stream_next_tick(self):
         """
@@ -130,30 +168,35 @@ class HistoricCSVPriceHandler(PriceHandler):
         well as updating the current bid/ask and inverse bid/ask.
         """
         try:
-            index, row = next(self.all_pairs)
+            index, row = next(self.cur_date_pairs)
         except StopIteration:
-            return
-        else:
-            getcontext().rounding = ROUND_HALF_DOWN
-            pair = row["Pair"]
-            bid = Decimal(str(row["Bid"])).quantize(
-                Decimal("0.00001")
-            )
-            ask = Decimal(str(row["Ask"])).quantize(
-                Decimal("0.00001")
-            )
+            # End of the current days data
+            if self._update_csv_for_day():
+                index, row = next(self.cur_date_pairs)
+            else: # End of the data
+                self.continue_backtest = False
+                return
+        
+        getcontext().rounding = ROUND_HALF_DOWN
+        pair = row["Pair"]
+        bid = Decimal(str(row["Bid"])).quantize(
+            Decimal("0.00001")
+        )
+        ask = Decimal(str(row["Ask"])).quantize(
+            Decimal("0.00001")
+        )
 
-            # Create decimalised prices for traded pair
-            self.prices[pair]["bid"] = bid
-            self.prices[pair]["ask"] = ask
-            self.prices[pair]["time"] = index
+        # Create decimalised prices for traded pair
+        self.prices[pair]["bid"] = bid
+        self.prices[pair]["ask"] = ask
+        self.prices[pair]["time"] = index
 
-            # Create decimalised prices for inverted pair
-            inv_pair, inv_bid, inv_ask = self.invert_prices(pair, bid, ask)
-            self.prices[inv_pair]["bid"] = inv_bid
-            self.prices[inv_pair]["ask"] = inv_ask
-            self.prices[inv_pair]["time"] = index
+        # Create decimalised prices for inverted pair
+        inv_pair, inv_bid, inv_ask = self.invert_prices(pair, bid, ask)
+        self.prices[inv_pair]["bid"] = inv_bid
+        self.prices[inv_pair]["ask"] = inv_ask
+        self.prices[inv_pair]["time"] = index
 
-            # Create the tick event for the queue
-            tev = TickEvent(pair, index, bid, ask)
-            self.events_queue.put(tev)
+        # Create the tick event for the queue
+        tev = TickEvent(pair, index, bid, ask)
+        self.events_queue.put(tev)

data/streaming.py

@@ -1,9 +1,11 @@
 from __future__ import print_function
 
 from decimal import Decimal, getcontext, ROUND_HALF_DOWN
-import requests
+import logging
 import json
 
+import requests
+
 from qsforex.event.event import TickEvent
 from qsforex.data.price import PriceHandler
 
@@ -19,6 +21,7 @@ class StreamingForexPrices(PriceHandler):
         self.events_queue = events_queue
         self.pairs = pairs
         self.prices = self._set_up_prices_dict()
+        self.logger = logging.getLogger(__name__)
 
     def invert_prices(self, pair, bid, ask):
         """
@@ -38,12 +41,13 @@ class StreamingForexPrices(PriceHandler):
 
     def connect_to_stream(self):
         pairs_oanda = ["%s_%s" % (p[:3], p[3:]) for p in self.pairs]
+        pair_list = ",".join(pairs_oanda)
         try:
             requests.packages.urllib3.disable_warnings()
             s = requests.Session()
             url = "https://" + self.domain + "/v1/prices"
             headers = {'Authorization' : 'Bearer ' + self.access_token}
-            params = {'instruments' : pairs_oanda, 'accountId' : self.account_id}
+            params = {'instruments' : pair_list, 'accountId' : self.account_id}
             req = requests.Request('GET', url, headers=headers, params=params)
             pre = req.prepare()
             resp = s.send(pre, stream=True, verify=False)
@@ -62,10 +66,12 @@ class StreamingForexPrices(PriceHandler):
                     dline = line.decode('utf-8')
                     msg = json.loads(dline)
                 except Exception as e:
-                    print("Caught exception when converting message into json\n" + str(e))
+                    self.logger.error(
+                        "Caught exception when converting message into json: %s" % str(e)
+                    )
                     return
                 if "instrument" in msg or "tick" in msg:
-                    print(msg)
+                    self.logger.debug(msg)
                     getcontext().rounding = ROUND_HALF_DOWN 
                     instrument = msg["tick"]["instrument"].replace("_", "")
                     time = msg["tick"]["time"]

event/event.py

@@ -10,15 +10,33 @@ class TickEvent(Event):
         self.bid = bid
         self.ask = ask
 
+    def __str__(self):
+        return "Type: %s, Instrument: %s, Time: %s, Bid: %s, Ask: %s" % (
+            str(self.type), str(self.instrument), 
+            str(self.time), str(self.bid), str(self.ask)
+        )
+
+    def __repr__(self):
+        return str(self)
+
 
 class SignalEvent(Event):
     def __init__(self, instrument, order_type, side, time):
         self.type = 'SIGNAL'
         self.instrument = instrument
         self.order_type = order_type
-        self.side = side        
+        self.side = side
         self.time = time  # Time of the last tick that generated the signal
 
+    def __str__(self):
+        return "Type: %s, Instrument: %s, Order Type: %s, Side: %s" % (
+            str(self.type), str(self.instrument), 
+            str(self.order_type), str(self.side)
+        )
+
+    def __repr__(self):
+        return str(self)
+
 
 class OrderEvent(Event):
     def __init__(self, instrument, units, order_type, side):
@@ -26,4 +44,13 @@ class OrderEvent(Event):
         self.instrument = instrument
         self.units = units
         self.order_type = order_type
-        self.side = side        
+        self.side = side
+
+    def __str__(self):
+        return "Type: %s, Instrument: %s, Units: %s, Order Type: %s, Side: %s" % (
+            str(self.type), str(self.instrument), str(self.units),
+            str(self.order_type), str(self.side)
+        )
+
+    def __repr__(self):
+        return str(self)

examples/mac.py

@@ -0,0 +1,29 @@
+from __future__ import print_function
+
+from qsforex.backtest.backtest import Backtest
+from qsforex.execution.execution import SimulatedExecution
+from qsforex.portfolio.portfolio import Portfolio
+from qsforex import settings
+from qsforex.strategy.strategy import MovingAverageCrossStrategy
+from qsforex.data.price import HistoricCSVPriceHandler
+
+
+if __name__ == "__main__":
+    # Trade on GBP/USD and EUR/USD
+    pairs = ["GBPUSD", "EURUSD"]
+    
+    # Create the strategy parameters for the
+    # MovingAverageCrossStrategy
+    strategy_params = {
+        "short_window": 500, 
+        "long_window": 2000
+    }
+   
+    # Create and execute the backtest
+    backtest = Backtest(
+        pairs, HistoricCSVPriceHandler, 
+        MovingAverageCrossStrategy, strategy_params, 
+        Portfolio, SimulatedExecution, 
+        equity=settings.EQUITY
+    )
+    backtest.simulate_trading()

execution/execution.py

@@ -5,6 +5,7 @@ try:
     import httplib
 except ImportError:
     import http.client as httplib
+import logging
 try:
     from urllib import urlencode
 except ImportError:
@@ -47,6 +48,7 @@ class OANDAExecutionHandler(ExecutionHandler):
         self.access_token = access_token
         self.account_id = account_id
         self.conn = self.obtain_connection()
+        self.logger = logging.getLogger(__name__)
 
     def obtain_connection(self):
         return httplib.HTTPSConnection(self.domain)
@@ -68,6 +70,6 @@ class OANDAExecutionHandler(ExecutionHandler):
             "/v1/accounts/%s/orders" % str(self.account_id), 
             params, headers
         )
-        response = self.conn.getresponse().read()
-        print(response)
+        response = self.conn.getresponse().read().decode("utf-8").replace("\n","").replace("\t","")
+        self.logger.debug(response)
         

logging.conf

@@ -0,0 +1,28 @@
+[loggers]
+keys=root,qsforex.trading.trading
+
+[handlers]
+keys=consoleHandler
+
+[formatters]
+keys=simpleFormatter
+
+[logger_root]
+level=DEBUG
+handlers=consoleHandler
+
+[logger_qsforex.trading.trading]
+level=DEBUG
+handlers=consoleHandler
+qualname=qsforex.trading.trading
+propagate=0
+
+[handler_consoleHandler]
+class=StreamHandler
+level=DEBUG
+formatter=simpleFormatter
+args=(sys.stdout,)
+
+[formatter_simpleFormatter]
+format=%(asctime)s - %(name)s - %(levelname)s - %(message)s
+datefmt=

performance/performance.py

@@ -26,7 +26,7 @@ def create_drawdowns(pnl):
 
     # Loop over the index range
     for t in range(1, len(idx)):
-        hwm.append(max(hwm[t-1], pnl[t]))
-        drawdown[t]= (hwm[t]-pnl[t])
-        duration[t]= (0 if drawdown[t] == 0 else duration[t-1]+1)
+        hwm.append(max(hwm[t-1], pnl.ix[t]))
+        drawdown.ix[t]= (hwm[t]-pnl.ix[t])
+        duration.ix[t]= (0 if drawdown.ix[t] == 0 else duration.ix[t-1]+1)
     return drawdown, drawdown.max(), duration.max()

portfolio/portfolio.py

@@ -2,6 +2,7 @@ from __future__ import print_function
 
 from copy import deepcopy
 from decimal import Decimal, getcontext, ROUND_HALF_DOWN
+import logging
 import os
 
 import pandas as pd
@@ -14,9 +15,9 @@ from qsforex.settings import OUTPUT_RESULTS_DIR
 
 class Portfolio(object):
     def __init__(
-        self, ticker, events, home_currency="GBP", leverage=20, 
-        equity=Decimal("100000.00"), risk_per_trade=Decimal("0.02"),
-        backtest=True
+        self, ticker, events, home_currency="GBP", 
+        leverage=20, equity=Decimal("100000.00"), 
+        risk_per_trade=Decimal("0.02"), backtest=True
     ):
         self.ticker = ticker
         self.events = events
@@ -28,7 +29,9 @@ class Portfolio(object):
         self.backtest = backtest
         self.trade_units = self.calc_risk_position_size()
         self.positions = {}
-        self.backtest_file = self.create_equity_file()
+        if self.backtest:
+            self.backtest_file = self.create_equity_file()
+        self.logger = logging.getLogger(__name__)
 
     def calc_risk_position_size(self):
         return self.equity * self.risk_per_trade
@@ -114,62 +117,77 @@ class Portfolio(object):
         if currency_pair in self.positions:
             ps = self.positions[currency_pair]
             ps.update_position_price()
-        out_line = "%s,%s" % (tick_event.time, self.balance)
-        for pair in self.ticker.pairs:
-            if pair in self.positions:
-                out_line += ",%s" % self.positions[currency_pair].profit_base
-            else:
-                out_line += ",0.00"
-        out_line += "\n"
         if self.backtest:
+            out_line = "%s,%s" % (tick_event.time, self.balance)
+            for pair in self.ticker.pairs:
+                if pair in self.positions:
+                    out_line += ",%s" % self.positions[pair].profit_base
+                else:
+                    out_line += ",0.00"
+            out_line += "\n"
             print(out_line[:-2])
-        self.backtest_file.write(out_line)
+            self.backtest_file.write(out_line)
 
-    def execute_signal(self, signal_event):       
-        side = signal_event.side
-        currency_pair = signal_event.instrument
-        units = int(self.trade_units)
-        time = signal_event.time
-        
-        # If there is no position, create one
-        if currency_pair not in self.positions:
-            if side == "buy":
-                position_type = "long"
+    def execute_signal(self, signal_event):
+        # Check that the prices ticker contains all necessary
+        # currency pairs prior to executing an order
+        execute = True
+        tp = self.ticker.prices
+        for pair in tp:
+            if tp[pair]["ask"] is None or tp[pair]["bid"] is None:
+                execute = False
+
+        # All necessary pricing data is available,
+        # we can execute
+        if execute:
+            side = signal_event.side
+            currency_pair = signal_event.instrument
+            units = int(self.trade_units)
+            time = signal_event.time
+            
+            # If there is no position, create one
+            if currency_pair not in self.positions:
+                if side == "buy":
+                    position_type = "long"
+                else:
+                    position_type = "short"
+                self.add_new_position(
+                    position_type, currency_pair, 
+                    units, self.ticker
+                )
+
+            # If a position exists add or remove units
             else:
-                position_type = "short"
-            self.add_new_position(
-                position_type, currency_pair, 
-                units, self.ticker
-            )
+                ps = self.positions[currency_pair]
 
-        # If a position exists add or remove units
+                if side == "buy" and ps.position_type == "long":
+                    add_position_units(currency_pair, units)
+
+                elif side == "sell" and ps.position_type == "long":
+                    if units == ps.units:
+                        self.close_position(currency_pair)
+                    # TODO: Allow units to be added/removed
+                    elif units < ps.units:
+                        return
+                    elif units > ps.units:
+                        return
+
+                elif side == "buy" and ps.position_type == "short":
+                    if units == ps.units:
+                        self.close_position(currency_pair)
+                    # TODO: Allow units to be added/removed
+                    elif units < ps.units:
+                        return
+                    elif units > ps.units:
+                        return
+                        
+                elif side == "sell" and ps.position_type == "short":
+                    add_position_units(currency_pair, units)
+
+            order = OrderEvent(currency_pair, units, "market", side)
+            self.events.put(order)
+
+            self.logger.info("Portfolio Balance: %s" % self.balance)
         else:
-            ps = self.positions[currency_pair]
-
-            if side == "buy" and ps.position_type == "long":
-                add_position_units(currency_pair, units)
-
-            elif side == "sell" and ps.position_type == "long":
-                if units == ps.units:
-                    self.close_position(currency_pair)
-                # TODO: Allow units to be added/removed
-                elif units < ps.units:
-                    return
-                elif units > ps.units:
-                    return
-
-            elif side == "buy" and ps.position_type == "short":
-                if units == ps.units:
-                    self.close_position(currency_pair)
-                # TODO: Allow units to be added/removed
-                elif units < ps.units:
-                    return
-                elif units > ps.units:
-                    return
-                    
-            elif side == "sell" and ps.position_type == "short":
-                add_position_units(currency_pair, units)
-
-        order = OrderEvent(currency_pair, units, "market", side)
-        self.events.put(order)
+            self.logger.info("Unable to execute order as price data was insufficient.")
         

portfolio/portfolio_test.py

@@ -212,7 +212,7 @@ class TestPortfolio(unittest.TestCase):
         )
         self.assertTrue(rpu)
         self.assertEqual(ps.units, Decimal("7000"))
-        self.assertEqual(self.port.balance, Decimal("99988.84"))
+        self.assertEqual(self.port.balance, Decimal("99988.83"))
 
     def test_close_position_long(self):
         position_type = "long"
@@ -265,7 +265,7 @@ class TestPortfolio(unittest.TestCase):
         cp = self.port.close_position(currency_pair)
         self.assertTrue(cp)
         self.assertRaises(ps)  # Key doesn't exist
-        self.assertEqual(self.port.balance, Decimal("100026.64"))
+        self.assertEqual(self.port.balance, Decimal("100026.63"))
 
     def test_close_position_short(self):
         position_type = "short"
@@ -312,13 +312,13 @@ class TestPortfolio(unittest.TestCase):
         )
         self.assertTrue(rpu)
         self.assertEqual(ps.units, Decimal("7000"))
-        self.assertEqual(self.port.balance, Decimal("99988.84"))
+        self.assertEqual(self.port.balance, Decimal("99988.83"))
 
         # Close the position
         cp = self.port.close_position(currency_pair)
         self.assertTrue(cp)
         self.assertRaises(ps)  # Key doesn't exist
-        self.assertEqual(self.port.balance, Decimal("99962.80"))
+        self.assertEqual(self.port.balance, Decimal("99962.77"))
 
 
 if __name__ == "__main__":

portfolio/position.py

@@ -24,7 +24,7 @@ class Position(object):
         ticker_cur = self.ticker.prices[self.currency_pair]
         if self.position_type == "long":
             self.avg_price = Decimal(str(ticker_cur["ask"]))
-            self.cur_price = Decimal(str(ticker_cur["bid"]))
+            self.cur_price = Decimal(str(ticker_cur["bid"]))    
         else:
             self.avg_price = Decimal(str(ticker_cur["bid"]))
             self.cur_price = Decimal(str(ticker_cur["ask"]))
@@ -83,11 +83,11 @@ class Position(object):
         ticker_cp = self.ticker.prices[self.currency_pair]
         ticker_qh = self.ticker.prices[self.quote_home_currency_pair]
         if self.position_type == "long":
-            remove_price = ticker_cp["ask"]
-            qh_close = ticker_qh["bid"]
-        else:
             remove_price = ticker_cp["bid"]
             qh_close = ticker_qh["ask"]
+        else:
+            remove_price = ticker_cp["ask"]
+            qh_close = ticker_qh["bid"]
         self.units -= dec_units
         self.update_position_price()
         # Calculate PnL
@@ -99,11 +99,9 @@ class Position(object):
         ticker_cp = self.ticker.prices[self.currency_pair]
         ticker_qh = self.ticker.prices[self.quote_home_currency_pair]
         if self.position_type == "long":
-            remove_price = ticker_cp["ask"]
-            qh_close = ticker_qh["bid"]
-        else:
-            remove_price = ticker_cp["bid"]
             qh_close = ticker_qh["ask"]
+        else:
+            qh_close = ticker_qh["bid"]
         self.update_position_price()
         # Calculate PnL
         pnl = self.calculate_pips() * qh_close * self.units

requirements.txt

@@ -7,7 +7,7 @@ pandas==0.16.1
 pyparsing==2.0.3
 python-dateutil==2.4.2
 pytz==2014.10
-requests==2.7.0
+requests==2.20.0
 scikit-learn==0.16.1
 scipy==0.15.1
 seaborn==0.5.1

scripts/generate_simulated_pair.py

@@ -0,0 +1,80 @@
+from __future__ import print_function
+
+import calendar
+import copy
+import datetime
+import os, os.path
+import sys
+
+import numpy as np
+import pandas as pd
+
+from qsforex import settings
+
+
+def month_weekdays(year_int, month_int):
+    """
+    Produces a list of datetime.date objects representing the
+    weekdays in a particular month, given a year.
+    """
+    cal = calendar.Calendar()
+    return [
+        d for d in cal.itermonthdates(year_int, month_int) 
+        if d.weekday() < 5 and d.year == year_int
+    ]
+
+
+if __name__ == "__main__":
+    try:
+        pair = sys.argv[1]
+    except IndexError:
+        print("You need to enter a currency pair, e.g. GBPUSD, as a command line parameter.")
+    else:
+        np.random.seed(42)  # Fix the randomness
+
+        S0 = 1.5000
+        spread = 0.002
+        mu_dt = 1400  # Milliseconds
+        sigma_dt = 100  # Millseconds
+        ask = copy.deepcopy(S0) + spread / 2.0
+        bid = copy.deepcopy(S0) - spread / 2.0
+        days = month_weekdays(2014, 1)  # January 2014
+        current_time = datetime.datetime(
+            days[0].year, days[0].month, days[0].day, 0, 0, 0,
+        )
+
+        # Loop over every day in the month and create a CSV file
+        # for each day, e.g. "GBPUSD_20150101.csv"
+        for d in days:
+            print(d.day)
+            current_time = current_time.replace(day=d.day)
+            outfile = open(
+                os.path.join(
+                    settings.CSV_DATA_DIR, 
+                    "%s_%s.csv" % (
+                        pair, d.strftime("%Y%m%d")
+                    )
+                ), 
+            "w")
+            outfile.write("Time,Ask,Bid,AskVolume,BidVolume\n")     
+            
+            # Create the random walk for the bid/ask prices
+            # with fixed spread between them
+            while True:
+                dt = abs(np.random.normal(mu_dt, sigma_dt))
+                current_time += datetime.timedelta(0, 0, 0, dt)
+                if current_time.day != d.day:
+                    outfile.close()
+                    break
+                else:
+                    W = np.random.standard_normal() * dt / 1000.0 / 86400.0
+                    ask += W
+                    bid += W
+                    ask_volume = 1.0 + np.random.uniform(0.0, 2.0)
+                    bid_volume = 1.0 + np.random.uniform(0.0, 2.0)
+                    line = "%s,%s,%s,%s,%s\n" % (
+                        current_time.strftime("%d.%m.%Y %H:%M:%S.%f")[:-3], 
+                        "%0.5f" % ask, "%0.5f" % bid,
+                        "%0.2f00" % ask_volume, "%0.2f00" % bid_volume
+                    )
+                    outfile.write(line)

scripts/test_performance.py

@@ -0,0 +1,35 @@
+"""
+This is a small helper script written to help debug issues
+with performance calculation, that avoids having to re-run
+the full backtest.
+
+In this case it simply works off the "backtest.csv" file that
+is produced from a backtest.py run.
+"""
+
+
+import os
+
+import pandas as pd
+
+from qsforex.performance.performance import create_drawdowns
+from qsforex.settings import OUTPUT_RESULTS_DIR
+
+
+if __name__ == "__main__":
+    in_filename = "backtest.csv"
+    out_filename = "equity.csv" 
+    in_file = os.path.join(OUTPUT_RESULTS_DIR, in_filename)
+    out_file = os.path.join(OUTPUT_RESULTS_DIR, out_filename)
+
+    # Create equity curve dataframe
+    df = pd.read_csv(in_file, index_col=0)
+    df.dropna(inplace=True)
+    df["Total"] = df.sum(axis=1)
+    df["Returns"] = df["Total"].pct_change()
+    df["Equity"] = (1.0+df["Returns"]).cumprod()
+
+    # Create drawdown statistics
+    drawdown, max_dd, dd_duration = create_drawdowns(df["Equity"])
+    df["Drawdown"] = drawdown
+    df.to_csv(out_file, index=True)

strategy/strategy.py

@@ -1,3 +1,5 @@
+import copy
+
 from qsforex.event.event import SignalEvent
 
 
@@ -18,7 +20,7 @@ class TestStrategy(object):
         self.invested = False
 
     def calculate_signals(self, event):
-        if event.type == 'TICK':
+        if event.type == 'TICK' and event.instrument == self.pairs[0]:
             if self.ticks % 5 == 0:
                 if self.invested == False:
                     signal = SignalEvent(self.pairs[0], "market", "buy", event.time)
@@ -52,39 +54,49 @@ class MovingAverageCrossStrategy(object):
         short_window=500, long_window=2000
     ):
         self.pairs = pairs
-        self.events = events
-        self.ticks = 0
-        self.invested = False
-        
+        self.pairs_dict = self.create_pairs_dict()
+        self.events = events      
         self.short_window = short_window
         self.long_window = long_window
-        self.short_sma = None
-        self.long_sma = None
+
+    def create_pairs_dict(self):
+        attr_dict = {
+            "ticks": 0,
+            "invested": False,
+            "short_sma": None,
+            "long_sma": None
+        }
+        pairs_dict = {}
+        for p in self.pairs:
+            pairs_dict[p] = copy.deepcopy(attr_dict)
+        return pairs_dict
 
     def calc_rolling_sma(self, sma_m_1, window, price):
         return ((sma_m_1 * (window - 1)) + price) / window
 
     def calculate_signals(self, event):
         if event.type == 'TICK':
+            pair = event.instrument
             price = event.bid
-            if self.ticks == 0:
-                self.short_sma = price
-                self.long_sma = price
+            pd = self.pairs_dict[pair]
+            if pd["ticks"] == 0:
+                pd["short_sma"] = price
+                pd["long_sma"] = price
             else:
-                self.short_sma = self.calc_rolling_sma(
-                    self.short_sma, self.short_window, price
+                pd["short_sma"] = self.calc_rolling_sma(
+                    pd["short_sma"], self.short_window, price
                 )
-                self.long_sma = self.calc_rolling_sma(
-                    self.long_sma, self.long_window, price
+                pd["long_sma"] = self.calc_rolling_sma(
+                    pd["long_sma"], self.long_window, price
                 )
             # Only start the strategy when we have created an accurate short window
-            if self.ticks > self.short_window:
-                if self.short_sma > self.long_sma and not self.invested:
-                    signal = SignalEvent(self.pairs[0], "market", "buy", event.time)
+            if pd["ticks"] > self.short_window:
+                if pd["short_sma"] > pd["long_sma"] and not pd["invested"]:
+                    signal = SignalEvent(pair, "market", "buy", event.time)
                     self.events.put(signal)
-                    self.invested = True
-                if self.short_sma < self.long_sma and self.invested:
-                    signal = SignalEvent(self.pairs[0], "market", "sell", event.time)
+                    pd["invested"] = True
+                if pd["short_sma"] < pd["long_sma"] and pd["invested"]:
+                    signal = SignalEvent(pair, "market", "sell", event.time)
                     self.events.put(signal)
-                    self.invested = False
-            self.ticks += 1
+                    pd["invested"] = False
+            pd["ticks"] += 1

trading/trading.py

@@ -1,5 +1,7 @@
 import copy
 from decimal import Decimal, getcontext
+import logging
+import logging.config
 try:
     import Queue as queue
 except ImportError:
@@ -30,25 +32,32 @@ def trade(events, strategy, portfolio, execution, heartbeat):
         else:
             if event is not None:
                 if event.type == 'TICK':
+                    logger.info("Received new tick event: %s", event)
                     strategy.calculate_signals(event)
                     portfolio.update_portfolio(event)
                 elif event.type == 'SIGNAL':
+                    logger.info("Received new signal event: %s", event)
                     portfolio.execute_signal(event)
                 elif event.type == 'ORDER':
+                    logger.info("Received new order event: %s", event)
                     execution.execute_order(event)
         time.sleep(heartbeat)
 
 
 if __name__ == "__main__":
+    # Set up logging
+    logging.config.fileConfig('../logging.conf')
+    logger = logging.getLogger('qsforex.trading.trading')
+
     # Set the number of decimal places to 2
     getcontext().prec = 2
 
-    heartbeat = 0.0  # Half a second between polling
+    heartbeat = 0.0  # Time in seconds between polling
     events = queue.Queue()
     equity = settings.EQUITY
 
-    # Trade "Cable"
-    pairs = ["GBPUSD"]
+    # Pairs to include in streaming data set
+    pairs = ["EURUSD", "GBPUSD"]
 
     # Create the OANDA market price streaming class
     # making sure to provide authentication commands
@@ -86,5 +95,7 @@ if __name__ == "__main__":
     price_thread = threading.Thread(target=prices.stream_to_queue, args=[])
     
     # Start both threads
+    logger.info("Starting trading thread")
     trade_thread.start()
+    logger.info("Starting price streaming thread")
     price_thread.start()