Book a Demo!
CoCalc Logo Icon
StoreFeaturesDocsShareSupportNewsAboutPoliciesSign UpSign In
AI4Finance-Foundation
GitHub Repository: AI4Finance-Foundation/FinRL
 Path: blob/master/examples/FinRL_Ensemble_StockTrading_ICAIF_2020.ipynb
726 views
Kernel: Python 3

Deep Reinforcement Learning for Stock Trading from Scratch: Multiple Stock Trading Using Ensemble Strategy

Tutorials to use OpenAI DRL to trade multiple stocks using ensemble strategy in one Jupyter Notebook | Presented at ICAIF 2020

Content

Part 1. Problem Definition

This problem is to design an automated trading solution for single stock trading. We model the stock trading process as a Markov Decision Process (MDP). We then formulate our trading goal as a maximization problem.

The algorithm is trained using Deep Reinforcement Learning (DRL) algorithms and the components of the reinforcement learning environment are:

  • Action: The action space describes the allowed actions that the agent interacts with the environment. Normally, a ∈ A includes three actions: a ∈ {−1, 0, 1}, where −1, 0, 1 represent selling, holding, and buying one stock. Also, an action can be carried upon multiple shares. We use an action space {−k, ..., −1, 0, 1, ..., k}, where k denotes the number of shares. For example, "Buy 10 shares of AAPL" or "Sell 10 shares of AAPL" are 10 or −10, respectively

  • Reward function: r(s, a, s′) is the incentive mechanism for an agent to learn a better action. The change of the portfolio value when action a is taken at state s and arriving at new state s', i.e., r(s, a, s′) = v′ − v, where v′ and v represent the portfolio values at state s′ and s, respectively

  • State: The state space describes the observations that the agent receives from the environment. Just as a human trader needs to analyze various information before executing a trade, so our trading agent observes many different features to better learn in an interactive environment.

  • Environment: Dow 30 consituents

The data of the single stock that we will be using for this case study is obtained from Yahoo Finance API. The data contains Open-High-Low-Close price and volume.

Part 2. Getting Started- Load Python Packages

2.1. Install all the packages through FinRL library

# ## install finrl library
!pip install wrds
!pip install swig
!pip install -q condacolab
import condacolab
condacolab.install()
!apt-get update -y -qq && apt-get install -y -qq cmake libopenmpi-dev python3-dev zlib1g-dev libgl1-mesa-glx swig
!pip install git+https://github.com/AI4Finance-Foundation/FinRL.git
Requirement already satisfied: wrds in /usr/local/lib/python3.10/site-packages (3.2.0) Requirement already satisfied: numpy<1.27,>=1.26 in /usr/local/lib/python3.10/site-packages (from wrds) (1.26.4) Requirement already satisfied: packaging<23.3 in /usr/local/lib/python3.10/site-packages (from wrds) (23.2) Requirement already satisfied: pandas<2.3,>=2.2 in /usr/local/lib/python3.10/site-packages (from wrds) (2.2.1) Requirement already satisfied: psycopg2-binary<2.10,>=2.9 in /usr/local/lib/python3.10/site-packages (from wrds) (2.9.9) Requirement already satisfied: scipy<1.13,>=1.12 in /usr/local/lib/python3.10/site-packages (from wrds) (1.12.0) Requirement already satisfied: sqlalchemy<2.1,>=2 in /usr/local/lib/python3.10/site-packages (from wrds) (2.0.29) Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/site-packages (from pandas<2.3,>=2.2->wrds) (2.9.0.post0) Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/site-packages (from pandas<2.3,>=2.2->wrds) (2024.1) Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.10/site-packages (from pandas<2.3,>=2.2->wrds) (2024.1) Requirement already satisfied: typing-extensions>=4.6.0 in /usr/local/lib/python3.10/site-packages (from sqlalchemy<2.1,>=2->wrds) (4.11.0) Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.10/site-packages (from sqlalchemy<2.1,>=2->wrds) (3.0.3) Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas<2.3,>=2.2->wrds) (1.16.0) WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv Requirement already satisfied: swig in /usr/local/lib/python3.10/site-packages (4.2.1) WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv ✨🍰✨ Everything looks OK! Collecting git+https://github.com/AI4Finance-Foundation/FinRL.git Cloning https://github.com/AI4Finance-Foundation/FinRL.git to /tmp/pip-req-build-xp3z2_tj Running command git clone --filter=blob:none --quiet https://github.com/AI4Finance-Foundation/FinRL.git /tmp/pip-req-build-xp3z2_tj Resolved https://github.com/AI4Finance-Foundation/FinRL.git to commit 7b2f30302e787e9276f52823c87b7c2ade4203cf Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Collecting elegantrl@ git+https://github.com/AI4Finance-Foundation/ElegantRL.git#egg=elegantrl (from finrl==0.3.6) Cloning https://github.com/AI4Finance-Foundation/ElegantRL.git to /tmp/pip-install-fh0kgi8h/elegantrl_9258375a89e84ebb973a8518f78e4a92 Running command git clone --filter=blob:none --quiet https://github.com/AI4Finance-Foundation/ElegantRL.git /tmp/pip-install-fh0kgi8h/elegantrl_9258375a89e84ebb973a8518f78e4a92 Resolved https://github.com/AI4Finance-Foundation/ElegantRL.git to commit 87cf325af4c0608f1e74941091a3bc277dfaf739 Preparing metadata (setup.py) ... done Requirement already satisfied: alpaca-trade-api<4,>=3 in /usr/local/lib/python3.10/site-packages (from finrl==0.3.6) (3.2.0) Requirement already satisfied: ccxt<4,>=3 in /usr/local/lib/python3.10/site-packages (from finrl==0.3.6) (3.1.60) Requirement already satisfied: exchange-calendars<5,>=4 in /usr/local/lib/python3.10/site-packages (from finrl==0.3.6) (4.5.3) Requirement already satisfied: jqdatasdk<2,>=1 in /usr/local/lib/python3.10/site-packages (from finrl==0.3.6) (1.9.3) Requirement already satisfied: pyfolio<0.10,>=0.9 in /usr/local/lib/python3.10/site-packages (from finrl==0.3.6) (0.9.2) Requirement already satisfied: pyportfolioopt<2,>=1 in /usr/local/lib/python3.10/site-packages (from finrl==0.3.6) (1.5.5) Requirement already satisfied: ray<3,>=2 in /usr/local/lib/python3.10/site-packages (from ray[default,tune]<3,>=2->finrl==0.3.6) (2.10.0) Requirement already satisfied: scikit-learn<2,>=1 in /usr/local/lib/python3.10/site-packages (from finrl==0.3.6) (1.4.1.post1) Requirement already satisfied: stable-baselines3>=2.0.0a5 in /usr/local/lib/python3.10/site-packages (from stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (2.3.0) Requirement already satisfied: stockstats<0.6,>=0.5 in /usr/local/lib/python3.10/site-packages (from finrl==0.3.6) (0.5.4) Requirement already satisfied: wrds<4,>=3 in /usr/local/lib/python3.10/site-packages (from finrl==0.3.6) (3.2.0) Requirement already satisfied: yfinance<0.3,>=0.2 in /usr/local/lib/python3.10/site-packages (from finrl==0.3.6) (0.2.37) Requirement already satisfied: pandas>=0.18.1 in /usr/local/lib/python3.10/site-packages (from alpaca-trade-api<4,>=3->finrl==0.3.6) (2.2.1) Requirement already satisfied: numpy>=1.11.1 in /usr/local/lib/python3.10/site-packages (from alpaca-trade-api<4,>=3->finrl==0.3.6) (1.26.4) Requirement already satisfied: requests<3,>2 in /usr/local/lib/python3.10/site-packages (from alpaca-trade-api<4,>=3->finrl==0.3.6) (2.31.0) Requirement already satisfied: urllib3<2,>1.24 in /usr/local/lib/python3.10/site-packages (from alpaca-trade-api<4,>=3->finrl==0.3.6) (1.26.18) Requirement already satisfied: websocket-client<2,>=0.56.0 in /usr/local/lib/python3.10/site-packages (from alpaca-trade-api<4,>=3->finrl==0.3.6) (1.7.0) Requirement already satisfied: websockets<11,>=9.0 in /usr/local/lib/python3.10/site-packages (from alpaca-trade-api<4,>=3->finrl==0.3.6) (10.4) Requirement already satisfied: msgpack==1.0.3 in /usr/local/lib/python3.10/site-packages (from alpaca-trade-api<4,>=3->finrl==0.3.6) (1.0.3) Requirement already satisfied: aiohttp<4,>=3.8.3 in /usr/local/lib/python3.10/site-packages (from alpaca-trade-api<4,>=3->finrl==0.3.6) (3.9.3) Requirement already satisfied: PyYAML==6.0.1 in /usr/local/lib/python3.10/site-packages (from alpaca-trade-api<4,>=3->finrl==0.3.6) (6.0.1) Requirement already satisfied: deprecation==2.1.0 in /usr/local/lib/python3.10/site-packages (from alpaca-trade-api<4,>=3->finrl==0.3.6) (2.1.0) Requirement already satisfied: packaging in /usr/local/lib/python3.10/site-packages (from deprecation==2.1.0->alpaca-trade-api<4,>=3->finrl==0.3.6) (23.2) Requirement already satisfied: setuptools>=60.9.0 in /usr/local/lib/python3.10/site-packages (from ccxt<4,>=3->finrl==0.3.6) (68.2.2) Requirement already satisfied: certifi>=2018.1.18 in /usr/local/lib/python3.10/site-packages (from ccxt<4,>=3->finrl==0.3.6) (2023.11.17) Requirement already satisfied: cryptography>=2.6.1 in /usr/local/lib/python3.10/site-packages (from ccxt<4,>=3->finrl==0.3.6) (42.0.5) Requirement already satisfied: aiodns>=1.1.1 in /usr/local/lib/python3.10/site-packages (from ccxt<4,>=3->finrl==0.3.6) (3.2.0) Requirement already satisfied: yarl>=1.7.2 in /usr/local/lib/python3.10/site-packages (from ccxt<4,>=3->finrl==0.3.6) (1.9.4) Requirement already satisfied: pyluach in /usr/local/lib/python3.10/site-packages (from exchange-calendars<5,>=4->finrl==0.3.6) (2.2.0) Requirement already satisfied: toolz in /usr/local/lib/python3.10/site-packages (from exchange-calendars<5,>=4->finrl==0.3.6) (0.12.1) Requirement already satisfied: tzdata in /usr/local/lib/python3.10/site-packages (from exchange-calendars<5,>=4->finrl==0.3.6) (2024.1) Requirement already satisfied: korean-lunar-calendar in /usr/local/lib/python3.10/site-packages (from exchange-calendars<5,>=4->finrl==0.3.6) (0.3.1) Requirement already satisfied: six in /usr/local/lib/python3.10/site-packages (from jqdatasdk<2,>=1->finrl==0.3.6) (1.16.0) Requirement already satisfied: SQLAlchemy>=1.2.8 in /usr/local/lib/python3.10/site-packages (from jqdatasdk<2,>=1->finrl==0.3.6) (2.0.29) Requirement already satisfied: thriftpy2>=0.3.9 in /usr/local/lib/python3.10/site-packages (from jqdatasdk<2,>=1->finrl==0.3.6) (0.4.20) Requirement already satisfied: pymysql>=0.7.6 in /usr/local/lib/python3.10/site-packages (from jqdatasdk<2,>=1->finrl==0.3.6) (1.1.0) Requirement already satisfied: ipython>=3.2.3 in /usr/local/lib/python3.10/site-packages (from pyfolio<0.10,>=0.9->finrl==0.3.6) (8.23.0) Requirement already satisfied: matplotlib>=1.4.0 in /usr/local/lib/python3.10/site-packages (from pyfolio<0.10,>=0.9->finrl==0.3.6) (3.8.4) Requirement already satisfied: pytz>=2014.10 in /usr/local/lib/python3.10/site-packages (from pyfolio<0.10,>=0.9->finrl==0.3.6) (2024.1) Requirement already satisfied: scipy>=0.14.0 in /usr/local/lib/python3.10/site-packages (from pyfolio<0.10,>=0.9->finrl==0.3.6) (1.12.0) Requirement already satisfied: seaborn>=0.7.1 in /usr/local/lib/python3.10/site-packages (from pyfolio<0.10,>=0.9->finrl==0.3.6) (0.13.2) Requirement already satisfied: empyrical>=0.5.0 in /usr/local/lib/python3.10/site-packages (from pyfolio<0.10,>=0.9->finrl==0.3.6) (0.5.5) Requirement already satisfied: cvxpy<2.0.0,>=1.1.19 in /usr/local/lib/python3.10/site-packages (from pyportfolioopt<2,>=1->finrl==0.3.6) (1.4.2) Requirement already satisfied: click>=7.0 in /usr/local/lib/python3.10/site-packages (from ray<3,>=2->ray[default,tune]<3,>=2->finrl==0.3.6) (8.1.7) Requirement already satisfied: filelock in /usr/local/lib/python3.10/site-packages (from ray<3,>=2->ray[default,tune]<3,>=2->finrl==0.3.6) (3.13.3) Requirement already satisfied: jsonschema in /usr/local/lib/python3.10/site-packages (from ray<3,>=2->ray[default,tune]<3,>=2->finrl==0.3.6) (4.21.1) Requirement already satisfied: protobuf!=3.19.5,>=3.15.3 in /usr/local/lib/python3.10/site-packages (from ray<3,>=2->ray[default,tune]<3,>=2->finrl==0.3.6) (4.25.3) Requirement already satisfied: aiosignal in /usr/local/lib/python3.10/site-packages (from ray<3,>=2->ray[default,tune]<3,>=2->finrl==0.3.6) (1.3.1) Requirement already satisfied: frozenlist in /usr/local/lib/python3.10/site-packages (from ray<3,>=2->ray[default,tune]<3,>=2->finrl==0.3.6) (1.4.1) Requirement already satisfied: aiohttp-cors in /usr/local/lib/python3.10/site-packages (from ray[default,tune]<3,>=2->finrl==0.3.6) (0.7.0) Requirement already satisfied: colorful in /usr/local/lib/python3.10/site-packages (from ray[default,tune]<3,>=2->finrl==0.3.6) (0.5.6) Requirement already satisfied: py-spy>=0.2.0 in /usr/local/lib/python3.10/site-packages (from ray[default,tune]<3,>=2->finrl==0.3.6) (0.3.14) Requirement already satisfied: opencensus in /usr/local/lib/python3.10/site-packages (from ray[default,tune]<3,>=2->finrl==0.3.6) (0.11.4) Requirement already satisfied: pydantic!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,<3 in /usr/local/lib/python3.10/site-packages (from ray[default,tune]<3,>=2->finrl==0.3.6) (2.6.4) Requirement already satisfied: prometheus-client>=0.7.1 in /usr/local/lib/python3.10/site-packages (from ray[default,tune]<3,>=2->finrl==0.3.6) (0.20.0) Requirement already satisfied: smart-open in /usr/local/lib/python3.10/site-packages (from ray[default,tune]<3,>=2->finrl==0.3.6) (7.0.4) Requirement already satisfied: virtualenv!=20.21.1,>=20.0.24 in /usr/local/lib/python3.10/site-packages (from ray[default,tune]<3,>=2->finrl==0.3.6) (20.25.1) Requirement already satisfied: grpcio>=1.42.0 in /usr/local/lib/python3.10/site-packages (from ray[default,tune]<3,>=2->finrl==0.3.6) (1.62.1) Requirement already satisfied: tensorboardX>=1.9 in /usr/local/lib/python3.10/site-packages (from ray[default,tune]<3,>=2->finrl==0.3.6) (2.6.2.2) Requirement already satisfied: pyarrow>=6.0.1 in /usr/local/lib/python3.10/site-packages (from ray[default,tune]<3,>=2->finrl==0.3.6) (15.0.2) Requirement already satisfied: fsspec in /usr/local/lib/python3.10/site-packages (from ray[default,tune]<3,>=2->finrl==0.3.6) (2024.3.1) Requirement already satisfied: joblib>=1.2.0 in /usr/local/lib/python3.10/site-packages (from scikit-learn<2,>=1->finrl==0.3.6) (1.4.0) Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/site-packages (from scikit-learn<2,>=1->finrl==0.3.6) (3.4.0) Requirement already satisfied: gymnasium<0.30,>=0.28.1 in /usr/local/lib/python3.10/site-packages (from stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (0.29.1) Requirement already satisfied: torch>=1.13 in /usr/local/lib/python3.10/site-packages (from stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (2.2.2) Requirement already satisfied: cloudpickle in /usr/local/lib/python3.10/site-packages (from stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (3.0.0) Requirement already satisfied: opencv-python in /usr/local/lib/python3.10/site-packages (from stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (4.9.0.80) Requirement already satisfied: pygame in /usr/local/lib/python3.10/site-packages (from stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (2.1.0) Requirement already satisfied: tensorboard>=2.9.1 in /usr/local/lib/python3.10/site-packages (from stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (2.16.2) Requirement already satisfied: psutil in /usr/local/lib/python3.10/site-packages (from stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (5.9.8) Requirement already satisfied: tqdm in /usr/local/lib/python3.10/site-packages (from stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (4.66.1) Requirement already satisfied: rich in /usr/local/lib/python3.10/site-packages (from stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (13.7.1) Requirement already satisfied: shimmy~=1.3.0 in /usr/local/lib/python3.10/site-packages (from shimmy[atari]~=1.3.0; extra == "extra"->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (1.3.0) Requirement already satisfied: pillow in /usr/local/lib/python3.10/site-packages (from stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (10.3.0) Requirement already satisfied: autorom~=0.6.1 in /usr/local/lib/python3.10/site-packages (from autorom[accept-rom-license]~=0.6.1; extra == "extra"->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (0.6.1) Requirement already satisfied: psycopg2-binary<2.10,>=2.9 in /usr/local/lib/python3.10/site-packages (from wrds<4,>=3->finrl==0.3.6) (2.9.9) Requirement already satisfied: multitasking>=0.0.7 in /usr/local/lib/python3.10/site-packages (from yfinance<0.3,>=0.2->finrl==0.3.6) (0.0.11) Requirement already satisfied: lxml>=4.9.1 in /usr/local/lib/python3.10/site-packages (from yfinance<0.3,>=0.2->finrl==0.3.6) (5.2.1) Requirement already satisfied: appdirs>=1.4.4 in /usr/local/lib/python3.10/site-packages (from yfinance<0.3,>=0.2->finrl==0.3.6) (1.4.4) Requirement already satisfied: frozendict>=2.3.4 in /usr/local/lib/python3.10/site-packages (from yfinance<0.3,>=0.2->finrl==0.3.6) (2.4.1) Requirement already satisfied: peewee>=3.16.2 in /usr/local/lib/python3.10/site-packages (from yfinance<0.3,>=0.2->finrl==0.3.6) (3.17.1) Requirement already satisfied: beautifulsoup4>=4.11.1 in /usr/local/lib/python3.10/site-packages (from yfinance<0.3,>=0.2->finrl==0.3.6) (4.12.3) Requirement already satisfied: html5lib>=1.1 in /usr/local/lib/python3.10/site-packages (from yfinance<0.3,>=0.2->finrl==0.3.6) (1.1) Requirement already satisfied: gym in /usr/local/lib/python3.10/site-packages (from elegantrl@ git+https://github.com/AI4Finance-Foundation/ElegantRL.git#egg=elegantrl->finrl==0.3.6) (0.26.2) Requirement already satisfied: pycares>=4.0.0 in /usr/local/lib/python3.10/site-packages (from aiodns>=1.1.1->ccxt<4,>=3->finrl==0.3.6) (4.4.0) Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/site-packages (from aiohttp<4,>=3.8.3->alpaca-trade-api<4,>=3->finrl==0.3.6) (23.2.0) Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/site-packages (from aiohttp<4,>=3.8.3->alpaca-trade-api<4,>=3->finrl==0.3.6) (6.0.5) Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/site-packages (from aiohttp<4,>=3.8.3->alpaca-trade-api<4,>=3->finrl==0.3.6) (4.0.3) Requirement already satisfied: AutoROM.accept-rom-license in /usr/local/lib/python3.10/site-packages (from autorom[accept-rom-license]~=0.6.1; extra == "extra"->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (0.6.1) Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.10/site-packages (from beautifulsoup4>=4.11.1->yfinance<0.3,>=0.2->finrl==0.3.6) (2.5) Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.10/site-packages (from cryptography>=2.6.1->ccxt<4,>=3->finrl==0.3.6) (1.16.0) Requirement already satisfied: osqp>=0.6.2 in /usr/local/lib/python3.10/site-packages (from cvxpy<2.0.0,>=1.1.19->pyportfolioopt<2,>=1->finrl==0.3.6) (0.6.4) Requirement already satisfied: ecos>=2 in /usr/local/lib/python3.10/site-packages (from cvxpy<2.0.0,>=1.1.19->pyportfolioopt<2,>=1->finrl==0.3.6) (2.0.13) Requirement already satisfied: clarabel>=0.5.0 in /usr/local/lib/python3.10/site-packages (from cvxpy<2.0.0,>=1.1.19->pyportfolioopt<2,>=1->finrl==0.3.6) (0.7.1) Requirement already satisfied: scs>=3.0 in /usr/local/lib/python3.10/site-packages (from cvxpy<2.0.0,>=1.1.19->pyportfolioopt<2,>=1->finrl==0.3.6) (3.2.4.post1) Requirement already satisfied: pybind11 in /usr/local/lib/python3.10/site-packages (from cvxpy<2.0.0,>=1.1.19->pyportfolioopt<2,>=1->finrl==0.3.6) (2.12.0) Requirement already satisfied: pandas-datareader>=0.2 in /usr/local/lib/python3.10/site-packages (from empyrical>=0.5.0->pyfolio<0.10,>=0.9->finrl==0.3.6) (0.10.0) Requirement already satisfied: typing-extensions>=4.3.0 in /usr/local/lib/python3.10/site-packages (from gymnasium<0.30,>=0.28.1->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (4.11.0) Requirement already satisfied: farama-notifications>=0.0.1 in /usr/local/lib/python3.10/site-packages (from gymnasium<0.30,>=0.28.1->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (0.0.4) Requirement already satisfied: webencodings in /usr/local/lib/python3.10/site-packages (from html5lib>=1.1->yfinance<0.3,>=0.2->finrl==0.3.6) (0.5.1) Requirement already satisfied: decorator in /usr/local/lib/python3.10/site-packages (from ipython>=3.2.3->pyfolio<0.10,>=0.9->finrl==0.3.6) (5.1.1) Requirement already satisfied: jedi>=0.16 in /usr/local/lib/python3.10/site-packages (from ipython>=3.2.3->pyfolio<0.10,>=0.9->finrl==0.3.6) (0.19.1) Requirement already satisfied: matplotlib-inline in /usr/local/lib/python3.10/site-packages (from ipython>=3.2.3->pyfolio<0.10,>=0.9->finrl==0.3.6) (0.1.6) Requirement already satisfied: prompt-toolkit<3.1.0,>=3.0.41 in /usr/local/lib/python3.10/site-packages (from ipython>=3.2.3->pyfolio<0.10,>=0.9->finrl==0.3.6) (3.0.43) Requirement already satisfied: pygments>=2.4.0 in /usr/local/lib/python3.10/site-packages (from ipython>=3.2.3->pyfolio<0.10,>=0.9->finrl==0.3.6) (2.17.2) Requirement already satisfied: stack-data in /usr/local/lib/python3.10/site-packages (from ipython>=3.2.3->pyfolio<0.10,>=0.9->finrl==0.3.6) (0.6.3) Requirement already satisfied: traitlets>=5.13.0 in /usr/local/lib/python3.10/site-packages (from ipython>=3.2.3->pyfolio<0.10,>=0.9->finrl==0.3.6) (5.14.2) Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/site-packages (from ipython>=3.2.3->pyfolio<0.10,>=0.9->finrl==0.3.6) (1.2.0) Requirement already satisfied: pexpect>4.3 in /usr/local/lib/python3.10/site-packages (from ipython>=3.2.3->pyfolio<0.10,>=0.9->finrl==0.3.6) (4.9.0) Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/site-packages (from matplotlib>=1.4.0->pyfolio<0.10,>=0.9->finrl==0.3.6) (1.2.1) Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/site-packages (from matplotlib>=1.4.0->pyfolio<0.10,>=0.9->finrl==0.3.6) (0.12.1) Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/site-packages (from matplotlib>=1.4.0->pyfolio<0.10,>=0.9->finrl==0.3.6) (4.51.0) Requirement already satisfied: kiwisolver>=1.3.1 in /usr/local/lib/python3.10/site-packages (from matplotlib>=1.4.0->pyfolio<0.10,>=0.9->finrl==0.3.6) (1.4.5) Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/site-packages (from matplotlib>=1.4.0->pyfolio<0.10,>=0.9->finrl==0.3.6) (3.1.2) Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/site-packages (from matplotlib>=1.4.0->pyfolio<0.10,>=0.9->finrl==0.3.6) (2.9.0.post0) Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/site-packages (from pydantic!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,<3->ray[default,tune]<3,>=2->finrl==0.3.6) (0.6.0) Requirement already satisfied: pydantic-core==2.16.3 in /usr/local/lib/python3.10/site-packages (from pydantic!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,<3->ray[default,tune]<3,>=2->finrl==0.3.6) (2.16.3) Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/site-packages (from requests<3,>2->alpaca-trade-api<4,>=3->finrl==0.3.6) (3.3.2) Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/site-packages (from requests<3,>2->alpaca-trade-api<4,>=3->finrl==0.3.6) (3.6) Requirement already satisfied: ale-py~=0.8.1 in /usr/local/lib/python3.10/site-packages (from shimmy[atari]~=1.3.0; extra == "extra"->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (0.8.1) Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.10/site-packages (from SQLAlchemy>=1.2.8->jqdatasdk<2,>=1->finrl==0.3.6) (3.0.3) Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.10/site-packages (from tensorboard>=2.9.1->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (2.1.0) Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.10/site-packages (from tensorboard>=2.9.1->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (3.6) Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/site-packages (from tensorboard>=2.9.1->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (0.7.2) Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/site-packages (from tensorboard>=2.9.1->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (3.0.2) Requirement already satisfied: ply<4.0,>=3.4 in /usr/local/lib/python3.10/site-packages (from thriftpy2>=0.3.9->jqdatasdk<2,>=1->finrl==0.3.6) (3.11) Requirement already satisfied: sympy in /usr/local/lib/python3.10/site-packages (from torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (1.12) Requirement already satisfied: networkx in /usr/local/lib/python3.10/site-packages (from torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (3.3) Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/site-packages (from torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (3.1.3) Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /usr/local/lib/python3.10/site-packages (from torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (12.1.105) Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /usr/local/lib/python3.10/site-packages (from torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (12.1.105) Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /usr/local/lib/python3.10/site-packages (from torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (12.1.105) Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /usr/local/lib/python3.10/site-packages (from torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (8.9.2.26) Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /usr/local/lib/python3.10/site-packages (from torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (12.1.3.1) Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /usr/local/lib/python3.10/site-packages (from torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (11.0.2.54) Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /usr/local/lib/python3.10/site-packages (from torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (10.3.2.106) Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /usr/local/lib/python3.10/site-packages (from torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (11.4.5.107) Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /usr/local/lib/python3.10/site-packages (from torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (12.1.0.106) Requirement already satisfied: nvidia-nccl-cu12==2.19.3 in /usr/local/lib/python3.10/site-packages (from torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (2.19.3) Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /usr/local/lib/python3.10/site-packages (from torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (12.1.105) Requirement already satisfied: triton==2.2.0 in /usr/local/lib/python3.10/site-packages (from torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (2.2.0) Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.10/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (12.4.127) Requirement already satisfied: distlib<1,>=0.3.7 in /usr/local/lib/python3.10/site-packages (from virtualenv!=20.21.1,>=20.0.24->ray[default,tune]<3,>=2->finrl==0.3.6) (0.3.8) Requirement already satisfied: platformdirs<5,>=3.9.1 in /usr/local/lib/python3.10/site-packages (from virtualenv!=20.21.1,>=20.0.24->ray[default,tune]<3,>=2->finrl==0.3.6) (4.1.0) Requirement already satisfied: gym-notices>=0.0.4 in /usr/local/lib/python3.10/site-packages (from gym->elegantrl@ git+https://github.com/AI4Finance-Foundation/ElegantRL.git#egg=elegantrl->finrl==0.3.6) (0.0.8) Requirement already satisfied: box2d-py==2.3.5 in /usr/local/lib/python3.10/site-packages (from gym[box2d]->elegantrl@ git+https://github.com/AI4Finance-Foundation/ElegantRL.git#egg=elegantrl->finrl==0.3.6) (2.3.5) Requirement already satisfied: swig==4.* in /usr/local/lib/python3.10/site-packages (from gym[box2d]->elegantrl@ git+https://github.com/AI4Finance-Foundation/ElegantRL.git#egg=elegantrl->finrl==0.3.6) (4.2.1) Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/site-packages (from jsonschema->ray<3,>=2->ray[default,tune]<3,>=2->finrl==0.3.6) (2023.12.1) Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.10/site-packages (from jsonschema->ray<3,>=2->ray[default,tune]<3,>=2->finrl==0.3.6) (0.34.0) Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/site-packages (from jsonschema->ray<3,>=2->ray[default,tune]<3,>=2->finrl==0.3.6) (0.18.0) Requirement already satisfied: opencensus-context>=0.1.3 in /usr/local/lib/python3.10/site-packages (from opencensus->ray[default,tune]<3,>=2->finrl==0.3.6) (0.1.3) Requirement already satisfied: google-api-core<3.0.0,>=1.0.0 in /usr/local/lib/python3.10/site-packages (from opencensus->ray[default,tune]<3,>=2->finrl==0.3.6) (2.18.0) Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/site-packages (from rich->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (3.0.0) Requirement already satisfied: wrapt in /usr/local/lib/python3.10/site-packages (from smart-open->ray[default,tune]<3,>=2->finrl==0.3.6) (1.16.0) Requirement already satisfied: importlib-resources in /usr/local/lib/python3.10/site-packages (from ale-py~=0.8.1->shimmy[atari]~=1.3.0; extra == "extra"->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (6.4.0) Requirement already satisfied: pycparser in /usr/local/lib/python3.10/site-packages (from cffi>=1.12->cryptography>=2.6.1->ccxt<4,>=3->finrl==0.3.6) (2.21) Requirement already satisfied: googleapis-common-protos<2.0.dev0,>=1.56.2 in /usr/local/lib/python3.10/site-packages (from google-api-core<3.0.0,>=1.0.0->opencensus->ray[default,tune]<3,>=2->finrl==0.3.6) (1.63.0) Requirement already satisfied: proto-plus<2.0.0dev,>=1.22.3 in /usr/local/lib/python3.10/site-packages (from google-api-core<3.0.0,>=1.0.0->opencensus->ray[default,tune]<3,>=2->finrl==0.3.6) (1.23.0) Requirement already satisfied: google-auth<3.0.dev0,>=2.14.1 in /usr/local/lib/python3.10/site-packages (from google-api-core<3.0.0,>=1.0.0->opencensus->ray[default,tune]<3,>=2->finrl==0.3.6) (2.29.0) Requirement already satisfied: parso<0.9.0,>=0.8.3 in /usr/local/lib/python3.10/site-packages (from jedi>=0.16->ipython>=3.2.3->pyfolio<0.10,>=0.9->finrl==0.3.6) (0.8.4) Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/site-packages (from markdown-it-py>=2.2.0->rich->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (0.1.2) Requirement already satisfied: qdldl in /usr/local/lib/python3.10/site-packages (from osqp>=0.6.2->cvxpy<2.0.0,>=1.1.19->pyportfolioopt<2,>=1->finrl==0.3.6) (0.1.7.post1) Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.10/site-packages (from pexpect>4.3->ipython>=3.2.3->pyfolio<0.10,>=0.9->finrl==0.3.6) (0.7.0) Requirement already satisfied: wcwidth in /usr/local/lib/python3.10/site-packages (from prompt-toolkit<3.1.0,>=3.0.41->ipython>=3.2.3->pyfolio<0.10,>=0.9->finrl==0.3.6) (0.2.13) Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.10/site-packages (from werkzeug>=1.0.1->tensorboard>=2.9.1->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (2.1.5) Requirement already satisfied: executing>=1.2.0 in /usr/local/lib/python3.10/site-packages (from stack-data->ipython>=3.2.3->pyfolio<0.10,>=0.9->finrl==0.3.6) (2.0.1) Requirement already satisfied: asttokens>=2.1.0 in /usr/local/lib/python3.10/site-packages (from stack-data->ipython>=3.2.3->pyfolio<0.10,>=0.9->finrl==0.3.6) (2.4.1) Requirement already satisfied: pure-eval in /usr/local/lib/python3.10/site-packages (from stack-data->ipython>=3.2.3->pyfolio<0.10,>=0.9->finrl==0.3.6) (0.2.2) Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/site-packages (from sympy->torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (1.3.0) Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/site-packages (from google-auth<3.0.dev0,>=2.14.1->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default,tune]<3,>=2->finrl==0.3.6) (5.3.3) Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/site-packages (from google-auth<3.0.dev0,>=2.14.1->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default,tune]<3,>=2->finrl==0.3.6) (0.4.0) Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/site-packages (from google-auth<3.0.dev0,>=2.14.1->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default,tune]<3,>=2->finrl==0.3.6) (4.9) Requirement already satisfied: pyasn1<0.7.0,>=0.4.6 in /usr/local/lib/python3.10/site-packages (from pyasn1-modules>=0.2.1->google-auth<3.0.dev0,>=2.14.1->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default,tune]<3,>=2->finrl==0.3.6) (0.6.0) WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

2.2. Check if the additional packages needed are present, if not install them.

  • Yahoo Finance API

  • pandas

  • numpy

  • matplotlib

  • stockstats

  • OpenAI gym

  • stable-baselines

  • tensorflow

  • pyfolio

2.3. Import Packages

import warnings
warnings.filterwarnings("ignore")

import pandas as pd
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
# matplotlib.use('Agg')
import datetime

%matplotlib inline
from finrl.config_tickers import DOW_30_TICKER
from finrl.meta.preprocessor.yahoodownloader import YahooDownloader
from finrl.meta.preprocessor.preprocessors import FeatureEngineer, data_split
from finrl.meta.env_stock_trading.env_stocktrading import StockTradingEnv
from finrl.agents.stablebaselines3.models import DRLAgent,DRLEnsembleAgent
from finrl.plot import backtest_stats, backtest_plot, get_daily_return, get_baseline

from pprint import pprint

import sys
sys.path.append("../FinRL-Library")

import itertools

2.4. Create Folders

import os
from finrl.main import check_and_make_directories
from finrl.config import (
    DATA_SAVE_DIR,
    TRAINED_MODEL_DIR,
    TENSORBOARD_LOG_DIR,
    RESULTS_DIR,
    INDICATORS,
    TRAIN_START_DATE,
    TRAIN_END_DATE,
    TEST_START_DATE,
    TEST_END_DATE,
    TRADE_START_DATE,
    TRADE_END_DATE,
)

check_and_make_directories([DATA_SAVE_DIR, TRAINED_MODEL_DIR, TENSORBOARD_LOG_DIR, RESULTS_DIR])

Part 3. Download Data

Yahoo Finance is a website that provides stock data, financial news, financial reports, etc. All the data provided by Yahoo Finance is free.

  • FinRL uses a class YahooDownloader to fetch data from Yahoo Finance API

  • Call Limit: Using the Public API (without authentication), you are limited to 2,000 requests per hour per IP (or up to a total of 48,000 requests a day).

class YahooDownloader: Provides methods for retrieving daily stock data from Yahoo Finance API

Attributes
----------
    start_date : str
        start date of the data (modified from config.py)
    end_date : str
        end date of the data (modified from config.py)
    ticker_list : list
        a list of stock tickers (modified from config.py)

Methods
-------
fetch_data()
    Fetches data from yahoo API

print(DOW_30_TICKER)

# TRAIN_START_DATE = '2009-04-01'
# TRAIN_END_DATE = '2021-01-01'
# TEST_START_DATE = '2021-01-01'
# TEST_END_DATE = '2022-06-01'
from finrl.meta.preprocessor.yahoodownloader import YahooDownloader
from finrl.config_tickers import DOW_30_TICKER

TRAIN_START_DATE = '2010-01-01'
TRAIN_END_DATE = '2021-10-01'
TEST_START_DATE = '2021-10-01'
TEST_END_DATE = '2023-03-01'

df = YahooDownloader(start_date = TRAIN_START_DATE,
                     end_date = TEST_END_DATE,
                     ticker_list = DOW_30_TICKER).fetch_data()
[*********************100%%**********************] 1 of 1 completed [*********************100%%**********************] 1 of 1 completed [*********************100%%**********************] 1 of 1 completed [*********************100%%**********************] 1 of 1 completed [*********************100%%**********************] 1 of 1 completed [*********************100%%**********************] 1 of 1 completed [*********************100%%**********************] 1 of 1 completed [*********************100%%**********************] 1 of 1 completed [*********************100%%**********************] 1 of 1 completed [*********************100%%**********************] 1 of 1 completed [*********************100%%**********************] 1 of 1 completed [*********************100%%**********************] 1 of 1 completed [*********************100%%**********************] 1 of 1 completed [*********************100%%**********************] 1 of 1 completed [*********************100%%**********************] 1 of 1 completed [*********************100%%**********************] 1 of 1 completed [*********************100%%**********************] 1 of 1 completed [*********************100%%**********************] 1 of 1 completed [*********************100%%**********************] 1 of 1 completed [*********************100%%**********************] 1 of 1 completed [*********************100%%**********************] 1 of 1 completed [*********************100%%**********************] 1 of 1 completed [*********************100%%**********************] 1 of 1 completed [*********************100%%**********************] 1 of 1 completed [*********************100%%**********************] 1 of 1 completed [*********************100%%**********************] 1 of 1 completed [*********************100%%**********************] 1 of 1 completed [*********************100%%**********************] 1 of 1 completed [*********************100%%**********************] 1 of 1 completed [*********************100%%**********************] 1 of 1 completed
Shape of DataFrame: (97013, 8)

Part 4: Preprocess Data

Data preprocessing is a crucial step for training a high quality machine learning model. We need to check for missing data and do feature engineering in order to convert the data into a model-ready state.

  • Add technical indicators. In practical trading, various information needs to be taken into account, for example the historical stock prices, current holding shares, technical indicators, etc. In this article, we demonstrate two trend-following technical indicators: MACD and RSI.

  • Add turbulence index. Risk-aversion reflects whether an investor will choose to preserve the capital. It also influences one's trading strategy when facing different market volatility level. To control the risk in a worst-case scenario, such as financial crisis of 2007–2008, FinRL employs the financial turbulence index that measures extreme asset price fluctuation.

 INDICATORS = ['macd',
               'rsi_30',
               'cci_30',
               'dx_30']

from finrl.meta.preprocessor.preprocessors import FeatureEngineer
fe = FeatureEngineer(use_technical_indicator=True,
                     tech_indicator_list = INDICATORS,
                     use_turbulence=True,
                     user_defined_feature = False)

processed = fe.preprocess_data(df)
processed = processed.copy()
processed = processed.fillna(0)
processed = processed.replace(np.inf,0)
Successfully added technical indicators Successfully added turbulence index 
len(state), len(data)
(7, 13)

Part 5. Design Environment

Considering the stochastic and interactive nature of the automated stock trading tasks, a financial task is modeled as a Markov Decision Process (MDP) problem. The training process involves observing stock price change, taking an action and reward's calculation to have the agent adjusting its strategy accordingly. By interacting with the environment, the trading agent will derive a trading strategy with the maximized rewards as time proceeds.

Our trading environments, based on OpenAI Gym framework, simulate live stock markets with real market data according to the principle of time-driven simulation.

The action space describes the allowed actions that the agent interacts with the environment. Normally, action a includes three actions: {-1, 0, 1}, where -1, 0, 1 represent selling, holding, and buying one share. Also, an action can be carried upon multiple shares. We use an action space {-k,…,-1, 0, 1, …, k}, where k denotes the number of shares to buy and -k denotes the number of shares to sell. For example, "Buy 10 shares of AAPL" or "Sell 10 shares of AAPL" are 10 or -10, respectively. The continuous action space needs to be normalized to [-1, 1], since the policy is defined on a Gaussian distribution, which needs to be normalized and symmetric.

stock_dimension = len(processed.tic.unique())
state_space = 1 + 2*stock_dimension + len(INDICATORS)*stock_dimension
print(f"Stock Dimension: {stock_dimension}, State Space: {state_space}")
Stock Dimension: 29, State Space: 175 
env_kwargs = {
    "hmax": 100,
    "initial_amount": 1000000,
    "buy_cost_pct": 0.001,
    "sell_cost_pct": 0.001,
    "state_space": state_space,
    "stock_dim": stock_dimension,
    "tech_indicator_list": INDICATORS,
    "action_space": stock_dimension,
    "reward_scaling": 1e-4,
    "print_verbosity":5

}

# buy_cost_list = sell_cost_list = [0.001] * stock_dimension
# num_stock_shares = [0] * stock_dimension
# env_kwargs = {
#     "hmax": 100,
#     "initial_amount": 1000000,
#     "num_stock_shares": num_stock_shares,
#     "buy_cost_pct": buy_cost_list,
#     "sell_cost_pct": sell_cost_list,
#     "state_space": state_space,
#     "stock_dim": stock_dimension,
#     "tech_indicator_list": INDICATORS,
#     "action_space": stock_dimension,
#     "reward_scaling": 1e-4
# }

Part 6: Implement DRL Algorithms

  • The implementation of the DRL algorithms are based on OpenAI Baselines and Stable Baselines. Stable Baselines is a fork of OpenAI Baselines, with a major structural refactoring, and code cleanups.

  • FinRL library includes fine-tuned standard DRL algorithms, such as DQN, DDPG, Multi-Agent DDPG, PPO, SAC, A2C and TD3. We also allow users to design their own DRL algorithms by adapting these DRL algorithms.

  • In this notebook, we are training and validating 3 agents (A2C, PPO, DDPG) using Rolling-window Ensemble Method (reference code)

rebalance_window = 63 # rebalance_window is the number of days to retrain the model
validation_window = 63 # validation_window is the number of days to do validation and trading (e.g. if validation_window=63, then both validation and trading period will be 63 days)

ensemble_agent = DRLEnsembleAgent(df=processed,
                 train_period=(TRAIN_START_DATE,TRAIN_END_DATE),
                 val_test_period=(TEST_START_DATE,TEST_END_DATE),
                 rebalance_window=rebalance_window,
                 validation_window=validation_window,
                 **env_kwargs)
# e_train_gym = StockTradingEnv(df = processed, **env_kwargs)
# agent = DRLAgent(e_train_gym)
# if_using_a2c = True
# model_a2c = agent.get_model("a2c")
# # if if_using_a2c:
# #   tmp_path = RESULTS_DIR + '/a2c'
# #   new_logger_a2c = configure(tmp_path, ["stdout", "csv", "tensorboard"])
# #   model_a2c.set_logger(new_logger_a2c)
# trained_a2c = agent.train_model(model=model_a2c,
#                              tb_log_name='a2c',
#                              total_timesteps=50000)

A2C_model_kwargs = {
                    'n_steps': 5,
                    'ent_coef': 0.005,
                    'learning_rate': 0.0007
                    }

PPO_model_kwargs = {
                    "ent_coef":0.01,
                    "n_steps": 2048,
                    "learning_rate": 0.00025,
                    "batch_size": 128
                    }

DDPG_model_kwargs = {
                      #"action_noise":"ornstein_uhlenbeck",
                      "buffer_size": 10_000,
                      "learning_rate": 0.0005,
                      "batch_size": 64
                    }

SAC_model_kwargs = {
    "batch_size": 64,
    "buffer_size": 100000,
    "learning_rate": 0.0001,
    "learning_starts": 100,
    "ent_coef": "auto_0.1",
}

TD3_model_kwargs = {"batch_size": 100, "buffer_size": 1000000, "learning_rate": 0.0001}




timesteps_dict = {'a2c' : 10_000,
                 'ppo' : 10_000,
                 'ddpg' : 10_000,
                 'sac' : 10_000,
                 'td3' : 10_000
                 }

df_summary = ensemble_agent.run_ensemble_strategy(A2C_model_kwargs,
                                                 PPO_model_kwargs,
                                                 DDPG_model_kwargs,
                                                 SAC_model_kwargs,
                                                 TD3_model_kwargs,
                                                 timesteps_dict)
============Start Ensemble Strategy============ ============================================ turbulence_threshold: 201.74162030011615 ======Model training from: 2010-01-01 to 2021-10-04 ======a2c Training======== {'n_steps': 5, 'ent_coef': 0.005, 'learning_rate': 0.0007} Using cuda device Logging to tensorboard_log/a2c/a2c_126_1 --------------------------------------- | time/ | | | fps | 83 | | iterations | 100 | | time_elapsed | 5 | | total_timesteps | 500 | | train/ | | | entropy_loss | -41.1 | | explained_variance | -0.589 | | learning_rate | 0.0007 | | n_updates | 99 | | policy_loss | -62.2 | | reward | -0.13443886 | | std | 0.998 | | value_loss | 3.54 | --------------------------------------- -------------------------------------- | time/ | | | fps | 93 | | iterations | 200 | | time_elapsed | 10 | | total_timesteps | 1000 | | train/ | | | entropy_loss | -41 | | explained_variance | -0.3 | | learning_rate | 0.0007 | | n_updates | 199 | | policy_loss | -58.5 | | reward | 0.42441055 | | std | 0.996 | | value_loss | 6.11 | -------------------------------------- -------------------------------------- | time/ | | | fps | 92 | | iterations | 300 | | time_elapsed | 16 | | total_timesteps | 1500 | | train/ | | | entropy_loss | -41.1 | | explained_variance | 0 | | learning_rate | 0.0007 | | n_updates | 299 | | policy_loss | -27.8 | | reward | -3.0476444 | | std | 0.998 | | value_loss | 2.41 | -------------------------------------- ------------------------------------- | time/ | | | fps | 95 | | iterations | 400 | | time_elapsed | 21 | | total_timesteps | 2000 | | train/ | | | entropy_loss | -41.1 | | explained_variance | -1.19e-07 | | learning_rate | 0.0007 | | n_updates | 399 | | policy_loss | 24 | | reward | 1.1522169 | | std | 0.999 | | value_loss | 2.67 | ------------------------------------- --------------------------------------- | time/ | | | fps | 97 | | iterations | 500 | | time_elapsed | 25 | | total_timesteps | 2500 | | train/ | | | entropy_loss | -41.1 | | explained_variance | 1.19e-07 | | learning_rate | 0.0007 | | n_updates | 499 | | policy_loss | -9.18 | | reward | 0.008832613 | | std | 0.999 | | value_loss | 3.65 | --------------------------------------- ------------------------------------- | time/ | | | fps | 96 | | iterations | 600 | | time_elapsed | 31 | | total_timesteps | 3000 | | train/ | | | entropy_loss | -41 | | explained_variance | 1.19e-07 | | learning_rate | 0.0007 | | n_updates | 599 | | policy_loss | -0.0801 | | reward | 0.8033523 | | std | 0.997 | | value_loss | 0.0839 | ------------------------------------- --------------------------------------- | time/ | | | fps | 97 | | iterations | 700 | | time_elapsed | 35 | | total_timesteps | 3500 | | train/ | | | entropy_loss | -41.1 | | explained_variance | 0.0154 | | learning_rate | 0.0007 | | n_updates | 699 | | policy_loss | -10.3 | | reward | 0.056252044 | | std | 0.999 | | value_loss | 0.266 | --------------------------------------- --------------------------------------- | time/ | | | fps | 96 | | iterations | 800 | | time_elapsed | 41 | | total_timesteps | 4000 | | train/ | | | entropy_loss | -41.2 | | explained_variance | 0.0216 | | learning_rate | 0.0007 | | n_updates | 799 | | policy_loss | 102 | | reward | -0.17337318 | | std | 1 | | value_loss | 7.78 | --------------------------------------- -------------------------------------- | time/ | | | fps | 97 | | iterations | 900 | | time_elapsed | 46 | | total_timesteps | 4500 | | train/ | | | entropy_loss | -41.1 | | explained_variance | 1.19e-07 | | learning_rate | 0.0007 | | n_updates | 899 | | policy_loss | -150 | | reward | -0.7243548 | | std | 1 | | value_loss | 16.4 | -------------------------------------- --------------------------------------- | time/ | | | fps | 96 | | iterations | 1000 | | time_elapsed | 51 | | total_timesteps | 5000 | | train/ | | | entropy_loss | -41.2 | | explained_variance | -0.0375 | | learning_rate | 0.0007 | | n_updates | 999 | | policy_loss | -510 | | reward | -0.63908553 | | std | 1 | | value_loss | 240 | --------------------------------------- ------------------------------------- | time/ | | | fps | 97 | | iterations | 1100 | | time_elapsed | 56 | | total_timesteps | 5500 | | train/ | | | entropy_loss | -41.2 | | explained_variance | 0.00788 | | learning_rate | 0.0007 | | n_updates | 1099 | | policy_loss | -248 | | reward | 3.7405448 | | std | 1 | | value_loss | 48.9 | ------------------------------------- -------------------------------------- | time/ | | | fps | 97 | | iterations | 1200 | | time_elapsed | 61 | | total_timesteps | 6000 | | train/ | | | entropy_loss | -41.1 | | explained_variance | 0.346 | | learning_rate | 0.0007 | | n_updates | 1199 | | policy_loss | -16.7 | | reward | -0.7040105 | | std | 1 | | value_loss | 0.666 | -------------------------------------- -------------------------------------- | time/ | | | fps | 97 | | iterations | 1300 | | time_elapsed | 66 | | total_timesteps | 6500 | | train/ | | | entropy_loss | -41.1 | | explained_variance | -0.0607 | | learning_rate | 0.0007 | | n_updates | 1299 | | policy_loss | 21.4 | | reward | 0.14980054 | | std | 0.999 | | value_loss | 0.792 | -------------------------------------- ------------------------------------- | time/ | | | fps | 97 | | iterations | 1400 | | time_elapsed | 71 | | total_timesteps | 7000 | | train/ | | | entropy_loss | -41.1 | | explained_variance | 0.15 | | learning_rate | 0.0007 | | n_updates | 1399 | | policy_loss | 148 | | reward | -1.891962 | | std | 0.999 | | value_loss | 17.6 | ------------------------------------- ------------------------------------- | time/ | | | fps | 97 | | iterations | 1500 | | time_elapsed | 77 | | total_timesteps | 7500 | | train/ | | | entropy_loss | -41.1 | | explained_variance | -0.0336 | | learning_rate | 0.0007 | | n_updates | 1499 | | policy_loss | 91.2 | | reward | -2.299666 | | std | 1 | | value_loss | 5.62 | ------------------------------------- ------------------------------------- | time/ | | | fps | 97 | | iterations | 1600 | | time_elapsed | 81 | | total_timesteps | 8000 | | train/ | | | entropy_loss | -41.1 | | explained_variance | 0 | | learning_rate | 0.0007 | | n_updates | 1599 | | policy_loss | -19.3 | | reward | 2.7779486 | | std | 0.999 | | value_loss | 13.9 | ------------------------------------- ------------------------------------- | time/ | | | fps | 97 | | iterations | 1700 | | time_elapsed | 86 | | total_timesteps | 8500 | | train/ | | | entropy_loss | -41.1 | | explained_variance | 0 | | learning_rate | 0.0007 | | n_updates | 1699 | | policy_loss | 36.7 | | reward | 5.3634834 | | std | 0.998 | | value_loss | 29.5 | ------------------------------------- -------------------------------------- | time/ | | | fps | 97 | | iterations | 1800 | | time_elapsed | 91 | | total_timesteps | 9000 | | train/ | | | entropy_loss | -41.1 | | explained_variance | 0.0493 | | learning_rate | 0.0007 | | n_updates | 1799 | | policy_loss | -60.4 | | reward | 0.34657776 | | std | 0.999 | | value_loss | 4.15 | -------------------------------------- ------------------------------------- | time/ | | | fps | 98 | | iterations | 1900 | | time_elapsed | 96 | | total_timesteps | 9500 | | train/ | | | entropy_loss | -41.1 | | explained_variance | -0.229 | | learning_rate | 0.0007 | | n_updates | 1899 | | policy_loss | 29.5 | | reward | 1.0733021 | | std | 1 | | value_loss | 2.12 | ------------------------------------- ------------------------------------- | time/ | | | fps | 97 | | iterations | 2000 | | time_elapsed | 102 | | total_timesteps | 10000 | | train/ | | | entropy_loss | -41.2 | | explained_variance | -1.19e-07 | | learning_rate | 0.0007 | | n_updates | 1999 | | policy_loss | 41.5 | | reward | 0.5164767 | | std | 1 | | value_loss | 1.16 | ------------------------------------- ======a2c Validation from: 2021-10-04 to 2022-01-03 a2c Sharpe Ratio: 0.12016203130695303 ======ddpg Training======== {'buffer_size': 10000, 'learning_rate': 0.0005, 'batch_size': 64} Using cuda device Logging to tensorboard_log/ddpg/ddpg_126_1 day: 2957, episode: 5 begin_total_asset: 1000000.00 end_total_asset: 4002721.16 total_reward: 3002721.16 total_cost: 6360.02 total_trades: 48784 Sharpe: 0.828 ================================= ======ddpg Validation from: 2021-10-04 to 2022-01-03 ddpg Sharpe Ratio: 0.23149939361322536 ======td3 Training======== {'batch_size': 100, 'buffer_size': 1000000, 'learning_rate': 0.0001} Using cuda device Logging to tensorboard_log/td3/td3_126_1 day: 2957, episode: 10 begin_total_asset: 1000000.00 end_total_asset: 5945616.62 total_reward: 4945616.62 total_cost: 2786.02 total_trades: 38732 Sharpe: 0.922 ================================= ======td3 Validation from: 2021-10-04 to 2022-01-03 td3 Sharpe Ratio: 0.12034224444593176 ======sac Training======== {'batch_size': 64, 'buffer_size': 100000, 'learning_rate': 0.0001, 'learning_starts': 100, 'ent_coef': 'auto_0.1'} Using cuda device Logging to tensorboard_log/sac/sac_126_1 day: 2957, episode: 15 begin_total_asset: 1000000.00 end_total_asset: 4555559.85 total_reward: 3555559.85 total_cost: 238769.98 total_trades: 66550 Sharpe: 0.861 ================================= ======sac Validation from: 2021-10-04 to 2022-01-03 sac Sharpe Ratio: 0.08822857821789602 ======ppo Training======== {'ent_coef': 0.01, 'n_steps': 2048, 'learning_rate': 0.00025, 'batch_size': 128} Using cuda device Logging to tensorboard_log/ppo/ppo_126_1 ---------------------------------- | time/ | | | fps | 108 | | iterations | 1 | | time_elapsed | 18 | | total_timesteps | 2048 | | train/ | | | reward | 1.2595656 | ---------------------------------- ----------------------------------------- | time/ | | | fps | 107 | | iterations | 2 | | time_elapsed | 38 | | total_timesteps | 4096 | | train/ | | | approx_kl | 0.016799435 | | clip_fraction | 0.204 | | clip_range | 0.2 | | entropy_loss | -41.2 | | explained_variance | -0.0259 | | learning_rate | 0.00025 | | loss | 3.89 | | n_updates | 10 | | policy_gradient_loss | -0.0262 | | reward | 1.0748519 | | std | 1 | | value_loss | 9.71 | ----------------------------------------- ----------------------------------------- | time/ | | | fps | 106 | | iterations | 3 | | time_elapsed | 57 | | total_timesteps | 6144 | | train/ | | | approx_kl | 0.014081008 | | clip_fraction | 0.135 | | clip_range | 0.2 | | entropy_loss | -41.3 | | explained_variance | 0.0159 | | learning_rate | 0.00025 | | loss | 13.2 | | n_updates | 20 | | policy_gradient_loss | -0.0211 | | reward | -0.25309741 | | std | 1 | | value_loss | 41.8 | ----------------------------------------- ----------------------------------------- | time/ | | | fps | 106 | | iterations | 4 | | time_elapsed | 77 | | total_timesteps | 8192 | | train/ | | | approx_kl | 0.014428517 | | clip_fraction | 0.148 | | clip_range | 0.2 | | entropy_loss | -41.3 | | explained_variance | -0.0206 | | learning_rate | 0.00025 | | loss | 30.4 | | n_updates | 30 | | policy_gradient_loss | -0.0176 | | reward | 3.6489613 | | std | 1.01 | | value_loss | 63.9 | ----------------------------------------- ----------------------------------------- | time/ | | | fps | 106 | | iterations | 5 | | time_elapsed | 95 | | total_timesteps | 10240 | | train/ | | | approx_kl | 0.018376704 | | clip_fraction | 0.215 | | clip_range | 0.2 | | entropy_loss | -41.4 | | explained_variance | 0.0272 | | learning_rate | 0.00025 | | loss | 5.93 | | n_updates | 40 | | policy_gradient_loss | -0.0251 | | reward | -0.09722769 | | std | 1.01 | | value_loss | 14.7 | ----------------------------------------- ======ppo Validation from: 2021-10-04 to 2022-01-03 ppo Sharpe Ratio: 0.30010210770044654 ======Best Model Retraining from: 2010-01-01 to 2022-01-03 ======Trading from: 2022-01-03 to 2022-04-04 ============================================ turbulence_threshold: 201.74162030011615 ======Model training from: 2010-01-01 to 2022-01-03 ======a2c Training======== {'n_steps': 5, 'ent_coef': 0.005, 'learning_rate': 0.0007} Using cuda device Logging to tensorboard_log/a2c/a2c_189_1 ---------------------------------------- | time/ | | | fps | 90 | | iterations | 100 | | time_elapsed | 5 | | total_timesteps | 500 | | train/ | | | entropy_loss | -41.1 | | explained_variance | 0.213 | | learning_rate | 0.0007 | | n_updates | 99 | | policy_loss | -82.9 | | reward | -0.023693616 | | std | 0.998 | | value_loss | 4.94 | ---------------------------------------- --------------------------------------- | time/ | | | fps | 97 | | iterations | 200 | | time_elapsed | 10 | | total_timesteps | 1000 | | train/ | | | entropy_loss | -41.1 | | explained_variance | 0 | | learning_rate | 0.0007 | | n_updates | 199 | | policy_loss | -64.3 | | reward | -0.38760617 | | std | 1 | | value_loss | 6.33 | --------------------------------------- -------------------------------------- | time/ | | | fps | 95 | | iterations | 300 | | time_elapsed | 15 | | total_timesteps | 1500 | | train/ | | | entropy_loss | -41.2 | | explained_variance | 0.0205 | | learning_rate | 0.0007 | | n_updates | 299 | | policy_loss | 54.3 | | reward | -4.1570683 | | std | 1 | | value_loss | 4.02 | -------------------------------------- --------------------------------------- | time/ | | | fps | 98 | | iterations | 400 | | time_elapsed | 20 | | total_timesteps | 2000 | | train/ | | | entropy_loss | -41.1 | | explained_variance | 0.00152 | | learning_rate | 0.0007 | | n_updates | 399 | | policy_loss | 91.3 | | reward | -0.37606463 | | std | 0.999 | | value_loss | 14.6 | --------------------------------------- -------------------------------------- | time/ | | | fps | 99 | | iterations | 500 | | time_elapsed | 25 | | total_timesteps | 2500 | | train/ | | | entropy_loss | -41 | | explained_variance | 0 | | learning_rate | 0.0007 | | n_updates | 499 | | policy_loss | -72.9 | | reward | -1.6318904 | | std | 0.996 | | value_loss | 4.7 | -------------------------------------- ------------------------------------- | time/ | | | fps | 98 | | iterations | 600 | | time_elapsed | 30 | | total_timesteps | 3000 | | train/ | | | entropy_loss | -41.1 | | explained_variance | 0 | | learning_rate | 0.0007 | | n_updates | 599 | | policy_loss | 250 | | reward | 6.0129476 | | std | 0.997 | | value_loss | 57.9 | ------------------------------------- --------------------------------------- | time/ | | | fps | 99 | | iterations | 700 | | time_elapsed | 35 | | total_timesteps | 3500 | | train/ | | | entropy_loss | -41 | | explained_variance | -0.138 | | learning_rate | 0.0007 | | n_updates | 699 | | policy_loss | -199 | | reward | -0.07948906 | | std | 0.996 | | value_loss | 24.3 | --------------------------------------- ------------------------------------- | time/ | | | fps | 98 | | iterations | 800 | | time_elapsed | 40 | | total_timesteps | 4000 | | train/ | | | entropy_loss | -41 | | explained_variance | 0.00106 | | learning_rate | 0.0007 | | n_updates | 799 | | policy_loss | 35.2 | | reward | 1.1223269 | | std | 0.995 | | value_loss | 1.4 | ------------------------------------- ------------------------------------- | time/ | | | fps | 98 | | iterations | 900 | | time_elapsed | 45 | | total_timesteps | 4500 | | train/ | | | entropy_loss | -41 | | explained_variance | 0.104 | | learning_rate | 0.0007 | | n_updates | 899 | | policy_loss | -61.6 | | reward | 0.8920734 | | std | 0.995 | | value_loss | 4.01 | ------------------------------------- --------------------------------------- | time/ | | | fps | 98 | | iterations | 1000 | | time_elapsed | 50 | | total_timesteps | 5000 | | train/ | | | entropy_loss | -41 | | explained_variance | 0 | | learning_rate | 0.0007 | | n_updates | 999 | | policy_loss | 11.4 | | reward | -0.36836326 | | std | 0.993 | | value_loss | 1.83 | --------------------------------------- ------------------------------------ | time/ | | | fps | 98 | | iterations | 1100 | | time_elapsed | 55 | | total_timesteps | 5500 | | train/ | | | entropy_loss | -41 | | explained_variance | 0 | | learning_rate | 0.0007 | | n_updates | 1099 | | policy_loss | -33.7 | | reward | 3.212918 | | std | 0.994 | | value_loss | 3.55 | ------------------------------------ ---------------------------------------- | time/ | | | fps | 99 | | iterations | 1200 | | time_elapsed | 60 | | total_timesteps | 6000 | | train/ | | | entropy_loss | -40.9 | | explained_variance | 0 | | learning_rate | 0.0007 | | n_updates | 1199 | | policy_loss | -6.33 | | reward | -0.099947825 | | std | 0.993 | | value_loss | 2.43 | ---------------------------------------- ------------------------------------- | time/ | | | fps | 98 | | iterations | 1300 | | time_elapsed | 65 | | total_timesteps | 6500 | | train/ | | | entropy_loss | -41 | | explained_variance | 0.439 | | learning_rate | 0.0007 | | n_updates | 1299 | | policy_loss | -32.1 | | reward | 2.2411277 | | std | 0.994 | | value_loss | 0.802 | ------------------------------------- ------------------------------------- | time/ | | | fps | 98 | | iterations | 1400 | | time_elapsed | 70 | | total_timesteps | 7000 | | train/ | | | entropy_loss | -41 | | explained_variance | 0.136 | | learning_rate | 0.0007 | | n_updates | 1399 | | policy_loss | 105 | | reward | -1.403822 | | std | 0.994 | | value_loss | 10 | ------------------------------------- ------------------------------------ | time/ | | | fps | 98 | | iterations | 1500 | | time_elapsed | 76 | | total_timesteps | 7500 | | train/ | | | entropy_loss | -40.9 | | explained_variance | -0.1 | | learning_rate | 0.0007 | | n_updates | 1499 | | policy_loss | 179 | | reward | 1.388676 | | std | 0.993 | | value_loss | 21.9 | ------------------------------------ -------------------------------------- | time/ | | | fps | 98 | | iterations | 1600 | | time_elapsed | 81 | | total_timesteps | 8000 | | train/ | | | entropy_loss | -40.9 | | explained_variance | 0 | | learning_rate | 0.0007 | | n_updates | 1599 | | policy_loss | 122 | | reward | -1.1750767 | | std | 0.993 | | value_loss | 11.6 | -------------------------------------- ------------------------------------- | time/ | | | fps | 98 | | iterations | 1700 | | time_elapsed | 86 | | total_timesteps | 8500 | | train/ | | | entropy_loss | -40.9 | | explained_variance | 1.19e-07 | | learning_rate | 0.0007 | | n_updates | 1699 | | policy_loss | -644 | | reward | 5.2921677 | | std | 0.993 | | value_loss | 260 | ------------------------------------- ------------------------------------- | time/ | | | fps | 98 | | iterations | 1800 | | time_elapsed | 91 | | total_timesteps | 9000 | | train/ | | | entropy_loss | -40.9 | | explained_variance | -1.19e-07 | | learning_rate | 0.0007 | | n_updates | 1799 | | policy_loss | 674 | | reward | 3.8561575 | | std | 0.993 | | value_loss | 382 | ------------------------------------- ------------------------------------ | time/ | | | fps | 98 | | iterations | 1900 | | time_elapsed | 96 | | total_timesteps | 9500 | | train/ | | | entropy_loss | -41 | | explained_variance | 0.107 | | learning_rate | 0.0007 | | n_updates | 1899 | | policy_loss | 4.04 | | reward | 1.178899 | | std | 0.995 | | value_loss | 1.83 | ------------------------------------ -------------------------------------- | time/ | | | fps | 98 | | iterations | 2000 | | time_elapsed | 101 | | total_timesteps | 10000 | | train/ | | | entropy_loss | -41 | | explained_variance | 0 | | learning_rate | 0.0007 | | n_updates | 1999 | | policy_loss | 91 | | reward | -0.9507761 | | std | 0.995 | | value_loss | 5.67 | -------------------------------------- ======a2c Validation from: 2022-01-03 to 2022-04-04 a2c Sharpe Ratio: -0.14881682635553525 ======ddpg Training======== {'buffer_size': 10000, 'learning_rate': 0.0005, 'batch_size': 64} Using cuda device Logging to tensorboard_log/ddpg/ddpg_189_1 day: 3020, episode: 5 begin_total_asset: 1000000.00 end_total_asset: 4134941.46 total_reward: 3134941.46 total_cost: 6531.55 total_trades: 35218 Sharpe: 0.730 ================================= ======ddpg Validation from: 2022-01-03 to 2022-04-04 ddpg Sharpe Ratio: -0.23323576348385744 ======td3 Training======== {'batch_size': 100, 'buffer_size': 1000000, 'learning_rate': 0.0001} Using cuda device Logging to tensorboard_log/td3/td3_189_1 day: 3020, episode: 10 begin_total_asset: 1000000.00 end_total_asset: 6393593.18 total_reward: 5393593.18 total_cost: 1548.11 total_trades: 66333 Sharpe: 1.008 ================================= ======td3 Validation from: 2022-01-03 to 2022-04-04 td3 Sharpe Ratio: -0.22726474272699887 ======sac Training======== {'batch_size': 64, 'buffer_size': 100000, 'learning_rate': 0.0001, 'learning_starts': 100, 'ent_coef': 'auto_0.1'} Using cuda device Logging to tensorboard_log/sac/sac_189_1 day: 3020, episode: 15 begin_total_asset: 1000000.00 end_total_asset: 4134489.29 total_reward: 3134489.29 total_cost: 268303.08 total_trades: 69219 Sharpe: 0.757 ================================= ======sac Validation from: 2022-01-03 to 2022-04-04 sac Sharpe Ratio: -0.12984590976077562 ======ppo Training======== {'ent_coef': 0.01, 'n_steps': 2048, 'learning_rate': 0.00025, 'batch_size': 128} Using cuda device Logging to tensorboard_log/ppo/ppo_189_1 ---------------------------------- | time/ | | | fps | 102 | | iterations | 1 | | time_elapsed | 19 | | total_timesteps | 2048 | | train/ | | | reward | 0.6000615 | ---------------------------------- ----------------------------------------- | time/ | | | fps | 100 | | iterations | 2 | | time_elapsed | 40 | | total_timesteps | 4096 | | train/ | | | approx_kl | 0.015176104 | | clip_fraction | 0.199 | | clip_range | 0.2 | | entropy_loss | -41.2 | | explained_variance | 0.0105 | | learning_rate | 0.00025 | | loss | 4.55 | | n_updates | 10 | | policy_gradient_loss | -0.0282 | | reward | -0.84197414 | | std | 1 | | value_loss | 9.05 | ----------------------------------------- ----------------------------------------- | time/ | | | fps | 101 | | iterations | 3 | | time_elapsed | 60 | | total_timesteps | 6144 | | train/ | | | approx_kl | 0.010344334 | | clip_fraction | 0.12 | | clip_range | 0.2 | | entropy_loss | -41.3 | | explained_variance | -0.00571 | | learning_rate | 0.00025 | | loss | 15.5 | | n_updates | 20 | | policy_gradient_loss | -0.0171 | | reward | -0.52176756 | | std | 1 | | value_loss | 44.1 | ----------------------------------------- ----------------------------------------- | time/ | | | fps | 101 | | iterations | 4 | | time_elapsed | 80 | | total_timesteps | 8192 | | train/ | | | approx_kl | 0.013163242 | | clip_fraction | 0.146 | | clip_range | 0.2 | | entropy_loss | -41.3 | | explained_variance | -0.00859 | | learning_rate | 0.00025 | | loss | 15.3 | | n_updates | 30 | | policy_gradient_loss | -0.0224 | | reward | -2.1821563 | | std | 1 | | value_loss | 45.5 | ----------------------------------------- ----------------------------------------- | time/ | | | fps | 100 | | iterations | 5 | | time_elapsed | 101 | | total_timesteps | 10240 | | train/ | | | approx_kl | 0.014408065 | | clip_fraction | 0.187 | | clip_range | 0.2 | | entropy_loss | -41.3 | | explained_variance | -0.0553 | | learning_rate | 0.00025 | | loss | 9.32 | | n_updates | 40 | | policy_gradient_loss | -0.0217 | | reward | 0.40127692 | | std | 1.01 | | value_loss | 16.7 | ----------------------------------------- ======ppo Validation from: 2022-01-03 to 2022-04-04 ppo Sharpe Ratio: -0.10733817017427076 ======Best Model Retraining from: 2010-01-01 to 2022-04-04 ======Trading from: 2022-04-04 to 2022-07-06 ============================================ turbulence_threshold: 201.74162030011615 ======Model training from: 2010-01-01 to 2022-04-04 ======a2c Training======== {'n_steps': 5, 'ent_coef': 0.005, 'learning_rate': 0.0007} Using cuda device Logging to tensorboard_log/a2c/a2c_252_1 -------------------------------------- | time/ | | | fps | 96 | | iterations | 100 | | time_elapsed | 5 | | total_timesteps | 500 | | train/ | | | entropy_loss | -41.2 | | explained_variance | 0 | | learning_rate | 0.0007 | | n_updates | 99 | | policy_loss | -38.8 | | reward | 0.15574819 | | std | 1 | | value_loss | 1.31 | -------------------------------------- -------------------------------------- | time/ | | | fps | 91 | | iterations | 200 | | time_elapsed | 10 | | total_timesteps | 1000 | | train/ | | | entropy_loss | -41.1 | | explained_variance | -0.0432 | | learning_rate | 0.0007 | | n_updates | 199 | | policy_loss | -27.8 | | reward | 0.48102596 | | std | 0.999 | | value_loss | 2.17 | -------------------------------------- -------------------------------------- | time/ | | | fps | 94 | | iterations | 300 | | time_elapsed | 15 | | total_timesteps | 1500 | | train/ | | | entropy_loss | -41.1 | | explained_variance | -0.0275 | | learning_rate | 0.0007 | | n_updates | 299 | | policy_loss | -32.8 | | reward | -2.3948188 | | std | 1 | | value_loss | 4.81 | -------------------------------------- -------------------------------------- | time/ | | | fps | 92 | | iterations | 400 | | time_elapsed | 21 | | total_timesteps | 2000 | | train/ | | | entropy_loss | -41.1 | | explained_variance | 0.113 | | learning_rate | 0.0007 | | n_updates | 399 | | policy_loss | -105 | | reward | -0.2406286 | | std | 1 | | value_loss | 12.2 | -------------------------------------- -------------------------------------- | time/ | | | fps | 94 | | iterations | 500 | | time_elapsed | 26 | | total_timesteps | 2500 | | train/ | | | entropy_loss | -41.2 | | explained_variance | 0 | | learning_rate | 0.0007 | | n_updates | 499 | | policy_loss | -83.1 | | reward | -1.0310625 | | std | 1 | | value_loss | 9.53 | -------------------------------------- ------------------------------------ | time/ | | | fps | 93 | | iterations | 600 | | time_elapsed | 32 | | total_timesteps | 3000 | | train/ | | | entropy_loss | -41.1 | | explained_variance | 1.19e-07 | | learning_rate | 0.0007 | | n_updates | 599 | | policy_loss | -2.08 | | reward | 9.404537 | | std | 1 | | value_loss | 15.6 | ------------------------------------ ------------------------------------- | time/ | | | fps | 92 | | iterations | 700 | | time_elapsed | 37 | | total_timesteps | 3500 | | train/ | | | entropy_loss | -41.1 | | explained_variance | -0.0019 | | learning_rate | 0.0007 | | n_updates | 699 | | policy_loss | -203 | | reward | 1.8093679 | | std | 1 | | value_loss | 27 | ------------------------------------- -------------------------------------- | time/ | | | fps | 93 | | iterations | 800 | | time_elapsed | 42 | | total_timesteps | 4000 | | train/ | | | entropy_loss | -41.1 | | explained_variance | 0 | | learning_rate | 0.0007 | | n_updates | 799 | | policy_loss | -155 | | reward | 0.29696387 | | std | 1 | | value_loss | 18.6 | -------------------------------------- -------------------------------------- | time/ | | | fps | 92 | | iterations | 900 | | time_elapsed | 48 | | total_timesteps | 4500 | | train/ | | | entropy_loss | -41.1 | | explained_variance | 5.96e-08 | | learning_rate | 0.0007 | | n_updates | 899 | | policy_loss | -86.9 | | reward | -1.6688259 | | std | 1 | | value_loss | 8.07 | -------------------------------------- ------------------------------------- | time/ | | | fps | 92 | | iterations | 1000 | | time_elapsed | 54 | | total_timesteps | 5000 | | train/ | | | entropy_loss | -41.1 | | explained_variance | -2.38e-07 | | learning_rate | 0.0007 | | n_updates | 999 | | policy_loss | 109 | | reward | 3.7493455 | | std | 1 | | value_loss | 15.6 | ------------------------------------- -------------------------------------- | time/ | | | fps | 91 | | iterations | 1100 | | time_elapsed | 60 | | total_timesteps | 5500 | | train/ | | | entropy_loss | -41.1 | | explained_variance | 0 | | learning_rate | 0.0007 | | n_updates | 1099 | | policy_loss | -399 | | reward | -1.8209955 | | std | 0.999 | | value_loss | 126 | -------------------------------------- -------------------------------------- | time/ | | | fps | 92 | | iterations | 1200 | | time_elapsed | 65 | | total_timesteps | 6000 | | train/ | | | entropy_loss | -41.1 | | explained_variance | 0 | | learning_rate | 0.0007 | | n_updates | 1199 | | policy_loss | -121 | | reward | -5.2913017 | | std | 0.998 | | value_loss | 24 | -------------------------------------- --------------------------------------- | time/ | | | fps | 91 | | iterations | 1300 | | time_elapsed | 70 | | total_timesteps | 6500 | | train/ | | | entropy_loss | -41.1 | | explained_variance | 0.139 | | learning_rate | 0.0007 | | n_updates | 1299 | | policy_loss | 164 | | reward | -0.32371435 | | std | 1 | | value_loss | 16.9 | --------------------------------------- ------------------------------------- | time/ | | | fps | 92 | | iterations | 1400 | | time_elapsed | 75 | | total_timesteps | 7000 | | train/ | | | entropy_loss | -41.1 | | explained_variance | -0.0498 | | learning_rate | 0.0007 | | n_updates | 1399 | | policy_loss | -12 | | reward | 0.8689141 | | std | 1 | | value_loss | 2.3 | ------------------------------------- --------------------------------------- | time/ | | | fps | 92 | | iterations | 1500 | | time_elapsed | 80 | | total_timesteps | 7500 | | train/ | | | entropy_loss | -41.1 | | explained_variance | 0 | | learning_rate | 0.0007 | | n_updates | 1499 | | policy_loss | -72.2 | | reward | -0.54522103 | | std | 1 | | value_loss | 3.61 | --------------------------------------- -------------------------------------- | time/ | | | fps | 92 | | iterations | 1600 | | time_elapsed | 86 | | total_timesteps | 8000 | | train/ | | | entropy_loss | -41.2 | | explained_variance | -2.38e-07 | | learning_rate | 0.0007 | | n_updates | 1599 | | policy_loss | -13.1 | | reward | -1.0812243 | | std | 1 | | value_loss | 0.389 | -------------------------------------- --------------------------------------- | time/ | | | fps | 93 | | iterations | 1700 | | time_elapsed | 91 | | total_timesteps | 8500 | | train/ | | | entropy_loss | -41.2 | | explained_variance | -1.19e-07 | | learning_rate | 0.0007 | | n_updates | 1699 | | policy_loss | -86.9 | | reward | -0.64294523 | | std | 1 | | value_loss | 5.99 | --------------------------------------- -------------------------------------- | time/ | | | fps | 92 | | iterations | 1800 | | time_elapsed | 97 | | total_timesteps | 9000 | | train/ | | | entropy_loss | -41.2 | | explained_variance | -1.19e-07 | | learning_rate | 0.0007 | | n_updates | 1799 | | policy_loss | 413 | | reward | 0.50728357 | | std | 1 | | value_loss | 127 | -------------------------------------- --------------------------------------- | time/ | | | fps | 93 | | iterations | 1900 | | time_elapsed | 102 | | total_timesteps | 9500 | | train/ | | | entropy_loss | -41.2 | | explained_variance | 0 | | learning_rate | 0.0007 | | n_updates | 1899 | | policy_loss | -55.7 | | reward | -0.07039916 | | std | 1 | | value_loss | 2.31 | --------------------------------------- -------------------------------------- | time/ | | | fps | 92 | | iterations | 2000 | | time_elapsed | 107 | | total_timesteps | 10000 | | train/ | | | entropy_loss | -41.2 | | explained_variance | 0 | | learning_rate | 0.0007 | | n_updates | 1999 | | policy_loss | -22.2 | | reward | -0.5126278 | | std | 1 | | value_loss | 0.961 | -------------------------------------- ======a2c Validation from: 2022-04-04 to 2022-07-06 a2c Sharpe Ratio: -0.25366636627181594 ======ddpg Training======== {'buffer_size': 10000, 'learning_rate': 0.0005, 'batch_size': 64} Using cuda device Logging to tensorboard_log/ddpg/ddpg_252_1 day: 3083, episode: 5 begin_total_asset: 1000000.00 end_total_asset: 5721518.02 total_reward: 4721518.02 total_cost: 6164.84 total_trades: 44663 Sharpe: 0.868 ================================= ======ddpg Validation from: 2022-04-04 to 2022-07-06 ddpg Sharpe Ratio: -0.22747056221011977 ======td3 Training======== {'batch_size': 100, 'buffer_size': 1000000, 'learning_rate': 0.0001} Using cuda device Logging to tensorboard_log/td3/td3_252_1 day: 3083, episode: 10 begin_total_asset: 1000000.00 end_total_asset: 4419882.67 total_reward: 3419882.67 total_cost: 1787.44 total_trades: 46447 Sharpe: 0.822 ================================= ======td3 Validation from: 2022-04-04 to 2022-07-06 td3 Sharpe Ratio: -0.2839424978995499 ======sac Training======== {'batch_size': 64, 'buffer_size': 100000, 'learning_rate': 0.0001, 'learning_starts': 100, 'ent_coef': 'auto_0.1'} Using cuda device Logging to tensorboard_log/sac/sac_252_1 day: 3083, episode: 15 begin_total_asset: 1000000.00 end_total_asset: 4927709.27 total_reward: 3927709.27 total_cost: 85026.93 total_trades: 55788 Sharpe: 0.808 ================================= ======sac Validation from: 2022-04-04 to 2022-07-06 sac Sharpe Ratio: -0.16022717745791382 ======ppo Training======== {'ent_coef': 0.01, 'n_steps': 2048, 'learning_rate': 0.00025, 'batch_size': 128} Using cuda device Logging to tensorboard_log/ppo/ppo_252_1 ----------------------------------- | time/ | | | fps | 105 | | iterations | 1 | | time_elapsed | 19 | | total_timesteps | 2048 | | train/ | | | reward | 0.54891366 | ----------------------------------- ----------------------------------------- | time/ | | | fps | 103 | | iterations | 2 | | time_elapsed | 39 | | total_timesteps | 4096 | | train/ | | | approx_kl | 0.016892547 | | clip_fraction | 0.209 | | clip_range | 0.2 | | entropy_loss | -41.2 | | explained_variance | -0.0182 | | learning_rate | 0.00025 | | loss | 5.17 | | n_updates | 10 | | policy_gradient_loss | -0.0246 | | reward | -0.46093306 | | std | 1 | | value_loss | 10.4 | ----------------------------------------- ----------------------------------------- | time/ | | | fps | 104 | | iterations | 3 | | time_elapsed | 58 | | total_timesteps | 6144 | | train/ | | | approx_kl | 0.013119971 | | clip_fraction | 0.109 | | clip_range | 0.2 | | entropy_loss | -41.2 | | explained_variance | -0.000288 | | learning_rate | 0.00025 | | loss | 28.6 | | n_updates | 20 | | policy_gradient_loss | -0.0202 | | reward | -9.113171 | | std | 1 | | value_loss | 57.9 | ----------------------------------------- ------------------------------------------ | time/ | | | fps | 102 | | iterations | 4 | | time_elapsed | 79 | | total_timesteps | 8192 | | train/ | | | approx_kl | 0.0148056755 | | clip_fraction | 0.147 | | clip_range | 0.2 | | entropy_loss | -41.3 | | explained_variance | -0.0199 | | learning_rate | 0.00025 | | loss | 47.2 | | n_updates | 30 | | policy_gradient_loss | -0.0206 | | reward | -0.5656307 | | std | 1.01 | | value_loss | 78.6 | ------------------------------------------ ----------------------------------------- | time/ | | | fps | 102 | | iterations | 5 | | time_elapsed | 99 | | total_timesteps | 10240 | | train/ | | | approx_kl | 0.027241705 | | clip_fraction | 0.268 | | clip_range | 0.2 | | entropy_loss | -41.3 | | explained_variance | -0.00487 | | learning_rate | 0.00025 | | loss | 8.49 | | n_updates | 40 | | policy_gradient_loss | -0.0307 | | reward | -0.5270617 | | std | 1.01 | | value_loss | 17.2 | ----------------------------------------- ======ppo Validation from: 2022-04-04 to 2022-07-06 ppo Sharpe Ratio: -0.23906732813732043 ======Best Model Retraining from: 2010-01-01 to 2022-07-06 ======Trading from: 2022-07-06 to 2022-10-04 ============================================ turbulence_threshold: 201.74162030011615 ======Model training from: 2010-01-01 to 2022-07-06 ======a2c Training======== {'n_steps': 5, 'ent_coef': 0.005, 'learning_rate': 0.0007} Using cuda device Logging to tensorboard_log/a2c/a2c_315_1 -------------------------------------- | time/ | | | fps | 102 | | iterations | 100 | | time_elapsed | 4 | | total_timesteps | 500 | | train/ | | | entropy_loss | -41.2 | | explained_variance | 0.0353 | | learning_rate | 0.0007 | | n_updates | 99 | | policy_loss | -80 | | reward | 0.24549441 | | std | 1 | | value_loss | 4.92 | -------------------------------------- ------------------------------------- | time/ | | | fps | 97 | | iterations | 200 | | time_elapsed | 10 | | total_timesteps | 1000 | | train/ | | | entropy_loss | -41.1 | | explained_variance | -0.126 | | learning_rate | 0.0007 | | n_updates | 199 | | policy_loss | -17.6 | | reward | 1.3894979 | | std | 1 | | value_loss | 3.75 | ------------------------------------- ------------------------------------- | time/ | | | fps | 96 | | iterations | 300 | | time_elapsed | 15 | | total_timesteps | 1500 | | train/ | | | entropy_loss | -41.2 | | explained_variance | -0.012 | | learning_rate | 0.0007 | | n_updates | 299 | | policy_loss | -91.5 | | reward | -2.606353 | | std | 1 | | value_loss | 8.84 | ------------------------------------- ------------------------------------- | time/ | | | fps | 97 | | iterations | 400 | | time_elapsed | 20 | | total_timesteps | 2000 | | train/ | | | entropy_loss | -41.2 | | explained_variance | -0.183 | | learning_rate | 0.0007 | | n_updates | 399 | | policy_loss | -10.2 | | reward | 1.6483078 | | std | 1 | | value_loss | 4.13 | ------------------------------------- --------------------------------------- | time/ | | | fps | 95 | | iterations | 500 | | time_elapsed | 26 | | total_timesteps | 2500 | | train/ | | | entropy_loss | -41.2 | | explained_variance | 0.0521 | | learning_rate | 0.0007 | | n_updates | 499 | | policy_loss | -51.2 | | reward | -0.46745828 | | std | 1 | | value_loss | 2.93 | --------------------------------------- ------------------------------------- | time/ | | | fps | 96 | | iterations | 600 | | time_elapsed | 31 | | total_timesteps | 3000 | | train/ | | | entropy_loss | -41.2 | | explained_variance | 0.0325 | | learning_rate | 0.0007 | | n_updates | 599 | | policy_loss | -2.87 | | reward | 10.919484 | | std | 1 | | value_loss | 8.18 | ------------------------------------- ------------------------------------- | time/ | | | fps | 94 | | iterations | 700 | | time_elapsed | 36 | | total_timesteps | 3500 | | train/ | | | entropy_loss | -41.3 | | explained_variance | 0.0171 | | learning_rate | 0.0007 | | n_updates | 699 | | policy_loss | -27.7 | | reward | 0.8419629 | | std | 1.01 | | value_loss | 0.606 | ------------------------------------- --------------------------------------- | time/ | | | fps | 95 | | iterations | 800 | | time_elapsed | 41 | | total_timesteps | 4000 | | train/ | | | entropy_loss | -41.4 | | explained_variance | -1.19e-07 | | learning_rate | 0.0007 | | n_updates | 799 | | policy_loss | -27.5 | | reward | -0.18779811 | | std | 1.01 | | value_loss | 0.786 | --------------------------------------- -------------------------------------- | time/ | | | fps | 93 | | iterations | 900 | | time_elapsed | 47 | | total_timesteps | 4500 | | train/ | | | entropy_loss | -41.3 | | explained_variance | 1.19e-07 | | learning_rate | 0.0007 | | n_updates | 899 | | policy_loss | -23.3 | | reward | 0.21865338 | | std | 1.01 | | value_loss | 1.68 | -------------------------------------- -------------------------------------- | time/ | | | fps | 94 | | iterations | 1000 | | time_elapsed | 52 | | total_timesteps | 5000 | | train/ | | | entropy_loss | -41.3 | | explained_variance | 1.19e-07 | | learning_rate | 0.0007 | | n_updates | 999 | | policy_loss | 30.7 | | reward | -0.8657269 | | std | 1.01 | | value_loss | 0.856 | -------------------------------------- ------------------------------------- | time/ | | | fps | 95 | | iterations | 1100 | | time_elapsed | 57 | | total_timesteps | 5500 | | train/ | | | entropy_loss | -41.4 | | explained_variance | 5.96e-08 | | learning_rate | 0.0007 | | n_updates | 1099 | | policy_loss | 75.6 | | reward | 3.6971135 | | std | 1.01 | | value_loss | 7.44 | ------------------------------------- ------------------------------------- | time/ | | | fps | 94 | | iterations | 1200 | | time_elapsed | 63 | | total_timesteps | 6000 | | train/ | | | entropy_loss | -41.3 | | explained_variance | 0 | | learning_rate | 0.0007 | | n_updates | 1199 | | policy_loss | -237 | | reward | 0.9042094 | | std | 1.01 | | value_loss | 34.5 | ------------------------------------- -------------------------------------- | time/ | | | fps | 95 | | iterations | 1300 | | time_elapsed | 68 | | total_timesteps | 6500 | | train/ | | | entropy_loss | -41.4 | | explained_variance | -0.503 | | learning_rate | 0.0007 | | n_updates | 1299 | | policy_loss | 58.3 | | reward | 0.20865634 | | std | 1.01 | | value_loss | 2.1 | -------------------------------------- -------------------------------------- | time/ | | | fps | 94 | | iterations | 1400 | | time_elapsed | 74 | | total_timesteps | 7000 | | train/ | | | entropy_loss | -41.3 | | explained_variance | -1.19e-07 | | learning_rate | 0.0007 | | n_updates | 1399 | | policy_loss | -72.9 | | reward | -1.3368342 | | std | 1.01 | | value_loss | 4.41 | -------------------------------------- ------------------------------------- | time/ | | | fps | 95 | | iterations | 1500 | | time_elapsed | 78 | | total_timesteps | 7500 | | train/ | | | entropy_loss | -41.4 | | explained_variance | 0 | | learning_rate | 0.0007 | | n_updates | 1499 | | policy_loss | -144 | | reward | 3.5003781 | | std | 1.01 | | value_loss | 14.4 | ------------------------------------- ------------------------------------- | time/ | | | fps | 94 | | iterations | 1600 | | time_elapsed | 84 | | total_timesteps | 8000 | | train/ | | | entropy_loss | -41.4 | | explained_variance | 1.19e-07 | | learning_rate | 0.0007 | | n_updates | 1599 | | policy_loss | -38.2 | | reward | 1.3683945 | | std | 1.01 | | value_loss | 1.89 | ------------------------------------- ------------------------------------- | time/ | | | fps | 95 | | iterations | 1700 | | time_elapsed | 89 | | total_timesteps | 8500 | | train/ | | | entropy_loss | -41.4 | | explained_variance | 0 | | learning_rate | 0.0007 | | n_updates | 1699 | | policy_loss | 48.4 | | reward | 1.4903697 | | std | 1.01 | | value_loss | 4.15 | ------------------------------------- -------------------------------------- | time/ | | | fps | 95 | | iterations | 1800 | | time_elapsed | 94 | | total_timesteps | 9000 | | train/ | | | entropy_loss | -41.4 | | explained_variance | -1.19e-07 | | learning_rate | 0.0007 | | n_updates | 1799 | | policy_loss | -31.4 | | reward | -1.6481568 | | std | 1.01 | | value_loss | 17.5 | -------------------------------------- --------------------------------------- | time/ | | | fps | 95 | | iterations | 1900 | | time_elapsed | 99 | | total_timesteps | 9500 | | train/ | | | entropy_loss | -41.4 | | explained_variance | 0 | | learning_rate | 0.0007 | | n_updates | 1899 | | policy_loss | 27.9 | | reward | -0.39915258 | | std | 1.01 | | value_loss | 0.507 | --------------------------------------- -------------------------------------- | time/ | | | fps | 95 | | iterations | 2000 | | time_elapsed | 104 | | total_timesteps | 10000 | | train/ | | | entropy_loss | -41.3 | | explained_variance | 0.159 | | learning_rate | 0.0007 | | n_updates | 1999 | | policy_loss | 58.6 | | reward | -1.4694927 | | std | 1.01 | | value_loss | 2.85 | -------------------------------------- ======a2c Validation from: 2022-07-06 to 2022-10-04 a2c Sharpe Ratio: -0.10266785475978492 ======ddpg Training======== {'buffer_size': 10000, 'learning_rate': 0.0005, 'batch_size': 64} Using cuda device Logging to tensorboard_log/ddpg/ddpg_315_1 day: 3146, episode: 5 begin_total_asset: 1000000.00 end_total_asset: 8621242.15 total_reward: 7621242.15 total_cost: 7912.99 total_trades: 39562 Sharpe: 1.023 ================================= ======ddpg Validation from: 2022-07-06 to 2022-10-04 ddpg Sharpe Ratio: -0.06187703782204383 ======td3 Training======== {'batch_size': 100, 'buffer_size': 1000000, 'learning_rate': 0.0001} Using cuda device Logging to tensorboard_log/td3/td3_315_1 day: 3146, episode: 10 begin_total_asset: 1000000.00 end_total_asset: 4437212.38 total_reward: 3437212.38 total_cost: 2996.04 total_trades: 31813 Sharpe: 0.776 ================================= ======td3 Validation from: 2022-07-06 to 2022-10-04 td3 Sharpe Ratio: -0.12530693561038414 ======sac Training======== {'batch_size': 64, 'buffer_size': 100000, 'learning_rate': 0.0001, 'learning_starts': 100, 'ent_coef': 'auto_0.1'} Using cuda device Logging to tensorboard_log/sac/sac_315_1 day: 3146, episode: 15 begin_total_asset: 1000000.00 end_total_asset: 3601294.13 total_reward: 2601294.13 total_cost: 234860.64 total_trades: 64468 Sharpe: 0.695 ================================= ======sac Validation from: 2022-07-06 to 2022-10-04 sac Sharpe Ratio: -0.16088947893289524 ======ppo Training======== {'ent_coef': 0.01, 'n_steps': 2048, 'learning_rate': 0.00025, 'batch_size': 128} Using cuda device Logging to tensorboard_log/ppo/ppo_315_1 ---------------------------------- | time/ | | | fps | 105 | | iterations | 1 | | time_elapsed | 19 | | total_timesteps | 2048 | | train/ | | | reward | 1.6173023 | ---------------------------------- ----------------------------------------- | time/ | | | fps | 103 | | iterations | 2 | | time_elapsed | 39 | | total_timesteps | 4096 | | train/ | | | approx_kl | 0.015555255 | | clip_fraction | 0.228 | | clip_range | 0.2 | | entropy_loss | -41.2 | | explained_variance | -0.0101 | | learning_rate | 0.00025 | | loss | 6.3 | | n_updates | 10 | | policy_gradient_loss | -0.0255 | | reward | 3.5431957 | | std | 1 | | value_loss | 13.2 | ----------------------------------------- ----------------------------------------- | time/ | | | fps | 104 | | iterations | 3 | | time_elapsed | 58 | | total_timesteps | 6144 | | train/ | | | approx_kl | 0.015205141 | | clip_fraction | 0.167 | | clip_range | 0.2 | | entropy_loss | -41.2 | | explained_variance | 0.00121 | | learning_rate | 0.00025 | | loss | 24.5 | | n_updates | 20 | | policy_gradient_loss | -0.0202 | | reward | 5.9619627 | | std | 1 | | value_loss | 81.8 | ----------------------------------------- ---------------------------------------- | time/ | | | fps | 103 | | iterations | 4 | | time_elapsed | 78 | | total_timesteps | 8192 | | train/ | | | approx_kl | 0.01795656 | | clip_fraction | 0.154 | | clip_range | 0.2 | | entropy_loss | -41.3 | | explained_variance | -0.00243 | | learning_rate | 0.00025 | | loss | 31.6 | | n_updates | 30 | | policy_gradient_loss | -0.0196 | | reward | 1.3367934 | | std | 1.01 | | value_loss | 49.2 | ---------------------------------------- 
df_summary

Part 7: Backtest Our Strategy

Backtesting plays a key role in evaluating the performance of a trading strategy. Automated backtesting tool is preferred because it reduces the human error. We usually use the Quantopian pyfolio package to backtest our trading strategies. It is easy to use and consists of various individual plots that provide a comprehensive image of the performance of a trading strategy.

unique_trade_date = processed[(processed.date > TEST_START_DATE)&(processed.date <= TEST_END_DATE)].date.unique()

df_trade_date = pd.DataFrame({'datadate':unique_trade_date})

df_account_value=pd.DataFrame()
for i in range(rebalance_window+validation_window, len(unique_trade_date)+1,rebalance_window):
    temp = pd.read_csv('results/account_value_trade_{}_{}.csv'.format('ensemble',i))
    df_account_value = df_account_value.append(temp,ignore_index=True)
sharpe=(252**0.5)*df_account_value.account_value.pct_change(1).mean()/df_account_value.account_value.pct_change(1).std()
print('Sharpe Ratio: ',sharpe)
df_account_value=df_account_value.join(df_trade_date[validation_window:].reset_index(drop=True))

df_account_value.head()

%matplotlib inline
df_account_value.account_value.plot()

7.1 BackTestStats

pass in df_account_value, this information is stored in env class

print("==============Get Backtest Results===========")
now = datetime.datetime.now().strftime('%Y%m%d-%Hh%M')

perf_stats_all = backtest_stats(account_value=df_account_value)
perf_stats_all = pd.DataFrame(perf_stats_all)

#baseline stats
print("==============Get Baseline Stats===========")
df_dji_ = get_baseline(
        ticker="^DJI",
        start = df_account_value.loc[0,'date'],
        end = df_account_value.loc[len(df_account_value)-1,'date'])

stats = backtest_stats(df_dji_, value_col_name = 'close')

df_dji = pd.DataFrame()
df_dji['date'] = df_account_value['date']
df_dji['dji'] = df_dji_['close'] / df_dji_['close'][0] * env_kwargs["initial_amount"]
print("df_dji: ", df_dji)
df_dji.to_csv("df_dji.csv")
df_dji = df_dji.set_index(df_dji.columns[0])
print("df_dji: ", df_dji)
df_dji.to_csv("df_dji+.csv")

df_account_value.to_csv('df_account_value.csv')

7.2 BackTestPlot


# print("==============Compare to DJIA===========")
# %matplotlib inline
# # S&P 500: ^GSPC
# # Dow Jones Index: ^DJI
# # NASDAQ 100: ^NDX
# backtest_plot(df_account_value,
#               baseline_ticker = '^DJI',
#               baseline_start = df_account_value.loc[0,'date'],
#               baseline_end = df_account_value.loc[len(df_account_value)-1,'date'])
df.to_csv("df.csv")
df_result_ensemble = pd.DataFrame({'date': df_account_value['date'], 'ensemble':df_account_value['account_value']})
df_result_ensemble = df_result_ensemble.set_index('date')

print("df_result_ensemble.columns: ", df_result_ensemble.columns)

print("df_trade_date: ", df_trade_date)
# df_result_ensemble['date'] = df_trade_date['datadate']
# df_result_ensemble['account_value'] = df_account_value['account_value']
df_result_ensemble.to_csv("df_result_ensemble.csv")
print("df_result_ensemble: ", df_result_ensemble)
print("==============Compare to DJIA===========")
result = pd.DataFrame()
# result = pd.merge(result, df_result_ensemble, left_index=True, right_index=True)

result = pd.merge(df_result_ensemble, df_dji, left_index=True, right_index=True)
print("result: ", result)
result.to_csv("result.csv")
result.columns = ['ensemble', 'dji']

%matplotlib inline
plt.rcParams["figure.figsize"] = (15, 5)
plt.figure()
result.plot()