Optimal Portfolio and Trading Strategy Using Machine Learning

1. Introduction

Investing can be generally viewed as a method of purchasing a portfolio of assets to gain predictable returns over a certain period. But Random Walk Theory [18] proclaims that the trend of a stock price cannot be predictable based on its past movement. All methods of predicting stock prices are futile in the long run. Another similar theory is the Efficient Market Hypothesis (EMH). EMH postulates that the stock market is efficient in reflecting the true value of the stock with the assumption that all relevant information is freely and widely available to investors. The higher the return, the higher the risk. However, [17] develops an adaptive market hypothesis that the risk premium varies over time and arbitrage opportunities do exist from time to time. On the other hand, EMH is also challenged by momentum investors. [24] mentions successful investors, such as George Soros and Warren Buffett, could use numerical and systematic approaches to beat the market. The common methods to sort and pick assets are fundamental analysis and technical analysis. Fundamental analysts seek to determine an asset’s proper value whereas technical analysts study stock charts to predict the direction of the future trend. With the advent of machine learning, a hybrid model [12] & [9] of applying machine learning to stock price forecasting using financial indicators (financial ratios and technical indicators) is prevalent. This paper presents how to create an optimal portfolio from Nasdaq semiconductors assets using fundamental analysis and form a trading strategy using technical analysis during the period from 2011 to 2018. For technical analysis, the forecast horizon is also considered. There are 5 different forecast horizons (1-day, 5-day, 7-day, 10-day, and 15-day). The portfolio contains 5 assets and is optimized by Monte Carlo simulation. The portfolio’s performance with the specified strategy is assessed by back-testing in 2019 and compared with the buy-and-hold strategy and portfolio without optimization strategy. The rest of this paper is organized as follows: Section 2 reviews the techniques and previous work in using fundamental analysis and technical analysis with machine learning algorithms for stock price forecasting; Section 3 describes the framework and flow modelling; Section 4 implements the models on the framework and presents the results; Section 5 evaluates the performance of the models; The conclusions and future work are presented in Section 6.

2. Related Works

The commonly used methods in forecasting trends and turning points in the stock market are fundamental analysis and technical analysis. Fundamental analysis is significant for 1 year and beyond, with longer time horizons, whereas technical analysis is significant for intraday, 1 week to 3 months trading [3]. Fundamental analysis generally uses financial ratios as predictors to forecast equity returns. The studies [5]; [15]; [21]; [20]; find that the price-to-earnings (PE) ratio causes a statistically significant impact on future stock prices. But [16] found that none of the financial ratios affected the stock prices of the stock exchange of Switzerland in 2015. Another finding by [4] is that some financial ratios could give strong positive and significant relationships to stock price behaviour and trends for the years 2005-2014 in the Kuwaiti financial market. A lot of articles show that applying machine learning approaches with financial ratios could improve the quality of forecasting stock price returns. [1] indicate that the prediction using the Logistic regression model with financial ratios would be up to 89.77 % accurate for the prediction of good and bad performance of stocks. [7] applies the multiple regression analysis with monthly financial ratios of four agriculture companies on the stock exchange of Thailand from June 2005 to June 2015 to show that the current ratios, net profit margin ratio, and total assets turnover ratio positively affect stock prices at the statistical significance level of 0.01 and debt to equity ratio negatively affects stock prices at the statistical significance level of 0.01. [6] conclude 7 machine learning techniques (Random Forest, AdaBoost, Kernel Factory, Neural Networks, Logistic Regression, Support Vector Machines, and K-Nearest Neighbour) with financial ratios of 5767 publicly listed European companies for stock price directions prediction one year ahead and find that Random Forest is the top performer. [20] also compares different machine learning techniques (decision trees, Support Vector Machines with Sequential Minimal Optimization, Random Trees, Random Forest, Logistic regression, Naïve Bayes and Bayesian Networks) and finds that Random Forest is the best performer with the highest F-Score of 0.751 [28]. On the other hand, the application of artificial neural network models and principal component analysis methods with 12 financial ratios plus 8 other financial variables can accurately predict stock prices on the Tehran Stock Exchange over a period from 2006 to March 2012 [22]. Regarding technical analysis, the technical indicators come from both the past movement of stock prices and the volume of trading to predict the future direction of stock prices. The direction provides the timing of buying or selling stocks to investors to make trading strategies. [14] compare 5 different supervised learning techniques (Support Vector Machine, Random Forest, K-Nearest Neighbor, Naïve Bayes, and Softmax) and find that the random forest algorithm performs the best for large datasets and Naïve Bayesian classifier is the best for small datasets. However, the accuracy lies between 50% and 70%. To increase the accuracy, hybrid models combining 2 or more learning techniques and deep learning are going to be drawn. [12] adopted 3 hybrid approaches (SVR with hierarchical clustering, SVR with principal component analysis, and SVR with genetic algorithms) for the prediction of the Shanghai-Shenzhen 300 index and found that SVR-HC outperforms. [2] use a Fuzzy Metagraph (FM) to classify and predict the prices of stocks listed on the Bombay Stock Exchange and have a satisfactory result with a very low-risk error [19]. With technical indicators as predictors, [13] compare 6 different models (Bat algorithm with extreme gradient boosting, random forest, extreme gradient boosting, linear regression, support vector machine, and artificial neural networks) for classifying the prices of Facebook and Apple stocks and obtain that BA-XGB outperforms with a maximum accuracy of 0.96.

3. Framework Design

The framework for creating an optimal portfolio and a trading strategy is shown in Figure 1. It consists of 4 parts: generate a portfolio (top 3 boxes); optimise the portfolio (on the right 3 boxes); form a trading strategy (2 grey boxes); and perform simulations of trading (2 green boxes).

The financial ratios used in this project come from the companies’ annual financial statements and their list is shown In Table 1 [8]. The regression models for predicting expected returns are depicted in Table 2.

When a portfolio of the 5 top highest-return assets is formed, optimizing the portfolio with the maximum risk-adjusted return by the Monte Carlo method is applied. The objective is to determine the weighting of each asset by getting the maximum value of the Sharpe ratio under a given level of risk. Regarding the grey boxes in Figure 1, the description of the technical indicators adopted is listed in Table 3 and the processes for predicting trading signals are shown in Figure 2. Training/ Validation data are the historical data from 2011 to 2018, whereas the test data is the transaction records in 2019 in the Nasdaq market. The cross-validation is based on the method shown in section 7 of Analytics Vidhya [11]. For each asset, there are 252 samples for 1 year and therefore we have 2,016 samples of training/validation data. We use 10 splits for each modelling and the ratio of training to validation is 7 to 3. The grid search classifications include random forest (RF), K-nearest Neighbours (KNN), Adaboost (ABT), gradient boosting (GBT), Support Vector Machine (SVM), and ensemble.

When a portfolio of the 5 top highest-return assets is formed, optimizing the portfolio with the maximum risk-adjusted return by the Monte Carlo method is applied. The objective is to determine the weighting of each asset by getting the maximum value of the Sharpe ratio under a given level of risk. Regarding the grey boxes in Figure 1, the description of the technical indicators adopted is listed in Table 3 and the processes for predicting trading signals are shown in Figure 2. Training/validation data are historical data from 2011 to 2018, whereas the test data is the transaction records from 2019 in the Nasdaq market. The cross-validation is based on the method shown in section 7 of Analytics Vidhya [11]. For each asset, there are 252 samples for 1 year and therefore we have 2,016 samples of training/validation data. We use 10 splits for each modelling and the ratio of training to validation is 7 to 3. The grid search classifications include random forest (RF), K-nearest Neighbours (KNN), Adaboost (ABT), gradient boosting (GBT), Support Vector Machine (SVM), and ensemble [27].

Back-testing is the simulation of portfolio trading in 2019 based on the predicted trading signals. There are 2 labels of trading signals: 1 indicates ‘buy’ and 0 indicates ‘sell’. Initially, a principle will be divided into 5 portions for the portfolio assets according to the optimization result. We will do one trade per trading day and each day in 2019. The trading price is the mean value of the highest and lowest price for convenient calculation. No commission, dividend, and other fees are involved. A faction of shares and short selling are not allowed. For each buy action, the invested money is less than 1 quarter of the portion principle. For each sell action, normally we will sell half of the stocks in hand unless there is the last share. The total trading days in 2019 are 252. Summing up the daily returns is the net result. The net return is the annual return for that year. We shall compare the results with the buy-and-hold strategy and the trading with equal weighting of the portfolio assets [29].

4. Implementation and Results

4.1. Creating a Portfolio

Explanatory features are the annual financial ratios and the response variable is the daily returns. The greater the number of features, the higher the possibility of degrading the performance of machine learning. In this project, principal component analysis (PCA) is the technique for dimensionality reduction. The results of PCA are shown in Figure 3. The screen plot shows that the first 8 components can explain almost 95% of the variance. The features adopted are ‘roa’, ‘roe’, ‘net_profit_margin’, ‘current ratio’, ‘quick ratio’, ‘debt ratio’, ‘debt to equity’ and ‘price to book’.

With the selected features, the data are standardized and then fed to 20 common regression models with the predictive performance shown in Figure 4. The minimum errors of models from different categories are chosen for grid search to get the optimal parameters. The 6 chosen models are ANN, SVR, Bayesian ridge regression, ridge regression, GBR, and XGBR. The predictive and the actual top 5 highest return assets are shown in Figure 5. Based on the minimum error, we choose GBR prediction. As a result, the portfolio contains AMD, ASML, QCOM, KLAC, and NVDA [30].

4.2. Portfolio Optimisation

The optimisation method adopted is called Monte Carlo simulation. A random generation of 10,000 combinations of weights for the 5 assets, the expected return of the portfolio can be determined by taking the mean value of the product of each asset weighting and its average yearly returns from 2011 to 2018. The distribution of different expected returns and expected volatility (standard deviation) is shown in Figure 6. The red dot in Figure 6 shows the maximum value of the Sharpe ratio. The coordinates of the red dot are 0.3134 and 0.4613. With this expected return, the weighting of each asset is 0.118075 (AMD), 0.291511 (ASML), 0.036439 (KLAC), 0.543872 (NVDA), and 0.010103 (QCOM).

4.3. Trading Signals Prediction

Technical indicators are explanatory features movement of stock prices in the response variable. The response variable has 2 labels: 1 (positive daily return) and 0 (negative daily return). There are 5 different forecast horizons (1-day, 5-day, 7-day, 10-day, and 15-day). As technical analysis aims to predict short-term price fluctuations, it had better to keep the forecast horizon to be less than 4 weeks (20 trading days). Therefore, it is appropriate to set the maximum forecast horizon to 15 days.

4.4. Feature Selection

To avoid degrading the machine learning predictive performance, some features without any correlation with the response variable may be deleted. We adopt PCA to reduce the dimensionality. The results of 6 features with the maximum variation (PC1) of PCA for the 5 assets are shown in Figure 7. Almost 90% variance can be explained by 6 principal components.

After feature reduction, the explanatory features undergo standardization and time series cross-validation (referred to in Section 3 of this report) before modelling.

4.5. Modelling

Six different machine learning techniques with grid search are applied to predict the stock price movement. The accuracy of different models of each asset under different forecast horizons for the validation dataset is shown in Figure 8. The blue cells indicate the highest accuracy for each asset.

According to the results of Figure 8, we choose AMD technical indicators with forecast horizons of 15-day, ASML technical indicators with forecast horizons of 10-day, KLAC technical indicators with forecast horizons of 15-day, NVDA technical indicators with forecast horizons of 15-day, and QCOM technical indicators with forecast horizons of 5-day as training data to predict the trading signals in 2019. The confusion matrices of the test dataset of the 5 assets are shown in Figure 9.

4.6. Simulation of Trading

With weighting obtained in Section 4.2 and the predicted signals in Section 4.3, we simulate trading in 2019. Figure 10 plots the simulation of trading. The net return is 0.406312 or 50.13%.

5. Evaluation of Performances and Limitations

5.1. Expected Returns by Machine Learning

We recall Figure 5, the root-mean-square error seems ineffective in assessing the predictive performance of asset returns because the ANN model has the largest error. But this model correctly predicts 4 out of 5 highest-returns assets. However, each model can predict AMD as one of the highest-return assets.

5.2. Optimisation

We recall Figure 6, the expected return of the portfolio is 0.4613, whereas Figure 10 shows 0.5013, with a difference of 8.7%.

5.3. Prediction of Trading Signal

When we use the results of Figure 9, the accuracy of prediction is determined as 0.66 (AMD), 0.74 (ASML), 0.58 (KLAC), 0.72 (NVDA), and 0.61 (QCOM). The precision values are calculated as 0.68 (AMD), 0.75 (ASML), 0.78 (KLAC), 0.73 (NVDA), and 0.61 (QCOM). For recall, the values are 0.96 (AMD), 0.98 (ASML), 0.67 (KLAC), 0.97 (NVDA), and 1.00 (QCOM).

5.4. Simulation

When we add buy-and-hold gain and equal-weight portfolio gain curves to Figure 10, the result is shown in Figure 11. It is observed that the buy-and-hold strategy outperforms, and the performance of the equal-weight portfolio is the worst. The net return of the buy-and-hold and equal-weight portfolio is 0.528 or 69.6% and 0.342 or 40.8% respectively. When the prediction of trading signals is correct, the daily gain of the curves is the same. When the prediction is incorrect the curves will move differently. As the prediction is about 60% correct, the curves are moving in the same pattern for around 60% period.

5.5. Limitations

Both financial ratios and technical indicators rely on historical data which may not necessarily reflect future performance. Past trends may not always recur. The selection of different combinations of ratios and indicators may have varying degrees of influence on stock prices. However, this study does not account for these differences or assign specific weights to them. Changes in accounting policies and procedures can influence the calculation of these ratios. It is unclear whether the companies in this study maintained consistent accounting policies from 2011 to 2019. Additionally, factors like government policies and interest rates can significantly affect investor sentiment in the stock market and finally affect the stock prices [35,36].

6. Conclusion

This research paper describes the ways to apply machine learning for predicting expected returns of assets that originate from the NASDAQ semiconductor industry using financial ratios as explanatory features, to optimize a portfolio that consists of 5 assets by using Monte Carlo simulation, to build classifiers for predicting stock price movements (up or down) using technical indicators as explanatory features, and to simulate trading in 2019 using our developed optimal portfolio and trading rules. Finally, we compare the trading performance of our optimal portfolio with the buy-and-hold strategy and the portfolio without optimization. The result shows that our portfolio has a gain of 50.1% whereas the buy-and-hold strategy has a gain of 69.6% and the portfolio without optimization has a gain of 40.8%. Although our portfolio cannot beat the benchmark, the result is quite encouraging (50.1% profit acceptable). In addition, the gain is higher than the expected return determined by optimisation. Feeding financial ratios into 6 different types of machine learning models for predicting the expected return of assets is implemented. The grid search for hyperparameter tuning is applied while modelling. The performance of the GBR technique is the best based on minimum errors [31,32]. If the purpose is to select the top 5 assets from all the Nasdaq semiconductor assets based on the predicted expected returns, the correctness rating of the models is at least 20% correct. That is not good but acceptable. It is likely that financial ratios can project the expected returns to a certain extent. When Monte Carlo simulation is applied to find the best asset allocation, a graph of efficient frontier is formed. When comparing the trading performance of the optimal portfolio with the equal-weight portfolio by back-testing, the optimal portfolio outperforms. Diversification is important to investing. Feeding technical indicators to 6 different types of modelling techniques is used to classify the stock price movement. Cross-validation and grid search are applied to reduce the effect of overfitting. The result shows that there is no indication of a relationship between the level of accuracy and forecast horizon. Evaluating model performance with the confusion matrix, accuracy ranges from 56% to 64%. These values alone do not indicate whether this investment strategy based on these predictions would be effective. However, the precision and recall values range from 0.61 to 0.78 and 0.67 to 1.00, respectively, suggesting that the predictions tend to favour price increases over decreases. Only back-testing is an effective way to evaluate the trading strategy. The simulation shows the net return would be 50.13% using this predicted strategy [25]. However, our portfolio cannot the buy-and-hold trading strategy. Anyway, we have a such high value of gain because of the upward trend of the stock market in 2019. It is recommended to have a similar approach for taking back-testing in a different year, especially in the downward trend market. Anyway, the accuracy of prediction is not high, only around 60%. A more sophisticated model is recommended for future investigation. Moreover, the addition of sentiment analysis to technical analysis may increase the predictive power in the potential for further investigation [10,24]. In actual investment, we would consider 3 states (buy, hold, and sell). In this project, we only have 2 states (buy and sell). Most investors are not frequent traders. They are holding shares most of the time. For further investigation, it is recommended to design a sophisticated simulation of trading [32,33,34].