GRUvader ‐ Sentiment Informed Stock Market Prediction

1. Introduction

The general public are increasingly interested in stock market prediction, more specifically, the traders, investors and decision makers [1]. This is due to their interest in making informed and strategic decisions on their financial stock business, investments and intentions to invest. The stock market influences the economic growth of every nation, enabling buying and selling of investments, which supports businesses and boosts the economy [2]. Therefore, an accurate stock market prediction is vital to understand the market and mitigate investment loss. Recent advancement in technology and machine learning (ML) techniques has improved stock market prediction [1]. However, due to the unpredictive nature of stock market, influence of external factors, high complexity and volatility (short-term fluctuations and long-term trends), having a more stable and accurate prediction of stock market is challenging. Traditional models like autoregressive integrated moving average (ARIMA) have shown effectiveness in short-term forecasting [3]. ARIMA is useful to capture trends and seasonality [4], however, limited in capturing complex and non-linear relationships. Furthermore, theories like the efficient market hypothesis (EMF) and random walk theory suggest that historical data is insufficient in predicting stock prices as the price changes randomly and are influenced by external factors such as financial news [5]. Improvements in AI have led to application of neutral network-based prediction models, such as recurrent neural network (RNN) and convolutional neural networks (CNN), which are effective with time series data but face challenges with unstructured (text) data [6,7]. Recent studies emphasize the importance of incorporating textual data, as investor sentiment or external events (for example, Brexit, COVID-19) significantly impact stock price movements [8].

Previous studies [9,10,11] have explored various ML models on stock prediction incorporating the text data sentiment. However, these models were proposed with low performance for predicting stock price. Further to this, our literature synthesis (presented in section 2), indicates that only few methodologies are robust to exogenous variables from sources such as the financial news, company news, market sentiment, public opinion, and investors sentiment. In addition, literature review findings suggests that despite the extensive studies on the application of ML algorithms in this context, optimal stock prediction models are limited. For example, ref [11] used a regression model to predict stock movement based on sentiment indicator but did not clarify which stock were analysed, no information was provided on exogenous features towards the behavior of the target variable. Additionally, there was no comparison conducted against other models, nor optimality was achieved.

To this end, our study aims to develop a stock price prediction model and determine the influence of public sentiment. To achieve this, the study formulates the problem as a regression task and utilise dataset collected by Yukhymenko [11]. By doing so, we present our results from the experimental comparison of ML techniques, including ARIMA in two-folds. First is the results of experiment conducted to compare the ML algorithms using the stock price dataset. Second is the results of experiment conducted that incorporates public sentiment. The latter experiment helps to provide information on the behavior of exogenous features towards stock price. Based on the comprehensive experimentation, we propose GRU_Vader, as the best-performing stock prediction model. The results of this study not only offer insights into the comparative effectiveness of different ML techniques but also provide valuable guidance for selecting optimal models based on specific data characteristics. Specially, the findings are identified the model with superior accuracy, providing useful insights for stock market stakeholders regarding the key drivers of stock prices. The remainder of this paper is organized into four primary sections. Section 2 focuses on the literature review, Section 3 outlines the methodology, the following sections presents the results, and the final section provides conclusions and recommendations for future research

2. Related Works

Stock price movement can be modelled in several ways. There are two popular concepts to this. The first is to model as a classification task where the direction of movement of stock prices can be predicted as “up” or “down”. Secondly, is to predict the stock prices by formulating the problem as a regression task. The former approach is good for decision making processes, especially when beneficiaries are only interested in stock movement indicator. A popular ML algorithm for this approach is logistic regression [12]. Whilst the latter approach is more beneficial as it provides an understanding of financial gains or losses in the form of estimate and therefore, it is more popular. Predicting stock price provides valuable insights into financial market trends, guides investment decisions, and supports the development of effective financial strategies. Studies like [13,14,15,16,17,18] highlighted the reliance of investors and portfolio managers on stock price predictions for marking informed investment decisions. Accurate stock price prediction allows investors to adopt their portfolios proactively, capitalize on emerging opportunities, and strategically manage risks, ultimately resulting in more robust investments outcome. Conventional statistical learning algorithms such as linear regression has been used to model stock price movement [19]. Further to this, other techniques like the moving average [20], and exponential smoothing [21] have also been applied. Moreover, the time series models such as generalised autoregressive conditional heteroskedasticity [22,23] and ARIMA [24] have been used extensively. However, these models have been criticized for their limitation to model complex patterns within the stock price dataset [25].

More recently, advancement in technology led to the development and application of ML algorithms for stock price prediction. These ML algorithms ranging from the traditional ML algorithms (K-nearest neighbour [26], support vector regressor [27], decision tree [28], random forest [29], XGboost [30]) to the deep learning algorithms (RNN [31], LSTM [32], BiLSTM [33], GRU [34], BiGRU [35], GAN [36]) have been used to model stock price movement. In a comparative experimental study, authors in [37] showed that feed-forward neural networks yield better results when compared to ARIMA and ARIMAX. Furthermore, long short-term memory (LSTM) network introduced by [38], is a type of RNN designed to handle both long-term and short-term dependencies effectively. Several studies demonstrated the effectiveness of the LSTM in stock prediction. For example, authors in [39] showed attention-based LSTM outperform conventional LSTMs in predicting stock market movements. Study [40] showed LSTM can be used to enhance predictive performance by leveraging both past and future sequence in time series. Similarly, authors in [41] combined Bi-LSTM with CNN, showing a 9% improvement in prediction performance compared to single pipeline models. Whilst authors in [42] proposed a simple GAN model with LSTM as a generator and CNN as the discriminator, incorporating 13 technical indicators, and demonstrated that genrative adversarial network (GAN) outperformed other models like LSTM, artificial neural network (ANN), support vector machine (SVM), and ARIMA in stock price prediction. Ref [43] used GAN with a LSTM generator and a multiple perceptron (MLP) discriminator, outperforming traditional model based on RMSE and MAE. Authors in [44] proposed gated recurrent unit (GRU) with LSTM and dropout layer to model stock price based on historical data from S&P 500. Their results demonstrated better performance compared to RNN, CNN, and other ensembled techniques. Further to this, study [45] indicated that integrating news data with stock data and feeding into GRU model led to enhanced prediction results, surpassing LSTM models in accuracy. Authors in [46] combined GRU and LASSO dimensionality reduction, achieving improved accuracy for stock price prediction using data from the Shanghai Composite Index.

More recently, authors in [47,48] emphasized on the influence of financial news and investors words on the stock price movement. They further stated stock price prediction is more challenging due to these constraints and as such researchers considered modelling stock price movement by integrating sentiment indicators and financial stock price dataset. The sentiment indicator can be achieved by performing sentiment analysis of the financial news or stakeholders/investors words. Sentiment analysis is the process of classifying emotions or opinion expressing natural language into categories such as positive, negative and neutral [49,50,51]. Authors in [40,47,48] assert that sentiment analysis of financial news or stakeholders/investors comments is crucial for forecasting stock trends. The importance of sentiment analysis for stock prediction has been emphasized in previous studies. For example, authors in [52] incorporated investor sentiment to predict the CS1300 index using LSTM. The LSTM model outperform the traditional time-series models for financial data. Moreover, authors in [53,54] demonstrated how incorporating lexicon-based sentimental analysis of financial news with historical stock data can effectively predict market trends for indices like DIJA and S&P 500. Similarly, [55] found that combining sentiment analysis with stock data for modelling stock, particularly CNN, produces the good results. More specifically, study [11] used GAN to model stock price however, their study failed to provide detail on the influence of sentiment indicator utilized. To conclude, it is worth stating that majority of these studies ignored the use of hyper parameter tunning. In addition, most existing studies did not provide explicit or comprehensive detail of the sentiment score implemented. As such it is difficult to assess the reliability and efficiency of the models.

3. Methodology

This paper proposes the use of GRU_vader for stock price prediction. By doing so, we compare GRU_vader to traditional time series model namely, ARIMA and state-of-the-art ML models. The selection of the ML algorithms is based on their performances shown in the stock market prediction literature, as reviewed in previous section. In subsequent sections, this study presents detail of the datasets, and our approach.

3.1. Dataset

This study used two datasets to model stock market prediction. First, is the stock price dataset and the second is the text data. Below, we provide an overview of the datasets.

3.1.1. Stock Price Dataset

The stock price dataset was collected by Yukhymenko [11]. The dataset has been used extensively in the literature [9,56] for modelling stock price. The dataset contains daily information for 25 companies from September 30, 2021, to September 29, 2022. The dataset includes 6300 observations over eight features namely, date, open, high, low, close, adjusted close, volume and stock name. The data were collected only on weekdays (Monday to Friday), and no missing values were found. The companies represented in the dataset are Tesla (TSLA), Microsoft (MSFT), Amazon (AMZN), Google (GOOG), and Apple (AAPL), Netflix (NFLX), Taiwan Semiconductor Manufacturing Company Limited (TSM), Coca Cola (KO), Ford (F), Procter & Gamble Company (PG), Costco (COST), Disney (DIS), Verizon (VZ), Salesforce (CRM), Intel (INTC), Boeing (BA), Blackstone (BX), Northrop Grumman Corporation (NOC), PayPal (PYPL), Enphase Energy (ENPH), NIO In (NIO), ZS, and XPeng Inc (ZPEV). Each company contribute approximately 252 records. Figure 1 below presents a view of the dataset.

3.1.2. Text Dataset

The text data used in this study are tweets, collected from Twitter (currently known as X) by Yukhymenk [11] which contains four attributes namely, date, tweet, stock name and company name as shown in Figure 2 below. The dataset contains 80,792 observations and for the purpose of this study, only tweets with more than 10 likes were selected.

3.2. Our Approach

This paper deployed GRU_vader as a hybrid approach for stock price prediction. GRU_vader combines the power of deep learning algorithm and lexicon based sentiment analysis for prediction. This paper explains the two components as follows.

The gated recurrent unit (GRU) is a type of RNN designed as a simple alternative to LSTM networks. GRU utilizes a gating mechanism that enables it to selectively update its hidden state at each time step, effectively learning both short and long-term dependencies in sequential data. This makes GRU effective for applications such as language translation, speech recognition, and timeseries forecasting. GRU comprises of two gates namely, reset gate and update gate to control the flow of information. The reset gate is used to determine how much of the previous hidden state should be forgotten. Whilst the update gate is used to control how much new information should be added to the hidden state. These gates allow GRUs to selectively retain or discard information, enabling effective learning of temporal pattern while using fewer parameters than LSTMs resulting in a simpler and computationally efficient model. Thus, the GRU can be mathematically represented as follows.

Reset Gate (r_t) = (W_r .[h_t-1, x_t] + b_r)

(1)

Update Gate (u_t) = σ(W_u .[h_t-1, x_t] + b_u)

(2)

Candidate Activations (h_t) = tanh(W_h.[r_t* h_t-1, x_t] + b_h)

(3)

Output (h_t) =(1-u_t)*h_t-1+u_t*h^_t

(4)

where:

x_t is the input vectors at time t.

h_t is the new hidden state at time t.

h_t-1 is the previous hidden state.

W_r, W_u, and W_h are the weights matrices for the reset gate, update gate and candidate activation respectively.

b_r, b_u, and b_h are the bias vectors for the reset gate, update gate and candidate activation respectively.

σ is the sigmoid activation function.

Tanh is the hyperbolic tangent activation function.

3.2.1. Sentiment Analysis

In this context, sentiment analysis is the process of classifying public or investors’ emotions and opinion expressed in text into categories such as positive, negative and neutral. There are two popular approaches to sentiment analysis namely, ML approach and lexicon-based approach [49,50,51]. The study utilises the lexicon-based approach as supervised ML approach requires large, labelled dataset for training purposes. Unfortunately, this is not readily available. Based on this background, we employ VADER lexicon to analyse the sentiment of the text data. This will help measure the exogenous features related to stock. VADER (Valence aware dictionary and sentiment reasoner) is specially designed for social media content and uses a lexicon and rule-based approach to analyse polarity and intensity of emotions. VADER lexicon comprises of 7,517 words and phrases, each of which are assigned sentiment scores ranging from -4 to +4. VADER algorithm breaks down text into tokens and based on the labels in the lexicon, it assigns an overall sentiment score (normalised compound score) at a range of -1 to +1.

3.2.2. Experimental Setup

The GRU was implemented in Python using keras library. In the GRU architecture, we used two GRU layers. The first layer is a GRU with 50 units, and the second layer is a dropout layer (20%) to prevent overfitting and an output layer to predict the adjusted closing price. Further to this, we performed hyperparameter tuning on the number of units, dropout rate, and learning rate using a random search approach to minimize validation loss. The optimal model configuration was selected based on performance with validation set, and training was enhanced using callback like EarlyStopping and ReduceLROnPlateau to improve effectively and avoid overfitting. The Adam Optimizer was chosen for training due to its adaptive learning rates and computational efficiency. Adam combines features of Adagrad and RmSProp, making it suitable for optimizing non-stationary objectives with sparse gradients, which helps achieve faster convergences and more stable parameters updates compared to traditional optimizers. Thus, we summarize our experimentation as follows and present the workflow in Figure 3.

• Firstly, we experimented with GRU without the applying sentiment indicator and compared results against other ML algorithms (LSTM, Bi-LSTM, Bi-GRU, and GAN) including the traditional time series model ARIMA.
• The tweets were pre-processed, and we performed sentiment analysis using VADER. Furthermore, we compared VADER results against AFINN and TextBlob results to determine the best performing lexicon for this purpose.
• Moreover, we calculate the correlation between Adjusted close price and sentiment score to identify which stock exhibits the best correlation using the sentiment score and the price movement. This measures the behaviour of exogenous features towards adjusted close price.
• Lastly, we experimented with the hybrid of GRU and the sentiment indicator and thus, compared results against other ML algorithms (LSTM, Bi-LSTM, Bi-GRU, and GAN) integrated with the sentiment scores including the traditional time series model ARIMA.

3.3. Evaluation Metrics I

For evaluation of the sentiment analysis lexicons, we selected 2000 tweets randomly and labelled them manually as positive, negative, or neutral to create a reference dataset. In cases of disagreement, the tweets were discussed within the team to agree on the label. Thus, the 2000 labelled tweets were used to validate the performance of the lexicons. The confusion matrix was used to evaluate the performance of the lexicons by comparing the actual values against the predicted results. This was useful to calculate statistical metrics namely, true positive (TP), true negative (TN), false positive (FP), and false negative (FN). These metrics are described as follows.

• True Positive (TP): The number of instances where the model correctly predicts a positive outcome.
• False Positive (FP): The number of instance where the model incorrectly predicts a positive outcome when it should be negative.
• True Negative (TN): The number of instance where the model correctly predicts a negative outcome.
• False Negative (FN): The number of instance where the model incorrectly predicts a negative outcome when it should be positive.

The statistical metrics are used to calculate the classification metrics namely, accuracy, precision, recall, and F1-score which are described as follows.

Accuracy: Measures how often the model correctly predicts the outcome.

$$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}={\displaystyle \frac{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}}{ TP+TN+FP+FN}}$$

(5)

Recall: Measures the actual positive instances which is correctly predicted by the model.

$$\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}={\displaystyle \frac{TP}{TP+FN}}$$

(6)

Precision: Measures the positive prediction that are actually correct.

$$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}={\displaystyle \frac{TP}{TP+FP}}$$

(7)

F1-score: Represents the harmonic means of precision and recall, providing a balance between two metrics.

$$\mathrm{F}1-\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}=2\mathrm{*}{\displaystyle \frac{Precision\mathrm{*}recall}{Precision+recall}}$$

(8)

3.4. Evaluation Metrics II

For evaluation of the regression models, we employed the r-squared, mean square error and the mean absolute error. These metrics can be described as follows.

Mean Absolute Error (MAE): The MAE measures the average of absolute differences between predicted and actual values, providing an intuitive of prediction errors.

$$\mathrm{M}\mathrm{A}\mathrm{E}={\displaystyle \frac{1}{N}}{\sum }_{t=1}^N|yi-yi^|$$

(9)

Mean Square Error (MSE): The MSE measures the average squared difference between predicted and actual, penalizing large errors more than MAE.

$$\mathrm{M}\mathrm{S}\mathrm{E}={\displaystyle \frac{1}{N}}{{\displaystyle {\sum }_{t=1}^N\left(\mathrm{y}\mathrm{i}-\mathrm{y}\mathrm{i}^\right)}}^2$$

(10)

Adjusted R-Square: The adjusted R² values based on the number of predictors, ensuring it accounts model complexity.

$$\mathrm{A}\mathrm{d}\mathrm{j}\mathrm{u}\mathrm{s}\mathrm{t}\mathrm{e}\mathrm{d}-{\mathrm{R}}^2=1-\left[{\displaystyle \frac{(1-R^2)(n-1)}{n-k-1}}\right]$$

(11)

4. Results

This section presents our experimental results in two-fold. First is the result of the lexicon based sentiment analysis and seconded by the stock market prediction results.

4.1. Sentiment Analysis Result

To choose the best NLP model for tweets sentiment analysis, 2000 tweets were randomly selected and manually labelled as positive, negative, or neutral. Thess manually labelled sentiments were than compared with the predictions from three models; AFINN, Text-Blob, and VADER. The models were evaluated based on the accuracy, precision, recall and F1 Score. The Table 1. show below.

4.2. Correlation

Figure 5 shows the correlation coefficient of various stocks. The results indicate META and TSLA have the highest correlation coefficient which are moderate. Others are as follows, TSLA (0.44), META (0.44), BA (0.33), PG (0.21), BA (0.33), NIO (0.29), AMD (0.30), AAPL (0.15), and PYPL (0.28). The correlation coefficient shows that, in general, the sentiment indicator provides a moderate to low impact on the price movement. However, it is worth stating that the analysis is based on daily tweets sentiment scores, but aggregating scores on a weekly or monthly basis may yield a stronger correlation. The volume of tweets as shown in Table 2 may also affect the correlation coefficient. We selected only sentiment indicator variable to assess whether the stock price has any impact with public sentiment. I

Table 2 indicates that the sentiment scores do not only depend on the volume of the tweets but also on the strength of the sentiment. For instance, even with a small volume of tweets (399), BA has a moderate correlation coefficient.

Figure 6 compares the risk and expected return of different stocks without taking sentimental into account. In this scenario, TSLA offers high returns but carries high risk, indicating significant volatility. On the other hand, stocks like PG and AAPL show lower risk and lower expected returns.

Figure 7 examines the effect of sentiment on risk and expected return. The correlation analysis reveals that TSLA and META have moderate correlations with sentiment scores, while AMD, BA, NIO, PG, and PYPL show weaker correlations. Figure 7 indicates that TSLA, AMD, NIO returns increases when sentimental is considered and change in risk is observed across other stocks as well. PG shows an increase in risk, which highlights the impact of sentiment on stock prediction. So, we need to consider TSLA, AMD, META, NIO, PG stocks which exhibits positive relation with sentiment. In our analysis using ML models to better understand the impact of sentimental scores.

4.4. Model Performance

This section presents the results of our experiment on stock prediction using various machine learning models, both without and with the incorporation of sentimental analysis. We evaluate the performance of the models using metrics such as Adjusted- R², MAE, MSE, and Accuracy. Table 3 and Table 4 summarizes the evaluation results.

The initial analysis focused on predicting stock price using historical stock dataset without incorporating sentiment indicator. Across all the stocks, the traditional ARIMA model underperformed compared to neural network models. For example, TSLA achieved an Adjusted-R² of -2.5265 and a MAE of 40.45, indicating poor predictive capability. Neural networks models have superior performance. For LSTM with hyperparameter tuning improved the Adjusted-R² to 0.2829 and reduced the MAE to 7.826. The GRU model with hyperparameter tuning further enhanced the results, with Adjusted-R² 0.39294 and a MAE of 7.122. The best performance was observed with GAN(LSTM+CNN) which achieved an adjusted-R² of 0.53 and MAE of 6.72, indicating a strong fit and accurate prediction. Similarly, the trends were observed for NIO, AMD, and META. The GAN models and GRU with hyperparameter tuning consistently performing with higher Adjusted- R² and lower MAE and MSE.

In the seconded experiment, we incorporated the sentiment scores as continous values rather than categoical as such, this forms a new merged dataset which is the finial Merged dataset used for fitting the models. For all the stocks, GRU model with hyper parameter tuning showed an improvement. For example, TSLA with Adjusted-R² increases from 0.39294 to 0.40526, and MAE decreased from 7.122 to 6.939. The accuracy also improved from 97.53% to 97.60%. We examined the improvement across the models in general and we observed there is a slight positive impact on several model predictive performance, especially the TSLA. However, the GAN models perform very poor. The Adjusted -R² dropped to negative values, and the MAE and MSE increased significantly, suggesting that GAN models may not effectively predict the trends with sentiment features.

For NIO, the inclusion of sentiment analysis led to notable improvements. The GRU model with hyperparameter tuning showed Adusted-R² of 0.50236 (up from 0.3959 without sentiment) and MAE decreased from 0.853 to 0.739. The accuracy improved to 96.14% highlights the beneficial effect of sentiment analysis on predicting NIO stock price.

For AMD, the GRU hyperparameter tuning showed enhanced performance with sentiment analysis. The Adjusted-R² increases from 0.89676 to 0.91619 and the MAE reduced from 3.146 to 2.705. The accuracy improved.

For META, the GRU hyperparameter tuning also perform good with sentimental analysis. The Adjusted-R² improved from 0.80416 to 0.81954 and MAE decreased from 4.546 to 4.307. The accuracy increased to 97.30% suggests that incorporating sentimental score enhances the model ability to predict stock price.

The results showed by incorporating sentiment analysis can enhance the predictive performance of certain models for stock prediction. The GRU_Vader model consistently outperform other models across multiple stocks. The improvement in Adjusted-R² and reduced MSE and MAE indicates the sentiment analysis provides additional valuable information, enabling the models to capture market dynamics more effectively. This is particularly evident for stocks like TSLA, META, NIO and AMD where public sentiment appears to have strong correlation with stock price movement. In addition, we have compared our result to existing results in the literature and we observed (as shown in Table 5) that GRU_vader yields improved results.

Furthermore, we produced residual plots to show the residuals (errors) of the models predictions compared to the actual values. The plots shown in figure 8 are residual plots generated for each stock (TSLA, META, NIO, and AMD) after applying the hyperparameter tuning with sentiment analysis to investigate the model performances. From Figure 8 we assess the performance of our GRU_Vader using the indicators below.

• QQ (Quantile-Quantile plots): the residuals should follow normal distribution, and thus the residuals should align closely with the reference line in QQ plot.
• Histograms: To visualize the distribution of residuals, a bell shaped histogram suggests the residuals are normally distributed.
• Observation plot: To check the randomness in the residuals, residuals should scatter randomly without following any patterns. The randomness indicates that model has effectively capture the trends.

5. Conclusion

This paper aims to develop stock price prediction model with improved performance. In addition, we examine the influence of exogenous features in the form of public sentiment. By doing so, several ML algorithms, including ARIMAX, LSTM, GRU, Bi-LSTM, Bi-GRU, and GAN, were used to predict stock price using both historic stock data and sentiment score derived from tweets. Our results justify that incorporating sentiment analysis significantly improves the performance of stock model prediction, particularly with GRU_vader. Across various stocks, including TSLA, NIO, META, AMD and PG, sentiment enhanced models yielded better accuracy and lower error rate compared to the model without sentiment scores.

To conclude, this study highlights the importance of exogenous features, such as tweets, in financial prediction and shows that sentiment indicators can play a crucial role in predicting stock price movement. Theoretically, this research provides evidence that sentiment analysis can enhance traditional stock prediction model. Further work should consider integrating stock specific language models and deep learning algorithms to examine the ability of the hybrid approach to handle long-term dependencies and complex patterns. Additionally, incorporating other forms of text data, such as news articles or financial reports, could further improve prediction accuracy. The study also emphasizes the need to extend this methodology to other sectors and explore its potential for identifying market trends during volatile condition.