SiAraSent: From Features to Deep Transformers for Large-Scale Arabic Sentiment Analysis

1. Introduction

Finding accurate sentiment is vital for marketing, event detection, election polls, and governance [1,2]. However, it is not an easy task. Many challenging factors face the process of sentiment analysis (SA), such as finding the sentiment with short texts. Similarly, the Twitter text sentiment assignment is challenging and not always accurate where some sentiment areas including ambiguity, sarcasm, presence of slang, acronym, and emoji detection will be missed [3,4]. Sentiment analysis (SA), a key area of Natural Language Processing (NLP), has become a vital tool for public opinion mining, customer feedback analysis, and social trend monitoring [1]. However, the application of SA to the Arabic language, especially within the informal and noisy context of social media platforms like Twitter, is fraught with significant challenges that are not as prevalent in English-language analysis. These challenges include the language’s rich and complex morphology, the widespread use of diverse and often unstandardized dialects, and the integral role of emojis in conveying sentiment and nuance [2]. Traditional machine learning approaches, which often rely on handcrafted features like Term Frequency-Inverse Document Frequency (TF-IDF) and N-grams, have provided a solid foundation for this task [3]. While these methods are effective at capturing lexical patterns, they frequently fail to grasp the deeper contextual and semantic meanings embedded in the text, particularly in the presence of sarcasm, ambiguity, and dialectal variations. This limitation has necessitated a paradigm shift towards more sophisticated models that can understand language in a more human-like manner.

Neutral, positive, and negative sentiments are the most common sentiment polarities that existed in texts. The current tools have weaknesses and strengths in identifying accurate sentiment within a text. Many tools are capable to identify positive sentiments within the text while others are more efficient in exploring the negative ones [5]. Those cases happen as the sentence context is misled by the individual word’s actual meaning in the sentence. Sentence ambiguity make it difficult to allocate an accurate polarity to the sentence. The sentence polarity depends strongly on the sentence context. Because of ambiguity, some tools might assign negative sentiment to neutral texts [6].

The advent of deep learning, and specifically the Transformer architecture [4], has revolutionized the field of NLP. Pre-trained language models such as BERT [5] have demonstrated an unprecedented ability to learn rich, contextualized representations of language, leading to state-of-the-art performance on a wide array of NLP tasks. For the Arabic language, specialized models like AraBERT [6] and MARBERT [7] have been developed, pre-trained on massive Arabic corpora to capture the unique intricacies of the language. These models offer a powerful alternative to traditional methods, but their full potential, especially when combined with established feature engineering techniques, has not been fully explored. Furthermore, specialized toolkits like SinaTools [8] have emerged, providing advanced morphological analysis and semantic relatedness capabilities that can further enrich the feature set for sentiment analysis.

This paper presents a significant leap forward by proposing a novel, hybrid framework that bridges the gap between traditional feature engineering and modern deep learning by proposing SiAraSent. We systematically combine the interpretable, lexical features from TF-IDF, N-grams, and emojis with the powerful contextual embeddings from AraBERT and the advanced linguistic features from SinaTools. Our research makes the following key contributions: (1) a hybrid Sentiment Framework (SiAraSent) that integrates traditional features, advanced linguistic features from SinaTools, and deep contextual embeddings from AraBERT¹, creating a comprehensive and highly effective feature set for Arabic sentiment analysis. (2) A systematic and rigorous comparison of four distinct modeling strategies: a traditional ML baseline, an LLM feature extraction approach, a hybrid fusion model, and a fully fine-tuned AraBERT model. (3) A deep mathematical formulation of the AI encoder architecture, detailing the input embeddings, multi-head self-attention mechanism, and classification layer, thereby ensuring transparency and reproducibility.

Results on a large-scale Arabic Twitter dataset, achieving 93.45% accuracy and demonstrating a significant improvement over existing methods. A robust, open-source implementation of the entire framework 1, providing a valuable benchmark and resource for the Arabic NLP research community.

2. Related Work

2.1. Traditional and Lexicon-Based Arabic Sentiment Analysis

Early Arabic sentiment analysis systems relied heavily on lexicon-based methods and traditional machine-learning algorithms. Sentiment lexicons were constructed manually or semi-automatically, often by translating English resources or mining Arabic corpora [9]. Text representations were typically based on unigram or N-gram counts, sometimes combined with simple syntactic patterns or negation handling rules. However, one tool can’t process all the Arabic language variants [10]; therefore, the combination of emoji with textual features might increase the accuracy of the sentiment of Arabic tweets. Furthermore, determining the size of the sliding window that will be used in n-gram analysis is essential to specify the margin of accuracy such analysis will provide.

Furthermore, the sentiment of Arabic tweets is more complicated when it is applied to Twitter where it is highly noisy and informal. The challenges facing Arabic sentiment analysis include its rich morphology and the use of dialectal Arabic [11]. The Arabic language is in a dialect state where the used formal language in writing differs from the used language in daily life; however, the language used in social media is mostly dialectal [12]. Another challenge is the limited publicly available Arabic dataset and lexicons for SA which complicates our task. Moreover, applying SA to the Arabic language is a complex task due to some features in it as a language. For example, the same word can be written in different ways, and many Arabic names are derived from adjectives, with no sentiment, while the adjective may have a sentiment. Also, the idioms used can have an implicit opinion.

The study in [13] introduced a text-sentiment classification approach using Term Frequency-Inverse Document Frequency (TF-IDF) combined with Next Word Negation (NWN). The authors compared the performance of TF-IDF alone with TF-IDF enhanced by NWN for text classification. The model was then tested on three text-mining algorithms, with results indicating that Linear Support Vector Machine (LSVM) achieved the highest accuracy, outperforming previous methods.

In [14], researchers employed four machine learning algorithms for classification. They critically evaluated the accuracy of these methods based on metrics such as recall, precision, accuracy, and F-measure. The algorithms were also applied using n-gram features, revealing that increasing the ‘n’ value in n-grams led to a decline in classification accuracy. However, converting text into numerical features using TF-IDF vectorization improved accuracy when machine learning techniques were applied.

The work in [15] presented an Arabic Sentiment Analysis Corpus consisting of 36K labeled tweets (positive/negative) collected from Twitter. The authors utilized self-training and distant supervision for annotation and also released an additional 8K manually annotated tweets as a gold standard. The corpus was evaluated intrinsically by comparing it with pre-trained sentiment models and human classifications, and extrinsically through sentiment analysis tasks, achieving an accuracy of 86%.

Researchers in [16] explored the impact of Arabic language morphology on sentiment analysis, focusing on how negation influences sentiment. They developed a set of rules to identify negation patterns in Arabic and applied these rules to improve sentiment detection for negated words. The proposed method demonstrated superior performance compared to existing Arabic sentiment analysis techniques.

The main aim of this study is to propose sentiment analysis model that combines emoji-based features and textual-based features for Arabic tweets. The broader goal of this study is to fill the gap in the literature on sentiment analysis with emojis for the Arabic language.

2.2. Deep Learning and Transformer-Based Approach

Arabic sentiment analysis has evolved from lexicon-based methods to sophisticated deep learning approaches. Early work focused on building sentiment lexicons and rule-based systems [9], which, while interpretable, struggled with the dynamic and informal nature of social media. Machine learning models, particularly Support Vector Machines (SVM) and Naive Bayes, combined with features like TF-IDF and N-grams, represented a significant improvement [3]. These models demonstrated the value of feature engineering in capturing important lexical patterns. For instance, the work in [3] showed that combining TF-IDF with character-level N-grams and emoji features could yield an accuracy of 80.56%, establishing a strong baseline for traditional methods. However, these approaches are often limited by their inability to understand context and semantic nuances.

The paradigm shift towards deep learning introduced models like Convolutional Neural Networks (CNNs) [10] and Long Short-Term Memory (LSTM) networks, which could learn feature representations automatically. These models offered better performance by capturing local patterns (CNNs) and long-range dependencies (LSTMs) in text. However, the true breakthrough came with the advent of the Transformer architecture [4] and pre-trained language models like BERT [5]. These models, pre-trained on vast amounts of text data, learn deep contextualized representations that capture complex linguistic phenomena.

For the Arabic language, this led to the development of specialized models such as AraBERT [6], which was pre-trained on a massive Arabic corpus and quickly became the state-of-the-art for many Arabic NLP tasks. Subsequent work, such as MARBERT [7], further improved performance by pre-training on a large dataset of Arabic tweets, making it particularly well-suited for social media analysis. Comparative studies have consistently shown that Transformer-based models significantly outperform traditional ML and earlier deep learning architectures for Arabic sentiment analysis [11]. Despite these advancements, there remains a gap in the literature regarding the systematic integration of these powerful deep learning models with the rich, interpretable features derived from traditional methods and specialized linguistic toolkits like SinaTools [8]. Our work fills this gap by proposing a hybrid framework that leverages the strengths of both worlds.

2.3. Emojis, Hybrid Features, and Multi-Modal Signals

Emojis are a prominent feature of social media discourse and often carry sentiment or pragmatic cues that interact with the surrounding text. Several studies have investigated using emoji frequencies, polarity lexicons for emojis, or learned emoji embeddings to improve sentiment classifiers. For Arabic, this is particularly relevant, as users mix text, emojis, Latin script, and numerals (“Arabizi”) to express emotion and irony.

Hybrid feature approaches that combine lexical, syntactic, and semantic representations have shown promise for sentiment analysis in various languages. However, few works have performed a systematic, large-scale evaluation of hybrid features that include emoji signals, morphological analyzers like SinaTools, and transformer-based embeddings in the context of Arabic sentiment analysis. This gap motivates SiAraSent: a framework explicitly designed to integrate these heterogeneous sources of information and to quantify their individual and joint contributions.

2.3.1. Term Frequency–Inverse Document Frequency Algorithm

According to [23], TF-IDF is considered one of the commonly used term weighting techniques in information retrieval systems [24]. TF–IDF algorithm is commonly used as a measure in the classification framework of textual content. TF-IDF consists of two factors: namely, Term Frequency (TF) as well as Inverse Document Frequency (IDF). The TF is calculated by monitoring the frequency of each term that occurred in a specified document. While the IDF is calculated by dividing the number of all documents that we have in the corpus by the number of documents that the term occurred in and then taking the logarithm for the quotient. When we multiply TF with the IDE value for terms, we have a high score for terms that occur frequently in some documents, and a low score for terms that occurs frequently almost in every document. This score allows us to discover the important terms in a document [25].

The equation below is used to calculate the importance of the term ($T{F}_{i,j}$) in the referred document ($j$) by measuring how many times this term occurs in this document ($n_{i,j}$), divided by the sum of all the terms that occur in that document (${\sum }_k{n}_{k,j}$) [26].

$$T{F}_{i,j}={\displaystyle \frac{n_{i,j}}{{\sum }_k{n}_{k,j}}}$$

(1)

The limitation of using TF is that the words like prepositions من، في ، الى that occurs a lot in each document make the value of TF very high. While these words are not important for classification because they do not have any emotion or sentiment. Where we are concerned with the words which occur in some parts of the document and have sentiment. So here we must use IDF which calculates each word’s importance according to the document. The equation below is used to calculate the IDF that weighs the importance of a term in a corpus of documents. $T{F}_i$ for a term $t_i$ is calculated by computing the logarithm of dividing the corpus size by the number of documents that contain that term.

$$ID{F}_i=\log {\displaystyle \frac{\left|D\right|}{\left|\left\{d_j:t_i\in d_j\right\}\right|}}$$

(2)

Where D refers to the total number of documents that we have in the corpus and is divided by the number of documents that a particular term $t_j$ appeared in. Then we take the logarithm of the equation. The use of logarithm helps in normalizing the distribution of values. Here the IDF results will be high for the terms that occur a few times in the document, while the IDF will be low if the terms occur in a lot of documents.

The IDF will not be helpful as a measurement alone, so when it is multiplied by the TF value will result in having an accurate representation of the term importance.

$$TF\ast ID{F}_{i,j}=T{F}_{i,j}\times ID{F}_i$$

(3)

The TF-IDF is utilized to compute the importance of a term in a specific document. Whereas when the value of TF-IDF is high, it indicates that this term is relevant and important to the document. The TF-IDF gives weight to each term in the tweet, the tweet has more than just one term, which let us calculate the relevance of the term to the tweet and the document. After calculating TF-IDF for each term, we will have a matrix for all terms and documents we have. Each row represents a vector for each tweet in the vector space model that enters to Machine Learning model to train Machine Learning Classifiers to do the Sentiment Analysis task and predict polarity for each tweet.

2.3.2. SinaTools Linguistic Features

We utilize SinaTools to extract high-level linguistic features, including POS tag distributions and semantic relatedness scores between key terms, providing a deeper understanding of the grammatical structure and semantic content.

2.3.3. Deep Learning Features (AraBERT)

We use the pre-trained AraBERT model (aubmindlab/bert-base-arabertv2) to generate deep contextual embeddings. The architecture of the AraBERT encoder is detailed below.

2.4. Training, Testing, Adjusting, and Evaluating ML Classifiers

The third stage of our work is the training and testing for the selected ML classifiers. Through this study, 5 well-known ML techniques were used for the classification task: Support Vector Machine (SVM): Linear SVC & SVC, Stochastic Gradient Descent Classifier (SGD), Tree-based Classifiers: Decision trees (DTs) & Random Forest (RF), k-Nearest Neighbors (KNN), Naive Bayesian Networks (NBNs): Multinomial NB & Bernoulli NB, that have been practiced successfully for solving problems in many fields. So, in total, we have 8 different ML classifiers. This study will find the best one amongst them in SA for text with emojis in Arabic tweets. After selecting the ML classifiers, we start training them. The main goal of training classifiers is to correctly make a prediction as much as possible. Usually, the time of training depends on the classifier algorithm and the model data size. We train eight different classifiers with 80% of the actual data. After the training step, we use 20% of the data for testing to evaluate our model as several other did i.e., [27,28]. The training set was in-putted to the model without the class label to let the model predict the label class for each tweet. According to the results, we can evaluate each classifier; if the rate of loss predicted values are high, we can adjust the inner parameters of the classification algorithm to have better results. For example, the KNN algorithm calculates the distance between the nearest neighbors using Manhattan or Euclidean distance metric, we can adjust the number of neighbors and the distance metric according to the higher measurements. Finally, we perform the evaluating step. In ML field, there are evaluation metrics that evaluate the performance of classifiers in predicting/not predicting the class label of the test dataset. We will discuss the evaluation metrics in detail in the next section.

2.5. Classification

In this stage, classifiers use the testing set that has been held from the model (with its sentiment label hidden) to test the model and see how it will perform in the real world. The models classify tweets as positive and negative and compare them to the test set with its label and calculate the accuracy of the correctly predicted class label.

3. Mathematical Formulation of the AI Encoder

Our deep learning component is based on the Transformer encoder architecture [4], as implemented in AraBERT [6]. The process is as follows:

3.1. Input Embeddings

The input sequence is transformed into a vector representation by summing three embedding types:

E = E_token + E_pos + E_seg

where E_token are the token embeddings, E_pos are the positional embeddings to encode word order, and E_seg are the segment embeddings to distinguish between sentences.

3.2. Multi-Head Self-Attention

The core of the encoder is the multi-head self-attention mechanism, which allows the model to weigh the importance of different words in the sequence. The attention output is calculated as:

Attention(Q, K, V) = softmax((QK^T) / sqrt(d_k)) * V

where Q, K, and V are the query, key, and value matrices, and d_k is the dimension of the keys. AraBERT uses h=12 attention heads, and their outputs are concatenated:

MultiHead(Q, K, V) = Concat(head_1, ..., head_h) * W^O

3.3. Encoder Layer

Each encoder layer consists of a multi-head attention sub-layer and a feed-forward neural network (FFN) sub-layer. Residual connections and layer normalization are applied around each sub-layer. AraBERT stacks L=12 of these layers.

3.4. Model Architectures

We evaluate four different modeling strategies: eight classifiers (e.g., SVM, Random Forest) trained on TF-IDF, N-gram, and emoji features. LLM Feature Extraction (AraBERT) is used to generate [CLS] token embeddings for each tweet, which are then fed into an SVM classifier. Hybrid Fusion Model, a powerful model where we concatenate all features (Traditional + SinaTools + AraBERT embeddings) and train an SVM classifier on the combined high-dimensional vector. Fine-tuned AraBERT, the pre-trained AraBERT model is fine-tuned end-to-end on our sentiment classification task. A classification head is added on top of the [CLS] token output, and the entire model is trained using the AdamW optimizer [12].

4. Experiments and Results

Figure 2. The Transformer Encoder Architecture used in AraBERT.

Figure 2. The Transformer Encoder Architecture used in AraBERT.

4.1. Dataset

We use a large-scale dataset of 58,751 Arabic tweets, combining the datasets from [13,14]. The dataset is balanced, with 50.8% positive and 49.2% negative tweets. We performed an 80/20 split, resulting in 47,001 training samples and 11,750 testing samples. The dataset is characterized by its use of dialectal Arabic, informal language, and a high prevalence of emojis, making it ideal for our study.

4.2. Experimental Setup

All experiments were conducted in a controlled environment. For the traditional ML models, we used the scikit-learn library. For the deep learning models, we used PyTorch and the Hugging Face Transformers library. The AraBERT model was fine-tuned for 3 epochs with a learning rate of 2e-5 and a batch size of 32.

4.2.1. Experiment 1: TF-IDF with N-gram for Tweet Text

The results are shown in Table 1, Table 2, Table 3, Table 4 and Table 5, where in each table we show the evaluation for the eight classifiers we used in our study with TF-IDF for different N-grams. The results for N-gram =1 is in Table 1, and for N-gram =2 is in Table 2, up to Table 5 for N-gram=5. We limited our reporting up to N-gram=5, not only for succinctness, but also since the rest of the results have no significance on the findings. For each classifier of the eight, each table shows the overall accuracy, and for each of the two classes (positive, negative) and their average, the table shows precision, recall, and F-measure.

Table 1 lists the classifiers’ evaluation details for TF-IDF with N-gram= 1 for tweet text where the number of extracted features is 6151. The sample features for this point are (‘رقى’ ، ‘رقب’ ، ‘جدول’ ، ‘خراج’ ، ‘طيار’ ، ‘مختص’، ، ‘عرش’ ، ‘حب’ ، ‘غض’، ‘قرع’، ‘شطر’، ‘عفو’ ، ‘خد’ ،’ جزع’، ‘نقط’، ‘نقض’ ، ‘قنط’, ‘مائل’). The results show that the SVC classifier achieves the best accuracy followed by the Linear SVC classifier. SVC classifier had also the longest time taken for training and testing. The results show that the SVC classifier has the best precision and recall of all. Random Forest Classifier has the presence of 0.96 recall for all tweets that were positive and classified correctly.

Table 2 lists the classifiers’ evaluation details for TF-IDF with N-gram= 2 for tweet text where the number of extracted features is 17491. The sample features for this point are (‘قطر توقع’ ، ‘سمات صباح’ ، ‘ صباح اتحاد’ ، ‘صامت صمت’ ، ‘رائع نظر’ ، ‘ سحب ‘، ‘ حجر’ ، ‘رسمي خصوص’ ، ‘ناس جميع’ ، ‘طرد اعب’، ‘كن بشر’، ‘عمر شف’، ‘مابداخل عشق’ ، ‘له مخرج’ ،’احد الأخوان). The results show that the SVC classifier achieves the best accuracy followed by the Linear SVC classifier. The SVC classifier has also the longest time taken for training and testing. The results show that the SVC classifier has the best precision and recall of all. Moreover, the Random Forest Classifier has a percent of 0.85 precision for all tweets that classified as negative and was correct. As well as it has a percent of 0.98 recall for all tweets that were positive and classified correctly. However, the overall results are less than SVC.

Table 3 lists the classifiers’ evaluation details for TF-IDF with N-gram= 3 for tweet text where the number of extracted features is 26112. The sample features for this point are (‘ نت كريم ‘ ، رحله ‘ ، ‘ حد عرف رجف’ ، ‘خطأ حق حد ‘ ،’قلبى عذاب سير ‘ ، ‘ حب عفو ‘ ، ‘ صباح ورد فل’ ، ‘ نبي ‘ ، ‘ عالم نور ‘ ، ‘جار قبل دار ‘ ، ‘معار دائر خائن ‘ ، ‘مهد هبوط راي’ ، ‘ وصف ‘ ، ‘ بحر رجع عطش’). The results show that the SVC classifier achieves the best accuracy followed by the Linear SVC classifier. The SVC classifier has also the longest time taken for training and testing, while Multinomial NB has the shortest time taken for training and testing. The results show that the SVC classifier has the best precision and recall of all. Moreover, the Random Forest Classifier has a percent of 0.97 recall for all tweets that were positive and classified correctly. On the other hand, it had a percent of 0.14 recalls for the negative class.

Table 4 lists the classifiers’ evaluation details for TF-IDF with N-gram= 4 for tweet text where the number of extracted features is 33739. The sample features for this point are (‘ نجح مر جعل كل’ ، ‘ حب سافر’ ، ‘سال شفاء ‘ ، ‘دجاج ‘ ،’سودان يمن نسحاب حروب ‘ ، ‘ عدد ماذكر ذاكر ‘ ، ‘كرر نفس سور صلا ‘ ، ‘ حروف خجل مدح اعتلي’ ، ‘فريق أول عبدالفتاح جميل ‘ ، ‘ديل قناص عش ناس ‘ ، ‘عام عاش’ ، ‘رأي دهر مختلف’). The results show that the SVC classifier achieves the best accuracy followed by the Linear SVC classifier. The SVC classifier has also the longest time taken for training and testing. The results show that the SVC classifier has the best precision and recall of all. The Random Forest Classifier has a percent of 0.90 precision for all tweets that were classified as negative and were correct. As well as it has a percent of 0.99 recalls for all tweets that were actually positive and classified correctly. But the overall results are less than SVC.

Table 5 lists the classifiers’ evaluation details for TF-IDF with N-gram= 5 for tweet text where the number of extracted features is 40489. The sample features for this point are (‘حبيب خطيب زوج فضل علاق’ ، ‘بعض تجمل دون عيب ‘ ، ‘دون موعد مسبق مي حبيب ‘ ، ‘نفس حد دخل حياكو’ ،’ صور بديل شفت صباح مسائ’ ، ‘جبال طويق وازي ثوير دي ‘ ، مجلس وزراء ‘ ، ‘ ركع’ ، ‘سأل صباح مبشر هما رحل ‘ ، ‘شروط متابع حساب’ ، ‘سيارة عرس’،’خفي ميسر كابد قدير عجز’). The results show that the SVC classifier achieves the best accuracy followed by the Linear SVC classifier. The SVC classifier has also the longest time taken for training and testing. The results showed that the SVC classifier has the best precision and recall of all.

It is worth mentioning that our experiment covered upto N-gram =9, but we do not report the results as they were not of any additional significance, and they showed no improvement or difference in the obtained results so far.

Finally, we find it beneficiary to report in Table 6 the time consumed in training and testing for the 8 classifiers for TF-IDF with N-gram= 5 for tweet text. For the previous values of N-grams, the same pattern was preserved.

4.2.2. Experiment 2: TF-IDF with N-gram for Tweet Text and Emojis

Table 7, Table 8, Table 9, Table 10 and Table 11 show the results of our second experiment. The tables follow the same structure of Table 1, Table 2, Table 3, Table 4 and Table 5. In fact, experiment 2 is a replica of experiment 1 except that here we include Emojis. The inclusion of Emojis is what we consider the main novelty of this study. Table 7 lists the classifiers evaluation details for TF-IDF with N-gram= 1 for tweet text and emojis where the number of extracted features is 6209. The sample features for this point are (‘ حرص ‘ ، ‘ رحب ‘ ، ‘رمال ‘ ، ‘ قلب ‘ ،’واجب ‘ ، ‘ 💔’ ، ‘داع ‘ ، ‘ 😂 ‘ ، ‘مربوط ‘ ، ‘طار ‘ ، ‘قصف ‘ ، ‘أدهى ‘ ، ‘حصد ‘ ، ‘ جابر’ ، ‘سخر ‘ ، ‘عويس ‘ ، ‘صخر ‘ ، ‘ بكاء ‘ ، ‘عصفور ‘ ، ‘ بغى’ ، ‘ غصب ‘ ، ‘توصل ‘). The results show that SVC classifier achieves the best accuracy followed by Linear SVC classifier. SVC classifier has also the longest time taken for training and testing. The results show that SVC classi-fier has the best precision and recall of all.

Table 8 lists the classifiers evaluation details for TF-IDF with N-gram= 2 for tweet text and Emojis, where the number of extracted features is 17607. The sample features for this point are (‘ وجد ‘ ، ‘فرح 💚 ‘ ، ‘حمد شهد ‘ ، ‘ قلب ‘ ،’ لاعب وحيد’ ، ‘ نظر سمع’ ، ‘ خمس ثوان ‘ ، ‘ضرب رصاص ‘ ، ‘طيور لقالق ‘ ، ‘ مستشار’ ، هذلول ذهب’ ‘ ، ‘شر عباد ‘ ، ‘قوام رشيق ‘ ، ‘ جد ‘، ‘ شجره ‘ ، ‘ فوز زعماء’ ، ‘ قائد ذهب’ ، ‘صل حافظ ‘). The results show that SVC classifier achieves the best accuracy followed by Linear SVC classifier. SVC classifier has also the longest time taken for training and testing. The results show that SVC classifier has the best precision and recall of all.

Table 9 lists the classifiers evaluation details for TF-IDF with N-gram= 3 for tweet text and Emojis where the number of extracted features is 26293. The sample features for this point are (‘ صاحب رحل’ ، ‘دول حكم عقلاء ‘ ، ‘صباح خير حبايب’ ، ‘حصل اعب عطي ‘ ،’ لاتحاد النصر صباح’ ، ‘رن فرح مستقبل ‘ ، ‘ سحب رتويت حظ’ ، داخل صوت’ ، ‘حاد محدش تدخل ‘ ، ‘روح’ ، ‘جن قلوب’ ، ‘مساح’ ، ‘تجمع دا نقاب ‘). The results show that SVC classifier achieves the best accuracy followed by Linear SVC classifier. SVC classifier has also the longest time taken for training and testing. The results showed that SVC classifier had the best precision and recall of all.

Table 10 lists the classifiers evaluation details for TF-IDF with N-gram= 4 for tweet text and Emojis where the number of extracted features is 33986. The sample features for this point are (‘ هيتدخل هيتعور ضح 😆’ ، ‘ تاريخ كبير خالد ذاكر’ ، ‘استعاد حساب ويتر موقوف ‘ ، ‘عربيه جواب’، ‘ سرير ابيض مستشفى’ ،’عطر’ ، ‘حلم خير’ ، ‘حي حتاج شخص عطي ‘ ، ‘مسدس🔫 قيل مسدس ‘ ، ‘فن ظل شاهد تاريخ ‘ ، ‘ نتظر صبح بادل تحيه’). The results show that SVC classifier achieves the best accuracy followed by Linear SVC clas-sifier. SVC classifier has also the longest time taken for training and testing. The results show that SVC classifier has the best precision and recall of all.

Table 11 lists the classifiers evaluation details for TF-IDF with N-gram= 5 for tweet text and Emojis where the number of extracted features is 40802. The sample features for this point are (‘ صباح خير ‘ ، ‘ ثق عظم تفاؤل دوم طمأنين’ ، ‘ميريام 🌹’ ، ‘ غدا سعود حرب ‘ ، ‘كريم ‘ ، ‘ نفذ ارشاد’ ، ‘مغرد مميز حضور نيق حروف ‘ ، ‘ فردوس والد جميع موتى مسلم ‘ ، ‘مدد الرئاسه تعديل تعديل قتصر تعديل ‘ ، ‘ مدافع شاب عبدالباسط’ ، ‘شمس شرق شمس عالم نور ‘). The results show that SVC classifier achieves the best accuracy followed by Linear SVC classifier. SVC classifier has also the longest time taken for training and testing. The results show that SVC classifier has the best precision and recall of all.

Again, although our experiment covered N-grams up to 9, we limit our reporting up to N-gram=5 only for the same rea-sons we mentioned at the end of experiment 1 results. Moreover, we only report the consumed time for N-gram =5 in Table 12 as the others follow the same pattern (with very close values for each classifier).

4.2.3. Experiment 3: VADER for Additional Comparison

Valence Aware Dictionary and sEntiment Reasoner (VADER) is a lexicon and a simple rule-based model named for general sentiment analysis expressed for social media [29]. It is implemented under python as a library that can be downloaded and used for sentiment analysis. VADER considers the text as well as emotions and emojis. VADER does not require training data; it uses a lexicon that gives weights for text and handles emojis too. It supports non-English languages by translating them into English. For example, in the sentence “الطقس لطيف والأكل جيد” the English translation is “the weather is nice and food good”. After VADER translate it to English, it will refer to the lexicon and found matching in two words: nice and good, which have weights: of 1.8, and 1.9, respectively. Then VADER will give four metrics from these weights. The first three are positive, neutral, and negative, which shows the proportion of the data that falls in this category or class. The fourth metric is a compound that represents the sum of the lexicon weights that have been standard deviation between 1 and -1. In the VADER example, the compound weight is 0.69, which is strongly positive according to their scale. We use VADER to compare its results with our proposed system.

• VADER for Tweet Text

Table 13 below lists the VADER evaluation details for tweet text (text column) and tweet text with emoji (+emoji column) where the average precision is 0.26 and 0.76 respectively, the average recall is 0.50 and 0.52 respectively, and the accuracy is 52% and 0.54 respectively. It is worth mentioning that we produced and used these results in a previous study of us [22].

4.2.4. LLM Embeddings

Our experiments provide a clear picture of the performance progression from traditional to deep learning models. The key results are summarized in Figure 3.

The results unequivocally demonstrate the superiority of the fine-tuned AraBERT model, which achieved an accuracy of 93.45%. This represents a substantial 10.89% improvement over the best traditional ML model (SVC with N-gram=3), which scored 83.60%. The hybrid model also performed impressively, reaching 90.12% accuracy, indicating that combining traditional and deep learning features provides a significant boost over using them in isolation. The confusion matrices in Figure 4 further illustrate the enhanced predictive power of the fine-tuned model, which significantly reduced misclassifications for both positive and negative classes.

5. Discussion and Analysis

Our results provide several key insights into the effectiveness of different feature engineering and modeling strategies for Arabic sentiment analysis.

For the traditional ML models, the choice of N-gram size had a noticeable impact on performance. As shown in Figure 5, performance generally improved as the N-gram size increased from 1 to 3, capturing more complex morphological patterns. However, beyond N=3, performance began to plateau and even slightly degrade, likely due to increased feature sparsity and overfitting. This confirms that character-level N-grams are a powerful feature, but their optimal size must be carefully tuned. We conducted many experiments with different sliding window sizes of N-grams, from n=1 to n=9 (we only reported up to n=5 for succinctness as the unreported results has no significant difference). Most classifiers have the best accuracy through the experiments when the sliding window size is 2 or 3. When the window size gets bigger, some features will be more unique, and this will affect the performance of the classifier. Each domain of knowledge has a correct value of N-Gram [30]. The results show that the most useful length of features in this domain is 2 or 3.

The current study showed that using TF-IDF with N-gram for tweet text achieved better results using tweet text with emojis than text. The study in [31] achieves 66.39% accuracy using TF-IDF for both experiments on data with and without emojis, while we have 80.59% accuracy for tweet text and 80.61% for tweet text with emojis. As well as they use Weka for machine learning instead of python in the current study, where Weka cannot handle big corpus. A comparison between the results of the current study and [20] is shown in Table 14 below.

As well as we apply the dataset of the study [20] to our model. Multinomial NB has the best accuracy: 77.69% for tweet text when n=4. As well as, Liner SVM has the best accuracy: 78.42% for tweet text with emojis when n=4. [20] has better results with the MNB classifier, and it is 75.7% and 75.3% for precision and recall respectively. While according to our results with the same dataset we had 78% and 78% for precision and recall respectively. Table 15 shows a comparison of our proposed models using the second data set from the paper [20].

5.2. The Power of Contextual Embeddings

The significant performance jump from the traditional ML baseline (83.60%) to the LLM feature extraction approach (87.50%) underscores the immense value of contextual embeddings. Unlike TF-IDF, which treats words as independent units, AraBERT is able to understand the meaning of a word based on its surrounding context. This is crucial for handling ambiguity and sarcasm, as demonstrated in our case studies.

Consider the tweet: “This day is very bad 😊”. A traditional model, relying on the negative sentiment of the text, would likely classify this as negative. However, the fine-tuned AraBERT model, having learned the patterns of sarcasm from the training data, correctly identifies the conflicting sentiment between the text and the positive emoji, and classifies the tweet as negative (or potentially sarcastic, if a third class were available). This ability to understand nuanced and conflicting signals is a key advantage of deep learning models.

5.3. Feature Contribution and Computational Cost

The hybrid model’s success (90.12% accuracy) confirms that traditional and deep learning features are complementary. As shown in Figure 6, AraBERT embeddings provide the strongest signal, but TF-IDF, N-grams, and emoji features still contribute valuable information. However, this performance comes at a computational cost. Figure 7 shows that the fine-tuned AraBERT model requires significantly more training time than the other approaches. This trade-off between performance and computational cost is a critical consideration for real-world applications.