An Efficient Hybrid Deep Learning Approach for Detecting Online Abusive Language

1. Introduction

The rise of the digital age and the Internet has fueled widespread use of platforms like social media and forums, enabling free expression. Nearly 45% of the global population uses social media, often becoming addicted. However, this has also led to harassment, bullying, and harmful behavior, such as hate speech, toxic comments, and sharing obscene content. Digital technologies make such behavior possible across various online platforms, including social media, messaging, and gaming sites [1]. Social media has become a major platform for hostile behavior, with 65% of parents worldwide acknowledging its prevalence. Additionally, one-third of adolescents who play mobile games have reported being victims of bullying [2]. According to [3], 44% of internet users in the United States have personally encountered various forms of online harassment, with 28% experiencing severe hostility.

Automatically detecting and analyzing online abuse text presents significant challenges due to the complexity of language, contextual ambiguity, the dynamic evolution of terminology, and the sheer volume of data. These challenges are further amplified in the detection of Abuse Material shared on the dark web, where privacy and anonymity are prioritized, making it difficult to trace perpetrators [4]. Additionally, offenders often employ sophisticated evasion techniques, such as the use of code words, slang, cryptic abbreviations, and other forms of linguistic obfuscation, to bypass detection mechanisms and conceal illicit activities [5]. Furthermore, legal and ethical constraints necessitate careful handling of sensitive data, limiting the extent to which automated systems can operate effectively. These factors collectively make the development of robust AI-driven detection methods both critical and highly challenging [6].

Machine learning (ML) and deep learning (DL) techniques have been widely adopted to develop abusive text detection models, including Support Vector Machines (SVM) [7], Random Forest [8], Decision Trees [9], Naïve Bayes (NB) [10], K-Nearest Neighbors (KNN) [11], Long Short-Term Memory (LSTM) [12], Bi-directional LSTM (Bi-LSTM) [13], Convolutional Neural Networks (CNN) [14], and Bidirectional Encoder Representations from Transformers (BERT) [15]. However, each individual ML/DL technique has limitations, such as sensitivity to noise, difficulty handling large-scale data, or an inability to capture complex relationships. Hybrid models that combine ML/DL techniques can leverage the strengths of multiple algorithms to address these weaknesses [16]. Therefore, we propose a hybrid DL model combining BERT, CNN, LSTM, and ReLU activation. This integration improves prediction accuracy, especially for complex or imbalanced datasets, enhances generalization, reduces overfitting, and adapts to a wider range of tasks. These benefits are particularly valuable in domains like abusive text detection, where language can be highly variable and nuanced.

The contributions of our research can be articulated as follows:

• Building an integrated dataset of 77,620 abusive and 272,214 non-abusive text samples from three sources to create a diverse, imbalanced collection that enables comprehensive model evaluation across real-world scenarios.
• Proposing a hybrid DL model that effectively captures abusive text in both English and Romanized scripts by integrating BERT, CNN, LSTM, and the ReLU activation function. The model achieves very high performance with a Precision of 0.991, Recall of 0.986, Accuracy of 0.995, F1-score of 0.989, and AUC of 0.992, as evaluated using five-fold cross-validation.
• Conducting a comprehensive comparison of the proposed model with traditional ML and standalone DL baselines using a diverse benchmark dataset comprising YouTube comments, forum discussions, and dark web posts, evaluated through 5-fold cross-validation across multiple performance metrics.

The structure of this paper is as follows. In Section 2, we review relevant research. Section 3 outlines our methodology, providing our hybrid DL model and its comprising modules in detail. In Section 4, we describe the dataset and evaluate both our model and selected baseline models using various metrics, followed by a detailed performance analysis. Finally, Section 5 concludes the paper and offers insights for future research directions.

2. Related Work

Statistical analysis of large datasets and CSV files helps uncover patterns and trends, enabling unbiased insights from raw data. Studies such as [17,18] utilized statistical methods to analyze abuse-related information. In [17], data analysis began with SPSS, followed by hierarchical logistic regression to assess three key outcomes: (1) sexual revictimization, (2) psychological dating violence, and (3) physical dating violence. Similarly, [18] examined the prevalence and contributing factors of sexual abuse among adolescent girls during the COVID-19 pandemic. While, natural language processing techniques can be used to extract features and content from abuse chats and conversations. In [19], the authors used Latent Dirichlet Allocation to detect tweets about child abuse and applied Conjunctive Analysis of Case Configurations, finding that longer tweets from users with smaller accounts, lacking URLs or images, were more likely to disclose abuse. However, these studies did not leverage the advantages of ML/DL techniques in detecting abuse texts.

Some studies have applied ML/DL techniques to classify abusive posts, such as [20,21,22,23,24]. However, [20,21,22] primarily focused on self-figure drawings or pornographic images, utilizing object categories, visual attention mechanisms, gender-related visual features (e.g., long hair, dresses), and image metrics (e.g., luminance, sharpness). In contrast, [23,24] worked with videos, combining facial recognition with other biometric modalities, such as speaker recognition and age estimation.

Similar to our task, several studies have applied ML/DL techniques to detect abusive text, such as [12,13,25,26,27,28]. In [25], the authors applied natural language inference alongside the BERT model to detect harmful communication strategies. They used a dataset of 6,771 chat messages sent by child sex offenders, sourced from platforms such as MySpace and Yahoo Instant Messenger. In [26], the authors proposed a model based on BERT, combined with a text tokenization method, to classify complaints across multiple dimensions and provide deeper insights into the dynamics of abuse. They analyzed 1,196 reports collected from the Colombian Child Hotline, covering topics such as grooming, sexual content disclosure, and cyberbullying. Meanwhile, [12] employed explainable artificial intelligence (XAI) techniques to identify early warning signs, aiming to raise awareness among individuals with limited prior knowledge of child sexual abuse. The study was based on responses to survey questions collected from 3,002 participants. In [27], Kaur et al. developed a robust abusive language detection model using a dataset of 14,200 English tweets. The approach leveraged DL architectures, specifically LSTM and Gated Recurrent Unit (GRU) networks, to capture contextual dependencies and sequential patterns for accurate classification.

Nearly, [28] introduced a hybrid feature selection approach combining the Enhanced Non-Dominated Sorting Genetic Algorithm II with XGBoost, aiming to optimize classification performance while reducing the number of selected features. The model was evaluated on the Levantine Hate Speech and Abusive dataset, consisting of 5,846 political Arabic tweets from Syria and Lebanon, labeled as normal, abusive, or hate speech. Finally, in [13], the authors applied feature selection techniques such as XGBoost and Multilayer Perceptron to enhance a BiLSTM model for detecting cyberbullying texts and identifying specific bullying types. The dataset comprised 15,294 microblogs containing personal cyberbullying attacks, categorized accordingly, along with 17,826 non-cyberbullying texts for comparison.

However, the ML/DL models for abusive text detection proposed in the aforementioned studies (i.e., [12,13,25,26,27,28]) differ from ours in both scope and methodological approach.

3. Methodology

In this paper, we propose a hybrid DL model for detecting online abusive comments. The model architecture, illustrated in Figure 1, is designed to predict the abuse polarity of a text and classify it accordingly. It consists of three main modules: Preprocessing, Feature Extraction and Classification. In the Preprocessing module, BERT is employed as the word embedding model to generate feature vectors. The Feature Extraction module integrates CNN and LSTM models to capture both local and sequential features of the text. Finally, the Classification module applies a ReLU activation function to produce the final output.

The purpose of this module combination is to enhance detection accuracy compared to single-model approaches when applied to complex data from diverse sources, although it requires longer computation time. The hybrid architecture leverages the complementary strengths of BERT, CNN, and LSTM: BERT captures the full contextual meaning of words through bidirectional analysis, CNN effectively extracts salient textual features, and LSTM retains past information via its cell states, enhancing the model’s ability to learn sequential dependencies.

3.1. Word Embedding Layer

The initial phase of our model involves transforming the input text into a sequence of word embeddings—dense vector representations that encode semantic relationships among words. In this work, we deployed a pre-trained BERT model tailored for the Online Abusive Comments (OAC) dataset. After fine-tuning its parameters, BERT serves as a feature extractor, generating contextualized embeddings for the proposed hybrid framework. The OAC dataset is passed through the BERT model to generate feature vectors, which are then fed into the subsequent layers of the hybrid architecture. The BERT-base-uncased architecture was used to produce contextualized embeddings with a hidden size of 768.

3.2. Convolutional Neural Networks Model

A Convolutional Neural Network (CNN) is a form of feedforward neural network that processes data sequentially from input to output, without any feedback loops. It employs a deep layered architecture [29], usually starting with convolutional and pooling (or subsampling) layers that extract and transform features from the input data. These are then followed by one or more fully connected layers that perform the final classification.

While the general architecture of CNNs is independent of the dimensionality of the input data, their implementation depends on it. The dimensionality determines how subsampling filters slide across the data. In natural language processing, a one-dimensional convolutional layer (1D CNN) with m filters is commonly applied, producing an m-dimensional feature vector for each document n-gram. These feature vectors are then aggregated using max-pooling, followed by a ReLU (Rectified Linear Unit) activation function. The resulting output is then forwarded to a fully connected (linear) layer, which performs the ultimate classification step, as illustrated in Figure 2. To capture local contextual patterns in BERT embeddings, a sequence of three 1D convolutional layers with 512, 256, and 128 filters, respectively, and a kernel size of 3 is applied.

Let $w_{i:n}\in {\mathbb{R}}^d$ denote the input text consisting of n words, where each word is represented as a d-dimensional embedding vector. The sequence of embeddings forms a $d\times n$ matrix that serves as the input to the convolutional layer, where filters are applied across the text.

For each l-word n-gram, we define:

$$c_i=[w_i,\dots ,w_{i+l-1}]\in {\mathbb{R}}^{(d\times l)},0\le i\le n-l$$

(1)

For each filter $f_j\in {\mathbb{R}}^{d\times l}$, the convolution operation is computed as the inner product $\langle c_i,f_j\rangle$. The resulting convolutional feature maps are collected into a matrix $F\in {\mathbb{R}}^{n\times m}$, where m is the number of filters.

A max-pooling operation is then applied across the n-gram dimension:

$$p_j=\underset{i}{max}\left(F_{ij}\right)$$

(2)

The pooled features are passed through a ReLU non-linearity to introduce activation.

Finally, the extracted feature representation is passed through a fully connected layer that produces a probability distribution across the target classes. The class corresponding to the highest probability is then chosen as the final prediction.

3.3. Long Short-Term Memory Model

The Long Short-Term Memory (LSTM) network is a specialized variant of the Recurrent Neural Network architecture [30]. Each LSTM unit comprises three main gates—the forget gate, input gate, and output gate—as well as input/output components and a memory cell that allows the model to learn and retain long-term dependencies. The structure of an LSTM block is illustrated in Figure 3. In our model, the extracted features are passed through an LSTM layer with a hidden size of 500 to effectively capture sequential dependencies.

The forget gate regulates the extent to which information from the previous cell state $c_{t-1}$ is preserved:

$$f_t=\sigma \left(W_f·\left[h_{t-1},x_t\right]+b_f\right)$$

(3)

The input gate determines the amount of new information to incorporate into the cell state, while the candidate cell state ${\tilde{c}}_t$ is calculated as follows:

$$i_t=\sigma (W_i[h_{t-1},x_t]+b_i),{\tilde{c}}_t=tanh(W_c[h_{t-1},x_t]+b_c)$$

(4)

The updated cell state is then:

$$c_t=f_t·c_{t-1}+i_t·{\tilde{c}}_t$$

(5)

Finally, the output gate controls the proportion of the cell state that is exposed to the hidden output:

$$o_t=\sigma (W_o[h_{t-1},x_t]+b_o),h_t=o_t·tanh\left(c_t\right)$$

(6)

Each gate has its own learnable weights and biases, enabling the network to determine how much past information to retain, how much new input to incorporate, and how much of the internal state to expose at each time step.

3.4. Fully Connected Layer and Output

The context vector produced by the LSTM layer with an attention mechanism is fed into a fully connected (dense) layer utilizing a ReLU activation function. This layer enhances the high-level features learned by the CNN and LSTM modules, enabling the model to capture more intricate and non-linear relationships within the data.

Next, the output from the dense layer is passed to the final output layer, which employs an appropriate activation function (e.g., softmax) to generate the abuse classification probabilities. The model is optimized using the categorical cross-entropy loss function, and the class label with the highest probability is selected as the final prediction.

4. Model Evaluation

4.1. Our Dataset

To evaluate model performance, our dataset includes 77,620 abusive and 272,214 non-abusive text samples, integrated from three datasets. This diverse and imbalanced collection reflects real-world conditions, where abusive content is considerably less frequent than normal content. It enhances the model’s generalization across different contexts, supports robust evaluation under realistic class distributions, and enables fair comparison with other imbalanced benchmark datasets. The combined datasets include:

1.

Darkweb Dataset: We created the Darkweb dataset comprising 4,600 samples extracted from 352,000 dark web forum posts collected in 2022. It includes 2,500 child sexual abuse (CSA)-related and 2,100 non-CSA samples. Among them, 2,000 CSA and 100 non-CSA samples contain at least one sexual abuse phrase, see more details in [5]. Figure 4 presents blurred word clouds depicting single words and two-word phrases associated with sexual abuse, which were extracted from the post contents.

2.

PAN12 Dataset1: Developed for the CLEF 2012 competition, this dataset aims to detect predatory behavior in online chats. It includes 198,054 conversations, consisting of 4,029 abusive and 194,025 non-abusive samples.

3.

Roman Urdu Dataset: Contains 147,180 YouTube comments, evenly divided into abusive (73,590) and non-abusive (73,590) classes [31].

4.2. Experimental Design and Metrics

To assess the model’s performance, we use accuracy (ACC), precision (P), recall (R), and F1-score (F1), which are computed from the confusion matrix values: true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN). These metrics are defined as follows: accuracy is computed by $ACC={\displaystyle \frac{TP+TN}{TP+FP+TN+FN}}$, precision by $P={\displaystyle \frac{TP}{TP+FP}}$, recall by $R={\displaystyle \frac{TP}{TP+FN}}$, and the F1-score by $F1={\displaystyle \frac{2·P·R}{P+R}}$ [32].

The AUC (Area Under the Curve) metric is also used to assess a model’s overall classification performance. It represents the area beneath the ROC curve, which graphs the True Positive Rate against the False Positive Rate at various threshold settings. Higher AUC values, approaching 1, signify greater classification accuracy and model effectiveness.

The experiments were carried out in a Kaggle Notebook environment featuring an NVIDIA Tesla T4 GPU with 16 GB of memory, of which up to 15 GB was available per session, alongside an Intel(R) Xeon(R) CPU @ 2.00 GHz with four logical cores. The setup used Python 3.10, with the PyTorch 2.5 framework and Hugging Face’s Transformers library for implementing and fine-tuning the BERT-based model. The environment also provided up to 30 GB of RAM and 57.6 GB of temporary storage, which proved sufficient for training, validating, and evaluating the model’s performance efficiently. We use 5-fold cross-validation during model development where the reported classification performance and execution times correspond to the averaged scores across the five runs.

4.3. Results and Discussion

The average Precision, F1-score, and AUC of the NB, LR, SVM, CNN, and LSTM models using TF-IDF ($tf.idf$) and Word2Vec ($w2v$) feature representations are presented in Figure 5, averaged across five-fold cross-validation. Overall, models leveraging Word2Vec embeddings generally outperform their TF-IDF counterparts across most evaluation metrics, reflecting the benefits of semantic representations in capturing contextual meaning. The only exception is the NB model, where ${\mathrm{NB}}_{tf.idf}$ achieves higher Precision (0.696) and F1-score (0.783) than ${\mathrm{NB}}_{w2v}$, although the latter records a slightly higher AUC (0.906 vs. 0.892), indicating a marginally better discriminative ability.

Among the remaining models, ${\mathrm{LR}}_{w2v}$, ${\mathrm{SVM}}_{w2v}$, ${\mathrm{CNN}}_{w2v}$, and ${\mathrm{LSTM}}_{w2v}$ consistently achieve superior results. Specifically, ${\mathrm{LR}}_{w2v}$ outperforms ${\mathrm{LR}}_{tfidf}$ in Precision (0.897 vs. 0.867), F1-score (0.864 vs. 0.787), and Accuracy (0.903 vs. 0.845). ${\mathrm{SVM}}_{w2v}$ also shows improvements over SVM$tfidf$ in F1-score (0.914 vs. 0.884) and AUC (0.937 vs. 0.847). ${\mathrm{CNN}}_{w2v}$ also surpasses ${\mathrm{CNN}}_{tfidf}$ across all three evaluation metrics (Precision: 0.819 vs. 0.693, F1-score: 0.759 vs. 0.442, AUC: 0.829 vs. 0.610), demonstrating the advantage of semantic embeddings. Similarly, ${\mathrm{LSTM}}_{w2v}$ achieves the best overall performance among all models, with a Precision of 0.94, F1-score of 0.922, and AUC of 0.935. These results highlight the effectiveness of DL and kernel-based models when combined with Word2Vec embeddings. Therefore, the best-performing models in this comparison are ${\mathrm{NB}}_{tf.idf}$, ${\mathrm{LR}}_{w2v}$, ${\mathrm{SVM}}_{w2v}$, ${\mathrm{CNN}}_{w2v}$, and ${\mathrm{LSTM}}_{w2v}$.

Figure 6 presents and conpares the confusion matrix components across our proposed hybrid model, BERT and the the best-performing ML/DL models based on the $tf.idf$ or $w2v$. The proposed hybrid model achieved the highest number of TP (15,307) and TN (54,305), indicating its superior ability to correctly identify both abusive and non-abusive comments. It also recorded the lowest FP (138) and FN (217), demonstrating a strong balance between precision and recall. In contrast, traditional models such as NB and LR showed weaker performance, with significantly higher FP and FN. For example, ${\mathrm{NB}}_{tf.idf}$ produced 6,070 FP and 1,619 FN, while ${\mathrm{CNN}}_{w2v}$ also struggled with 4,546 FN. BERT and ${\mathrm{LSTM}}_{w2v}$ performed considerably better, reducing misclassifications and achieving stronger overall accuracy; however, our model consistently outperformed them in all four metrics.

Table 1 summarizes the experimental results across all models. The proposed hybrid model achieved the best overall performance, with the highest Precision (0.991), Recall (0.986), Accuracy (0.995), F1-score (0.989), and AUC (0.992). This indicates its strong ability to correctly identify both abusive and non-abusive comments with minimal errors. BERT and ${\mathrm{LSTM}}_{w2v}$ also performed competitively, achieving high precision and recall but slightly lower accuracy and F1-score compared to the hybrid model. Traditional ML models, including ${\mathrm{NB}}_{tf.idf}$, ${\mathrm{LR}}_{w2v}$, and ${\mathrm{SVM}}_{w2v}$, delivered moderate performance, with NB showing the lowest precision (0.696) and ${\mathrm{CNN}}_{w2v}$ yielding the weakest overall results. In terms of computational efficiency, ${\mathrm{NB}}_{tf.idf}$ and ${\mathrm{LR}}_{w2v}$ had the fastest training and prediction times, while DL models, particularly BERT and the hybrid model, required substantially longer processing times due to their complexity. Overall, the hybrid model offers the best trade-off between classification performance and reliability, albeit at the cost of increased computational time. These results highlight the effectiveness of integrating DL architectures in a hybrid framework to enhance both sensitivity and specificity in online abusive comment detection.

Additionally, the Hybrid model also achieves the highest performance on the ROC curve, as shown in Figure 7, with an averaged AUC of 99.2%. The curve almost touches the top-left corner, indicating an optimal balance between the true positive rate and the false positive rate.

5. Conclusions and Future Work

This paper presents an efficient hybrid deep learning model that combines BERT, CNN, and LSTM architectures with a ReLU activation function to detect abusive language across multiple online platforms. By effectively integrating these components, the model captures semantic, contextual, and sequential dependencies in text, enabling robust classification of abusive and non-abusive content. Evaluation on a large, diverse, and imbalanced dataset—including YouTube comments, online forum posts, and dark web content—demonstrated its effectiveness under realistic conditions.

Experimental results showed that the proposed hybrid model outperformed traditional ML and standalone DL methods. It achieved strong results with a Precision of 0.991, Recall of 0.986, Accuracy of 0.995, F1-score of 0.989, and AUC of 0.992, confirming its high discriminative ability and robustness. Although computationally more demanding than simpler models, this trade-off remains acceptable for real-world applications requiring reliable performance. Overall, the hybrid model provides an effective and flexible approach to abuse detection, serving as a strong foundation for future research on content moderation and digital safety.

Future work will focus on extending this framework to multilingual and code-mixed text, improving interpretability through XAI methods, applying Ontology [33,34], exploiting Knowledge Graph [35,36] and incorporating multimodal [37] features such as images or user metadata. Additionally, optimizing computational efficiency via model compression or pruning will support real-time deployment in large-scale social media monitoring systems [38].