Distilled Bidirectional Encoder Representations from Transformers (DistilBERT) for Multi-Class Mental Health Detection fromWeb-Scraped Reddit Posts

1. Introduction

The integration of artificial intelligence (AI) into healthcare and social technologies has introduced new opportunities to support individuals experiencing mental health challenges [1,2]. With the growing accessibility of large language models and natural language processing (NLP) frameworks, it has become increasingly feasible to analyze digital communication patterns and identify early indicators of psychological distress [3]. Social platforms such as Reddit, where users frequently express their emotions, struggles, and personal reflections under pseudonymous identities, provide a valuable source of naturally occurring text that can reveal meaningful linguistic and emotional patterns [2,4].

Prior research has shown that language use can reflect cognitive and emotional states associated with depression, anxiety, stress, and related conditions. Transformer-based NLP models, particularly those built upon pre-trained architectures such as BERT and DistilBERT, have demonstrated strong capabilities in sentiment and intent classification tasks [5,6,7]. However, significant questions remain regarding the accuracy, interpretability, and ethical feasibility of using such systems for mental health assessment [8,9]. If sufficiently reliable, automated text classification could assist practitioners and researchers in detecting signs of mental distress earlier, potentially facilitating faster intervention and improved access to care.

The present study investigates whether modern NLP models can serve as a viable foundation for mental health detection tools by assessing their classification accuracy on a large corpus of Reddit posts. The project focuses on developing an offline machine-learning pipeline comprising two Python-based components: a Reddit data-collection script using the PRAW API to gather posts from multiple mental-health-related subreddits, and a Jupyter Notebook framework for text preprocessing, model training, and performance evaluation. This setup ensures that all operations, data cleaning, model fine-tuning, and inference occur in a secure, offline environment to preserve data integrity and privacy.

The primary aim of this research is to measure the model’s predictive accuracy and determine whether such NLP techniques could form the basis for practical, ethically responsible systems that assist in identifying individuals who may be experiencing mental health concerns. By emphasizing quantitative evaluation and reproducible experimentation, the study seeks to contribute to the broader discussion on how AI can responsibly complement traditional psychological research and support mental health awareness.

The objectives of this study are to:

1.

To develop and evaluate the classification performance of a fine-tuned DistilBERT model on unseen Reddit posts using accuracy, precision, recall, and F1-score metrics.

2.

To assess the practical applicability of the model for potential use in mental health monitoring or early detection tools.

3.

To demonstrate that mental health text classification can be conducted entirely offline using open-source frameworks, ensuring reproducibility and data privacy.

4.

To examine the ethical implications and technical limitations of employing AI for mental health prediction, emphasizing transparency and responsible use.

2. Literature Review

2.1. Social Media as a Source for Mental Health Analysis

The rapid growth of social media platforms has created unprecedented opportunities for large-scale analysis of mental health using natural language processing (NLP) and machine learning (ML) techniques. Platforms such as Reddit, Twitter, and Facebook have become spaces where individuals openly express emotions, struggles, and personal experiences related to mental health. As a result, researchers have increasingly leveraged social media text as a proxy signal for psychological distress, depression, anxiety, and suicidal ideation.

Reddit, in particular, has been widely adopted in mental health research due to its pseudonymous structure and topic-focused communities (subreddits), which encourage candid self-disclosure. A systematic review by Yeskuatov et al. [2] highlights Reddit as one of the most frequently used platforms for detecting suicidal ideation, owing to the richness of its long-form textual content and community-moderated structures. Similarly, Hridoy et al. [4] demonstrated the effectiveness of web-scraped Reddit datasets for detecting major depressive disorder (MDD) using ensemble learning techniques.

Large-scale observational studies further support the feasibility of AI-powered mental health monitoring using social media signals. Mansoor and Ansari [3] developed a multimodal deep learning framework that detected early signs of mental health crises across multiple social media platforms with high accuracy, reporting measurable lead times prior to expert identification. These findings reinforce the potential of NLP-based models for early-stage intervention and population-level monitoring, while emphasizing that such systems should serve as supportive tools rather than clinical replacements.

2.2. Machine Learning Approaches to Mental Health Detection

Early research in automated mental health detection relied heavily on traditional machine learning classifiers, including Support Vector Machines (SVMs), Random Forests (RFs), Decision Trees (DTs), and Naïve Bayes. These models typically use handcrafted features such as term frequency–inverse document frequency (TF-IDF), sentiment scores, and psycholinguistic lexicons.

ShamsEldin et al. [10] evaluated multiple classical ML algorithms for predicting depression, anxiety, and stress using psychometric datasets and reported high classification accuracy, particularly with SVM-based approaches. Similarly, Shoaib et al. [7] applied SVM classifiers to BERT-generated embeddings for depression detection in social media posts, achieving strong performance on Twitter and Reddit datasets.

Although traditional ML models showed promise, systematic reviews have identified several limitations, including dependence on feature engineering, limited contextual understanding, and reduced generalizability across datasets [1,9]. These limitations motivated a shift toward deep learning architectures capable of capturing semantic and contextual nuances in language.

2.3. Deep Learning and Transformer-Based Architectures

Deep learning approaches, including Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) networks, and Gated Recurrent Units (GRUs), significantly advanced text-based mental health classification. Hybrid architectures combining CNN and LSTM components have demonstrated improved contextual modeling capabilities [11].

However, the introduction of transformer-based models marked a major advancement in NLP for mental health detection. Models such as BERT, RoBERTa, XLNet, and DistilBERT leverage self-attention mechanisms to capture long-range dependencies and contextual relationships more effectively than traditional sequential models.

Kasanneni et al. [5] reported that transformer-based models consistently outperformed classical ML methods in detecting early signs of mental health concerns from social media data, with accuracies exceeding 95%. Kokane et al.[6] compared multiple transformer architectures and found that DistilBERT achieved strong performance with improved computational efficiency compared to larger transformer models. Similarly, Islam et al.[12] introduced a fine-tuned RoBERTa-based multiclass classifier for mental health disorders and demonstrated superior macro-F1 performance in multi-class settings.

Recent work by Baydili et al.[13] further demonstrated the effectiveness of deep learning combined with feature selection techniques for detecting depression and suicidal tendencies across multiple datasets. These studies collectively justify the selection of DistilBERT in the present research, as it balances computational efficiency with strong contextual modeling performance, making it suitable for multi-class mental health classification tasks.

2.4. Multi-Class Classification and Disorder Overlap

While many early studies focused on binary classification (depressed vs. non-depressed), more recent research has explored multi-class mental health detection, reflecting the complex and overlapping nature of psychological conditions. Islam et al.[12] highlighted semantic overlap between depression, anxiety, and stress categories, noting frequent misclassifications due to shared linguistic markers. Taskynbayeva and Gutoreva [9] similarly observed that anxiety prediction models often struggle with overlapping symptom descriptions and class imbalance.

Kim et al.[14] emphasized that fine-grained classification tasks, such as severity detection or differential diagnosis, require models capable of capturing subtle contextual differences. These findings underscore the importance of macro-averaged evaluation metrics and weighted loss functions for multi-class mental health detection, as implemented in the present study.

2.5. Ethical Considerations and Explainability

Despite strong predictive performance, ethical considerations remain central to AI-based mental health systems. Privacy, consent, bias mitigation, and transparency are recurring concerns in the literature. Harvey et al.[15] emphasize that NLP applications in mental health must adhere to ethical governance frameworks and account for participant risk. Mansoor and Ansari [3] also highlight the need for careful integration with existing mental health services to prevent misuse or stigmatization.

Explainable AI (XAI) has emerged as an important direction for increasing trust in automated mental health systems. Ahmed et al. [8] incorporated SHAP and LIME explanations to interpret depression severity classification models, demonstrating how model transparency can enhance clinical interpretability. These studies suggest that while high accuracy is valuable, responsible deployment requires interpretability and human oversight.

2.6. Summary and Research Gap

Overall, prior literature establishes that:

• Social media text contains measurable linguistic indicators of mental health conditions.
• Transformer-based models outperform traditional ML approaches in contextual understanding.
• Multi-class classification introduces challenges due to symptom overlap and class imbalance.
• Ethical and interpretability considerations are essential for real-world deployment.

However, many studies focus either on binary depression detection or small subsets of disorders. There remains a need for reproducible, privacy-conscious, offline frameworks capable of multi-class mental health classification across diverse disorder categories using efficient transformer architectures [5,12]. The present study addresses this gap by fine-tuning DistilBERT for eleven-category classification using a large Reddit dataset, while emphasizing reproducibility, offline implementation, and ethical considerations.

3. Methodology

3.1. Study Design

This project was designed as an offline, exploratory machine learning study aimed at evaluating the feasibility of using natural language processing (NLP) models to analyze mental health discourse collected from Reddit, which aligns with prior studies utilizing Reddit for mental health detection [2,4]. Unlike traditional user studies, this research did not involve direct human participants; instead, it relied exclusively on publicly available, anonymized textual data [8].

The workflow consisted of two integrated Python components:

1.

A Reddit data collection script, which retrieved posts from multiple subreddits using the PRAW API client and securely stored credentials in a .env file.

2.

A Jupyter Notebook pipeline, which preprocessed, filtered, and cleaned the dataset, fine-tuned the model, and evaluated its classification performance on unseen data.

This design ensured that all experimental steps—data collection, preparation, training, and analysis—were executed offline to protect data privacy and maintain full control over reproducibility.

3.2. Data Collection

Data were collected from a curated list of 26 subreddits representing various mental health topics and a small number of neutral control communities. Examples include r/Anxiety, r/depression, r/MentalHealth, r/stopdrinking, r/ADHD, and control sources such as r/AskReddit and r/dataisbeautiful. The script used the PRAW library to retrieve up to 1,000 posts from each subreddit across multiple categories (hot, new, rising) and time filters (day, week, month, year, all).

Duplicate posts were identified by their unique post IDs and removed to ensure data uniqueness. Each record retained only the post body (selftext) and subreddit label; metadata such as author names, timestamps, or URLs were excluded to maintain anonymity. The final dataset contained approximately 80,000 unique posts saved to a local CSV file.

Figure 1. Architecture Diagram.

Figure 1. Architecture Diagram.

3.3. Data Preprocessing and Cleaning

All text data were imported into the Jupyter Notebook for preprocessing. The following cleaning operations were applied:

• Normalization: lowercasing, punctuation removal, HTML tag stripping, and whitespace normalization using regular expressions.
• Noise filtering: posts with very short text or empty content were discarded.
• Keyword filtering: domain-specific keyword lists were used to retain posts relevant to each subreddit (e.g., “panic attack” for anxiety, “withdrawal” for addiction, “insomnia” for sleep disorders).

Each subreddit was then mapped to a corresponding higher-level mental health category based on its primary topic. For example, r/addiction and r/stopdrinking were mapped to addiction_disorder; r/Anxiety and r/Anxietyhelp to anxiety_disorder; and control groups like r/AskReddit to no_disorder. This mapping produced 11 final categories: addiction_disorder, anxiety_disorder, bipolar_disorder, conduct_disorder, depression_disorder, neurodevelopmental_disorder, no_disorder, obsession_compulsion_disorder, personality_disorder, sleep_disorder, and stress_disorder.

3.4. Model Training and Validation

The cleaned dataset was encoded and split into training, validation, and testing sets using an 80–10–10 stratified partition. Label encoding was applied for single-label classification. Class imbalance was addressed by computing inverse-frequency weights for each class and passing them into the loss function.

A pre-trained transformer model, DistilBERT-base-uncased, was fine-tuned using a custom WeightedTrainer class built on the Hugging Face Trainer API. The model configuration specified:

• Maximum sequence length: 32 tokens
• Dropout probabilities: 0.3 for hidden and attention layers
• Optimizer: AdamW ($2\times {10}^{-5}$ learning rate, $0.01$ weight decay)
• Batch size: 128 (training and evaluation)
• Epochs: 4
• Loss function: weighted CrossEntropyLoss

Mixed-precision (fp16) was enabled automatically on the GPU to accelerate computation. The model’s best checkpoint was determined by the highest macro-averaged F1-score achieved on the validation set.

3.5. Data Analysis

Model performance was assessed using both the evaluate and scikit-learn libraries. Metrics included accuracy, macro-F1, and weighted-F1 scores, as well as class-level precision and recall. A confusion matrix was generated to analyze misclassifications among similar categories, particularly between stress, anxiety, and depression. Qualitative inspection of attention weights confirmed that emotionally charged terms (e.g., “panic”, “insomnia”, “worthless”, “drinking”) strongly influenced model predictions.

3.6. Ethical Considerations

All data used in this study were publicly available Reddit posts collected in compliance with Reddit’s content policy and academic research standards. No identifiable user data was stored or analyzed. The entire pipeline was executed offline to prevent exposure of external data. The research adhered to responsible AI principles by avoiding individual-level predictions, emphasizing aggregate-level insights, and acknowledging the ethical boundaries of mental health classification in automated systems.

4. Simulation

The simulation was conducted entirely within a Jupyter Notebook environment to evaluate the feasibility and performance of the proposed NLP-based mental health classifier. All processes from data preprocessing to model training and testing were executed offline using the dataset collected by the custom Python scraper. The complete workflow integrated data ingestion, cleaning, tokenization, model fine-tuning, and performance evaluation within a reproducible experimental setup [5,12].

4.1. Simulation Setup

Experiments were performed on a local workstation running Python 3 with libraries including PyTorch, transformers, pandas, and scikit-learn. GPU acceleration (CUDA) was enabled automatically when available. Random seeds were fixed at 42 for reproducibility. The cleaned Reddit dataset contained approximately 80,000 posts across 26 subreddits, which were mapped to 11 high-level mental health categories. Data were stratified into training, validation, and testing sets using an 80–10–10 split.

4.2. Model Configuration

The experiment employed the DistilBERT-base-uncased transformer as the underlying language model for single-label text classification. Tokenization was performed using the corresponding pretrained tokenizer, with a maximum sequence length of 32 tokens to balance context retention and computational efficiency. To enhance regularization, both the hidden and attention dropout probabilities were increased to 0.3 in the model configuration. The model was fine-tuned for four epochs with the following hyperparameters:

• Optimizer: AdamW ($\mathrm{learning}\mathrm{rate}=2\times {10}^{-5}$, weight decay = 0.01)
• Batch size: 128 (train and evaluation)
• Epochs: 4
• Evaluation strategy: End of each epoch
• Metric for best model selection: Macro F1-score

4.3. Training and Loss Optimization

A custom WeightedTrainer class was implemented to incorporate class imbalance handling directly into the training loop. The trainer used a weighted CrossEntropyLoss, with class weights inversely proportional to their frequency in the dataset. This ensured that minority categories (e.g., stress_disorder and depression_disorder) contributed proportionally to gradient updates. Mixed-precision (fp16) training was enabled on the GPU to reduce memory usage and accelerate training.

4.4. Performance Monitoring

Throughout training, the model tracked and logged the validation loss, accuracy, and macro-F1 score after each epoch. The best checkpoint, determined by the highest macro-F1, was automatically saved and later reloaded for evaluation on the held-out test set. During evaluation, metrics were computed using both evaluate and scikit-learn libraries. Quantitative performance was summarized using accuracy, macro-F1, and weighted-F1, while the qualitative evaluation included confusion matrix analysis and inspection of misclassified samples.

4.5. Evaluation Procedure

The final evaluation was performed on the 10% test set, which the model had not encountered during training or validation. The trained classifier achieved an overall accuracy of 77%, a macro-averaged F1 Score of 0.736, and a weighted F1 Score of 0.772 across 11 mental health categories. Predictions were further analyzed to identify overlapping or ambiguous cases, particularly between stress, anxiety, and depression classes. Attention-weight visualizations confirmed that emotionally salient words (e.g., “panic”, “insomnia”, “worthless”, “drinking”) were key decision features during inference.

4.6. Reproducibility

The trained model and tokenizer were saved locally as ./mental_health_classifier for future inference and replication. The notebook also included a lightweight predict() function enabling direct text input and on-device inference, allowing verification of model behavior on unseen statements or curated test cases.

5. Results

The results of this study demonstrate the overall performance and classification behavior of the fine-tuned NLP model when identifying various mental health categories from Reddit posts. The model was trained and evaluated using an 80–10–10 split, with the final 10% reserved for testing unseen data. The goal was to evaluate whether the model could accurately distinguish between posts corresponding to specific mental disorders and general (non-disorder) content.

5.1. Overall Performance

The trained transformer model achieved an overall accuracy of 77.0%, with a macro-averaged F1 Score of 0.736 and a weighted F1 Score of 0.772. These values indicate that the model maintained a strong balance between precision and recall across multiple classes, despite the complexity and linguistic variability inherent in mental-health-related text.

Table 1. Model Performance on Test Set.

Class	Precision	Recall	F1-Score
Addiction Disorder	0.798	0.838	0.818
Anxiety Disorder	0.746	0.723	0.734
Bipolar Disorder	0.814	0.696	0.750
Conduct Disorder	0.656	0.629	0.642
Depression Disorder	0.531	0.605	0.566
Neurodevelopmental Disorder	0.704	0.724	0.714
No Disorder	0.985	0.988	0.986
Obsessive-Compulsive Disorder	0.805	0.754	0.779
Personality Disorder	0.707	0.712	0.710
Sleep Disorder	0.839	0.862	0.850
Stress Disorder	0.525	0.582	0.552
Accuracy			0.770
Macro Average	0.737	0.738	0.736
Weighted Average	0.775	0.770	0.772

Table 1. Model Performance on Test Set.

Class	Precision	Recall	F1-Score
Addiction Disorder	0.798	0.838	0.818
Anxiety Disorder	0.746	0.723	0.734
Bipolar Disorder	0.814	0.696	0.750
Conduct Disorder	0.656	0.629	0.642
Depression Disorder	0.531	0.605	0.566
Neurodevelopmental Disorder	0.704	0.724	0.714
No Disorder	0.985	0.988	0.986
Obsessive-Compulsive Disorder	0.805	0.754	0.779
Personality Disorder	0.707	0.712	0.710
Sleep Disorder	0.839	0.862	0.850
Stress Disorder	0.525	0.582	0.552
Accuracy			0.770
Macro Average	0.737	0.738	0.736
Weighted Average	0.775	0.770	0.772

5.2. Class-Level Analysis

Performance varied across mental health categories. The model performed exceptionally well at detecting no disorder posts, achieving an F1 Score of 0.986, indicating near-perfect classification of neutral or non-mental-health-related content. Strong performance was also observed for sleep disorder (0.850 F1), addiction disorder (0.818 F1), and bipolar disorder (0.750 F1), suggesting that linguistic markers for these categories are more distinct or consistently expressed in the dataset.

In contrast, lower F1-scores were observed for depression disorder (0.566 F1) and stress disorder (0.552 F1), indicating greater semantic overlap and greater variability in how users describe these conditions. Such overlap may lead to misclassifications where general stress or depressive expressions are misinterpreted as other disorders, or vice versa.

5.3. Confusion Matrix Analysis

An analysis of the confusion matrix revealed that most misclassifications occurred between closely related categories such as stress, anxiety, and depression. This behavior aligns with known linguistic and psychological correlations between these conditions, as they often share overlapping vocabulary and emotional tone. Conversely, the model showed strong discriminative ability in separating the no disorder class from all other categories.

5.4. Qualitative Observations

Manual review of misclassified samples highlighted that subtle, context-dependent expressions of mood (e.g., humor, sarcasm, or ambiguous emotional phrasing) frequently contributed to errors. Posts describing multiple co-occurring symptoms also challenged the model’s single-label classification framework.

Attention-weight visualizations confirmed that emotionally charged or symptom-descriptive words (e.g., “panic,” “insomnia,” “worthless,” “drinking”) were heavily weighted during inference, supporting the model’s ability to identify linguistic indicators of specific disorders.

5.5. Summary of Findings

Overall, the results indicate that the model effectively distinguishes between diverse mental health categories, achieving reliable performance across multiple conditions. With an overall accuracy of 77% and balanced macro-averaged metrics, the simulation demonstrates that NLP-based models can feasibly support early-stage detection or triage systems for mental health analysis, provided that their limitations in contextual understanding and class ambiguity are appropriately addressed. The achieved accuracy of 77% is competitive with prior multi-class classification studies [1,12]. Lower performance in stress and depression categories also reflects linguistic overlap previously identified in related studies [9,14].

6. Conclusion

This study evaluated the feasibility of using natural language processing (NLP) models to identify indicators of mental health conditions in online discourse. By developing and testing a controlled, offline Jupyter Notebook framework, this research demonstrated that transformer-based architectures can achieve strong performance in classifying diverse mental health categories using real-world Reddit data.

The model achieved an overall accuracy of 77%, with macro and weighted F1-scores of 0.736 and 0.772, respectively. These results indicate that the model was capable of distinguishing between multiple forms of mental health expressions, with particularly strong precision and recall for conditions such as addiction disorder, sleep disorder, and no disorder. Although reduced performance was observed for categories like depression and stress disorder, this variation likely reflects the high degree of linguistic and emotional overlap present in natural mental health discussions.

Overall, these findings suggest that NLP-based systems can serve as the foundation for early-detection tools or digital support systems to improve access to mental health resources. A model with this level of accuracy could, for example, assist practitioners, educators, or community organizations in identifying at-risk individuals earlier, enabling faster intervention and support. However, the practical application of such technology must be accompanied by rigorous ethical guidelines to prevent misclassification, bias, or misuse of sensitive data.

These findings also support prior research indicating that transformer-based NLP models can effectively classify mental health discourse [5,13]. While AI cannot replace clinical expertise, it can serve as a supplementary early detection tool [2,3]. Ethical safeguards and explainability remain essential for responsible deployment [8,9].

Future work should explore methods to improve contextual understanding, such as integrating multi-label classification and context-aware embeddings, and expand the dataset to include more balanced and representative samples. Additionally, collaboration with mental health professionals will be critical to the validation of the interpretability and responsible deployment of AI-assisted mental health analysis.

In conclusion, this study demonstrates that machine learning models, when properly trained, evaluated, and ethically implemented, hold significant promise in aiding mental health assessment and awareness. While AI cannot replace clinical expertise, it can serve as a valuable supplementary tool in identifying patterns of concern and promoting timely, data-informed mental health support in the digital age.