From Patient Emotion Recognition to Provider Understanding: A Multimodal Data Mining Framework for Clinical Counseling Communication Analysis

1. Introduction

1.1. Data Mining Challenges in Therapeutic Communication

The computational analysis of therapeutic communication poses substantial challenges for contemporary data mining systems. Unlike conventional text or speech classification tasks, therapeutic interactions are characterized by extreme multi-label structure, severe class imbalance, and heterogeneous multimodal signals that must be jointly modeled to capture clinically meaningful patterns [1,2]. In psychotherapy settings, individual utterances may simultaneously express multiple emotional or behavioral states, resulting in high-dimensional label spaces with long-tailed distributions that violate assumptions underlying standard supervised learning approaches [3].

Class imbalance is particularly acute in this domain. While common affective states such as concern or neutrality occur frequently, clinically salient behaviors and emotions such as rupture markers, avoidance, or reflective listening appear sparsely, often at ratios exceeding 50:1 relative to dominant classes [4,5]. Naive optimization toward overall accuracy or micro-averaged metrics leads to systematic under-detection of these rare but pedagogically critical phenomena [6]. Effective therapeutic communication analysis therefore requires data mining strategies that explicitly address imbalance through class-aware optimization and threshold calibration [7,8].

In addition, therapeutic communication is inherently multimodal, integrating lexical content, vocal prosody, pacing, and turn-taking dynamics [9,10]. Textual cues alone are often insufficient to distinguish between superficially similar utterances that differ in affective intent, while acoustic signals without semantic grounding can be ambiguous [11,12]. Integrating heterogeneous modalities introduces further challenges related to temporal alignment, modality-specific noise, and the need to learn cross-modal dependencies rather than simple feature concatenation [13,14]. These characteristics collectively place therapeutic communication at the intersection of several open problems in applied data mining.

Table 1 summarizes the datasets, annotation strategies, and data mining challenges addressed in this work.

1.2. From Patient Monitoring to Bidirectional Clinical Analysis Systems

Prior work in patient-focused emotion recognition has predominantly addressed unidirectional detection of client emotional states for monitoring and crisis intervention purposes [15,16,17,18]. Existing approaches have primarily focused on developing sophisticated multi-label classification models for recognizing client emotional states from counseling interactions, achieving strong technical performance on benchmark datasets [19,20,21]. Prior studies and empirical evidence indicate that patient emotion recognition, while technically sophisticated and potentially valuable for monitoring applications, provides incomplete analytic utility for comprehensive interaction analysis [22,23].

Contemporary systems can identify when clients experience emotions such as anxiety and sadness, but this information alone does not indicate whether therapeutic responses were appropriate, whether expressed empathy was sufficient, whether vocal tone conveyed supportiveness, or whether the level of directiveness matched client needs [24,25]. Recognition of client emotional states represents only one component of therapeutic interaction. The subsequent phase involves assessing whether provider responses effectively address identified emotional states through coordinated verbal and nonverbal communication [26,27].

However, existing computational approaches rarely model provider behaviors alongside patient affect within a unified analytic framework, limiting their usefulness for interaction-level analysis and behavioral interpretation. The therapeutic alliance between clinician and client has emerged across decades of meta-analytic research as a primary predictor of positive treatment outcomes, independent of specific therapeutic modalities employed [28,29,30]. Yet computational approaches to modeling therapist communicative behaviors remain critically underdeveloped, particularly for naturalistic clinical data reflecting the complexity and variability of real-world practice [31,32].

Bidirectional modeling that encompasses both patient emotional states and provider communicative behaviors addresses fundamental limitations in existing approaches. Modeling therapist affective and communicative behaviors requires capturing subtle, context-dependent patterns where the same verbal content can convey vastly different therapeutic meanings depending on prosodic delivery [33,34]. The statement indicating difficulty can be spoken with warm, empathic prosody characterized by reduced speech rate, softer volume, and falling intonation contours, conveying genuine therapeutic presence. Alternatively, identical verbal content delivered with flat, perfunctory prosody characterized by monotone pitch, constant rate, and lack of vocal affect conveys substantially different meaning that clients may perceive as dismissive or inattentive [35].

1.3. Research Contributions

This work advances data mining techniques for extreme multi-label classification, class imbalance handling, and multimodal fusion through three primary contributions. First, we introduce frequency-stratified class weighting combined with dynamic per-class threshold optimization for multi-label classification under severe imbalance [36,37]. This approach achieves substantial improvement in macro-F1 score compared to baseline multi-label classification, maintaining strong performance on classes with limited training examples while preventing gradient collapse for minority classes.

Second, we establish an automated processing pipeline for analyzing real-world psychotherapy sessions that addresses the scarcity of naturalistic clinical data suitable for computational modeling [38]. The pipeline incorporates speaker diarization [39], automatic speech recognition [40], temporal segmentation, automated annotation [41], and quality filtering, processing hundreds of sessions into thousands of annotated communication segments. This reproducible methodology establishes viability of leveraging publicly available clinical content for research purposes while respecting privacy constraints.

Third, we develop a cross-modal attention architecture that learns content-dependent prosodic associations for behavior recognition in multimodal settings [42,43]. Implementing scaled dot-product attention between text and audio representations [44], the architecture achieves macro-F1 of 0.91 on controlled human-annotated data, comparable to inter-rater reliability reported among trained human annotators in psychotherapy process research [45,46]. The architecture outperforms simple concatenation-based fusion substantially, demonstrating that explicit modeling of cross-modal interactions provides benefits for recognizing coordinated multimodal behaviors [47]. On automatically annotated naturalistic data, the architecture achieves macro-F1 of 0.71, quantifying the performance gap between controlled and real-world annotation regimes.

These contributions advance data mining methods for extreme multi-label learning under severe class imbalance, multimodal fusion with heterogeneous signal quality, and scalable analysis of complex interaction data in naturalistic settings. The complete bidirectional framework is illustrated in Figure 1.

2. Materials and Methods

An overview of the complete system architecture and data processing pipelines is shown in Figure 1.

2.1. Human Annotation Protocol

We established a three-stage annotation protocol for all datasets. Stage one involved schema development through collaborative discussion sessions with clinical annotators. Stage two performed full dataset annotation where each sample received independent review by multiple annotators. Stage three addressed schema refinement and consolidation based on label distributions and clinical validity.

For CounselChat, three licensed clinical psychologists with an average of eight years of clinical experience independently annotated 1,482 counseling interactions across 25 emotion categories. Initial inter-annotator agreement achieved Cohen’s kappa of 0.72. Final inter-rater reliability achieved Fleiss’ kappa of 0.78 on a validation subset of 200 interactions. For DAIC-WOZ, two clinical psychologists independently annotated approximately 8,400 utterances across 11 emotion categories, achieving Cohen’s kappa of 0.69. For controlled HOPE, a single licensed clinical psychologist with 12 years of experience annotated approximately 12,500 therapist utterances across 25 PQS dimensions, achieving Fleiss’ kappa of 0.76 on a validation subset of 1,000 utterances. Controlled human-labeled data anchors evaluation throughout this work.

2.2. Patient-Side: Multi-Label Classification with Imbalance Handling

The patient-side emotion recognition pipeline is shown on the left side of Figure 1.

2.2.1. Problem Formulation

This design choice was motivated by the need to capture co-occurring emotional states. We formulate patient emotion recognition as multi-label classification where each interaction receives predictions for L emotion labels simultaneously. ClinicalBERT [53] serves as the base encoder, producing 768-dimensional contextual embeddings from the [CLS] token. A linear classification head maps to L output dimensions with sigmoid activation enabling independent probabilistic estimates.

2.2.2. Frequency-Stratified Class Weighting

This design choice was motivated by the need to address severe class imbalance. We implement position-weighted binary cross-entropy loss:

$$\mathcal{L}=-\sum _{i=1}^L{w}_i[y_{i}log\left(p_i\right)+(1-y_i)log(1-p_i)]$$

(1)

where weights combine square root inverse frequency with stratified multipliers:

$$w_i=m_i\times \sqrt{{\displaystyle \frac{N_{neg,i}}{N_{pos,i}}}}$$

(2)

Multipliers $m_i$ equal 2.0 for extremely rare categories, 1.5 for moderately rare categories, and 1.0 for common categories.

2.2.3. Dynamic Threshold Optimization

For each label, we select thresholds maximizing validation F1:

$${\tau }_i^{*}=arg\underset{\tau \in [0.1,0.9]}{max}F{1}_i\left(\tau \right)$$

(3)

2.2.4. Multimodal Extension

For DAIC-WOZ, we extracted acoustic features including fundamental frequency, mean energy, and thirteen mel-frequency cepstral coefficients using librosa [55]. Late fusion combines independent text and audio classifiers at the decision level:

$$p_{fused}=\alpha ·p_{text}+(1-\alpha )·p_{audio}$$

(4)

where $\alpha$ is optimized on validation data.

Table 2. Patient-Side Model Configurations.

Component	CounselChat	DAIC-WOZ Early	DAIC-WOZ Late
Base encoder	ClinicalBERT	ClinicalBERT	ClinicalBERT + MLP
Fusion	N/A	Concatenation	Weighted averaging
Loss	Weighted BCE	Standard BCE	Independent BCE
Optimizer	AdamW, 1e-5	AdamW, 1e-5	Text: 1e-5; Audio: 1e-3
Batch size	8/16	8/16	Text: 8/16; Audio: 16/32
Epochs	5, patience=3	5	Text: 5; Audio: 20
Fusion weight	N/A	N/A	$\alpha$=0.75

Table 2. Patient-Side Model Configurations.

Component	CounselChat	DAIC-WOZ Early	DAIC-WOZ Late
Base encoder	ClinicalBERT	ClinicalBERT	ClinicalBERT + MLP
Fusion	N/A	Concatenation	Weighted averaging
Loss	Weighted BCE	Standard BCE	Independent BCE
Optimizer	AdamW, 1e-5	AdamW, 1e-5	Text: 1e-5; Audio: 1e-3
Batch size	8/16	8/16	Text: 8/16; Audio: 16/32
Epochs	5, patience=3	5	Text: 5; Audio: 20
Fusion weight	N/A	N/A	$\alpha$=0.75

2.3. Provider-Side: Cross-Modal Attention for Real-World Data

The provider-side processing and cross-modal architecture are shown on the right side of Figure 1.

2.3.1. YouTube Data Processing Pipeline

This design choice was motivated by the scarcity of naturalistic clinical data. We process 330 YouTube psychotherapy sessions through an automated pipeline: (1) yt-dlp extracts audio as 16 kHz mono WAV, (2) pyannote.audio [39] performs speaker diarization, (3) therapist-only audio is reconstructed, (4) Whisper-large-v3 [40] generates timestamped transcripts, (5) temporal chunking creates 10-second windows with 2-second stride, (6) Claude Sonnet 4 [41] annotates six communication styles (Neutral, Reflective, Empathetic, Supportive, Validating, Transitional), and (7) quality filtering removes ambiguous segments. Large language models are used only for annotation; all predictive models are trained independently on annotated data. The pipeline successfully processes 278 sessions yielding 14,086 annotated segments.

2.3.2. Controlled HOPE Dataset

Using the annotation protocol described above, we process 178 controlled HOPE sessions [56] with session-level PQS ratings. We retrieved audio-video recordings, performed forced alignment for utterance segmentation, and obtained approximately 12,500 therapist utterances. A single psychologist annotated utterances across 25 PQS dimensions.

2.3.3. Cross-Modal Attention Architecture

This design choice was motivated by the need to model coordinated verbal-prosodic patterns. Text encoder DeBERTa-v3-base [58] and audio encoder WavLM-base-plus [43] project to shared 256-dimensional space. Cross-modal attention implements scaled dot-product attention [44]:

$$Q=h_t{W}_Q,K=h_a{W}_K,V=h_a{W}_V$$

(5)

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}\right)V$$

(6)

$$h_{\mathrm{attn}}=\mathrm{LayerNorm}(h_t+\mathrm{MultiHead}(Q,K,V))$$

(7)

$$h_{\mathrm{fused}}=\mathrm{LayerNorm}(h_{\mathrm{attn}}+\mathrm{FFN}\left(h_{\mathrm{attn}}\right))$$

(8)

Training employs multi-stage strategy: warmup (2 epochs), fine-tuning (10 epochs), and threshold optimization.

Table 3. Provider-Side Model Configuration.

Component	Controlled HOPE	YouTube HOPE
Text Encoder	DeBERTa-v3-base (184M)	DeBERTa-v3-base
Audio Encoder	WavLM-base-plus (95M)	WavLM-base-plus
Shared Dimension	d=256, h=8	d=256, h=8
Training	Warmup 2 epochs; Fine-tune 10 epochs	Single-stage 20 epochs
Batch Size	8 (effective 32)	8
Output Classes	25 PQS dimensions	6 communication styles

Table 3. Provider-Side Model Configuration.

Component	Controlled HOPE	YouTube HOPE
Text Encoder	DeBERTa-v3-base (184M)	DeBERTa-v3-base
Audio Encoder	WavLM-base-plus (95M)	WavLM-base-plus
Shared Dimension	d=256, h=8	d=256, h=8
Training	Warmup 2 epochs; Fine-tune 10 epochs	Single-stage 20 epochs
Batch Size	8 (effective 32)	8
Output Classes	25 PQS dimensions	6 communication styles

3. Results

3.1. Patient-Side Emotion Recognition Performance

As shown in Table 4, the single-label formulation achieved micro-F1 of 0.30 and macro-F1 of 0.12. Multi-label formulation without class weights yielded micro-F1 of 0.13 and macro-F1 of 0.12. Incorporating class weighting improved performance to micro-F1 of 0.52 and macro-F1 of 0.53. The final model with dynamic thresholds achieved micro-F1 of 0.65, macro-F1 of 0.74, and subset accuracy of 0.34.

Ablation experiments isolated component contributions. Removing dynamic thresholds reduced performance by 0.13 points. Removing class weights reduced performance by 0.62 points. Reducing training epochs reduced performance by 0.09 points.

3.2. Provider-Side Behavior Recognition Performance

Table 5 presents provider-side model performance across architectures. On controlled HOPE data, cross-modal fusion achieved micro-F1 of 0.93, macro-F1 of 0.91, and Cohen’s kappa of 0.87, comparable to inter-rater reliability reported among trained human annotators in psychotherapy process research. On YouTube data, the architecture achieved micro-F1 of 0.86 and macro-F1 of 0.71.

Table 6 presents per-dimension performance on controlled HOPE data. Warmth and Supportiveness achieved F1 of 0.94, Empathy achieved 0.92, Silence and Listening achieved 0.91, Validation achieved 0.90, Reassurance achieved 0.86, Directiveness achieved 0.85, Interpretation achieved 0.80, and Challenge and Confrontation achieved 0.77.

Table 7 presents per-style performance on YouTube HOPE data. Neutral achieved F1 of 0.934 with 327 support examples. Transitional achieved 0.834 with 211 examples. Reflective achieved 0.833 with 137 examples. Empathetic achieved 0.600 with 12 examples. Supportive achieved 0.561 with 18 examples. Validating achieved 0.500 with 8 examples.

4. Discussion

4.1. Controlled Versus Real-World Data Quality

The 20 percentage point performance gap between controlled HOPE (macro-F1 0.91) and YouTube HOPE (macro-F1 0.71) reveals systematic differences in data quality regimes. These findings are consistent with the impact of annotation methodology, where expert human psychologist review provides higher label reliability than automated large language model annotation [59]. Audio quality variation between controlled studio recordings and heterogeneous YouTube content introduces additional noise. Label complexity reduction from fine-grained expert taxonomies to coarser functional categories affects discriminative capacity. Segmentation methodology differs between utterance-level forced alignment providing natural linguistic boundaries and fixed temporal windows potentially fragmenting therapeutic statements.

These patterns suggest that automated annotation pipelines, while enabling large-scale data acquisition, introduce systematic performance degradation compared to expert human labeling. The magnitude of this degradation quantifies the cost of transitioning from controlled to naturalistic settings. For data mining applications prioritizing scale over precision, this trade-off may be acceptable. For applications requiring high reliability, hybrid approaches combining automated preprocessing with selective human review merit investigation [38].

4.2. Class-Aware Optimization for Extreme Imbalance

The observed six-fold macro-F1 improvement demonstrates the effectiveness of frequency-stratified weighting combined with dynamic threshold optimization for multi-label classification under severe imbalance. These results suggest that two-component weighting schemes balancing gradient emphasis with training stability outperform single-component approaches. The square root transformation moderates extreme weight values while stratified multipliers provide targeted amplification for different rarity tiers [4,5].

Dynamic per-class threshold optimization contributes substantial additional gains. This pattern aligns with theoretical expectations that imbalanced multi-label tasks exhibit heterogeneous calibration characteristics across classes, making fixed threshold approaches suboptimal [6,7]. The observed threshold variation reflects different optimal operating points on class-specific precision-recall curves. For data mining practitioners addressing extreme imbalance, these findings suggest that threshold tuning should be considered standard practice rather than optional refinement.

Comparison with alternative approaches provides context. SMOTE oversampling [36] creates synthetic minority samples but struggles with multi-label co-occurrence patterns where synthesizing realistic combinations of concurrent labels proves challenging. Focal loss [53] emphasizes hard examples but does not explicitly address frequency imbalance. Standard class-balanced loss employs inverse frequency without stratification, potentially producing training instability. These findings are consistent with the hypothesis that multi-component approaches combining frequency weighting, sublinear scaling, and threshold tuning provide more robust solutions for extreme multi-label imbalance.

4.3. Fusion Strategy Selection for Heterogeneous Modalities

The differential performance of fusion strategies across patient and provider tasks reveals task-dependent optimal architectures. For patient emotions, late fusion with learned weight alpha of 0.75 outperforms early fusion. This pattern aligns with scenarios where modalities contribute relatively independently and exhibit different noise characteristics [47]. Text provides strong signal through clinical language patterns while audio exhibits substantial variability from recording quality issues. Late fusion allows text-dominant combination while preserving some complementary acoustic information.

For provider behaviors, cross-modal attention achieves substantial gains over both concatenation and late fusion. These findings are consistent with the hypothesis that therapist communication involves coordinated multimodal patterns where verbal content and prosodic delivery interact to convey meaning [33,42]. The same verbal statement delivered with different prosodic characteristics expresses different therapeutic intentions. Cross-modal attention mechanisms enable learning which acoustic patterns are relevant for specific textual content rather than treating modalities as independent contributors [9].

Attention weight visualization reveals interpretable patterns. For empathic statements, attention concentrates on prosodic features indicating vocal warmth including reduced speech rate, softer volume, and falling pitch contours. For directive interventions, attention shifts to features indicating confident guidance including steady speech rate and rising terminal pitch. These patterns align with communication theory regarding coordinated verbal and prosodic signaling of affective intent [11,12,34]. For data mining applications, these findings suggest that fusion strategy should match the causal structure of modality interactions rather than applying uniform architectures across tasks.

4.4. Scalability and Deployment Considerations

The YouTube processing pipeline demonstrates practical viability for moderate-scale analysis. Diarization accuracy exceeded 92 percent on validation samples [39]. Automatic speech recognition achieved word error rate of 11.3 percent [40]. Processing requires approximately 15 minutes per 45-minute session. Training requires 8 to 12 hours on NVIDIA A100 GPUs. Inference latency remains under 2 seconds per segment. These computational characteristics enable batch processing of hundreds of sessions and real-time analysis of streaming interactions.

The 84 percent success rate for YouTube video retrieval and processing identifies technical bottlenecks. Failures result from hosting changes, access restrictions, and quality issues. For large-scale deployment, robust error handling and retry mechanisms would improve yield. Audio quality variation introduces performance degradation compared to controlled recordings. For applications requiring consistent reliability, quality assessment mechanisms could filter low-quality segments during preprocessing.

4.5. Methodological Perspective

From a methodological perspective, the proposed framework should be interpreted as a contribution to data mining for extreme multi-label learning and multimodal fusion under heterogeneous noise, rather than as a deployed clinical or evaluative system.

4.6. Limitations and Future Directions

Automated annotation reliability limits YouTube model performance. These findings are consistent with systematic label noise introduced by large language model annotation compared to expert human review [59]. Rare class performance suffers from limited training data. Support of 8 to 18 examples for certain communication styles provides insufficient learning signal [61,65]. Temporal context remains unexploited as utterance-level processing ignores conversational dynamics and session-level progression [62,66]. Cross-domain generalization requires validation as models trained on one therapeutic approach may not transfer to different modalities. Video modality exclusion represents missed opportunity as facial expressions convey substantial affective information [63,64].

Future research directions include semi-supervised learning combining automated annotation with selective expert review. Few-shot learning approaches could improve rare class generalization from limited examples. Session-level temporal modeling through recurrent or transformer architectures could capture therapeutic progression. Multimodal expansion incorporating visual features would provide comprehensive analysis. Cross-cultural validation would establish generalizability across diverse contexts.

5. Conclusions

We present a comprehensive bidirectional framework for clinical counseling analysis addressing patient emotion recognition and provider behavior assessment using real-world data. The framework advances multi-label classification, class imbalance handling, and multimodal fusion through three key contributions.

First, frequency-stratified class weighting with dynamic per-class threshold optimization achieves substantial macro-F1 improvement over baseline multi-label classification [5,7]. The approach maintains reasonable performance on classes with limited training examples while preventing gradient collapse for minority classes.

Second, an automated processing pipeline for YouTube psychotherapy sessions addresses the scarcity of naturalistic clinical data [39,40,41]. The pipeline processes hundreds of sessions into thousands of annotated segments, establishing reproducible methodology for leveraging publicly available clinical content.

Third, cross-modal attention architecture learns content-dependent prosodic associations for multimodal behavior recognition [42,43,58]. The architecture achieves macro-F1 of 0.91 on controlled human-annotated data, comparable to inter-rater reliability reported among trained human annotators. On YouTube data, the architecture achieves macro-F1 of 0.71, quantifying the performance gap between controlled and real-world annotation regimes.

The framework establishes viability of analyzing naturalistic therapy data from publicly available sources. Cross-modal attention mechanisms learn interpretable patterns. Automated annotation pipelines validated against human expert standards provide methodology for transitioning from controlled to real-world clinical data. Future work should focus on semi-supervised learning, temporal modeling, and longitudinal validation. These advances will enable comprehensive platforms supporting interaction-level analysis essential for computational study of therapeutic communication.