A Multimodal Pain Sentiment Analysis System Using Ensembled Deep Learning Approaches for IoT-Enabled Healthcare Framework

1. Introduction

In recent years, modern security systems have undergone significant transformation due to the revolutionary technology of the Internet of Things (IoT) [1]. IoT-enabled analytics and automation systems, which rely on software, sensors, and electronic devices, collect and transmit data over the Internet without requiring human intervention. Wireless networks powered by these advanced systems improve security and operational efficiency across various sectors, including healthcare, business, retail, military, and transportation. In healthcare, IoT plays a crucial role in automating patient monitoring and nurse call systems. For example, patient monitoring systems process data from smart devices such as thermometers, ECGs, pulse oximeters, and sphygmomanometers to predict health conditions and alert medical staff to initiate appropriate treatments. Building on these advancements, this study proposes a multimodal pain sentiment analysis system for remote patient monitoring, utilizing IoT-enabled medical sensors [2].

Multimodal sentiment analysis systems combine data from multiple sources: text, audio, and videos to assess sentiment. These are used to analyze customer opinions, feedback, and experiences. For example, suppose a customer surveys their experience with a product or service. In that case, a sentiment analysis system can analyze the customer’s words, tone of voice, facial expressions, and body language to determine the customer’s sentiment. The system can then categorize the sentiment as positive, negative, or neutral. In addition, the system can analyze the customer’s emotions, such as happiness, anger, and frustration. These emotions allow companies to gain more insights into customer sentiment and make informed decisions on improving their products and services. Here, multimodal pain sentiment analysis is extracting sentiment from a combination of textual and non-textual data sources related to pain. The healthcare framework encompasses medical professionals, hospitals, doctors, nurses, external patient clinics, clinical tests, telemedicine, medical devices, and machinery, as well as health insurance services [3]. Recent advancements in technology have significantly enhanced healthcare by introducing electronic and mobile-based health facilities. Mobile health (m-health) systems now provide patients with daily health support, enabling continuous monitoring and assistance [3]. These electronic healthcare systems utilize speech recognition, gesture movement, voice, and facial expression analysis to remotely assess patients’ health status. Gesture-based interaction and identification have become increasingly important in human-computer interaction (HCI), making e-healthcare systems more intuitive and accessible for doctors and nurses [4]. The proposed sentiment analysis system enriches these e-healthcare solutions, offering more cost-effective and efficient healthcare services. Furthermore, essential requirements such as authenticity, accessibility, security, advanced tools, and privacy have been carefully integrated into the system to ensure its reliability and effectiveness.

In recent years, computer usage has had a hugely significant impact since its advent. Web surfing, watching videos, playing games, online teaching and examination, e-commerce, and storing and retrieving data are some activities involving computer usage. As a result, Human-Computer Interaction (HCI) has grown as one of the most exciting domains in the field of research [5]. Thus, HCI is categorized as the live research areas [6]. Web services control various cloud-based devices as a platform of the Internet of Things (IoT) coming into the scenario along with HCI. In the last few years, IoT has earned immense popularity. The IoT-based devices are applicable in smart homes, traffic congestion control, and automatic driving. Along with these applications, the IoT also works with Artificial Intelligence (AI) in the healthcare infrastructure. The healthcare industry is considered a significant sector from the employment point of view, revenue, and health-associated convenience, and it contributes to a healthy civilization in smart cities. For patients who are non-verbal, the American Society provides an example of how to estimate pain emotion. The description shows a hierarchical structure for pain estimation, considering facial representation-dependent pain estimate as a legitimate method. Payen et al. [7] concluded that a pain valuator-based facial expression could be used for seriously ill, unflappable patients. A pain estimation analysis on aged patients with substantial mental sickness and due to growing mental illness, they are not able to communicate verbally, so the facial expressions are proper to assess the appearance of the pain of those particular patients. A pain valuator-based facial expression could be employed for highly ill, unflappable patients, according to Payen et al. [7]. An analysis of facial expressions in elderly individuals with severe mental disease is appropriate for estimating their level of discomfort because these patients are unable to express their feelings orally due to their worsening mental illness. McGuire and colleagues [8] came to the conclusion that individuals in the non-intellectual population may not recognize their discomfort or receive adequate treatment, mainly if they are unable to communicate their annoyance effectively.

A strong correlation exists between behavioral responses—such as grimacing, closing eyelids, and wincing—and procedural pain, as shown in studies involving patients [9]. Facial expressions, according to [10], effectively convey unpleasant emotions, making them essential indicators for assessing pain. Sentiment analysis systems in real-world applications remotely detect and interpret patients’ facial expressions to determine emotional states [3]. Pain assessment in clinical settings plays a crucial role in delivering appropriate care and evaluating its effectiveness [11]. The gold standard for pain reporting often involves patient self-reports using verbal numerical scales and visual analogue scales [12]. Certain vulnerable groups—such as those who are physiologically immobile, terminally ill [8], critically ill but coherent [7], experiencing psychological distress [13], or suffering from cancers affecting the neck, head, or brain require reliable technological support to provide consistent pain alerts, alleviating the burden on healthcare professionals. Other sentiment analysis systems have been developed for conditions like Parkinson’s disease, depression, intent analysis, facial emotion recognition, and contextual semantic search. These systems analyze text, image, and video data using deep learning techniques to classify emotions and identify patterns. The primary contributions of this work are outlined as follows:

• Deep learning-based frameworks utilizing handcrafted features are employed in the proposed audio-based pain sentiment analysis system.
• Several top-down deep learning frameworks are developed using Convolutional Neural Network (CNN) architectures to extract discriminative features from facial regions for improved feature representation.
• Performance enhancements are achieved through various experiments, including batch vs. epoch comparisons, data augmentation, progressive image resizing, hyperparameter tuning, and transfer learning with pre-trained deep models.
• Post-classification fusion techniques are applied to improve system accuracy. Classification scores from different deep learning models are combined to reduce susceptibility to challenges such as variations in age, pose, lighting conditions, and noisy artifacts.
• The proposed system’s applications can be extended to detect mental health conditions such as stress, anxiety, and depression, contributing to treatment assessment within a smart healthcare framework.
• Finally, a multimodal pain sentiment analysis system is constructed by fusing classification scores from both image-based and audio-based sentiment analysis systems.

This paper is structured as follows: Section 2 provides an overview of the related work. Section 3 presents the implementation details of the proposed system. Experimental evaluation and analysis of the system are discussed in Section 4. Finally, the conclusions are outlined in Section 5.

2. Related Work

Numerous automatic techniques for identifying pain are insufficiently proficient in identifying pain based on facial representations. In the field of pain recognition systems from facial expression, some well-known feature extraction techniques, such as Active Appearance Models and Active Shape Model [14], Local Binary Pattern (LBP) [15], and Gabor wavelets [16], are used. It has been noted that the approaches now in use are related to facial representation-based pain detection. Specific techniques rely on deep learning, whereas others are focused on non-deep learning [17]. In recent years, significant progress has been made in this particular field of study. Regarding categorization, it has been noted that the retrieved face features [18] influence classification. A few popular feature extraction approaches perform this feature extraction task. The classification tasks are carried out using popular supervised machine learning techniques such as K-Nearest Neighbour [19], Support Vector Machine [20], and Logistic Regression [21].

A method for identifying acoustic pain detection in newborns based on audio features has been proposed in [22]. To explore the relationship between subjective pain reports and measurable biosignals in human speech, prosody was developed by Oshrat et al. [23]. In [24], an automatic evaluation system for pain based on paralinguistic speech was introduced, aiming to improve the objectivity and accuracy of pain diagnosis. A multimodal pain intensity recognition system using both audio and video channels was presented in [25]. Hossain [26] developed a multimodal sentiment analysis system for assessing patient states within a healthcare framework, using speech and facial expressions. Zeng et al. [27] conducted a psychological perspective survey on human emotion perception through audio, visual, and spontaneous expressions. Finally, Hong et al. [28] explored an improved method for automatic pain level recognition, incorporating the pain site as an auxiliary task.

In computer vision, features extracted by different CNN frameworks are utilized for different purposes like emotion recognition, object identification, etc. It has also been noticed that CNN dragged features perform better than handwrought features. In the last few years, growth in deep learning-based research activity has been found, and the outcome of this research activity has become the prominent solution to various cutting-edge problems. The classification of pain using the CNN framework was achieved in [29]. The proposed sentiment analysis system also leverages the high performance of CNN models, incorporating frameworks like GoogleNet [30] and AlexNet [31]. Liu et al. [32] introduced a hybrid neural network model combining CNN and BiLSTM to analyze emotions. A comparative study on sentiment analysis using bio-inspired algorithms was published by Yadav and Vishwakarma [33]. In [34], Hermawan et al. applied the concept of "Triplet Loss for Pain Detection" to classify pain based on different facial expressions. Haque et al. [35] proposed a two-part method, where the first part uses a 2D-CNN architecture to extract frame-wise features for pain recognition, and the second part employs Long Short-Term Memory (LSTM) to estimate temporal relations among the frames for sequence-level pain recognition. Menchetti et al. developed a pain detection system using two-stage deep learning networks: the first stage uses a Deep Neural Network (DNN) to extract facial landmarks, while the second stage employs a Convolutional Neural Network (CNN) to detect various Action Units [36]. Bandar et al. [37] proposed a feature learning-based classification method for pain detection.

Algorithms based on deep learning can enlighten internal dimmed patterns in complex datasets, which plays a vital role in feature extraction and classification [31]. Dashtipur et al. proposed a Multimodal Framework based on context awareness on Persian Sentiment Analysis [38]. Based on the movie review comments on emotion analysis, Sagum et al.[39] have proposed a sentiment measurement technique. Rustam et al. [40] compared the performances of various supervised machine-learning techniques based on Sentiment Analysis of COVID-19 tweets. The pain or anguish experienced when exposed to sound is called acoustic or auditory pain. This discomfort may appear in some ways, such as (i) Hyperacusis, An increased sensitivity to common environmental noises that can cause pain or discomfort even at low decibel levels; (ii) Phonophobia, Severe dread of loud noises or avoidance of them; frequently coupled with physical discomfort and anxiety; (iii) Tinnitus: This condition causes ringing, buzzing, or other phantom noises to be perceived in the ears; however, it is not always uncomfortable [41]. It has also been demonstrated that the acoustic elements of music, such as rhythm, speed, and harmony, will establish the connection between pain relief and music therapy.

Ghazal et al. [42] introduced a Principal Component Analysis (PCA) method to reduce the dataset and extract the most significant facial features, followed by the application of a CNN for face recognition. In their survey, Rasha M. et al. [43] identified and reviewed various pain detection methods, including multitask neural networks, CNN-LSTMs, and cumulative attributes. Additionally, they proposed a method for comparing feature extraction techniques, such as Skin Conductance Response (SCR) and Skin Conductance Level (SCL), to extract various features like root mean square, mean value of local maxima-minima, and mean absolute value for pain detection [44]. Patrick et al. [45] introduced a pain detection method that incorporated both Uni-Modal and Multi-Modal concepts. Eshan et al. [46] compared a Random Forest classifier using Facial Activity Descriptors with the performance of a deep learning-based approach for pain detection. Kornprom et al. [47] proposed a pain estimation method along with a pain intensity metric, using tools such as the Visual Analog Scale, Observer Rating Scale, Affective Motivational Scale, and Sensory Scale. Leila et al. [48] developed a specific model for automatic pain detection using the VGG-Face model on the UNBC dataset. Jiang et al. [49] conducted experiments employing the GLA-CNN approach for pain detection. Finally, Ehsan et al. [50] proposed a pain classification system using video data from the X-ITE dataset.

3. Proposed Methodology

This section outlines the implementation of the proposed pain sentiment analysis system. Facial expressions play a significant role in detecting pain sentiment, with the features of these expressions contributing to the identification of pain intensity. In cases where facial features are unavailable, audio-based sentiment features are utilized. Additionally, as facial expressions may vary from person to person due to the perception of pain, audio-based information can assist in determining the pain level. Therefore, combining both modalities enhances the performance of the proposed pain detection system. While several established pain detection algorithms exist, there is still room for improvement in pain recognition techniques. In the current landscape, cloud-dependent mobile applications that rely on streaming data and live video processing are gaining popularity. These applications typically consist of two components: the front-end, which runs on mobile devices, and the back-end, which operates in the cloud. The cloud provides enhanced data processing and computational capabilities. Our proposed method allows for the execution of complex applications on devices with existing resources and potential, optimizing performance. This section describes a sentiment analysis system for estimating pain intensity based on facial expressions. The research integrates cache and computing facilities to provide essential modules for an electronic healthcare system. The proposed system is divided into five key components for implementation, with the block diagram of the pain recognition system shown in Figure 1.

This block diagram of our proposed system shows that there will be a cloud network capable of storing some data. Various devices from outside the system will communicate through that network. Our database is also connected to the cloud network. Our database is mainly consisting of images, videos, and audio. Before classification, images are fetched from the database, preprocessed, and reshaped based on the algorithm. The core implementation of the proposed system is as follows: once the preprocessing is completed, the images are separated into training-testing segments and sent to two different $CNN$ architectures. These $CNN$ architectures are trained with the training images separately. Once the training is completed, the prediction model is obtained. The prepared prediction models generate scores corresponding to the input sample. Then, based on the score level fusion technique, the final decision is made regarding the outcome of the proposed system. The obtained prediction model is connected to the network so that the pain’s intensity can be predicted from external remote devices for the patients. The detailed implementation of the proposed system is elaborated on in the subsections below.

3.1. Audio-Based PSAS

In the digital world, audio-based media content is kept in the form of signal processing, where the dealing, processing, and manipulation of audio signals are performed for various applications such as speaker recognition, gender classification, age classification, disease detection, entertainment, and health care. Converting analogue signals to digital signals in audio files is much more challenging among the several types of media content, such as texts, images, videos, and audio. The chances of unwanted noise introduction and unbalanced time-frequency distribution make the audio-based recognition systems much more challenging. In this work, an audio-based speaker’s sentiment analysis has been performed to detect the intensity levels of their body-part pain sentiments. The purpose of an audio noise reduction system is to remove noise from audio signals, such as speech. There are two types of these systems: complementary and non-complementary. Before properly recording the audio stream, complementary noise reduction entails compressing it. Conversely, single-ended type, or non-complementary, noise reduction works well for lowering noise levels [51]. Digital and analogue devices are vulnerable to noise because of specific characteristics. It is possible to extract different aspects more precisely when noise is removed from audio signals. In this work, we have utilized filtered audio signals to extract features for the audio-based pain sentiment analysis system (PSAS). Here, identifying pain sentiment levels is based on analyzing the types of emotions a person has during human-computer-based interaction in the healthcare system. Fortunately, facial expressions are analyzed or recognized in image or video-based emotion recognition. Still, in real-time scenarios, video-based media might record the vocal emotions of a patient in the online healthcare framework. Then, there will be the requirement for audio-based emotion recognition systems. Here, the feature representation using the deep learning-based approach for audio-based classification has been demonstrated in Figure 2.

This work considers a speaker’s audio recording for the audio-based pain sentiment analysis. The audio data is in the form of a waveform audio file containing a byte sequence representing some audio signals over time. Hence, analyzing these sequences of bytes in terms of numeric feature representation, some feature extraction techniques are employed for feature computation from the waveform audio files. These feature extraction techniques are described as follows:

• Statistical audio features: Here, the audio signal initially undergoes a frequency domain signal analysis using fast Fourier transformation (FFT) [52]. Then, the computed frequencies are used to calculate descriptive statistics such as mean, median, standard deviations, quartiles, and kurtosis. Further, the magnitude of derived frequency components is used to compute Energy, and the amplitude is used to calculate Root Mean Square Energy.
• Mel Frequency Cepstral Coefficients (MFCC): This is a well-known audio-based feature computation technique [53] where the derived frequency components from FFT undergo to compute the logarithm of the amplitude spectrum. Then, the Mel scale is introduced to the logarithm of the amplitude spectrum, discrete cosine transformations (DCT) are applied on the Mel scale, and finally, 2-13 DCT coefficients are kept. The rest are discarded during feature computations.
• Spectral Features: Various spectral features like spectral contrast, spectral centroid, and spectral bandwidth are analyzed from the audio files. These spectral features [54] are related to the spectrogram of that audio files. The spectrogram represents the frequency intensities over time. It is measured from the squared magnitude of the Short-Time Fourier Transform (STFT) [55], which is obtained by computing FFT over successive signal frames.

So, using these above feature extraction techniques, $d_1$-dimensional statistical features ($f_1$), $d_2$-dimensional MFCC features ($f_2$), and $d_3$-dimensional spectral features ($f_3$) are extracted from each speech-audio. Then, these feature vectors are concatenated such that $f_{audio}=(f_1,f_2,f_3)$. Now, $f_{audio}\in {\mathbf{R}}^{1\times d}$, $d=d_1+d_2+d_3$, undergoes to the proposed deep-learning based feature representation followed by classification for speech-audios. During feature representation, d-dimensional feature vector is transformed to 512-dimensional features, then 256 to 128-dimensional features. Finally, during classifications, maps into 3-class audio-pain problems. The list of parameters required for the audio-based deep learning technique is demonstrated in Table 1.

3.2. Image-Based PSAS

During the image acquisition phase, noise is often introduced, which reduces the textural information and detail of facial patterns essential for pain sentiment recognition. This noise can negatively impact the experiment’s accuracy. In the preprocessing phase, a color image $\mathcal{I}$ is processed to extract the facial region for pain emotion detection. This research employs a tree-structured part model [56], which is effective for various face poses. For frontal faces, the method identifies a total of sixty-eight landmark points, while for profile faces, it identifies thirty-nine. Once the facial landmarks are detected, the extracted facial portion undergoes an image normalization process. Bilinear image interpolation is applied to normalize the extracted face portion to an image size of $N\times N$. After normalization, feature computation tasks are performed. Several deep learning approaches are utilized to extract valuable and discriminating features. Different deep learning architectures are applied individually to the same dataset, and the results from both architectures are fused to make the final decision in the proposed system. The process of detecting the facial region for the proposed pain identification system is illustrated in Figure 3.

3.3. Image-Based Feature Representation Using Proposed Deep Learning Architectures

Visual feature extraction provides a reliable and consistent way to distinguish between various objects. In tasks such as emotion classification and facial expression identification, effective feature representation plays a critical role [57]. It is not feasible to directly classify facial emotion-based pain levels using raw, unprocessed features. In this work, strong and efficient local-to-global feature representation schemes are applied for both feature synthesis and representation. Convolutional Neural Network (CNN) architectures have successfully extracted a range of distinct features from images and videos through various deep learning techniques [58]. In CNNs, different kernels process images using mixed convolution operations [59]. After convolution, the network employs a pooling technique with weight sharing to optimize network training parameters. While statistical and structural-based methods in computer vision exist, state-of-the-art CNN feature representation techniques are highly effective for facial expression-based pain sentiment analysis. These general CNN features provide superior performance by handling articulation and occlusion in face images captured in unconstrained settings [58]. They enhance performance by overcoming the challenges of occlusion and articulation in face images taken in uncontrolled environments. The proposed system’s performance includes a sentiment analysis component that analyzes facial expressions, balancing deep learning features with variables that influence the effectiveness of these features.

The proposed feature representation schemes describe the two primary functions of the CNN baseline model: feature extraction and classification. The architecture of the proposed CNN models consists of seven to eight deep image perturbation layers. These layers utilize the ReLU activation function during convolution operations, followed by max-pooling and batch normalization to extract features. The extracted feature maps are then passed through two fully connected flattened layers for classification tasks. The performance of the proposed CNN model has been enhanced through progressive image resizing techniques, which add new levels to the model. These techniques help reduce overfitting and address imbalanced data issues, while also minimizing the computational resources required. Proper learning of weights is crucial, as only the pre-trained weights from the previous layers are utilized. In addition, techniques such as activation functions, batch normalization, image augmentation, regularization methods (e.g., mix-up optimizer and label smoothing), and other regularization techniques are employed to further optimize the model. At the core of the CNN architecture is the convolution operation, which plays a fundamental role in feature extraction [57]. The convolution operation involves combining two functions to produce a third, where features are extracted from the images. A $t\times t$ sized kernel or filter is used to perform the convolution over the input image, producing feature maps. This process involves sliding the filter over the image and applying non-linearity, followed by matrix multiplication at each location and adding the result to the feature map. To further enhance the performance of the proposed recognition system, fine-tuning of parameters is carried out, and score-based fusion techniques are applied for improved results.

The Convolution Neural Network (CNN) architectures based on deep learning are being used here [30] to perform feature learning and classify facial region images into two or three pain-level class problems. The performance of an emotion recognition system based on facial expressions is severely compromised by over-fitting, which leads to expression alterations such as changes in head pose, age, identity, and illumination. The proposed approach emphasizes these issues and restores the CNN model’s computational complexity. The shape and texture information can be extracted from the images with the hybrid nature of the proposed $CNN$ architectures. The recommended system’s feature description uses textual details and shapes to give both geometrical and appearance features. The primary drawback of the CNN model, which is thought to be significant computation power loss, can be mitigated by using Graphical Processing Units. The Graphical Processing Units are primarily designed to handle a high volume of computation, including countless levels of representation and concepts for handling transfer learning, pretraining, data patterns, and techniques for fine-tuning CNN models. In this work, during training the proposed $CNN$ architectures, some heavy databases are used to train the architectures by adjusting the weights of the network based on the number of defined pain-class problems. So, this research proposes two Convolutional Neural Network (CNN) architectures, such as $CN{N}_A$ and $CN{N}_B$. The respective trained models generated from these $CNN$ architectures are $f_{CNN-A}$, and $f_{CNN-B}$. For a better understanding and transparency regarding the models, the specifics of these architectures with their use of input-output hidden layers, convoluted image output shape, and parameters produced at each layer are further described. These are shown in Figure 4 and Figure 5, respectively. Similarly, Table 2 and Table 3 describe the list of parameters required for training these $CNN$ architectures.

The convolution operation is the core operator in this network and serves as the fundamental structural block of the CNN architecture. This operation is responsible for extracting features from images. To apply the convolution operation to the input image F, a filter or kernel of size $t\times t$ is used to generate feature maps. The convolution operation is performed by sliding the filter across the input image I, applying non-linearity at each location. Element-wise matrix multiplication is carried out, and the results are aggregated to form a feature map element corresponding to the center of the mask pattern. To improve the performance of the proposed CNN architectures, fine-tuning of parameters is applied. Max-pooling is used to calculate the maximum value for strips of the feature map, resulting in a down-sampled feature map. The max-pool layer serves multiple purposes: (i) it ensures uniform translation, (ii) reduces computational overheads, and (iii) enhances the generation of discriminative features from the input image I. The $CN{N}_A$ architecture comprises six convolution blocks. Each block contains a series of layers, including a Convolutional layer, a MaxPooling layer, a BatchNormalization layer, and a Dropout layer. These layers are followed by a flattened layer. The network also includes three fully connected blocks, each with two Dense layers and a Dropout layer. The final output layer consists of a Dense layer. During compilation, the ’Adam’ optimizer is employed for optimization.

• Convolutional Layer: The convolutional layer generates feature maps Z by applying a kernel or mask of size $t\times t$ to perform a convolution operation on the input image I. During this process, the kernel slides across the input image F, associating each region of the image with a non-linear activation function. At each location, element-wise matrix multiplication is performed, and the resulting data is summed to form an element in the feature map corresponding to the central region of the mask pattern. The convolutional layers with N (e.g., 32, 64, 128, 256, 512, 1024, etc.) are implemented as $Z^{\left[l\right]\left[M\right]}=\mathrm{ReLU}(W^{\left[l\right]\left[M\right]}*I+b^{\left[l\right]\left[M\right]})$, where M denotes the number of convolutional layers in the CNN architecture, with $M=8$, $W^{\left[l\right]\left[M\right]}$ represents the weights of the M-th convolutional layer, * indicates the convolution operation, and $b^{\left[l\right]\left[M\right]}$ corresponds to the biases of the M-th convolutional layer.
• Max-Pooling Layer: The max-pooling operation contributes to fine-tuning the parameters of the proposed CNN architecture, enhancing its performance. This operation involves calculating the maximum values within segments of the feature map, followed by the creation of down-sampled feature maps. Max-pooling offers several advantages, including uniform translation, reduced computational overhead, and improved selection of discriminative features from the input image I. If $Z^{\left[l\right]\left[M\right]}$ represents the output feature map from the input image F, the max-pooling operation is applied as $P^{\left[l\right]\left[M\right]}=\mathrm{MaxPooling}(Z^{\left[l\right]\left[M\right]},\mathrm{pool}\mathrm{size}=(2,2))$.
• Batch Normalization Layer: Batch normalization acts as a controller, enabling the model to utilize higher learning rates, which are beneficial for performing image classification tasks [60]. Specifically, it stabilizes gradients across the network by mitigating the effects of parameter scales or initial values.
• Dropout Layer: In $CN{N}_A$ architectures, the dropout layer serves as a regularization technique to prevent overfitting. It compels the network to develop more robust and generalized features during training by randomly deactivating a subset of neurons. By reducing the dependency between neurons, the dropout layer enhances model performance and generalization on unseen inputs, leading to more efficient and reliable $CNN$ architectures.
• Flatten Layer: In $CN{N}_A$, the flatten layer acts as a bridge between the convolutional and fully connected layers. It transforms the multi-dimensional feature maps into a one-dimensional vector, making them compatible with the subsequent fully connected layers. By converting complex spatial data into a linear format, the flatten layer enables the network to capture higher-level patterns and relationships, which are essential for tasks such as classification and decision-making.
• Fully Connected Layer: In every $CN{N}_A$ architecture, the fully connected layers are essential for performing classification. They transform the complete feature maps into isolated vectors, which are then used to generate class scores. This classification process produces a final vector whose size is determined by the number of class labels. The weight values in the dense layer help establish the connection between the input and output layers. The input to this layer is the output of the flatten layer, denoted by F, and the output of the R-th fully connected layer is represented by $A^{\left[l\right]\left[R\right]}=\mathrm{Activation}(W^{\left[l\right]\left[R\right]}F+b^{\left[l\right]\left[R\right]})$. The fully connected layer also incorporates the dropout layer for regularization.
• Output Layer: The output layer is responsible for classification, and the Softmax activation function is employed for this purpose. It utilizes the fully connected layer once more. The final output of this layer is denoted by Y, and its input, which comes from the fully connected layer, is represented by $A^{\left[l\right]}$. The output is given by the equation $Y=\mathrm{Softmax}(W^{[l+1]}{A}^{\left[l\right]}+b^{[l+1]})$, where $W^{[l+1]}$ are the weights, and $b^{[l+1]}$ are the biases associated with this layer.

After training, the trained model $f_{CNN-A}$ is derived from the $CN{N}_A$ architecture. In the higher feature representation scheme, another $CN{N}_B$ architecture is proposed, which is composed of eight blocks, where each block is also composed of a convolution layer, max-pooling, batchNormalization, and dropout layers and, after that, three flatten layers. Each flatter layer comes after One dense layer, one batch normalization, the activation function "Relu," and one dropout layer. Lastly, this architecture contains the final dense layer. An Adam optimizer is used during the architecture compilation process. After training, the trained model $f_{CNN-B}$ is derived from the $CN{N}_B$ architecture. The performance of these $CN{N}_A$ and $CN{N}_B$ architectures are affected by several factors to improve not only the performance but also the efficacy of the derived model with better feature abstraction schemes for the proposed image-based pain sentiment analysis system.

(i) The scheme’s first factor is progressive image resizing ($Schem{e}_1$). The input layers in $Schem{e}_1$ will continuously obtain the image as input data. Convolution operations, on the other hand, will be carried out by numerous masks or filter banks. Various convolved images, or feature maps, are obtained based on the specific filters [61]. Convolution operations will perform more as image size increases, and the network will need to perform enormous computations as the number of filters rises. The parameters in these filter sets will be considered as refined weights. In terms of max-pooling layers, the number of parameters in the network similarly affects this layer’s computational barrier. As a result, it has been found that when patterns are analyzed from multiple image resolutions, the $CNN$ models perform better regarding feature representation of images using deep learning techniques. Further analysis of hidden patterns in the feature maps is made possible by adding more layers to the architecture as image resolution increases. Motivated by these findings, we have implemented several multi-resolution facial image techniques using different CNN architectures and layers. During feature representations, in this work, two different image resolutions of the normalized facial region F have been considered for $Schem{e}_1$ such that the $F_{k\times k}$ is down-sampled to $F_{k_1\times k_1}$, and $F_{k_2\times k_2}$, whereas $k>k_2>k_1$.

(ii) The second factor is $Schem{e}_2$, which pertains to the Transfer Learning and Fine Tuning effects that influence the performance of the proposed CNNs. Transfer learning enables further development of discriminant features required for the PSAS, which demands higher-order feature representation. In this process, certain layers and their corresponding parameters are frozen while others are retrained to reduce the computational load. Transfer learning allows a model trained for one problem to be effectively applied to other related problems. In our proposed model, we leverage a transfer learning technique that integrates convolutional neural network (CNN) deep learning. To address the same problem, the new model utilizes the collection of layers and weights from a pre-trained model. This approach helps reduce training time and minimizes generalization error.

(iii) The third factor, $Schem{e}_3$, involves the deployment of Fusion techniques. In this approach, the trained models generate features or scores, which are then transformed into other sets of features or scores based on their values. These fusion techniques are applied to combine the classification scores produced by the trained deep learning models, improving the performance of the Pain Sentiment Recognition System (PSRS). The fusion of test sample scores is performed using methods that fall under two categories: post-classification (b) and pre-classification (a) techniques. In this work, post-classification methods, such as Sum-of-score, Product-of-score, and Weighted-sum-of-score, are used, as the trained deep learning models are already employed. Consider two trained deep learning models, ${\zeta }_1$ and ${\zeta }_2$, which solve an L-class problem and generate classification scores $l_1\in {\mathbb{R}}^{1\times L}$ and $l_2\in {\mathbb{R}}^{1\times L}$ for a test sample a. The fusion methods are applied as follows:

• (i) The Sum-of-score is defined as $l_s=(l_1+l_2)\in {\mathbb{R}}^{1\times L}$,
• (ii) The Product-of-score is defined as $l^p=(l_1\times l_2)\in {\mathbb{R}}^{1\times L}$,
• (iii) The Weighted-sum-of-score is defined as $l^w=({\rho }_1\times l_1+{\rho }_2\times l_2)\in {\mathbb{R}}^{1\times L}$,

where ${\rho }_1$ and ${\rho }_2$ are the weights assigned to models ${\zeta }_1$ and ${\zeta }_2$, respectively, such that $\rho ={\rho }_1+{\rho }_2$. Experimentally, it has been observed that when the performance of ${\zeta }_1$ exceeds that of ${\zeta }_2$, then ${\rho }_1>{\rho }_2$, and vice versa.

4. Experiments

The proposed multimodal pain sentiment analysis system has been implemented using Python in this research. The implementation was conducted on a system equipped with an Intel Core i5 processor (3.20GHz) and 16GB of RAM, running the Windows 10 operating system. Several essential Python libraries were utilized during the implementation, including Keras [62] and Theano [63]. These packages facilitated the development of deep learning-based convolutional neural network (CNN) architectures and techniques, which are central to the proposed system. In this work, facial image regions were employed to extract critical information that aids in determining the pain-level intensity of individual subjects.

4.1. Databases Used

The proposed system has been evaluated using one audio-based dataset, the "Variably Intense Vocalizations of Affect and Emotion Corpus (VIVAE)" [64], and two image-based datasets: the UNBC-McMaster shoulder pain expression archive dataset [65] and the 2D Face Set database with pain expressions, known as D2 [66]. The UNBC-McMaster shoulder pain expression archive dataset consists of 129 participants (66 female and 63 male), all of whom experience shoulder pain identified by physiotherapists at a clinic. The videos in this dataset were recorded at McMaster University. Participants in this study were assumed to be suffering from conditions such as bone spur, tendinitis, rotator cuff injuries, bursitis, arthritis, impingement syndromes, subluxation, capsulitis, dislocation, all contributing to their shoulder pain. From these videos, images were extracted and labeled according to pain intensity, ranging from ’No pain’ to ’high-intensity pain’ classes. In this research, the images were categorized into two or five pain classes. For the 3-class problem, the following labels were used: Non-Aggressive (NAG) for No-pain (${\mathcal{R}}_1$), Covertly Aggressive (CAG) for Low-pain (${\mathcal{R}}_2$), and Overtly Aggressive (OAG) for High-pain (${\mathcal{R}}_3$). Some image samples from the UNBC-McMaster dataset are shown in Figure 6, and the dataset’s description, including the 2-class and 3-class problems, is provided in Table 4. The second dataset used is the 2D Face Set Database with Pain Expression ($2DFPE$) [66], which contains 599 images from 13 women and 10 men. This dataset primarily involves a 2-class problem, with 298 images labeled as ’No-Pain’ (${\mathcal{P}}_1$) and the remaining 298 images labeled as ’Pain’ (${\mathcal{P}}_2$). A description of this database is provided in Table 4, and several sample images from the 2DFPE database are shown in Figure 7.

The BioVid Heat Pain Database [67] is the third dataset used in this research, and it is based on video samples. The dataset consists of twenty videos per class and five distinct classes for each of the 87 subjects (persons). The five distinct data categories in the dataset are Part A through Part E, with Part A being selected for this work. From the list of 87 subjects, five subjects were chosen for Part A, with 50 videos considered for each subject. As there are five different classes for each subject, ten videos of each class were selected. In total, 250 videos were chosen, each lasting 5.5 seconds and containing 138 frames. The frames or images from each video were extracted and assigned one of five distinct labels, ranging from "No pain" to "extreme intensity pain." These labels correspond to the following pain intensity categories: baseline (${\mathcal{S}}_1$), Intensity of ’Pain-Level 1’ (${\mathcal{S}}_2$), Intensity of ’Pain-Level 2’ (${\mathcal{S}}_3$), Intensity of ’Pain-Level 3’ (${\mathcal{S}}_4$), and Intensity of ’Pain-Level 4’ (${\mathcal{PI}}_5$). Example images from the BioVid Heat Pain Database are shown in Figure 8. A description of the BioVid Heat Pain Dataset, including its correlation to image samples for specific pain classes, is provided in Table 4.

The fourth database is a speech audio database, the Variably Intense Vocalizations of Affect and Emotion Corpus (VIVAE) [64]. This VIVAE database is composed of audio files of non-speech emotion vocalizations of human beings. There are 1085 audio files in this database, while eleven speakers have participated in giving their speeches to this database. Each speaker expresses their fear, anger, surprise, pain, achievements, and sexual pleasure emotions. Each speaker’s emotion has been recorded into four intensities: ’ low’, `moderate’, `peak’, and `strong’. Since this work is related to detecting pain intensities among Non-Aggressive (NAG) (No-pain) (${\mathcal{T}}_1$), Covertly Aggressive (CAG) (Low-pain) (${\mathcal{T}}_2$), and Overtly Aggressive (OAG) (High-pain) (${\mathcal{T}}_3$) classes, so, for convenience, `low’, and `moderate’ are kept in NAG class, `peak’ in CAG class, `strong’ in OAG class. At the same time, all four emotions are categorized into these three classes based on their intensities. Hence, this database contains 530 audio of NAG, 272 audio of CAG, and 282 audio of OAG classes. So, different training-testing sets are employed during experiments to test the proposed system.

4.2. Results for Image-Based PSAS

There are two specific steps in the feature learning with the classification process of an image-based pain sentiment analysis system (PSAS): (i) image preprocessing and (ii) feature learning with classification. The facial region is extracted as the region of interest during this image preprocessing step. After that, the image size is changed to $128\times 128\times 3$ and $192\times 128\times 3$ for the normalized facial region $\mathcal{F}$. Then, the training dataset’s transformed facial images are subjected to the proposed CNN architectures (Figure 4) and (Figure 5), where the input image for the $CN{N}_A$ architecture is $128\times 128\times 3$, and the input image for the $CN{N}_B$ architecture is $192\times 192\times 3$. Activities such as feature computation and pain sentiment categorization are trained into the CNN architectures. The concentration on training the CNN architectures has been done to understand these architectures’ functions better. On the other hand, the trained model $f_{CNN-A}$ and $f_{CNN-B}$ go through to find out how well the system works with test samples. The $CN{N}_A$ architecture with a (50-50%) training-testing strategy is the starting point of the experiment. It is employed for the 2DFPE Database (2-class Pain levels) and UNBC Database (3-class Pain levels) databases.

• $Schem{e}_1$ experiments: This experiment is associated with parameter learning, which builds on batches and epochs and is fundamental to every CNN architecture. Both batch vs. epoch factors impact the architecture’s learning capacity when samples are trained inside the network. In any deep CNN model, one of the most critical tasks is to learn the weight parameters. As a result, this work has established a trade-off between 16 batches and 1500 epochs, enhancing the proposed pain sentiment analysis system’s (PSAS) performance. An experiment comparing batch and epoch for the proposed system is displayed in Figure 9 using the UNBC 2-class dataset, and it can be seen from Figure 9 that performance improves with batch size 16 and epoch variations ranging from 1000 to 1500.
  Here, in the $Schem{e}_1$, experiments are also performed to introduce multi-resolution facial images by utilizing progressive image resizing. The introduction of this scheme needs the use of multi-resolution image analysis in order to benefit from progressive image scaling, such that the image of two different sizes are trained via the $CN{N}_A$ and $CN{N}_B$ architectures. In comparison to the $CNN$ architecture performance, the $Schem{e}_2$ offers the following benefits: (i) networks can be trained with images ranging from low to high resolution; (ii) hierarchical feature representations corresponding to each image can be obtained with improved texture pattern discrimination; and (iii) overfitting problems can be minimized. Figure 10 shows the multi-resolution image analysis performance for the proposed PSAS employing for 2DFPE Database (2-class Pain levels), UNBC Database (2-class Pain levels), UNBC Database (3-class Pain levels), and BioVid Heat Pain Dataset (5-class Pain levels) databases. From this figure, it has been observed that the derived $f_{CNN}^{Schem{e}_1}$ model resolves the overfitting issue and improves the proposed system’s performance.
• $Schem{e}_2$ experiments: This experiment is about measuring the effectiveness of introducing the Transfer-learning with fine-tuning of parameters when two new CNN models are being trained on the same domain of data (i.e., images). Fine-tuning refers to taking into account the weights of one proposed CNN architecture. It shortens the training period, which leads to better results. Transfer learning is a machine learning technique in which a model that is suggested for one job is utilized as a foundation for a different model on a different task. Different transfer learning methodologies have been applied here: (i) in the first approach, we used the refreshed models, i.e., we trained both CNN architectures $CN{N}_A$ and $CN{N}_B$ from scratch using the relevant image sizes; (ii) in the second approach, we used the retrained model of both architectures. We can see that CNN architectures perform better when employing these retrain models. Table 5 shows the CNN architectures’ performance. The accuracy and F1-score are the basis for the reported performances. The results in Table 5 indicate that Approach-1 improves PSAS performance.
• $Schem{e}_3$ experiments: This experiment begins with the application of multiple Fusion techniques on the classification scores generated by the trained $CN{N}_A$ and $CN{N}_B$ architectures. Deep-learning features are utilized to obtain the classification results, and a post-classification approach is applied for fusion. The respective CNN models generate the classification scores, which are then subjected to the Weighted Sum-rule, Product-rule, and Sum-rule procedures for post-classification fusion. The fused performance of the proposed system, resulting from various combinations of fusion methods, is presented in Table 6. The fusion results demonstrate that the proposed system performs better with the Product-rule fusion approach, using both CNN architectures, compared to the Weighted Sum-rule, Sum-rule, and Product-rule techniques. The proposed system achieves an accuracy of 85.15% for the 2-Class UNBC, 83.79% for the 3-Class UNBC, and 77.41% for the 2-Class 2DFPE database. These performance results serve as a reference point in the section below.
• Comparison with existing System for Image-based PSAS: The performance of the proposed system has been compared with several current state-of-the-art techniques across the 2-Class 2DFPE database, the 2-Class and 3-Class UNBC-McMaster shoulder pain databases, and the 5-Class BioVid Heat Pain database. Feature extraction for the local binary pattern (LBP) and Histogram of Oriented Gradient (HoG) procedures, as utilized in [68] and [69], has been employed. Each image’s feature representation approach uses a size of $191\times 192$ pixels, and the images are divided into nine separate blocks for both the LBP and HoG approaches. From each block, a 256-dimensional LBP feature vector [69] and an 81-dimensional HoG feature vector [68] are extracted, resulting in 648 HoG features and 2304 LBP features per image. For deep feature extraction, methods like ResNet50 [70], Inception-v3 [71], and VGG16 [72] were employed, with each using an image size of $192\times 192$. Transfer learning techniques were applied, where each network was independently trained using samples from two or three class problems. Features were then extracted from the trained networks, and neural network-based classifiers were used for test sample classification. Competing approaches such as those by Lucey et al. [65], Anay et al. [73], and Werner et al. [74] were also implemented using the same image size of $192\times 192$. The results for all these competing techniques were obtained using identical training and testing protocols as the proposed system. These results are summarized in Table 7, which demonstrates that the proposed system outperformed the competing techniques.
  Hence, Table 7 presents the performance comparison for the image-based sentiment analysis system using the 2-class and 3-class pain sentiment analysis system. All experiments were conducted under the same training and testing protocols. Quantitative comparisons were made between the proposed system’s two-class classification model (A) and the competing methods: Anay et al. [73] (B), HoG [68] (C), Inception-v3 [71] (D), LBP [69] (E), Lucey et al. [65] (F), ResNet50 [70] (G), VGG16 [72] (H), and Werner et al. [74] (I). The statistical comparison of the performances between the proposed system (A) and the competing methods ({B,C,D,E,F,G,H,I}) is shown in Table 8. A one-tailed t-test [75] was conducted using the t-statistic to analyze the results. The one-tailed test assumes two hypotheses: (i) H0 (Null-hypothesis): ${\mu }_A\le {\mu }_{\{B,C,D,E,F,G,H,I\}}$ (indicating that the proposed system (A) performs on average less than or equal to the competing method’s performance); (ii) H1 (Alternative-hypothesis): ${\mu }_A>{\mu }_{\{B,C,D,E,F,G,H,I\}}$ (indicating that the proposed system (A) performs better on average than the competing methods). The null hypothesis is rejected if the p-value is less than 0.05, indicating that the proposed method (A) performs better than the competing methods. For the quantitative comparison, each test dataset for the employed databases was divided into two sets. For the $D_2$ database, there are 298 test image samples (Set-1: 149, Set-2: 149). For the UNBC (2-Class/3-Class) problem, there are 24,199 test samples (Set-1: 12,100, Set-2: 12,099). The performances of the proposed method and the competing methods were evaluated on these test sets. Table 8 shows the results of the quantitative comparisons for the employed databases. From this table, it is evident that in every comparison between the proposed system and the competing methods, the alternative hypothesis is accepted and the null hypothesis is rejected, confirming that the proposed method (A) outperforms the competing methods.

4.3. Results for Audio-Based PSAS

In the case of the audio-based pain sentiment analysis system (PSAS), the feature representation using the Deep learning-based approach (discussed in Section 3.5 (Figure 2)) has been employed. Here, the above-discussed audio-based features such as Statistical audio features, Mel Frequency Cepstral Coefficients (MFCC), and Spectral Features are employed, extracted from each speech-audio using Librosa [76] python package. This package is used for audio, speech, and music analysis. Here, 11 statistical features, 128 MFCC features, and 224 spectra features are extracted using this package. During experimentations, each set of these features and a combination of these features are experimented with individually correspond to machine learning classifiers such as Logistic regression, K-nearest neighbour, decision tree, and support vector machine concerning 50-50% and 75-25% training-testing sets. Each individual and combined feature set is employed for the deep learning-based feature representation scheme. Table 9 demonstrates the performance of the proposed audio-based pain sentiment analysis system corresponding to machine learning classifiers.

The performances reported in Table 9 show that in most cases, the Decision tree classifier has obtained better performance due to each employed feature type for both 50-50% and 75-25% training-testing protocols. Apart from these, it has also been observed that the impact of MFCC and spectra features is higher than statistical-based audio-frequency features. Hence, 128-MFCC and 224-spectra features are concatenated for further experiments to make a 352-dimensional feature vector corresponding to each speech audio. This is an input to our proposed deep-learning architecture of the audio-based sentiment analysis system. Here, the experiment for deep-learning architecture has been carried out using 32 batch sizes with 1-50 epoch variations, and the performance of the proposed pain sentiment analysis system (PSAS) has been demonstrated in Figure 11. From this figure, it has been observed that for batch size 32 and epoch 100, performance is better. So, this work considers these batch sizes and the number of epochs for further audio-based PSAS.

Hence, the performance reported in Table 9 and Figure 11 has been observed that using machine learning classifiers, the decision tree obtains better performance corresponds to all employed audio-based features. In contrast, according to deep features in both Figure 11(a), Figure 11(b), the proposed audio-based sentiment analysis has obtained better performance for 32 Batch-size and 100 Epochs. So, the outcomes of these two experiments are reported in Figure 12, where decision-tree and deep-learning-based performance correspond to different training-testing protocols. From Figure 12, it has been observed that deep-learning-based features obtain better performance for the proposed audio-based sentiment analysis system.

4.4. Results for Multimodal PSAS

Three experiments are performed in the multimodal pain sentiment analysis system (PSAS). In the first experiment, the classification scores of 149 NAG-class image samples of $2DFPE$ are fused with 149 NAG-class audio samples of VIVAE; similarly, the classification scores of 136 OAG-class image samples of $2DFPE$ with 136 OAG-class audio samples of VIVAE, are fused to obtain the performance of the proposed multimodal sentiment analysis system using $2DFPE$ and VIVAE databases. In the second experiment, the classification scores of 265 NAG-class image samples of UNBC-McMaster are fused with 265 NAG-class audio samples of VIVAE; the classification scores of 136 CAG-class image samples of UNBC-McMaster with 136 CAG-class audio samples of VIVAE, and the classification scores of 141 OAG-class image samples of UNBC-McMaster with 141 OAG-class audio samples of VIVAE are fused to obtain the performance of the proposed 3-class multimodal sentiment analysis system using UNBC-McMaster and VIVAE databases. In the third experiment, the classification scores of 260 Pain-lavel 1 class image samples of BioVid are fused with 260 low-intensity class audio samples of VIVAE; the classification scores of 260 Pain-lavel 2 class image samples of BioVid are fused with 260 moderate-intensity class audio samples of VIVAE; the classification scores of 260 Pain-lavel 3 class image samples of BioVid are fused with 260 strong-intensity class audio samples of VIVAE; the classification scores of 260 Pain-lavel 4 class image samples of BioVid are fused with 260 peak-intensity class audio samples of VIVAE. Here, (50-50%) training-testing protocol has been used for all employed databases. The performance of these multimodal sentiment analysis systems is shown in Figure 13, Figure 14, and Figure 15, respectively. From these figures, it has been observed that the performance of the proposed system is outstanding for the decision-level (majority voting) fusion technique.

Table 10 demonstrates the computational complexity analysis of the proposed system. Here, the complexity analysis has been performed corresponding to the different derived models for the image-based and audio-based pain sentiment analysis system, the number of parameters required for training these models, and the time required to detect the pain based on image and audio data of the patient. Pain Sentiment Analysis is important in the era of electronic healthcare, which uses e-mobile and smart healthcare systems. The research in this work requires the patients’ facial images and speech audio data, which will be analyzed to detect the patients’ pain levels. However, these data might suffer from different challenging issues. So, to enhance the performance, (i) the combined effect of image and audio data is incorporated into the multimodal pain sentiment analysis system, and (ii) some cutting-edge technologies, such as deep learning algorithms and their variants, are used in this work.

5. Conclusion

This paper proposes a multimodal sentiment analysis system for pain detection using facial expressions and speech-audio data in an IoT-enabled healthcare system. The work is divided into five main components: (i) image preprocessing to extract facial patterns as regions of interest, (ii) image-based feature learning and classification using two distinct CNN architectures, (iii) enhancement of CNN performance via transfer learning, fine-tuning, and progressive image resizing, (iv) speech-audio feature learning and classification using both handcrafted and deep features, and (v) fusion of classification scores from image-based and audio-based samples to improve overall system performance. The experiments address 2-class, 3-class, and 5-class pain recognition problems. For the 2-class problem, the 2DFPE and UNBC-McMaster shoulder pain datasets are used, with the classes being ’Pain’ and ’No-Pain.’ For the 3-class problem, the 3-class image-based UNBC-McMaster dataset and the audio-based VIVAE dataset are considered, with the classes ’Non-Aggressive (NAG) (No-pain),’ ’Covertly Aggressive (CAG) (Low-pain),’ and ’Overtly Aggressive (OAG) (High-pain).’ The 5-class problem involves classes: ’No-Pain’, ’Pain-Level 1’, ’Pain-Level 2’, ’Pain-Level 3’, and ’Pain-Level 4.’ The experimental results demonstrate that the proposed system outperforms existing state-of-the-art methods. Additionally, fusion of the image-based and audio-based samples further enhances the system’s performance for the 2-class, 3-class, and 5-class pain recognition tasks. In our research, challenges such as the availability of synchronized data across images, videos of facial expressions, and audio are encountered. Our system addresses this by enabling simultaneous data acquisition from different modalities, which adds complexity to the implementation. While deploying image and audio-based technologies on the web is feasible, implementing them on hardware remains a significant challenge.