CNN-Based Facial Expression Recognition with Simultaneous Consideration of Inter-class and Intra-class Variations

1. Introduction

Facial expressions have been used as critical and natural signals to represent human emotions and intentions. Therefore, various facial expression recognition (FER) methods have been studied and applied to fields, such as virtual reality (VR) [1], human-robot interaction (HRI) [2], and advanced driver assistant systems (ADAS) [3].

Typical FER methods include three stages: (1) face component detection, (2) feature point extraction, and (3) facial expression classification. Facial-component detection involves extracting a facial region from an input image to obtain features such as the eyes and nose from the detected facial components. More recently, studies have shown that feature extraction can be classified into spatial [4,5], and temporal feature extraction [6]. Generally, the expression classifier and feature extraction are vital for the accuracy of FER. Many developments have been made to exploit facial expression (FE) classification, including the Bayesian Classifier [7], Hidden Markov Model (HMM) [8], Adaboost [9], and Support Vector Machine (SVM) [10]. Fig. 1 shows the details of the conventional FER process.

Recent developments in deep learning have achieved significant advancements in computer vision and image processing [11,12,13]. Among the deep-learning methods, the Convolutional Neural Network (CNN) has been proven capable of reducing the dependency on analytical models and preprocessing techniques by enabling “end-to-end” direct learning from input images. For example, feature extraction and recognition are jointly learned using deep-learning methods [14,15,16].

FER is highly sensitive to intra-class variation according to age and gender, illuminance, and facial pose [17]. In addition, because FER datasets are limited and small, operating a CNN to extract the salient features that represent the facial expressions from the facial image is problematic. Several methods have been explored to overcome this problem [18]. Examples of this are the transfer learning method [19] for solving the overfitting problem in training datasets, and the ensemble architectures [20] and hybrid variant input approaches [21] for extracting discriminative features. Notably, most of these approaches primarily concentrated on designing new deep-learning architectures and overlooked the loss function. Additionally, the limited training datasets remain a challenge in improving FER performance.

One method of extracting salient features from limited datasets is to change the traditional loss function of the CNN architecture to reduce the intra-class variation and increase the inter-class variation of the deep features, thereby creating discriminative features. Typically, CNN-based FER optimizes the softmax loss function, which seeks to penalize misclassified samples, encouraging the distinction of features between different classes. The softmax layer is crucial for ensuring that the learned features of various classes remain distinguishable. However, severe intra-class variation remains challenging. Advanced loss functions can be used to address this problem. Generally, advanced loss functions are divided into two categories: Angular-distance-based method (L-Softmax [22], AM-Softmax [23]), Euclidean-distance-based method (contrastive loss [24], triplet loss [25], center loss [26]).

The angular-distance-based losses have made the learned features potentially separable with a larger angular/cosine distance. These losses were reformulated based on the original softmax loss, allowing inter-class separability and intra-class compactness between learned features. However, these loss functions was difficult to converge when trained with complex datasets such as that of FER.

Furthermore, Euclidean distance-based methods have embedded the input images in the Euclidean space to decrease intra-class variation and increase inter-class variation. Contrastive and triplet losses increased memory load and training time owing to the complex recombination of training samples. Center loss updated the center by reducing the distance between the deep features and their corresponding class centers. Nevertheless, it disregarded inter-class variation, thus limiting the FER performance improvement.

To summarize, the existing loss functions for CNN-based FER have the following drawbacks: (1) difficulty in convergence with the complex training dataset, (2) the high memory consumption and training time, and (3) the disregard of inter-class similarity.

Given the above analysis, this study presents a variant loss to minimize the distance between the deep features and their corresponding class centers as well as maximize the distances of deep features with their non-corresponding class centers, and the distances between different class centers. Fig. 2 illustrates the concept of the proposed loss function. Finally, the proposed loss function was assessed on four well-known benchmark facial expression databases: the Cohn-Kanade Plus (CK+) [27], the Oulu-CASIA [28], MMI [29], and FER2013 [30] databases. The contributions of this study can be summarized as follows:

• A new loss function is proposed to simultaneously consider inter- and intra-class variations, which enables CNN-based FER methods to achieve impressive performance.

• A new loss function can be easily optimized with various CNN architectures on diverse databases to learn the discriminative power of deep features for the FER problem.

• Comprehensive experiments on benchmark databases are conducted to prove that the auxiliary CNN architectures trained with the proposed loss function performed much better than with existing loss functions.

The remainder of this paper is organized into four sections. Section II summarizes previous loss functions and auxiliary CNN architectures. Section III describes the proposed loss function that simultaneously considers intra- and inter-class variations. Section IV analyzes the simulation results, and Section V states the conclusions.

2. Related Work

2.1. Previous Loss Functions

The softmax loss is obviously good at increasing the inter-class variation but unable to decrease the intra-class variation. To tackle this problem, several loss functions have been introduced to reduce the intra-class variation. Most representatively, L-Softmax loss [22] is an improvement over the conventional softmax loss, enabling inter-class separability and intra-class compactness between learned features. With an adjustable margin value, L-softmax could determine an adaptable learning task with flexible difficulty for CNNs. It also prevented overfitting problems to leverage the powerful learning capacity of deep and wide architectures. Nevertheless, when the training dataset has various subjects, the convergence of L-Softmax will be tougher than the softmax loss. AM-Softmax loss [23] used an additive margin strategy to the target logit of softmax loss with feature and weights normalized. Although it was intuitively appealing and more interpretable than the L-Softmax [22], the selection of the margin hyperparameter was challenging.

Contrastive [24] and triplet losses [25] adopted pair training technique. In particular, the contrastive loss included negative and positive pairs. Its gradients attracted positive pairs and repelled negative ones. Whereas, triplet loss reduced the distance between an anchor and a positive sample, and increased the distance between an anchor and a negative sample of a different identity. The training procedure for these losses was still challenging owing to the selection of effective training samples. Center loss [26] decreased intra-class variations during training by penalizing the distances between deep features and their corresponding class centers. By relying solely on training CNNs with center loss, the deep features and class centers might deteriorate to zero. Moreover, the center loss was minimal, and discriminative features could not be achieved. Thus, the center loss should be jointly supervised with the softmax loss during training. However, each identity’s center doubled the memory storage of the last CNN layer.

Range loss [31] was proposed to effectively use the whole long-tailed dataset in the training procedure. The range loss was optimized jointly with softmax loss as supervisory signals to train CNNs. However, the optimization strategy could be challenging because softmax loss requires uniform distribution among all the classes, and the ability to improve inter-class differences within each mini-batch was restricted. Marginal loss [32] could decrease the intra-class variances and enlarge the inter-class distances by focusing on the marginal samples. The marginal combined with a softmax loss to jointly supervise the learning of CNN, the discriminative capacity of deep features could be greatly improved for efficient face recognition. Even so, the age variance restriction in the training data could significantly reduce the performance when there was a large year gap.

Overall, while existing loss functions achieved promising performance, there is still much room for improvement. To this end, this study proposes variant loss to minimize the distance between the deep features and their corresponding class centers as well as maximize the distances of deep features with their non-corresponding class centers and the distances between different class centers. The proposed loss function is easy to adopt in CNN-based FER methods and achieves outstanding performance.

2.2. Auxiliary CNN Architectures

Given an input image or feature, classification models predict specific labels. In this study, six popular CNN architectures are trained using various loss functions to evaluate the feasibility of the proposed loss function. First, AlexNet [33] has eight layers comprising five convolutional layers and three fully connected layers combined with dropout techniques. Its simplicity and moderate depth made its training fast.

To improve the classification performance, InceptionNet [34] was designed based on the Inception module, which aggregated four parallel branches: three convolution branches with different kernel sizes (1×1, 3×3, and 5×5) and a max-pooling branch. InceptionNet contained 22 layers, including nine Inception modules stacked on top of each other. This design increased the width of the network and adaptability to various scales.

Additionally, the deep learning networks suffer from a vanishing gradient problem that impedes accuracy. ResNet [35] was proposed to add skip connections from the input to the output of the convolutional layer to address these problems. The residual block contained two 3×3 convolutional layers, each followed by Batch Norm and ReLU activate function. ResNet-18 was selected to train with the comparative loss function in this study. Fig. 3 shows the ResNet-18 architecture, whereas ResBlock is shown in Fig. 4.

DenseNet [36] proposed dense blocks and transition layers. Dense blocks concatenated the output of the previous layer as the input of the next. In this way, a feed-forward nature could be maintained. However, the number of channels would be increased when concatenating layers. The transition was used to control the size of the features by 1×1 convolution. Moreover, the height and width of features were reduced through the average pooling layer.

Recently, MobileNetV3 [37] can be applied to mobile and embedded devices owing to its lightweight. It was based on a combination of hardware-aware network architecture search (NAS) algorithm and squeeze-and-excitation (SE) module [38]. The block-wise search algorithm (MnasNet [39]) was employed to identify global network structures, and then the layer-wise search algorithm (NetAdapt [40]) was sequentially employed to adjust individual layers. MobileNetV3 inserted the SE module to build channel-wise attention. The hard-sigmoid function was utilized to substitute the conventional sigmoid in the SE module for more efficient calculation. In addition, the hard-swish function was adopted instead of ReLU for non-linearity improvement.

Finally, ResNeSt [41] is an improved version of ResNet. It combined channel-wise attention with multi-path representation into a unified Split-Attention block. These Split-Attention blocks were stacked to follow the concept of residual learning from the ResNet model [35]. This architecture enhanced learned feature representations for multiple high-level vision tasks, including object detection, image classification, and semantic segmentation. Moreover, it was reported that ResNeSt enabled the acceleration of training with computationally efficient.

3. Proposed Method

As aforementioned, a variant loss is proposed to minimize the distance between the deep features and their corresponding class centers as well as maximize the distances of deep features with their non-corresponding class centers, and the distances between different class centers. The new loss function is expressed as follows:

$$L_v=\frac{1}{2}\sum _{i=1}^M(\parallel F\left(x_i\right)-c_{y_i}{\parallel }_2^2+\frac{{\lambda }_1}{{ϵ}_1+{\displaystyle \sum _{j=1,j\ne y_i}^N}{\parallel F\left(x_i\right)-c_j\parallel }_2^2})+\frac{{\lambda }_2}{{ϵ}_2+{\displaystyle \sum _{m\in N}}{\displaystyle \sum _{\begin{array}{c}n\in N\\ n\ne m\end{array}}}{\parallel c_m-c_n\parallel }_2^2},$$

(1)

where $y_i$, $x_i\in {\mathbb{R}}^d$ are the ordinary label and input images of i-$th$ sample facial expressions, respectively; d is dimension features. $F(.)$ expresses the feature extraction from the CNNs; $c_{y_i}$$\in {\mathbb{R}}^d$ denotes the $y_i$-$th$ class center of the deep features from the CNNs with the same label class $y_i$. M is the number of training data in the batch size; N is the number of classes; $c_j,c_m$, and $c_n\in {\mathbb{R}}^d$ are the j-$th$, m-$th$, and n-$th$ class centers of deep features, respectively. ${ϵ}_1$ and ${ϵ}_2$ are tolerance parameters that guarantee that the denominator is higher than zero; and ${\lambda }_1$, ${\lambda }_2$ are the hyperparameters used for balancing these loss terms.

The first term is similar to the center loss and tends to reduce the distance between the deep features and their corresponding class centers. The second and third terms tend to increase the distance between the deep features and their non-corresponding class centers and between class centers, respectively. By minimizing the proposed loss function, the intra-class variations of the deep features are reduced, whereas the inter-class variations continue to increase.

The proposed loss function is applied to the batch data in each iteration to train it with the softmax loss. The overall loss function for training the CNN is computed as the sum of the weights of the softmax and variant losses. It is expressed as follows:

$$L=L_s+\lambda L_v,$$

(2)

where $\lambda$ is a hyperparameter used for balancing the softmax and the variant losses. The overall system of the CNNs using the proposed loss is illustrated in Fig. 5.

In this method, the network parameters include CNN parameters W and the softmax loss parameters $\theta$ are updated on mini-batches. Only the gradient of $L_s$ is needed to update the softmax loss parameters $\theta$ because the $L_v$ does not affect it. The gradient of overall loss is used to update W. The gradient of $L_v$ with respect to the $F\left(x_i\right)$ is calculate as follows:

$$\begin{array}{cc}{\displaystyle \frac{d{L}_v}{dF\left(x_i\right)}}=(F\left(x_i\right)-c_{y_i})-{\lambda }_1\frac{{\displaystyle \sum _{j=1,j\ne y_i}^N}(F\left(x_i\right)-c_j)}{{({ϵ}_1+{\displaystyle \sum _{j=1,j\ne y_i}^N}{\parallel F\left(x_i\right)-c_j\parallel }_2^2)}^2}\hfill & \\ & \hfill -{\lambda }_2\frac{{\displaystyle \sum _{\begin{array}{c}m=y_i\\ n\in N\\ n\ne m\end{array}}}(c_{y_i}-c_n)-{\displaystyle \sum _{\begin{array}{c}m\in N\\ n=y_i\\ n\ne m\end{array}}}(c_m-c_{y_i})}{({ϵ}_2+{\displaystyle \sum _{m\in N}\sum _{\begin{array}{c}n\in N\\ n\ne m\end{array}}}\parallel c_m-c_n{{\parallel }_2^2)}^2}.\end{array}$$

(3)

In addition, the class center is calculated by averaging the features in the same class and updating in each iteration. The centers are updated as follows:

$$c_k^{t+1}=c_k^t+\alpha \nabla c_k,$$

(4)

where $\alpha$ is the learning rate of class centers.

The update of the k-$th$ class center can be computed as a derivative of the overall loss with respect to the class center $c_k$ :

$$\begin{array}{c}\nabla c_k=\frac{{\displaystyle \sum _{\begin{array}{c}i=1\end{array}}^M}\delta (y_i,k)(c_k-F\left(x_i\right))}{1+{\displaystyle \sum _{\begin{array}{c}i=1\end{array}}^M}\delta (y_i,k)}-{\lambda }_1\frac{{\displaystyle \sum _{\begin{array}{c}i=1\end{array}}^M}(1-\delta (y_i,k))(c_k-F\left(x_i\right))}{({ϵ}_1+{\displaystyle \sum _{j=1,j\ne y_i}^N}\left|\right|F\left(x_i\right)-c_j{\left|\right|}_2^2{)}^2}\\ -{\lambda }_2\frac{{\displaystyle \sum _{m\in N}}\delta (m,k){\displaystyle \sum _{\begin{array}{c}n\in N\\ n\ne m\end{array}}}(c_k-c_n)}{({ϵ}_2+{\displaystyle \sum _{m\in N}\sum _{\begin{array}{c}n\in N\\ n\ne m\end{array}}}\parallel c_m-c_n{{\parallel }_2^2)}^2(1+{\displaystyle \sum _{m\in N}}\delta (m,k))},\end{array}$$

(5)

where $\delta (y_i,k)$ and $\delta (m,k)$ are defined as

$$\delta (y_i,k)=\left\{\begin{array}{c}1;y_i=k\hfill \\ 0;y_i\ne k\hfill \end{array}\right.,$$

(6)

$$\delta (m,k)=\left\{\begin{array}{c}1;m=k\hfill \\ 0;m\ne k\hfill \end{array}\right..$$

(7)

CNNs can be trained utilizing standard stochastic gradient descent (SGD). The hyperparameters of CNNs contain: a batch size M, the number of training iterations T, the learning rates of the weight parameter $\mu$, the learning rates of the class centers $\alpha$, and balanced terms of the loss functions $\lambda$, ${\lambda }_1$, ${\lambda }_2$. First, the parameters of the CNNs are initialized W, the softmax loss parameters $\theta$, and class center $c_k$. In each iteration, M training images $x_i$ are passed into the CNNs to obtain the output of the last fully-connected layer $F\left(x_i\right)$ in each batch. The overall loss of the model and derivative of the loss functions with respect to the output of the last fully-connected layers $F\left(x_i\right)$ are calculated to update the parameters of the CNNs. $\theta$ is independent of the overall loss; therefore, only the softmax loss is considered. Whereas the gradient of overall loss is used to update W. The update process of W and $\theta$ are separated with difference derivatives. Finally, the derivatives of the overall loss with respect to class center $c_k$ are calculated to update the class centers $c_k$ with the learning rate of the class center $\alpha$. The training process for CNNs with proposed loss is interpreted in Algorithm 1.

Algorithm 1 Training process for CNNs with proposed loss

Input: Training images $x_i$, batch size M, number of training iterations T, learning rates of weight parameter $\mu$, learning rate of class centers $\alpha$, hyper-parameters $\lambda$, ${\lambda }_1$, ${\lambda }_2$.      Initialization: the CNNs parameters W, the softmax loss parameters $\theta$, the class centers $c_k$, the iteration t=0. 1: while$t\le T$do   2:     Calculate the deep features, the output of the last fully-connected layers $F\left(x_i\right)$ of M input images in one mini-batch.   3:     Calculate the overall loss as in (2):    4:     $L=L_s+\lambda L_v$   5:     Calculate the gradients for each input i by:    6:     ${\displaystyle \frac{d{L}^t}{dF{\left(x_i\right)}^t}}={\displaystyle \frac{d{L}_s^t}{dF{\left(x_i\right)}^t}}+\lambda {\displaystyle \frac{d{L}_v^t}{dF{\left(x_i\right)}^t}}$   7:     Update parameters $\theta$ by:   8:     ${\theta }^{t+1}$ = ${\theta }^t-\mu {\displaystyle \frac{d{L}^t}{d{\theta }^t}}={\theta }^t-\mu {\displaystyle \frac{d{L}_s^t}{d{\theta }^t}}$   9:     Update parameters W by:   10:     $W^{t+1}=W^t-\mu {\displaystyle \frac{d{L}^t}{d{W}^t}}=W^t-\mu {\displaystyle \sum _{\begin{array}{c}i\end{array}}^M}{\displaystyle \frac{d{L}^t}{dF{\left(x_i\right)}^t}}{\displaystyle \frac{dF{\left(x_i\right)}^t}{d{W}^t}}$   11:     Update $c_k$ for k-th class center: $c_k^{t+1}=c_k^t-\alpha \nabla c_k$   12:     $t=t+1$   13: end while    End of the algorithm: The CNNs parameters W, the softmax loss parameters $\theta$

4. Experiments

4.1. Experimental Setup

The performance of the proposed method was evaluated based on four benchmark facial expression databases: three from a laboratory environment, namely, Cohn-Kanade Plus (CK+) [27], Oulu-CASIA [28], MMI [29]; and one from a wild environment, FER2013 [30]. A 10-fold cross-validation strategy was employed for model evaluation, especially focusing on scenarios with small and imbalanced datasets such as CK+, MMI, and Oulu-CASIA. Each of these databases was strictly divided into 90% as a training set and 10% allocated as a testing set. Furthermore, FER is a large-scale dataset; the training and evaluation processes were conducted on its provided datasets. Several sample images derived from these databases are illustrated in Fig. 6. The details of the databases and the number of images for each emotion are presented in Table 1.

To minimize the variations in the face scale and in-plane rotation, the face was detected and aligned from the original database using the OpenCV library with Haar–Cascade detection [42]. The aligned facial images were resized to 64×64 pixels. Moreover, intensity equalization was used to enhance the contrast in facial images. A data augmentation technique was used to overcome the restricted number of training images in the FER problem. Furthermore, the facial images were flipped, and each one and its corresponding flipped image was rotated at -15, -10, -5, 5, 10, and 15 °. The training databases were augmented 14 times using original, flipped, six-angle, and six-angle-flipped images. The rotated facial images are shown in Fig. 7.

The proposed loss function was compared with softmax, center [26], range [31], and marginal losses[32] using the same CNN architectures to demonstrate the effectiveness of the proposed loss function. The experiment was conducted in a subject-independent scenario. The CNN architectures were processed with 64 images in each batch. The training was performed using the standard SGD technique to optimize the loss functions. The hyper-parameter $\lambda$ was used to balance the softmax and variant losses. ${\lambda }_1$ and ${\lambda }_2$ were utilized to balance among these losses in the variant loss, and $\alpha$ controlled the learning rate of the class center $c_k$. All of these factors affect the performance of our model. In this experiment, the values $\lambda$=0.001, ${\lambda }_1$=0.4, ${\lambda }_2$=0.6, ${ϵ}_1$=${ϵ}_2$=10e-3 were empirically selected for the proposed loss. For the center, marginal, and range losses, $\lambda$ was set to 0.001. An Intel^®Xeon^® CPU E5-2620 v2 @ 2.10GHz ×12 48GB RAM and an NVIDIA RTX 3090 GPU were used to implement the proposed method in a Pytorch framework.

4.2. Experimental Results

1) Results on Cohn-Kanade Plus (CK+) database: The CK+ is a representative laboratory-controlled database for FER. It comprises 593 image sequences collected from 123 participants. 327 of these image sequences have one of seven emotion labels: anger, contempt, disgust, fear, happiness, sadness, and surprise, from 118 subjects. Each image sequence starts with a neutral face and ends with the peak emotion. To collect additional data, the last three frames of each sequence were collected and associated with the provided labels. Therefore, a database containing 981 experimental images was constructed. The images were primarily grayscale and digitized to a 640×490 or 640×480 resolution.

The average recognition precision of the methods based on the loss functions and CNN architectures is listed in Table 2. The accuracy of the proposed loss function was superior to that of the others for all six CNN architectures. For the same loss functions, the accuracy of ResNet was the highest, followed by those of MobileNetV3, ResNeSt, InceptionNet, AlexNet, and DenseNet. Overall, the proposed loss produced an average recognition accuracy of 94.89% for the seven expressions using the ResNet.

Table 3 presents the confusion matrix of the ResNet, which was optimized using the proposed loss function. The accuracy of the contempt, disgust, happiness, and surprise labels was significant. Notably, the happiness percentage was the highest at 99.5%, followed closely by surprise, disgust, and contempt at 98.4%, 97.7%, and 93.4%, respectively. The proportions of the anger, fear, and sadness labels were inferior to these emotions because of their visual similarity.

2) Results on Oulu-CASIA database: The Oulu-CASIA database includes 2,880 image sequences obtained from 80 participants using a visible light (VIS) imaging system under normal illumination conditions. Six emotion labels were assigned to each image sequence: anger, disgust, fear, happiness, sadness, and surprise. Like the Cohn-Kanade Plus database, the image sequence started with a neutral face and ended with the peak emotion. For each image sequence, the last three frames were collected as the peak frames of the labeled expression. The imaging hardware was operated at 25 fps with an image resolution of 320×240 pixels.

The average recognition accuracy of the methods is listed in Table 4. The performance of the proposed loss function was comparable to that of previous ones. Specifically, the proposed loss achieved an average recognition accuracy of 77.61% for the six expressions using the ResNet architectures.

Table 5 presents the confusion matrix of ResNet trained with the proposed loss function. The accuracy of the happiness and surprise labels increased, with the former achieving 92.1% and the latter gaining 84.0%. The accuracy for anger, disgust, fear, and sadness was inferior, obtained 66.2%, 70.5%, 76.3%, and 76.5%, respectively.

3) Results on MMI database: The laboratory-controlled MMI database comprises 312 image sequences collected from 30 participants. 213 image sequences were labeled with six facial expressions: anger, disgust, fear, happiness, sadness, and surprise. Moreover, 208 sequences from 30 participants were captured in frontal view. The spatial resolution was 720×576 pixels, and the videos were recorded at 24 fps. Unlike the Cohn-Kanade Plus and Oulu-CASIA databases, the MMI database features image sequences labeled by the onset-apex. Therefore, the sequences started with a neutral expression, peaked near the middle, and returned to a neutral expression. The location of the peak expression frame was not provided. Furthermore, the MMI database presented challenging conditions, particularly in the case of large interpersonal variations. Three middle frames were chosen as the peak expression frames in each image sequence to conduct a subject-independent cross-validation scenario.

Table 6 lists the average recognition accuracy of the methods. Our loss function outperformed all the other loss functions by a certain margin. Specifically, the proposed loss achieved an average recognition accuracy of 67.43% for the six expressions using the MobileNetV3 architecture.

Table 7 presents the percentages in the confusion matrix of the MobileNetV3 optimized with the proposed loss function. The accuracy for all emotions was under 80.0%, except for happiness and surprise, which obtained 89.7% and 81.3%, respectively. This may be due to the number of images in each class. An instance of this is fear, which had the fewest labels and whose accuracy was a low 31.0%. Similar results were also confirmed for the accuracy of anger, disgust, and sadness.

4) Results on FER2013 database: FER2013 is a large-scale, unconstrained database automatically collected by the Google image search API. It includes 35,887 images with a relatively low resolution of 48×48 pixels, which are labeled with one of seven emotion labels: anger, disgust, fear, happiness, sadness, surprise, and neutral. The training set comprises 28,709 examples. The public test consists of 3,589 examples; the remaining 3,589 images are used as a private test set.

Table 8 lists all the methods’ average recognition accuracy. The accuracy of the proposed loss function greatly exceeds that of the others in all CNN architectures except AlexNet. The proposed loss achieved a peak average recognition accuracy of 61.05% for the seven expressions using the ResNeSt architecture.

The confusion matrix of ResNeSt, which was trained with the proposed loss function, is present in Table 9. The happiness percentage was highest at 80.7%, followed by the surprise at 77.4%. The others obtained relatively low prediction ratios.

4.3. Training Time

The training time is essential for evaluating the computational complexity of deep learning networks with specific loss functions. This section compares the training time of the auxiliary CNN architectures with the existing and proposed loss functions. Notably, all loss functions were trained on a single GPU. Depending on the dataset and network architecture, the number of iterations was empirically set to achieve optimal convergence with the corresponding loss function. As presented in Table 10, the softmax loss trained the fastest because it only uses one term in the mathematical function, followed closely by the center and proposed loss functions. Meanwhile, the range and marginal loss functions required longer training times among the compared methods because their complex mathematical functions produced a time-consuming backpropagation process. In summary, only softmax and center loss were marginally faster than the proposed method. However, the proposed method achieved superior performance compared with these loss functions. Therefore, the proposed method is computationally efficient and meets the practical requirements.

5. Conclusions

Although loss functions can drive network learning, it has received little attention for promoting FER performance. This study presents a new loss function that allows simultaneous consideration of inter- and intra-class variations to be applied to CNN architecture for facial expression recognition. More specifically, this loss minimizes the distance between the deep features and their corresponding class centers as well as maximizes the distances of deep features with their non-corresponding class centers and the distances between different class centers. In addition, the proposed loss improves the testing accuracy of the benchmark FER database compared to several other loss functions. Overall, this study demonstrates that choosing optimal loss functions strongly affects the performance of deep learning networks, even when maintaining their architecture. We hope the proposed loss function can be applied to other high-level vision tasks.